Setting up Routing Rules using CRDs

Overview

Custom Resource Definitions (CRDs) are used to extend the Kubernetes APIs server with additional schemas. To know more, click here.

AMKO uses the AMKOCluster to configure a federation to member clusters.

AMKO uses the following custom resources to configure the GSLB services in the GSLB leader site:

• GSLBConfig
• GlobalDeploymentPolicy
• GSLBHostRule

If AMKO is installed via helm, it by default creates one instance of each type in the avi-system namespace. To view these objects:

$ kubectl get amkocluster amkocluster-federation -n avi-system
NAME                       AGE
amkocluster-federation     45m

Note: Only one instance of each GDP and GSLBConfig is supported and AMKO will ignore other instances.

GSLBConfig (gs)

The GSLBConfig adds GSLB configuration. The parameters of this object are discussed below:

apiVersion: "avilb.k8s.io/v1alpha1"
kind: "GSLBConfig"
metadata:
  name: "amko-gc"
  namespace: "avi-system"
spec:
  gslbLeader:
    credentials: gslb-avi-secret
    controllerVersion: 20.1.1
    controllerIP: 10.10.10.10
  memberClusters:
    - clusterContext: cluster1-admin
    - clusterContext: cluster2-admin
  refreshInterval: 1800
  logLevel: "INFO"
  useCustomGlobalFqdn: false

Here,

• apiVersion: The api version for this object is avilb.k8s.io/v1alpha1
• kind: The object kind is GSLBConfig
• name: Can be anything, but it has to be specified in the GDP object
• namespace: By default, this object must be created in avi-system
• gslbLeader.credentials: A secret object has to be created for (helm install does that automatically) the GSLB Leader cluster. The username and password have to be provided as part of this secret object.
• gslbLeader.controllerVersion: The version of the GSLB leader cluster.
• gslbLeader.controllerIP: The GSLB leader IP address or the hostname along with the port number, if any.
• memberClusters: The Kubernetes/OpenShift cluster contexts which are part of this GSLB cluster.
• refreshInterval: This is an internal cache refresh time interval, on which syncs up with the AVI objects and checks if a sync is required.
• logLevel: Define the log level that the amko pod prints. The supported values are INFO, DEBUG, WARN and ERROR
• useCustomGlobalFqdn: If set to true, AMKO will look for AKO HostRules to derive the GslbService name using the local to global fqdn mapping. If set to false (default case), AMKO ignores AKO HostRules and uses the default way of deriving GslbService names by just looking at the local fqdn in the ingress/route/service type LB.
• downResponse: Specifies the response to the client query when the GSLB service is DOWN. If this field is absent, the GSLB service will be configured with GSLB_SERVICE_DOWN_RESPONSE_NONE. See Down Response Settings for more details.

Notes

• Only one GSLBConfig object is allowed.
• If using helm install, a GSLBConfig object is created by picking up values from the values.yaml file.
• During helm delete, the GSLBConfig that holds the UUID of the current instance is deleted. Hence a cleanup of stale GSLB services, if any, is required at the controller before re-installing AMKO.

GlobalDeploymentPolicy

The GlobalDeploymentPolicy CRD enables selecting Kubernetes/OpenShift objects based on certain rules.
The selection policy applies to all the clusters which are mentioned in the GDP object.

An example for GlobalDeploymentPolicy is as shown below:

apiVersion: "amko.k8s.io/v1alpha2"
kind: "GlobalDeploymentPolicy"
metadata:
  name: "global-gdp"
  namespace: "avi-system"   // a cluster-wide GDP
spec:
  matchRules:
    appSelector:
      label:
        app: gslb
    namespaceSelector:
      label:
        ns: gslb
  matchClusters:
    - cluster: cluster1-admin    // cluster names are kubernetes cluster contexts
    - cluster: cluster2-admin
  trafficSplit:
    - cluster: cluster1
      weight: 8
      priority: 2
    - cluster: cluster2
      weight: 2
      priority: 3
  ttl: 10
  healthMonitorRefs:
  - my-health-monitor1
  sitePersistenceRef: gap-1
  poolAlgorithmSettings:
    lbAlgorithm: GSLB_ALGORITHM_ROUND_ROBIN

Here,

• namespace: namespace of this object must be avi-system.
• matchRules: This enables selecting objects using either application labels (configured as labels on Ingress/Route objects) or via namespace labels (configured as labels on the namespace objects). MatchRules are defined as:
  

  
    matchRules:
        appSelector:                       // application selection criteria
          label:
            app: gslb                       // kubernetes/openshift label key-value
        namespaceSelector:                 // namespace selection criteria
          label:
            ns: gslb                        // kubernetes/openshift label key-value
    

  A combination of appSelector and namespaceSelector will decide which objects will be selected for GSLB service consideration.

  • appSelector: Selection criteria only for applications:
    • Label: Is used to match the ingress/service type load balancer labels (key:value pair)
  • namespaceSelector: Selection criteria only for namespaces:
    • Label: Is used to match the namespace labels (key:value pair).

  AMKO supports the following combination for GDP match rules:

  appSelector	namespaceSelector	Result
  Yes	Yes	Select all objects satisfying appSelector and from the namespaces satisfying the namespaceSelector
  No	Yes	Select all objects from the selected namespaces (satisfying namespaceSelector)
  Yes	No	Select all objects satisfying the appSelector criteria from all namespaces
  No	No	No objects selected (default action)

  For example,
  

  • Select objects with label app:gslb from all the namespaces:
    

  
    matchRules:
        appSelector:
          label:
            app: gslb
    

  • Select objects with label app:gslb and from namespaces labelled ns:prod:

  
    matchRules:
        appSelector:
          label:
            app: gslb
        namespaceSelector:
          label:
            ns: prod
    

• matchClusters: List of clusters on which the above matchRules will be applied on. The member object of this list are cluster contexts of the individual Kubernetes/ OpenShift clusters.

• trafficSplit: Is required if we want to route a percentage of traffic to objects in a given cluster. Weights for these clusters range from 1 to 20. The fieldtrafficSplit can also be used to prioritize certain clusters before others.
  trafficSplit can also be used to prioritize certain clusters before others. The maximum value for priority is 100 and default is 10. If two clusters are given a priority of 20 and a third cluster is added with a priority of 10. The third cluster will not be routed any traffic unless both cluster1 and cluster2 (with priority 20) are down.

• ttl: Use this flag to set the Time-To-Live value. The value can range from 1-86400 seconds. This determines the frequency with which clients need to obtain fresh steering information for client requests. If none is specified in the GDP object, the value defaults to the one specified in the DNS application profile.

• healthMonitorRefs: Provide federated custom health monitors. If this option is used and refs are specified, the default path based health monitoring will be deleted for the GslbServices. If no custom health monitors are specified, AMKO sets the default health monitors for all GslbServices.

• sitePersistenceRef: Provide an Application Persistence Profile ref (pre-created in Avi Controller). This has to be a federated profile. Follow the steps provided here to create a federated Application Persistence Profile on the Avi Controller. If no reference is provided, Site Persistence is disabled.

• poolAlgorithmSettings: Provide the GslbService pool algorithm settings. Refer to Pool Algorithm Settings to know more. If this field is absent, the default is assumed as Round Robin algorithm.
  

• downResponse: Specifies the response to the client query when the GSLB service is DOWN. If this field is absent, the GSLB service will be configured with GSLB_SERVICE_DOWN_RESPONSE_NONE. See Down Response Settings for more details.

Notes

• Only one GDP object is supported.

• If using helm install, a GDP object is created by picking up values from values.yaml file. This GDP object can be edited to modify their selection of objects.

• trafficSplit, ttl, sitePersistence and healthMonitorRefs provided in the GDP object are applicable on all the GslbServices. These properties, however, can be overridden via GSLBHostRule created for a GslbService.

• Site Persistence, if specified, will only be enabled for the GslbServices which have secure ingresses or secure routes as the members and will be disabled for all other cases.

GSLBHostRule

GSLBHostRule enables overriding certain properties of a specific GslbService object on the Avi Controller created by AMKO.

An example of a GSLBHostRule is as follows:

apiVersion: amko.vmware.com/v1alpha1
kind: GSLBHostRule
metadata:
  name: gslb-host-rule-1
  namespace: avi-system
spec:
  fqdn: foo.avi.internal
  sitePersistence:
    enabled: true
    profileRef: "gap-1"  # only enabled for secure ingresses/routes
  thirdPartyMembers:
  - site: non-avi-site
    vip: 10.10.10.10
    publicIP: 122.162.150.96
  healthMonitorRefs:
  - hm1
  - hm2
  trafficSplit:
  - cluster: k8s
    weight: 15
    priority: 10
  - cluster: oshift
    weight: 5
    priority: 10
   publicIP:
  - cluster: k8s
    ip: 160.10.1.1
  - cluster: oshift
    ip: 170.11.1.1
  ttl: 30

Here,

• namespace: The namespace of this object must be avi-system.
• fqdn: FQDN of the GslbService
• sitePersistence: Enable Site Persistence for client requests. Set the enabled flag as true and add a profileRef for a pre-created Application Persistence Profile created on the Avi Controller. Follow the steps provided here to create a federated Application Persistence Profile on the Avi Controller.
  Note: Site persistence is disabled on GslbServices created for insecure Ingresses/Routes, irrespective of this field. If this field is not provided in GSLBHostRule, the site persistence property will be inherited from the GDP object.
• thirdPartyMembers: To add one or more third party members to a GS from a non-avi site (third party site) for the purpose of maintenance, specify a list of those members. For each member, provide the site name in site and IP address in vip. Refer Associating Third-Party Services with Third-Party Sites to know more.
  Optional publicIP in IPv4 format can be added if the vip IP is private and not accessible by the client network. For more information, see NAT-aware Public-Private GSLB Configuration.
  Note: To add third party members, set the enable flag in sitePersistence to False for this object. If site persistence is enabled for a GSLB Service, third party members can not be added.
  The site must be added to the GSLB leader as a third party site before adding more members here.
• healthMonitorRefs: If a GslbService requires some custom health monitoring, the user can create a federated custom health monitor in the Avi Controller and provide the ref(s) here. To add a custom health monitor, follow the steps given in Configuring GSLB Health Monitors using Avi UI. If no custom health monitor refs have been added, the custom health monitors from the GDP object will be inherited.
• trafficSplit: Specify traffic steering to member clusters/sites. The traffic is then split proportionately between two different clusters. Weight for each cluster must be provided between 1 to 20. If not added, GDP object’s traffic split applies on this GslbService.
• publicIP: An optional public IP address (IPv4) can be specified for each site. This field is used to host the public IP address for the VIP, which gets NAT’ed to the private IP by a firewall. For more information, see NAT-aware Public-Private GSLB Configuration.
• ttl: Override the default ttl value specified on the GDP object using this field.
• poolAlgorithmSettings: Provide the GslbService pool algorithm settings. Refer to Pool Algorithm Settings to know more. If this field is absent, the default is assumed as Round Robin algorithm.
• downresponse: Specifies the response to the client query when the GSLB service is DOWN. If this field is absent, GDP’s down response settings would get applied on the GslbService. Refer to down response settings for details.

Pool Algorithm Settings

The pool algorithm settings for GslbService(s) can be specified via the GDP or a GSLBHostRule objects. The GslbService uses the algorithm settings to distribute the traffic accordingly. To set the required settings, use the following fields:

poolAlgorithmSettings:
    lbAlgorithm:
    hashMask:
    geoFallback:
      lbAlgorithm:
      hashMask:

lbAlgorithm is used to specify the name of the algorithm. The supported algorithms are:

• GSLB_ALGORITHM_CONSISTENT_HASH (needs the hash mask in the hashMask field).
• GSLB_ALGORITHM_GEO (needs the fallback algorithm settings to be specified in geoFallback fields)
• GSLB_ALGORITHM_ROUND_ROBIN (by default)
• GSLB_ALGORITHM_TOPOLOGY

If GSLB_ALGORITHM_GEO is set as the main algorithm, specify the geoFallback settings. geoFallback.lbAlgorithm can have either of the two values:

• GSLB_ALGORITHM_CONSISTENT_HASH (needs the hash mask in geoFallback.hashMask)
• GSLB_ALGORITHM_ROUND_ROBIN

For more details on the algorithm that best fits your needs and the configuration on the Avi Controller, refer to Load Balancing Algorithms for GSLB Pool Members.

Down Response Settings

Down Response specifies the response to the client query when the GSLB service is DOWN. The down response settings for GslbService(s) can be specified via the GDP or GSLBHostRule objects. To following fields must be used to set the down response:

  downResponse:
    type:
    fallbackIP: # required only when the type is set as GSLB_SERVICE_DOWN_RESPONSE_FALLBACK_IP

where,

• The field type is used to specify the type of response from DNS service towards the client when the GSLB service is DOWN. The Supported types are:
  
  • GSLB_SERVICE_DOWN_RESPONSE_NONE - No response to the client query when the GSLB service.
  • GSLB_SERVICE_DOWN_RESPONSE_ALL_RECORDS - Respond with all the records to the client query when the GSLB.
  • GSLB_SERVICE_DOWN_RESPONSE_FALLBACK_IP - Respond with the given fallback IP address to the client query when GSLB service is down.
  • GSLB_SERVICE_DOWN_RESPONSE_EMPTY - Respond with an empty response to the client query when the GSLB service is down.
• The field fallbackIP is the fallback IP address to use in response to the client query when the GSLB service is down.

Caveats

• Site Persistence cannot be enabled for the GslbServices which have insecure ingresses or routes as the members.

Date	Change Summary
June 23, 2023	Updated the article for AMKO version 1.10.1
February 03, 2022	Updated the article for AMKO version 1.9.1
October 10, 2022	Updated the article for AMKO version 1.8.1
January 25, 2022	Published the article for AMKO version 1.6.1
July 29, 2021	Updated the article with `AMKOCluster` to configure Federation (version 1.4.2)