Setting up Routing Rules using CRDs

Overview

Custom Resource Definitions (CRDs) are used to extend the Kubernetes APIs server with additional schemas. To know more, click here.

The Avi Kubernetes Operator (AKO) supports some CRD objects (installed through helm). The CRDs are relevant to:

Operators:
Users of this category:

Are aware of the Avi-related semantics
Have access to the Avi Controller
Manage the lifecycle of AKO

Developers:
Users of this category:

Are owners of microservices deployed in Kubernetes
Are assumed to know basic routing principles but may not know the specifics of Avi attributes

Advantages of CRDs

Some load balancers allow configuration options via annotations.

The following are the advantages of using CRDs:

Versioning: CRDs allow AKO to version fields appropriately due to the dependency on the Avi Controller Versions. In general, this helps preserve unique states across various deployment versions.
Syntactical Validations: CRDs can be used to verify syntax at the time of creation of the CRD object. This saves API cost and facilitates quicker feedback using a combination of field constraints and effective status messages.
Role segregation: CRDs can benefit from the Role-based access control (RBAC) policies of Kubernetes and allow stricter access to a group of users.

Types of CRDs Supported in AKO

AKO categorizes the CRDs into:

Layer 4: These CRD objects are used to express layer 4 traffic routing rules.
Layer 7: These CRD objects are used to express layer 7 traffic routing rules.
Infrastructure: These CRD objects are used to control Avi’s infrastructure components like Ingress class, SE group properties etc.

Layer 7 CRDs

The following are the Layer 7 CRDs currently available in AKO:

HostRule
HTTPRule
SSORule

HostRule

The Operator is the primary users of the HostRule CRD. The CRD is used to express additional virtual host properties. The virtual host FQDN is matched from either the Kubernetes ingress or OpenShift route-based objects.

A sample HostRule CRD looks as shown below:


apiVersion: ako.vmware.com/v1alpha1
kind: HostRule
metadata:
  name: my-host-rule
  namespace: red
spec:
  virtualhost:
    fqdn: foo.region1.com # mandatory
    fqdnType: Exact
    enableVirtualHost: true
    tls: # optional
      sslKeyCertificate:
        name: avi-ssl-key-cert
        type: ref
        alternateCertificate:
          name: avi-ssl-key-cert2
          type: ref
      sslProfile: avi-ssl-profile
      termination: edge
    gslb:
      fqdn: foo.com
      includeAliases: false
    httpPolicy: 
      policySets:
      - avi-secure-policy-ref
      overwrite: false
    datascripts:
    - avi-datascript-redirect-app1
    wafPolicy: avi-waf-policy
    applicationProfile: avi-app-ref
    icapProfile: 
    - avi-icap-ref
    analyticsProfile: avi-analytics-ref
    errorPageProfile: avi-errorpage-ref
    analyticsPolicy: # optional
      fullClientLogs:
        enabled: true
        throttle: HIGH
      logAllHeaders: true
    tcpSettings:
      listeners:
      - port: 8081
      - port: 6443
        enableSSL: true
      loadBalancerIP: 10.10.10.1
    aliases: # optional
    -  bar.com
    -  baz.com

Usage of HostRule

HostRule CRD can be created in a given namespace where the operator requires better control. This section explains the details and associated rules of using each field of the HostRule CRD.

HostRule to Virtual Service Matching Using FQDN/FQDNType

A given HostRule is applied to a virtual service if the virtual service hosts the FQDN mentioned in the HostRule CRD. This FQDN must exactly match the one the virtual service is hosting. However, to simplify the user experience and provide an easy way to apply a HostRule to an individual or multiple virtual services, the FQDNType field can be employed. It provides great flexibility when it comes to specifying the string in the FQDN field.

With the FQDN Type, one of the three matching criteria listed below can be specified:

Exact: Matches the string character by character to the virtual service FQDNs in an exact match fashion.
Wildcard: Matches the string to multiple virtual service FQDNs and matches the FQDNs with the provided string as the suffix. The string must start with a * to qualify for wildcard matching.
```
    fqdn: *.alb.vmware.com
    fqdnType: Wildcard
  
```
Contains: Matches the string to multiple virtual service FQDNs and matches the FQDNs with the provided string as a substring of any possible FQDNs programmed by AKO.
```
      fqdn: Shared-VS-L7-1
    fqdnType: Contains
  
```

The fqdnType field defaults to Exact.

Enable/Disable Virtual Host

HostRule CRD can be used to enable/disable corresponding virtual services created by AKO on Avi. This removes any virtual host related configuration from the data plane (Avi Service Engines) in addition to disabling traffic on the virtual host/FQDN.

enableVirtualHost: false

This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The default value is true.

Express httppolicyset Object Refs

HostRule CRD can be used to express httppolicyset references.

Note: The httppolicyset objects should be pre-created in the Avi Controller.


     httpPolicy: 
       policySets:
       - "avi-secure-policy-ref"
       overwrite: false

The httppolicyset currently is only applicable for secure FQDNs and cannot be applied for insecure routes. The order of evaluation of the httppolicyset rules is in the same order they appear in the CRD definition. The list of httppolicyset rules is always intepreted as an AND operation.

AKO currently uses httppolicyset objects on the SNI virtual services to route traffic based on host/path matches. These rules are always at a lower index than the httppolicyset objects specified in the CRD object. In case a user would want to overwrite all httppolicyset objects on a SNI virtual service with the ones specified in the HostRule CRD, the overwrite flag can be set to true. The default value for overwrite is false.

Express WAF Policy Object Refs

HostRule CRD can be used to express WAF policy references. Create the WAF policy object in the Avi Controller prior to the CRD creation as follows:

wafPolicy: "avi-waf-policy"

Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. WAF policies are useful when deep layer 7 packet filtering is required.

Express Custom Application Profiles

HostRule CRD can be used to express application profile references. Create the application profile reference in the Avi Controller prior to the CRD creation. The application profile should be of Type APPLICATION_PROFILE_TYPE_HTTP.

applicationProfile: "my-app-ref"

Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The application profiles can be used for various HTTP/HTTP2 protocol settings.

Express Custom ICAP profile

HostRule CRD can be used to express a single ICAP profile reference per host. The ICAP profile reference should have been created in the Avi Controller prior to this CRD creation.
icapProfile: - avi-icap-ref

This property can be applied for both secure and insecure hosts through EVH parent and child Virtual Services, SNI child Virtual Services and dedicated virtual services. The ICAP Profile can be used for transporting HTTP traffic to 3rd party services for processes such as content sanitization and antivirus scanning.

Express Custom Analytics Profiles

HostRule CRD is used to express analytics profile references.

Create the analytics profile reference in the Avi Controller prior to this CRD creation.

analyticsProfile: avi-analytics-ref`

Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The analytics profiles can be used for various Network/HTTP/Health score analytics settings, log processing, etc.

Express Custom Error Page Profiles

HostRule CRD can be used to express error page profile references.
Create the error page profile reference in the Avi Controller prior to this CRD creation.

errorPageProfile: avi-errorpage-ref

Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The error page profiles can be used to send a custom error page to the client generated by the proxy.

Express DataScripts

HostRule CRD can be used to express error DataScript references.
Create the DataScript references in the Avi Controller prior to this CRD creation.


    datascripts:
    - avi-datascript-redirect-app1

Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The DataScripts can be used to apply custom scripts to data traffic. The order of evaluation of the DataScripts is in the same order they appear in the CRD definition.

Express TLS Configuration

For the Avi Kubernetes operator to control the TLS termination from a privileged namespace, the HostRule CRD can be created in such a namespace.


    tls:
      sslKeyCertificate:
        name: avi-ssl-key-cert
        type: ref
        alternateCertificate:
          name: avi-ssl-key-cert2
          type: ref
      sslProfile: avi-ssl-profile
      termination: edge

Here, name refers to an Avi object with the Type as ref. Alternatively, we also support a kubernetes Secret to be specified where the sslkeyandcertificate object is created by AKO using the Secret.


    tls:
      sslKeyCertificate:
        name: k8s-app-secret
        type: secret
      termination: edge

Note: Currently, only Edge is supported as the type of termination.

An alternateCertificate option is provided in case the application needs to be configured to provide multiple server certificates, typically when trying to configure both RSA and ECC signed certificates. The NSX Advanced Load Balancer Controller allows a virtual service to be configured with two certificates at a time, one each of RSA and ECC. This enables Avi Controller to negotiate the optimal algorithm or cipher with the client. If the client supports ECC, in that case the ECC algorithm is preferred, and RSA is used as a fallback in cases where the clients do not support ECC.

The sslProfile, additionally, can be used to determine the set of SSL versions and ciphers to accept for SSL/TLS terminated connections. If the sslProfile is not defined, AKO defaults to the sslProfile System-Standard-PFS defined in NSX Advanced Load Balancer.

Configure GSLB FQDN

A GSLB FQDN can be specified within the HostRule CRD. This is only used if AKO is used with AMKO and not otherwise.


 gslb:
      fqdn: foo.com

This additional FQDN inherits all the properties of the root FQDN specified under the the virtualHost section. Use this flag if you would want traffic with a GSLB FQDN to get routed to a site local FQDN. For example, in the above CRD, the client request from a GSLB DNS will arrive with the host header as foo.com to the VIP hosting foo.region1.com in region1. This CRD property would ensure that the request is routed appropriately to the backend service of foo.region1.com

This knob is currently supported only with the SNI model and not with Enhanced Virtual Hosting model.

The includeAliases is used by AMKO. Whenever a GSLB FQDN is provided, and the useCustomGlobalFqdn is set to true in AMKO, a GSLB Service is created for the GSLB FQDN instead of the local FQDN(hostname). For more information, see Deriving GslbService FQDNs.

When this flag is set to false, the Domain Name of the GSLB Service is set to the GSLB FQDN.

When this flag is set to true in addition to the GSLB FQDN, AMKO adds the FQDNs mentioned under aliases to domain names of the GSLB Service.

Configure Aliases for FQDN

The Aliases field adds the ability to have multiple FQDNs configured under a specific route/ingress for the child VS instead of creating the route/ingress multiple times.


aliases:
    - bar.com
    - baz.com

This list of FQDNs inherits all the properties of the root FQDN specified under the virtualHost section. Traffic arrives with the host header as bar.com to the VIP hosting foo.region1.com and this CRD property would ensure that the request is routed appropriately to the backend service of foo.region1.com.

Aliases field must contain unique FQDNs and must not contain GSLB FQDN or the root FQDN. Users must ensure that the fqdnType is set as Exact before setting this field.

Configure Analytics Policy

The HostRule CRD can be used to configure analytics policies such as enable/disable non-significant logs, throttle the number of non-significant logs per second on each SE, enable/disable logging of all headers, etc.:


      analyticsPolicy:
        fullClientLogs:
          enabled: true
          throttle: HIGH
        logAllHeaders: true

The throttle will be in effect only when enabled is set to true. The possible values of throttle are DISABLED (0), LOW (50), MEDIUM (30) and HIGH (10).

AKO sets the duration of logging the non-significant logs to infinity by default.
It is recommended to disable the non-significant logs when it is no longer required.

Configure TCP Settings

The TCP Settings section is responsible for configuring Parent virtual service specific parameters using the HostRule CRD. The tcpSettings block, in addition to any other parameters provided in the HostRule, is only applied to Parent VSes and dedicated VSes. The tcpSettings block does not have any effect on child VSes.

In order to consume TCP setting configurations for parent VSes, the HostRule must be matched to a Shared/Dedicated VS FQDN, using the existing fqdn field in HostRule. Where dedicated virtual services are created corresponding to a single application, Shared virtual services host multiple application FQDNs. Therefore, in order to apply a HostRule to a dedicated VS, users can simply provide the application FQDN in the HostRule fqdn field. For Shared virtual services however, users can either provide the AKO programmed shared virtual service FQDN, or utilize the fqdnType: Contains parameter with the Shared virtual service name itself.


    fqdn: foo.com     # dedicated VS
    fqdnType: Exact
    tcpSetting:
      listeners:
      - port: 6443
        enableSSL: true


    fqdn: Shared-VS-L7-1.admin.avi.com    # AKO configured Shared VS fqdn
    fqdnType: Exact
    tcpSetting:
      loadBalancerIP: 10.10.10.1


    fqdn: Shared-VS-L7-1      # bound for clusterName--Shared-VS-L7-1
    fqdnType: Contains
    tcpSetting:
      loadBalancerIP: 10.10.10.1

Custom Ports:
In order to overwrite the ports opened for the virtual services created by AKO, users can provide the port details under the listeners setting. The ports mentioned under this section overwrites the default open ports, 80 and 443 (SSL enabled). This is applicable only for Shared or Dedicated virtual services.


    
      listeners:
      - port: 80
      - port: 8081
      - port: 6443
        enableSSL: true

Note: At least one of the ports that are mentioned in the setting must have the enableSSL field set to true.

L7 Static IP
The loadBalancerIP field can be used to provide a valid, preferred IPv4 address for L7 virtual services created for the Shared or Dedicated virtual service. The preferred IP must be part of the IPAM configured for the Cloud, and must not overlap with any other IP addresses already in use. In case of misconfigurations, AKO fails to configure the virtual service appropriately throwing and ERROR log for the same.


    tcpSettings:
      loadBalancerIP: 10.10.10.199

Note: The HostRule CRD is not aware of the misconfigurations while it is being created, therefore the HostRule will be Accepted nonetheless.

Status Messages

The status messages are used to give instant feedback about the reference objects specified in the HostRule CRD.

Following are some of the sample status messages:

Accepted HostRule Object


$ kubectl get hr
NAME                 HOST                  STATUS     AGE
secure-waf-policy    foo.avi.internal      Accepted   3d3h

A HostRule is accepted only when all the reference objects specified inside it exist in the Avi Controller.

A Rejected HostRule Object


$ kubectl get hr
NAME                     HOST                  STATUS     AGE
secure-waf-policy-alt    foo.avi.internal      Rejected   2d23h

The reason for rejection can be obtained from the status:


status:
error: duplicate fqdn foo.avi.internal found in default/secure-waf-policy-alt
status: Rejected

Caveats

Converting Insecure FQDNs to Secure

The HostRule CRD can be used to convert an insecure host FQDN to a secure one. This is done by specifying a TLS section in the CRD object. The sslKeyCertificate is provided for the FQDN, will override all sslkeyandcertificates generated for the FQDN. This is useful if:

The operator wants to convert an insecure ingress FQDN to secure.
The operator wants to override any existing secrets for a given host FQDN and define TLS termination semantics.

Certificate Precedence

If the ingress object specifies a Secret for SNI termination and the HostRule CRD also specifies a sslKeyCertificate for the same virtualhost, then the sslkeycertificate in the HostRule CRD will take precedence over the Secret object associated with the Ingress.

HostRule Deletion

If a HostRule is deleted, all the settings for the FQDNs are withdrawn from the Avi controller.

HostRule Admission

A HostRule CRD is only admitted if all the objects referenced in it, exist in the Avi Controller. If after admission the object references are deleted out-of-band, then AKO does not re-validate the associated HostRule CRD objects. The user needs to manually edit or delete the object for new changes to take effect.

Duplicate FQDN rules

Two HostRule CRDs cannot be used for the same FQDN information across namespaces. If AKO finds a duplicate FQDN in more than one HostRules, AKO honors the first HostRule that gets created and rejects the others. In case of AKO reboots, the CRD that gets honored might not be the same as the one honored earlier.

HTTP Rule

The developers are the primary users of the HTTPRule CRD. The path matching rules in the ingress or route objects define traffic routing rules to the microservices. The HTTPRule CRD can be used as a complimentary object to control additional layer 7 properties like algorithm, hash, and tls re-encrypt use cases.

A sample HTTPRule object is as shown below:


apiVersion: ako.vmware.com/v1alpha1
kind: HTTPRule
metadata:
   name: my-http-rule
   namespace: purple-l7
spec:
  fqdn: foo.avi.internal
  paths:
  - target: /foo
    healthMonitors:
    - my-health-monitor-1
    - my-health-monitor-2
    loadBalancerPolicy:
      algorithm: LB_ALGORITHM_CONSISTENT_HASH
      hash: LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS
    tls: ## This is a re-encrypt to pool
      type: reencrypt # Mandatory [re-encrypt]
      sslProfile: avi-ssl-profile
      destinationCA:  |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----

Note: The HTTPRule only applies to paths in the Ingress/Route objects which are specified in the same namespace as the HTTPRule CRD.

Usage of the HTTPRule CRD

The HTTPRule CRD does not have any Avi specific semantics. Hence, you are free to express your preferences using this CRD without any knowledge of the Avi objects. Each HTTPRule CRD must be bound to a FQDN (both secure or insecure) to subscribe to rules for a specific hostpath combinations.

Express Load Balancer Algorithm

The load balancer policies are a predefined set of values to choose from.
Currently, the following values are supported for load balancer policy:

LB_ALGORITHM_CONSISTENT_HASH
LB_ALGORITHM_CORE_AFFINITY
LB_ALGORITHM_FASTEST_RESPONSE
LB_ALGORITHM_FEWEST_SERVERS
LB_ALGORITHM_LEAST_CONNECTIONS
LB_ALGORITHM_LEAST_LOAD
LB_ALGORITHM_ROUND_ROBIN

To configure the load balancer policy for a given ingress path,


  - target: /foo 
    loadBalancerPolicy:
      algorithm: LB_ALGORITHM_FEWEST_SERVERS

This rule is applied all paths matching /foo and subsets of /foo/xxx.

To know more, refer to the Load Balancing Algorithm article.

The hash field is used when the algorithm is selected as LB_ALGORITHM_CONSISTENT_HASH. Otherwise, it is not applicable. Similarly, a hostHeader field is used only when the hash is selected as LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER.

A sample setting with these fields is shown below:


  - target: /foo 
    loadBalancerPolicy:
      algorithm: LB_ALGORITHM_CONSISTENT_HASH
      hash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      hostHeader: foo

The hostHeader is disregarded if it is specified in any other case. The hash field is disregarded if the algorithm is not LB_ALGORITHM_CONSISTENT_HASH.

Express Application Persistence Profile

HTTPRule CRD can be used to express application persistence profile references. Create the application persistence profile reference in the Avi Controller prior to this CRD creation.


  applicationPersistence: cookie-userid-persistence

The application persistence profile can be used to maintain stickiness to a server instance based on cookie values, headers, and so on for a desired duration of time.

Express Health Monitors

The HTTPRule CRD can be used to express health monitor references.
Create the health monitor reference in the Avi Controller prior to this CRD creation.

To express health monitor references, use:


healthMonitors:
  - my-health-monitor-1
  - my-health-monitor-2

The health monitors can be used to verify server health. A server (Kubernetes pods in this case) is marked as UP only when all the health monitors return successful responses. Health monitors provided here overwrite the default health monitor configuration set by AKO, that is, System-TCP for HTTP/TCP traffic and System-UDP for UDP traffic based on the ingress/service configuration.

Reencrypt Traffic to the Services

While AKO can terminate TLS traffic, it also provides an option where the users can choose to re-encrypt the traffic between the Avi SE and the backend application server. The following option is provided for reencrypt one is by providing a raw certificate using destinationCA or by providing an Avi PKI Profile reference using the pkiProfile field:


    tls: ## This is a re-encrypt to pool
      type: reencrypt # Mandatory [re-encrypt]
      sslProfile: avi-ssl-profile
      destinationCA:  |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----
    tls: ## This is a re-encrypt to pool
      type: reencrypt # Mandatory [re-encrypt]
      sslProfile: avi-ssl-profile
      pkiProfile: avi-pki-profile

In case of reencrypt, if the destinationCA is specified in the HTTP Rule CRD, as shown in the example, a corresponding PKI profile is created for that pool (host path combination). Also Note that only one of PKI profile or destination CA can be provided to configure reencrypt for a pool corresponding to the host path backend Service.

Status Messages

The status messages are used to give instant feedback on whether a HTTPRule CRD was accepted or rejected.

Example of a HTTP Rule


$ kubectl get httprule
NAME            HOSTRULE                     STATUS     AGE
my-http-rules   default/secure-waf-policy    Accepted   5h34m

Avi Infra Setting

Avi Infra Setting provides a way to segregate Layer-4/Layer-7 virtual services to have properties based on different underlying infrastructure components, like Service Engine Group, intended VIP Network etc.

A sample Avi Infra Setting is as shown below:


apiVersion: ako.vmware.com/v1alpha1
kind: AviInfraSetting
metadata:
  name: my-infra-setting
spec:
  seGroup:
    name: compact-se-group
  network:
    vipNetworks:
      - networkUUID: dvportgroup-2167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
        cidr: 10.10.10.0/24
        v6cidr: 2002::1234:abcd:ffff:c0a8:101/64
    nodeNetworks:
      - networkName: node-network-10-10-20-0-24
        cidrs:
        - 10.10.20.0/24
    enableRhi: true
    listeners:
      - port: 8081
        enableHTTP2: false
        enableSSL: false
      - port: 6443
        enableSSL: true
    bgpPeerLabels:
      - peer1
      - peer2
  l7Settings:
    shardSize: MEDIUM

Avi Infra Setting with Services, Ingress or Routes

Avi Infra Setting is a cluster scoped CRD and can be attached to the intended Services, Kubernetes Ingresses and OpenShift Routes as shown below.

Attaching Avi Infra Setting to Services

Avi Infra setting resources can be attached to Services using Gateway APIs or simply by using annotations.

Using Gateway API
Gateway APIs provide interfaces to structure Kubernetes service networking.

AKO supports Gateway APIs via the servicesAPI flag in the values.yaml. For more information on how AKO integrates with Gateway API, see Gateway and Gateway Class.

The Avi Infra Setting resource can be attached to a Gateway Class object, via the .spec.parametersRef as shown below:


apiVersion: networking.x-k8s.io/v1alpha1
kind: GatewayClass
metadata:
  name: avi-gateway-class
spec:
  controller: ako.vmware.com/avi-lb
  parametersRef:
    group: ako.vmware.com
    kind: AviInfraSetting
    name: my-infrasetting

Using Annotation
In case the servicesAPI flag is not set to True, and AKO is not watching over the Gateway APIs, Services of Type LoadBalancer can specify the Avi Infra Setting using an annotation as shown below:


 annotations:
    aviinfrasetting.ako.vmware.com/name: "my-infrasetting"

Attaching Avi Infra Setting to Ingress

Avi Infra Settings can be applied to Ingress resources, using the IngressClass construct. IngressClass provides a way to configure Controller-specific load balancing parameters and applies these configurations to a set of Ingress objects. AKO supports listening to IngressClass resources in Kubernetes version 1.19+. The Avi Infra Setting reference can be provided in the Ingress Class as shown below:


apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: avi-ingress-class
spec:
  controller: ako.vmware.com/avi-lb
  parameters:
    apiGroup: ako.vmware.com
    kind: AviInfraSetting
    name: my-infrasetting

Attaching Avi Infra Setting to Routes

Avi Infra Settings with OpenShift Routes, via the annotation shown below:


 annotations:
    aviinfrasetting.ako.vmware.com/name: "my-infrasetting"

Using Avi Infra Setting

Configure Service Engine Group

The AviInfraSetting CRD can be used to configure Service Engine Groups for virtual services created corresponding to Services/Ingresses/ Routes. Ensure the Service Engine Group is created and configured in the Avi Controller prior to this CRD creation.


  seGroup:
      name: compact-se-group

AKO tries to configure labels on the SEG specified in the AviInfraSetting resources, which enables static route syncing on the member SEs. AKO configures the labels only when the SEGs do not have any other labels configured already. In case AKO finds the SEG configured with some different labels, the AviInfraSetting resource would be Rejected.

Note: Once deployed, the member SEs will not reflect any changes related to label additions or updates on the SEGs.

Therefore, label based static route syncing will not work on already deployed SEs. Ensure that the SEGs have no member SEs deployed before specifying the SEG in the AviInfraSetting resource to properly configure static route syncing.

Configure VIP Networks

The AviInfraSetting CRD can be used to configure VIP networks for virtual services created corresponding to Services/Ingresses/Openshift Routes. Ensure the network is present in the Avi Controller prior to this CRD creation.

Starting with AKO version 1.10.3, networkUUID field has been introduced as part of vipNetworks setting to choose appropriate network. User can specify either networkName or networkUUID while specifying the network.


vipNetworks: - networkName: vip-network-10-10-10-0-24 cidr: 10.10.10.0/24


    network:
      vipNetworks:
        - networkUUID: dvportgroup-2167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
          cidr: 10.10.10.0/24

As shown in the examples, each unique network can be specified using either networkName or networkUUID with appropriate cidr.

Note:
If there are duplicate network entries at the Controller side, AKO recommends to use networkUUID while choosing vipNetwork for new aviinfrasetting object creation.

Note: Multiple network names can be added to the CRD (only in case of AWS cloud). The Avi virtualservices flag will acquire a VIP from each of these specified networks. If Avi fails to allocate a VIP due to IP exhaustion, this results in complete failure of entire request. This is same as vip allocation failures in single VIP.

AKO 1.5.1 updates the schema to provide VIP network information in AviInfraSetting CRD. For further details, refer to the Upgrade Notes.

Configure Pool Placement Networks

AviInfraSetting CRD can be used to configure Pool Placement Settings on the AKO created Pools in order for the Service Engines to place the Pools on appropriate interfaces.
With AKO 1.10.3, networkUUID field has been introduced as part of nodeNetworks setting to choose appropriate network. User can specify either networkName or networkUUID while specifying the network.


        network:
          nodeNetworks:
            - networkName: node-network-10-10-20-0-24
              cidrs:
              - 10.10.20.0/24


        network:
          nodeNetworks:
            - networUUID: dvportgroup-4167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
              cidrs:
              - 10.10.20.0/24

As shown in the examples, each unique network can be specified using either networkName or networkUUID with appropriate cidrs.

Note: If there are duplicate network entries at the Controller side, AKO recommends to use networkUUID while choosing nodeNetworks for new aviinfrasetting object creation.

If two Kubernetes clusters have overlapping Pod CIDRs, the service engine needs to identify the right gateway for each of the overlapping CIDR groups. This is achieved by specifying the right placement network for the pools that helps the Service Engine place the pools appropriately.

Enable/Disable Route Health Injection

Avi Infra Setting CRD is used to enable/disable Route Health Injection (RHI) on the virtual services created by AKO.


network:
      enableRhi: true

This overrides the global enableRHI flag for the virtual services corresponding to the Avi Infra Setting.

Enable/Disable Public IP

The AviInfraSetting CRD can be used to enable/disable Public IP on the virtual services created by AKO.


    network:
      enablePublicIP: true

Note: Enabling Public IP is only supported for public clouds.

Custom Ports:
To customize the ports opened for L7 Shared or dedicated virtual services created by AKO, you can provide the port details under the listeners setting. The ports mentioned under this section will be added to the virtual service along with the default open ports, 80 and 443 along with the ports mentioned in the HostRule. The settings mentioned in the avi infrasetting will always take precedence over default and hostrule properties.


      listeners:
      - port: 80
        enableHTTP2: false
        enableSSL: false
      - port: 6443
        enableSSL: true

Configure BGP Peer Labels for BGP Virtual Services

AviInfraSetting CRD can be used to configure BGP Peer labels for BGP virtual services. AKO configures the virtual sevices with the appropriate peer labels, only when enableRHI is enabled, either during AKO installation via helm chart’s values.yaml or via the AviInfraSetting CRD itself. If not set to true, the AviInfraSetting resource will be marked Rejected.


    bgpPeerLabels:
      - peer1
      - peer2

Use Dedicated VIP for Ingress

AviInfraSetting CRD can be used to allocate a dedicated vip per Ingress FQDN.


    l7Settings:
      shardSize: DEDICATED

For the subset of ingresses, that refer to an ingress class which in turn refers to an AviInfraSetting CRD setting that has shardSize as DEDICATED, will get VIP per Ingress FQDN.

For passthrough routes/ingresses, setting l7Settings:shardSize present in AviInfrasetting CRD overrides setting L7Settings.passthroughShardSize present in values.yaml.

Note: The value DEDICATED is not supported when AviInfra CRD is applied to the passthrough route/ingress.

Configure IPv6

Avi Infra Setting CRD can be used to enable IPv6, IPv4 or both IPv4 and IPv6 VIPs on the virtual services created by AKO.
Note: This feature is currently under Tech-Preview.


    network:
      vipNetworks:
        - networkName: vip-network-10-10-10-0-24
          cidr: 10.10.10.0/24
          v6cidr: 2002::1234:abcd:ffff:c0a8:101/64

SSORule CRD

SSORule CRD can be used to configure OAuth or SAML specific properties for L7 virtual services that are created by the AKO from Kubernetes Ingress and Openshift Route objects. The SSORule CRD specifies an fqdn field, which is used to attach the SSORule to a virtual service. A given SSORule is applied to a virtual service if the virtual service hosts the fqdn specified in the SSORule CRD object.

The SSORule CRD can be used to configure either OAuth/OIDC properties or SAML service provider properties, along with the SSO policy. A sample SSORule CRD object with OAuth/OIDC properties looks like this:


apiVersion: ako.vmware.com/v1alpha2
kind: SSORule
metadata:
  name: my-ssorule
  namespace: default
spec:
  fqdn: my-ssorule.test.com
  oauthVsConfig:
    cookieName: OAUTH_XYZ
    cookieTimeout: 120
    logoutURI: https://my-ssorule.test.com/oauth/logout
    oauthSettings:
    - appSettings:
        clientID: my-client-id
        clientSecret: my-oauth-secret
        oidcConfig:
          oidcEnable: true
          profile: true
          userinfo: true
        scopes: ["scope1"]
      authProfileRef: okta-oauth
      resourceServer:
        accessType: ACCESS_TOKEN_TYPE_OPAQUE
        introspectionDataTimeout: 60
        opaqueTokenParams:
          serverID: my-server-id
          serverSecret: my-oauth-secret
    redirectURI: https://my-ssorule.test.com/oauth/callback
    postLogoutRedirectURI: https://my-ssorule.test.com/oauth/postLogoutRedirectURI
  ssoPolicyRef: oauth

A sample SSORule CRD object with SAML properties looks like this:


apiVersion: ako.vmware.com/v1alpha2
kind: SSORule
metadata:
  name: my-ssorule
  namespace: default
spec:
  fqdn: my-ssorule.test.com
  samlSpConfig:
    authnReqAcsType: SAML_AUTHN_REQ_ACS_TYPE_NONE
    cookieName: saml-cookie
    cookieTimeout: 120
    entityID: my-ssorule.test.com
    singleSignonURL: https://my-ssorule.test.com/sso/acs/
    signingSslKeyAndCertificateRef: my-test-app-secret
    useIdpSessionTimeout: false
  ssoPolicyRef: saml-app

NOTE: The SSORule CRD is supported only for Avi Controller version 22.1.3 and above. It only supports the configuration of OAuth and SAML SSO protocols currently. Also, the SSORule CRD is only supported for Enhanced Virtual Hosting (EVH). When the shard virtual service size is LARGE or MEDIUM or SMALL, the OAuth and SAML settings will only be configured on the EVH child virtual services.

Specific Usage of SSORule CRD

SSORule CRD can be created in a given namespace where the operator desires to have more control. The section below walks over the details and associated rules for using each field of the SSORule CRD.

SSORule to VS Matching Using FQDN

A given SSORule is applied to a virtual service if the VS hosts the fqdn mentioned in the SSORule CRD. This FQDN must exactly match the one the virtual service is hosting.

fqdn: my-ssorule.test.com

Express OAuth/OIDC Configuration for Virtual Service

The SSORule CRD can be used to configure the OAuth/OIDC properties of an L7 virtual service. The OAuth/OIDC configuration needs to be specified in the oauthVsConfig field.


    oauthVsConfig:
    cookieName: OAUTH_XYZ
    cookieTimeout: 120
    logoutURI: https://my-ssorule.test.com/oauth/logout
    oauthSettings:
    - appSettings:
        clientID: my-client-id
        clientSecret: my-oauth-secret
        oidcConfig:
          oidcEnable: true
          profile: true
          userinfo: true
        scopes: ["scope1"]
      authProfileRef: okta-oauth
      resourceServer:
        accessType: ACCESS_TOKEN_TYPE_OPAQUE
        introspectionDataTimeout: 60
        opaqueTokenParams:
          serverID: my-server-id
          serverSecret: my-oauth-secret
    redirectURI: https://my-ssorule.test.com/oauth/callback
    postLogoutRedirectURI: https://my-ssorule.test.com/oauth/postLogoutRedirectURI

The properties under the oauthVsConfig field are discussed in detail below:

The cookieName property can be used to express the name of HTTP cookie for an authorized session. If this field is not specified, NSX Advanced Load Balancer will create the authorized session cookie with a random name.

cookieName: OAUTH_XYZ

The cookieTimeout property can be used to specify the HTTP cookie timeout in minutes for an authorized session. It supports values from 1 to 1440 and defaults to 60.

cookieTimeout: 120

Express Logout URI for OAuth

The logoutURI property can be used to express the URI that will trigger OAuth logout.

logoutURI: https://my-ssorule.test.com/oauth/logout

Express Application and IDP settings for OAuth/OIDC

The oauthSettings property can be used to express the application specific OAuth configuration and Identity Provider (IDP) settings for OAuth/OIDC.
The application specific properties are specified as a list of appSettings properties.

Express Client ID

The clientID property can be used to express the client ID for the application. It is an application specific identifier registered with the authorization server or Identity Provider (IDP).

clientID: my-client-id

Express Client Secret

The clientSecret field can be used to express the client secret for the application. It is an application specific identifier secret registered with the authorization server, or IDP. Since clientSecret is a sensitive field in NSX Advanced Load Balancer, AKO requires it to be specified inside a Kubernetes Secret object. So, this clientSecret field should be the name of the Kubernetes secret object that specifies the actual client secret value (Base64 encoded) in the clientSecret data field. This Kubernetes secret object should be created in the same namespace as the SSORule.

clientSecret: my-oauth-secret

A sample Kubernetes secret object, with the actual value for client and secret (Base64 encoded) in the clientSecret data field is shown below:


  apiVersion: v1
  data:
    clientSecret: bXktY2xpZW50LXNlY3JldA==
    serverSecret: bXktc2VydmVyLXNlY3JldA==
  kind: Secret
  metadata:
    name: my-oauth-secret
  type: Opaque

Express OpenID Connect Configuration

The oidcConfig property can be used to express OpenID Connect specific configuration.


  oidcConfig:
    oidcEnable: true
    profile: true
    userinfo: true

oidcEnable: If set to true, adds OpenID as one of the scopes enabling the OpenID Connect flow.
profile: If set to true, will allow fetching profile information by enabling profile scope. It is set to true by default.
userinfo: If set to true, will allow fetching profile information from the Userinfo Endpoint.

Express Scope for OAuth

The scopes property can be used to express the scope for OAuth to give limited access to the application.

scopes: ["scope1"]

Express Authentication Profile for OAuth

SSORule CRD can be used to express the authentication profile reference for OAuth. The authentication profile must be created in the AVI Controller before referring to it.

authProfileRef: okta-oauth

The Auth Profile should specify all the endpoints and configurations associated with the Identity Provider (IDP) and will be used for validating users.

Express Resource Server OAuth Configuration

The accessType property can be used to express the access token type. The access token type can either be opaque or JWT.

accessType: ACCESS_TOKEN_TYPE_OPAQUE
Or
accessType: ACCESS_TOKEN_TYPE_JWT

Express Introspection Data Timeout

The introspectionDataTimeout property can be used to set the lifetime of the cached introspection data in minutes. It supports values from Zero (0) to 1440 and defaults to Zero (0). However, it will be set only if the access token type is opaque.

introspectionDataTimeout: 60

Express Opaque Token Parameters

The opaqueTokenParams property should be specified if the accessType property is ACCESS_TOKEN_TYPE_OPAQUE. It can be used to express the validation parameters to be used when the access token type is opaque. This Kubernetes secret object should be created in the same namespace as the SSORule.
opaqueTokenParams: serverID: my-server-id serverSecret: my-oauth-secret

The serverID property can be used to express the server ID for the resource server. It is the resource server specific identifier registered with the authorization server or Identity Provider(IDP), and is used to validate against the introspection endpoint when the access token type is opaque.

The serverSecret field can be used to express the server secret for the resource server. It is a resource server specific identifier secret registered with the authorization server, or IDP. Since serverSecret is a sensitive field in NSX Advanced Load Balancer, AKO requires it to be specified inside a Kubernetes Secret object. So, this serverSecret field should be the name of the Kubernetes secret object that specifies the actual server secret value (Base64 encoded) in the serverSecret data field. This Kubernetes secret object should be created in the same namespace as the SSORule. The server and client secrets can be specified in the same or different Kubernetes objects, as already shown in Express Client Secret.

Express Redirect URI for OAuth

SSORule CRD can be used to express the redirect URI that is specified in the request to the authorization server or Identity Provider (IDP). The redirect URI is the callback entry point of the application where the authorization server sends the user once the application has been successfully authorized and granted an authorization code.

redirectURI: https://my-ssorule.test.com/oauth/callback

Express Post Logout Redirect URI for OAuth

SSORule CRD can be used to express the post-logout redirect URI for the application. It is the URI to which the Identity Provider (IDP) will redirect after application logs out of the IDP.

postLogoutRedirectURI: https://my-ssorule.test.com/oauth/postLogoutRedirectURI

Express Application Specific SAML configuration

SSORule CRD can be used to express the application specific configuration, i.e., SAML service provider configuration for the L7 virtual service. The SAML service provider configuration needs to be specified in the samlSpConfig field.


  samlSpConfig:
    acsIndex: 64
    authnReqAcsType: SAML_AUTHN_REQ_ACS_TYPE_INDEX
    cookieName: saml-cookie
    cookieTimeout: 120
    entityID: my-ssorule.test.com
    singleSignonURL: https://my-ssorule.test.com/sso/acs/
    signingSslKeyAndCertificateRef: my-test-app-secret
    useIdpSessionTimeout: false

The properties under the samlSpConfig field are discussed in detail below.

Express Assertion Consumer Service Index

The acsIndex property can be used to express the index to be used in the AssertionConsumerServiceIndex attribute of the authentication request. It will be set only if the authnReqAcsType is set to SAML_AUTHN_REQ_ACS_TYPE_INDEX. It supports values from Zero (0) to 64.
acsIndex: 64

Express Assertion Consumer Service Type for Authentication Request

The authnReqAcsType property can be used to express the assertion consumer service type for authentication requests. It will determine the ACS attributes that will be set in the authentication request. It supports the following three values:

Value	Description
SAML_AUTHN_REQ_ACS_TYPE_NONE	No ACS attributes will be set in the SAML authentication request
SAML_AUTHN_REQ_ACS_TYPE_URL	The `AssertionConsumerServiceURL attribute` will be set in the SAML authentication request. The ACS URL should be equal to the single sign on URL set for the virtual service
SAML_AUTHN_REQ_ACS_TYPE_INDEX	The `AssertionConsumerServiceIndex` attribute of the SAML authentication request will be set with the value specified in the acsIndex property

The cookieName property can be used to express the name of HTTP cookie for an authenticated session. If this field is not specified, NSX Advanced Load Balancer will create the authenticated session cookie with a random name.
cookieName: saml-cookie

The cookieTimeout property can be used to specify the timeout for HTTP cookie in minutes for an authenticated session. It supports values from 1 to 1440 and defaults to 60.

cookieTimeout: 120

Express Entity IP for SAML Service Provider Application

The entityID property can be used to express the globally unique entity ID for the SAML Service Provider application. The SAML application entity ID on the Identity Provider (IDP) should match this.

entityID: my-ssorule.test.com

Express SslKeyAndCertificateRef for SAML Service Provider Application

The signingSslKeyAndCertificateRef property can be used to express SslKeyAndCertificate reference for the SAML Service Provider application. The SslKeyAndCertificate must be created in the NSX Advanced Load Balancer Controller before referring to it. The service provider will use this SSL certificate to sign requests going to the Identity Provider (IDP) and decrypt the assertions coming from IDP.
signingSslKeyAndCertificateRef: my-test-app-secret

Express Single Sign on URL for SAML

The singleSignonURL property can be used to express the single sign on URL for the application. It specifies the endpoint to receive the authentication response and the destination endpoint to be configured for the application on the IDP. If the authnReqAcsType is set to SAML_AUTHN_REQ_ACS_TYPE_URL, this endpoint will be sent in the AssertionConsumerServiceURL attribute of the authentication request.

singleSignonURL: https://my-ssorule.test.com/sso/acs/

Express IDP Control for Service Provider Session Timeout

The useIdpSessionTimeout property can be used to enable the Identity Provider (IDP) to control the duration in which the Service Provider (SP) session can exist through the SessionNotOnOrAfter field in the AuthNStatement of the SAML Response.
useIdpSessionTimeout: false

Express SSO Policy for the Virtual Service

The SSORule CRD can be used to express the SSO policy reference for the virtual service. The SSO Policy must be created in the AVI Controller before referring to it. The SSO policy can be of type OAUTH or SAML, depending on the SSO protocol that needs to be configured for the virtual service.
ssoPolicyRef: my-sso-policy

NOTE: The oauthVsConfig and samlSpConfig are one of fields, and only one of them can be configured since a virtual service can specify either OAuth or SAML as the SSO protocol.

Status Messages

The status messages are used to give instantaneous feedback to the users about the reference objects specified in the SSORule CRD.

Following are some sample status messages:


$ kubectl get ssorule
NAME         STATUS     AGE
my-sso-rule   Accepted   3d5s

Accepted SSORule Object


$ kubectl get ssorule
NAME         STATUS     AGE
my-sso-rule   Accepted   3d5s

An SSORule is accepted only when all the reference objects specified inside it exist in the NSX Advanced Load Balancer Controller.

Rejected SSORule object


$ kubectl get ssorule
NAME            STATUS     AGE
my-sso-rule-alt  Rejected   2d23h

The detailed rejection reason can be obtained from the status:


  status:
    error: authprofile "okta-oauth" not found on controller
    status: Rejected

Caveats

SSORule is only supported for Enhanced Virtual Hosting (EVH)

SSORule CRD is only supported for Enhanced Virtual Hosting (EVH). The OAuth and SAML settings will only be configured on virtual services when EVH is enabled. When shard virtual service size is LARGE or MEDIUM or SMALL, the OAuth and SAML settings will only be configured on the EVH child virtual services.

SSORule may get rejected on OpensShift if `useDefaultSecretsOnly` is true in the AKO configuration

An SSORule, created on an OpenShift cluster to configure OAuth settings for a virtual service, may get rejected if useDefaultSecretsOnly is set to “true” in AKO configuration. This is because AKO will only handle default secrets in the namespace where AKO is installed.

SSORule deletion

If an SSORule is deleted, all the settings for the FQDNs are withdrawn from the Avi controller.

SSORule admission

An SSORule CRD is only admitted if all the objects referenced in it exist in the AVI Controller. If, after admission, the object references are deleted out-of-band, then AKO does not re-validate the associated SSORule CRD objects. The user needs to manually edit or delete the object, for new changes to take effect.

Duplicate FQDN rules

Two SSORule CRDs cannot be used for the same FQDN information across namespaces. If AKO finds a duplicate FQDN in more than one SSORule, AKO honours the first SSORule that gets created and rejects the others. In the case of AKO reboots, the CRD that gets honoured might not be the same as the one honoured earlier.

L4Rule CRD

L4Rule CRD is created in a given namespace where the operator desires to have more control. The topic discusses the details and associated rules of using each field of the L4Rule CRD.

L4 Rule CRD can be used to modify the default properties of the L4 virtual service and the pools created from a service of Type LoadBalancer. Service of type LoadBalancer has to be annotated with the name of the CRD to attach the CRD to the service.

A sample L4 Rule CRD is as shown below:


  apiVersion: ako.vmware.com/v1alpha2
  kind: L4Rule
  metadata:
    name: my-l4-rule
    namespace: green
  spec:
    analyticsProfileRef: Custom-Analytics-Profile
    analyticsPolicy:
      fullClientLogs:
        enabled: true
        duration: 0
        throttle: 30
    applicationProfileRef: Custom-L4-Application-Profile
    loadBalancerIP: "49.20.193.207"
    performanceLimits:
      maxConcurrentConnections: 105
      maxThroughput: 100
    networkProfileRef: Custom-Network-Profile
    networkSecurityPolicyRef: Custom-Network-Security-Policy
    securityPolicyRef: Custom-Security-Policy
    vsDatascriptRefs:
    - Custom-DS-01
    - Custom-DS-02
    backendProperties:
    - port: 80
      protocol: TCP
      enabled: true
      applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
      healthMonitorRefs:
      - Custom-HM-01
      - Custom-HM-02
      lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
      lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      lbAlgorithmConsistentHashHdr: "custom-string"
      sslProfileRef: Custom-SSL-Profile
      sslKeyAndCertificateRef: Custom-Key-And-Certificate
      pkiProfileRef: Custom-PKI-Profile
      analyticsPolicy:
        enableRealtimeMetrics: true
      minServersUp: 1

Note: The L4 Rule CRD must be configured in the same namespace as the service of type LoadBalancer.

Attaching L4Rule to LoadBalancer type of Services

An L4 Rule is applied to a virtual service and pool created from the LoadBalancer type of Services when the l4 rule is attached to the service. An L4Rule can be attached by annotating the service with the name of the L4Rule CRD with ako.vmware.com/l4rule as the key and name of the l4 rule crd as the value.


  metadata:
    annotations:
      ako.vmware.com/l4rule: <name-of-the-l4-rule-crd>

Consider the following example showing a service my-service of type LoadBalancer annotated with an L4Rule my-l4-rule.


  apiVersion: v1
  kind: Service
  metadata:
    name: my-service
    annotations:
      ako.vmware.com/l4rule: my-l4-rule
  spec:
    selector:
      app.kubernetes.io/name: MyApp
    ports:
      - protocol: TCP
        port: 80
        targetPort: 9376
    clusterIP: 10.0.171.239
    type: LoadBalancer

Express Custom Analytics Profiles

L4Rule CRD can be used to express analytics profile references. The analytics profile reference should have been created in the AVI Controller before the CRD creation.
analyticsProfile: Custom-Analytics-Profile The analytics profiles can be used for various Network/Health Score analytics settings, log processing, etc.

Configure Analytics Policy

The L4Rule CRD can be used to configure analytics policies such as enable/disable non-significant logs, throttle the number of non-significant logs per second on each SE, and the duration for which the system should capture the logs.


    analyticsPolicy:
      fullClientLogs:
        enabled: true
        duration: 0
        throttle: 30

Here, the field throttle is configured as shown below:

For throttle to take effect, the value of enabled has to be set to True.
Range: 0 to 65535
Default value: 10
To deactivate throttling, set the value of throttle to 0

By default, the AKO sets the duration of logging the non-significant logs to 30 minutes. Configure the duration as Zero (0) to capture the logs indefinitely.

Express Custom Application Profiles

L4 Rule CRD can be used to express application profile references. The application profile must be created in the NSX Advanced Load Balancer Controller before referring to it. The application profile should be of TYPE of APPLICATION_PROFILE_TYPE_L4.

applicationProfile: Custom-L4-Application-Profile

Currently, the application profile of type APPLICATION_PROFILE_TYPE_L4 is supported. The application profile can be used to enable PROXY Protocol, rate limit the connections from a client, etc.

Express Custom Load Balancer IP

The loadBalancerIP field can be used to provide a valid preferred IPv4 address for L4 virtual services. The preferred IP must be part of the IPAM configured for the cloud, and must not overlap with any other IP addresses already in use. In case of any misconfigurations whatsoever, AKO would fail to configure the virtual service appropriately throwing an ERROR log for the same.

loadBalancerIP: "49.20.193.207"

NOTE: The L4Rule CRD is not aware of any misconfigurations during its creation process, and as a result, the L4Rule will still be marked as Accepted.

Configure Performance Limits

The L4 Rule CRD can be used to configure the performance limit settings such as maximum concurrent client connections allowed, and maximum throughput per second for all clients allowed through the client side.


    performanceLimits:
      maxConcurrentConnections: 105
      maxThroughput: 100

The maxConcurrentConnections and maxThroughput supports values from 0 to 65535.

Express Custom Network Profile

The L4 Rule CRD can be used to express a custom network profile. The network profile can be used to configure either TCP/UDP proxy settings or TCP/UDP fast path settings. The network profile must be created in the AVI Controller before referring to it.

networkProfileRef: Custom-Network-Profile

The AKO defaults the network profile to System-TCP-Proxy.

NOTE: The network profile settings are dependent on the license configured in the NSX Advanced Load Balancer Controller.

Express Custom Network Security Policy

The L4 Rule CRD can be used to express a custom network security policy. The Network security policy can be configured with rules to allow/deny/rate limit connections from a single or group of IP addresses, etc.

networkSecurityPolicyRef: Custom-Network-Security-Policy

The Network Security Policy must be created in the NSX Advanced Load Balancer Controller before referring to it.

Express Custom Security Policy

The L4Rule CRD can be used to express a custom Security Policy. Security Policy is applied to the traffic of the virtual service, and it is used to specify various configuration information used to perform Distributed Denial of Service (DDoS) attacks detection and mitigation.

securityPolicyRef: Custom-Security-Policy

The Security Policy must be created in the AVI Controller before referring to it.

Express Custom DataScripts

The L4Rule CRD can be used to express DataScripts references. The DataScripts references should have been created in the AVI Controller before the CRD creation.


    vsDatascriptRefs:
    - Custom-DS-01
    - Custom-DS-02

The DataScripts can be used to apply custom scripts to data traffic. The order of evaluation of the DataScripts is in the same order they appear in the CRD definition.

Configure Backend Properties

The backendProperties section in the L4Rule can be used to configure pool settings such as custom health monitors, application persistence profiles, LB algorithms, etc. The L4Rule CRD identifies the pools based on the port and protocol, and AKO applies the configuration to it. AKO logs a warning if the port and protocol do not match the service’s port and protocol configurations.

A sample backendProperties looks like this:


    backendProperties:
    - port: 80
      protocol: TCP
      enabled: true
      applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
      healthMonitorRefs:
      - Custom-HM-01
      - Custom-HM-02
      lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
      lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      lbAlgorithmConsistentHashHdr: "custom-string"
      sslProfileRef: Custom-SSL-Profile
      sslKeyAndCertificateRef: Custom-Key-And-Certificate
      pkiProfileRef: Custom-PKI-Profile
      analyticsPolicy:
        enableRealtimeMetrics: true
      minServersUp: 1

Note: The fields port and protocol are mandatory and AKO uses these fields to identify the pool. The port and protocol must equal the service’s port and protocol.

AKO defaults the fields enabled to true and lbAlgorithm to LB_ALGORITHM_LEAST_CONNECTIONS.

Enable/ Disable Pool

The field enabled in the L4Rule can be used to enable/disable pools attached to an L4 virtual service. By default, the value of the field is true and the user has to set the value to false to disable the pool.

enabled: true # or false

Express Custom Application Persistence Profile

The L4Rule CRD can be used to express a custom Application Persistence Profile reference. The Application Persistence Profile reference should have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

applicationPersistenceProfileRef: Custom-Application-Persistence-Profile

Express Custom Health Monitors

L4Rule CRD can be used to express custom health monitor references. The health monitor reference should have been created in the NSX Advanced Load Balancer Controller before the CRD creation.


      healthMonitorRefs:
      - Custom-HM-01
      - Custom-HM-02

The health monitors can be used to verify server health. A server (Kubernetes pods in this case) will be marked UP only when all the health monitors return successful responses. Health monitors provided here overwrite the default health monitor configuration set by AKO i.e. System-TCP for TCP traffic and System-UDP for UDP traffic based on the service configuration.

Configure LB Algorithm

The L4Rule CRD can be used to select suitable LB algorithms to effectively distribute traffic across healthy servers. The LB algorithm may be used for distributing TCP and UDP connections across servers.

A sample LB algorithm configuration is shown below:


      lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
      lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      lbAlgorithmConsistentHashHdr: "custom-string"

The lbAlgorithm allows a predefined set of values, and the user can choose the desired one. Presently the following values are supported for the field lbAlgorithm:

LB_ALGORITHM_LEAST_CONNECTIONS
LB_ALGORITHM_ROUND_ROBIN
LB_ALGORITHM_FASTEST_RESPONSE
LB_ALGORITHM_CONSISTENT_HASH
LB_ALGORITHM_LEAST_LOAD
LB_ALGORITHM_FEWEST_SERVERS
LB_ALGORITHM_RANDOM
LB_ALGORITHM_FEWEST_TASKS
LB_ALGORITHM_NEAREST_SERVER
LB_ALGORITHM_CORE_AFFINITY
LB_ALGORITHM_TOPOLOGY

For more information on pool algorithms, see Load Balancing Algorithms.

The lbAlgorithmHash field is used only when the algorithm is chosen as LB_ALGORITHM_CONSISTENT_HASH. The following values are supported for lbAlgorithmHash:

LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS
LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS_AND_PORT
LB_ALGORITHM_CONSISTENT_HASH_URI
LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_STRING
LB_ALGORITHM_CONSISTENT_HASH_CALLID

The lbAlgorithmConsistentHashHdr field is used only when the lbAlgorithmHash is chosen as LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER.

Express Custom SSL Profile

The custom SSL profile can be used to configure the desired set of SSL versions and ciphers to accept SSL/TLS terminated connections.

sslProfileRef: Custom-SSL-Profile

The custom SSL profile should have been created in the AVI Controller before the CRD creation.

Express Custom SSL Key And Certificate

The L4Rule CRD can be used to express a custom SSL key and certificate reference. The service engines present this certificate to the backend servers. The custom SSL key and certificate should have been created in the NSX Advanced Controller before the CRD creation.
sslKeyAndCertificateRef: Custom-Key-And-Certificate

Express Custom PKI Profile

The L4Rule CRD can be used to express a custom PKI profile reference. Once configured, the AVI controller validates the SSL certificate present by a server against the custom PKI Profile configured in the CRD. The custom PKI Profile must be created in the AVI Controller before referring to it.
pkiProfileRef: Custom-PKI-Profile

Configure Analytics Policy

The L4Rule CRD can be used to configure the analytics settings for the pool. Set the enableRealtimeMetrics to true/false to enable/disable real-time metrics for server and pool metrics.


      analyticsPolicy:
        enableRealtimeMetrics: true # or false

Configure Minimum Servers UP

The L4Rule CRD can be used to configure the minimum number of servers in the UP state for marking the pool UP.

minServersUp: 1

Note: The value given must be equal to or less than the number of health monitors attached to the pool.

Status Messages

The status messages are used to give instantaneous feedback about the reference objects specified in the L4Rule CRD.

Following are some sample status messages:

Accepted L4Rule Object


$ kubectl get l4rule
NAME         STATUS     AGE
my-l4-rule   Accepted   3d5s

An L4Rule is accepted only when all the reference objects specified inside it exist in the AVI Controller.

Rejected L4Rule Object


$ kubectl get l4rule
NAME            STATUS     AGE
my-l4-rule-alt  Rejected   2d23h

The detailed reason for rejection can be obtained from the status:


status:
error: applicationprofile "My-L4-Application" not found on controller
status: Rejected

Caveats

The L4Rule CRD with load balancer IP can be shared among services only when the services contain the ako.vmware.com/enable-shared-vip annotation.

L4Rule Deletion

If an L4Rule is deleted, the L4 VSes and Pools in the AVI controller will be configured with the default values.

L4Rule Admission

An L4Rule CRD is only admitted if all the objects referenced in it, exist in the NSX Advanced Load Balancer Controller. If after admission the object references are deleted out-of-band, then AKO does not re-validate the associated HostRule CRD objects. The user needs to manually edit or delete the object for new changes to take effect.

Document Revision History

Date	Change Summary
September 26, 2022	Updated Aviinfrasetting support in passthrough virtual service
April 31, 2021	Added the sections for Configure VIP Networks, Enable/Disable Public IP, and Configure BGP Peer Labels for BGP Virtual Services supported in AKO version 1.5.1
April 28, 2021	Added the section for Avi Infra Setting CRD, supported in AKO version 1.4.1
December 18, 2020	Updated the CRDs supported in AKO version 1.3.1
July 22, 2020	Published the article for Custom Resource Definitions

Setting up Routing Rules using CRDs

Overview

Advantages of CRDs

Types of CRDs Supported in AKO

Layer 7 CRDs

HostRule

Usage of HostRule

HostRule to Virtual Service Matching Using FQDN/FQDNType

Enable/Disable Virtual Host

Express httppolicyset Object Refs

Express WAF Policy Object Refs

Express Custom Application Profiles

Express Custom ICAP profile

Express Custom Analytics Profiles

Express Custom Error Page Profiles

Express DataScripts

Express TLS Configuration

Configure GSLB FQDN

Configure Aliases for FQDN

Configure Analytics Policy

Configure TCP Settings

Status Messages

Caveats

Converting Insecure FQDNs to Secure

Certificate Precedence

HostRule Deletion

HostRule Admission

Duplicate FQDN rules

HTTP Rule

Usage of the HTTPRule CRD

Express Load Balancer Algorithm

Express Application Persistence Profile

Express Health Monitors

Reencrypt Traffic to the Services

Status Messages

Example of a HTTP Rule

Avi Infra Setting

Avi Infra Setting with Services, Ingress or Routes

Attaching Avi Infra Setting to Services

Attaching Avi Infra Setting to Ingress

Attaching Avi Infra Setting to Routes

Using Avi Infra Setting

Configure Service Engine Group

Configure VIP Networks

Configure Pool Placement Networks

Enable/Disable Route Health Injection

Enable/Disable Public IP

Configure BGP Peer Labels for BGP Virtual Services

Use Dedicated VIP for Ingress

Configure IPv6

SSORule CRD

Specific Usage of SSORule CRD

SSORule to VS Matching Using FQDN

Express OAuth/OIDC Configuration for Virtual Service

Express Name for Authorized Session Cookie

Express Timeout for Authorized Session Cookie

Express Logout URI for OAuth

Express Application and IDP settings for OAuth/OIDC

Express Client ID

Express Client Secret

Express OpenID Connect Configuration

Express Scope for OAuth

Express Authentication Profile for OAuth

Express Resource Server OAuth Configuration

Express Introspection Data Timeout

Express Opaque Token Parameters

Express Redirect URI for OAuth

Express Post Logout Redirect URI for OAuth

Express Application Specific SAML configuration

Express Assertion Consumer Service Index

Express Assertion Consumer Service Type for Authentication Request

Express Name for Authenticated Session Cookie

Express Timeout for Authenticated Session Cookie

Express Entity IP for SAML Service Provider Application

Express SslKeyAndCertificateRef for SAML Service Provider Application

Express Single Sign on URL for SAML

Express IDP Control for Service Provider Session Timeout

Express SSO Policy for the Virtual Service

Status Messages

Accepted SSORule Object

SSORule may get rejected on OpensShift if `useDefaultSecretsOnly` is true in the AKO configuration