NSX Advanced Load Balancer 22.1.X Release Notes

Issues Resolved in 22.1.5 Patch Releases

Issues Resolved in 22.1.5-2p2

Release Date: 19 January 2024

• AV-185059: CSR certificates managed through the certificate management profile get stuck in a renewal loop, leading to repeated renewal attempts every few seconds and resulting in anticipated failures.
• AV-185882: Unable to update secure channel root certificate when the cloud is not “No Orchestrator” or SEs are running in the system.
• AV-192901: Updating passwords in vCenter can transition the Avi vCenter cloud to failed state.
• AV-193663: Metrics Manager’s database connections with Postgres are unclosed, causing a connection leak.
• AV-195223: Name resolution on the Controller fails.
• AV-195595: External log streaming to a servers or load balancers which erroneously responds to simplex log stream causes Service Engine memory growth, eventually leading to SE crash.
• AV-195716: Although licenses are available in Pulse, changing the Bandwidth Type of SE Group in the Cloud Services tier failed.
• AV-196914: VSVIP objects having same the IP address may cause SE to fail.
• AV-197046: IPAM allocation for A records with multiple subnets will fail when the first subnet is exhausted.
• AV-197350: When the connection gets reset by rsyslog server, the SE log agent buffer becomes full and the logs are not streamed even after the connection is restored.

Issues Resolved in 22.1.5-2p2

• AV-197350: Symptoms: Log streaming stops whenever the streaming endpoint restarts Workarounds: Modify analytics profile of the VS (e.g. change rate of logs streaming )
• AV-197157: NA
• AV-197046: IPAM allocation for A records with multiple subnets will fail when the first subnet is exhausted.
• AV-196914: VSVIP objects having same the IP address may cause SE to fail.
• AV-195716: Although licenses are available in Pulse, changing the Bandwidth Type of SE Group in the Cloud Services tier failed.
• AV-195595: External log streaming to a servers or load balancers which erroneously responds to simplex log stream causes Service Engine memory growth, eventually leading to SE crash.
• AV-195223: The Name resolution on the Controller fails as DNS configuration in system settings configuration fails. Workarounds: Remove and Reconfigure DNS in system settings and save
• AV-193663: Metrics Manager’s database connections with Postgres are unclosed, causing a connection leak.
• AV-192901: Updating passwords in vCenter can transition the Avi vCenter cloud to failed state.
• AV-185882: We have a mechanism to work around the must-check to update the entire certificate chain including the root certificate.

Problem statement: The customer would like to change the root certificate when the system is in NSX-T environment. They need a way to manually pass the no SE and no orchestrator mustcheck and let the setup working. And also need the instructions on how to update all the related secure channel cerficate on SE and cloud content library to bring back the system.

• AV-185059: CSR certificates managed through the certificate management profile get stuck in a renewal loop, leading to repeated renewal attempts every few seconds and resulting in anticipated failures.

Issues Resolved in 22.1.5-2p1

Release Date: 06 December 2023

• AV-188363: In LSC hosts, when configuring Mellanox devices in combination with Broadcom components, the ring size computation logic can cause initialization errors and stall the SE during connection to the Controller.
• AV-190126: Using Broadcom NIC as management with Mellanox Nic for datapath causes issues in bringing up the NIC.
• AV-190461: Frequent updates to string groups attached to a DataScript, that also makes repeated calls to avi.stringgroup functions may result in failures in StringGroup lookups.
• AV-190475: se_dp crash happens with GRO code signature in stack trace, in a rare case scenario.
• AV-191149: Objsync may cause memory build-up and might lead to Out-of-Memory eventually on the SE caused by objsync peer connection failures either due to port 9001 or 4001 not being open in DFW in nsx-t or no management plane connectivity between SEs.
• AV-192083: Failure in Objsync connection over management interfaces between SEs might lead to memory exhaustion.
• AV-192951: Unable to use Infoblox DNS and Infoblox IPAM profiles when they are handled by different Infoblox instances.
• AV-192508: When an IPv4 address is added while the IP address “type” is set to IPv6 for Pool-Servers, it leads to se_agent crash.
• AV-193221: Missing support for Outbound NAT with source port preserved for UDP flows.

What’s New in 22.1.5

Release Date: 11 October 2023
To refer to the upgrade checklist, click here.

• An application level timer is introduced to ensure DNS requests close within a defined time limit, thus preventing any kind of Slowloris attacks on the DNS virtual service .

Issues Resolved in 22.1.5

• AV-171793: Intermittently, virtual service logs may not load or exhibit delay in loading.

• AV-179018: Service Engines might not get placed in configured datastore in Service Engine Group if content lib is enabled in cloud configuration.

• AV-179167: False alerts stating, “100% of total licensed Service Engine service cores used.”, displayed when license consumption is greater than the license capacity of the recently added license unit.

• AV-179869: When a GSLB service is configured to return all the records if it is down, if this GSLB service has multiple CNAME records in it, only one of CNAME records will be included in the ‘down’ response.

• AV-179893: A discrepancy between the timeline of federated queue and the timeline used during the subscribe operation, triggers the repetitive cycle of Sync and Subscribe operations, resulting in high bandwidth utilization.

• AV-179916: Replication from the leader site to follower site stalls when a file fails to download even if the subsequent downloads are successful.

• AV-180062: The IP Address/ FQDN field under Client Logs in Analytics Profile does not accept hostnames as valid input through the UI.

• AV-180173: When HTTP Cookie Persistence is used, and there are longstanding connections, and if the virtual service configuration is changed, then for the subsequent requests over the connection, the persistent cookies are not honored, and a different backend server can get selected.

• AV-180535: In virtual service logs, the location of origin of the Client IP address is unavailable through the UI and DataScripts.

• AV-180654: WAF PSM duplicate Rule ID generated owing to the number of URI params restricted to 10,000.

• AV-181710: If a virtual service is in a fault state due to issues with a WAF policy, and if this WAF policy has Positive Security Model (PSM) groups configured, and if these groups were updated after the WAF policy entered the fault state, then deleting the WAF policy can cause SE failure.

• AV-181723: Unable to assign an SNAT IP to an SNI parent virtual service that is attached to a content switching rule pool.

• AV-181805: Issue with accounting related to memory management in the Controller for memory held in buffers and caches.

• AV-181840: Security Manager failure when DNS servers are either not configured or not reachable.

• AV-182114: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.

• AV-182499: In the DPDK mode, NSX Advanced Load Balancer does not support the NIC model used by the host. As a result, the traffic for the VLAN interface configured with the Mellanox interface fails to work.

• AV-182702: The Prometheus-metrics API endpoint intermittently provides empty responses within a one-hour timeframe.

• AV-182827: Updating credentials in vCenter Cloud through the UI fails.

• AV-182830: L4 SSL DataScripts with collect API in the request or response events may cause SE failure.

• AV-183138: Long requests with SAML authentication can cause SE failure.

• AV-183400: HTTP request header size greater than 4K with ICAP deployment enabled can cause Service Engine failure.

• AV-183885: If an HTTP/1.0 header arrives without a host header, which is NULL (permissible in HTTP/1.0), and this header is internally processed for comparison with GS domain names, it results in SE failure.

• AV-184154: vNICs in a No-Access setup had is_avi_internal set to True, causing VNIC IP updates to not persist.

• AV-184189: The cloud name changes to “TRUSTED” or “UNTRUSTED” in the GUI instead of the actual name.

• AV-184284: Duplicated network names in the UI cause inability to uniquely identify a network.

• AV-184734: NSX Advanced Load Balancer AWS S3 backup failing when using only IAM roles with S3 bucket permissions.

• AV-184809: In the NSX Advanced Load Balancer UI, the message “No pools configured” is displayed although the pool has pool groups configured under it.

• AV-184853: Disabling the virtual service having VIP as SNAT configuration when two virtual services are sharing the same VIP, can leave the other VS non-functional.

• AV-185279: Unable to edit a Cloud of type GCP in the UI if the optional Routes field is missing.

• AV-185506: If an NXDomain DoS attack is detected, the Service Engine may experience memory leakage.

• AV-185604: When configuring a TCP request for a health monitor of type TCP with user-defined settings including get or post strings, the system automatically appends HTTP/1.0 and \r\n\r\n to the TCP request.

• AV-180910: When a GSLB service member is configured as FQDN and GSLB service is using external health monitors, member FQDN will be passed as an environmental variable to external health monitor script.

• AV-181918: After upgrading to 22.1.3-2p4, the user-defined cloud name changes to a default vCenter cloud name.

• AV-182892: AWS cloud-specific information is not displayed in the Clouds page (Infrastructure > Clouds) in the NSX Advance Load Balancer UI.

• AV-186355: DNS resolution for pool FQDN may result in failure when the response is big enough (greater than 512 bytes) to trigger the resolution to happen through TCP transport.

• AV-186806: Service Engine fails during a pool update followed by the deletion and reconfiguration of a child virtual service.

• AV-186925: Service Engine might fail when it receives traffic in which the Ethernet header and the other headers arrive in separate packets.

• AV-185604: When configuring a TCP request for a health monitor of type TCP with user-defined settings including get or post strings, the system automatically appends HTTP/1.0 and \r\n\r\n to the TCP request.

• AV-187052: CRS overrides are not being added when the CRS version is updated in the WAF Policy modal.
  Workaround: Save the uncommitted WAF Policy settings before updating.

• AV-187301: If a virtual service is in a fault state due to issues with a WAF policy, and if this WAF policy has Positive Security Model (PSM) groups configured, and if these groups were updated after the WAF policy entered the fault state, then deleting the WAF policy can cause SE failure.

• AV-187301: When incorrect credentials are provided to the Avi Terraform provider, it initiates the creation of resources that have already been created.

• AV-187523: Configuration replication does not work for uncommon federated objects between leader and follower if they belong to different versions.

• AV-187919: SE failure when client sends an invalid HTTP/2 header.

• AV-188363: In LSC hosts, when configuring Mellanox devices in combination with Broadcom components, the ring size computation logic can cause initialization errors and stall the SE during connection to the Controller.

• AV-188419: In application logs the location of origin of the Client IP address is unavailable through the UI and DataScripts.

• AV-188464: Modifying the pool configuration through the GUI on an NSX-T cloud with Security Groups as Server definitions can lead to the removal of pool members until the next discovery sync occurs. This issue occurs even when the existing pool configuration is not modified, but just saved via UI.

• AV-188919: If the vm_uuid file is edited or saved manually, it can result in the generation of an extra newline at the end, which may lead to image upload failures. These failures can potentially be attributed to host resolution issues.

• AV-189340: The se_log_agent fails with the error message, “SE crashed with fatal error” for external log streaming over TCP/TLS.

Key Changes in 22.1.5

• In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

• Support for TCP Proxy Protocol (Enable Proxy Protocol is selected) when an L4 application profile is used as an override application profile.

• If a GSLB service member is monitored by multiple sites through a health monitor proxy, in the sites that rely on remote status from the health monitor proxy sites, the member will be marked UP if at least one health monitor proxy site reports the status as UP.

• When a GSLB service member is configured as FQDN and GSLB service is using external health monitors, member FQDN will be passed as an environmental variable to external health monitor script.

• Support for Enhanced Datapath mode for NSX-T Data Center: Select the Enhanced Datapath mode when preparing the ESXi hosts as Transport Nodes. NSX Advanced Load Balancer seamlessly adapts to the Enhanced Datapath mode in NSX. It is recommended to use the ENS interrupt mode for better performance. See NSX-T documentation for different modes and pre-requisites for more information.

• Starting 08 May 2023, some NSX editions include NSX Advanced Load Balancer Enterprise with a ratio of 1 NSX Advanced Load Balancer unit per 250 NSX CPU cores. Starting with version 22.1.5, the license keys generated as part of the specified entitlements will be recognized and decoded natively by the Controller. To learn more about the specific editions in which these entitlements are included, see NSX Editions and Feature Guide .

Known Issues in 22.1.5

• AV-187931: When System-SCTP-Proxy TCP/UDP Profile is selected as network profile for virtual services, a port range cannot be specified under Service Ports. If a port range is configured, only the first port within the specified range handles traffic.

• AV-190003: High CPU utilization may be observed in NSX-T based cloud environments (check using the show cpuusage controller command)
  Workaround: For optimal CPU utilization by cloud connector routines/processes on the system, use the configuration shown below:

  
  configure cloud <cloud_name>
  autoscale_polling_interval 600
  configure controller properties
  cloud_reconcile_interval 10
  cloud_discovery_interval 15
  attach_ip_retry_limit 1
  vs_se_attach_ip_fail 240
  detach_ip_retry_limit 1
  

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.5

Refer to this section before initiating upgrade.

• Upgrade to NSX Advanced Load Balancer to 22.1.5 is only supported from the following versions:
  

• Version 20.1.1 through 20.1.9

• Version 21.1.1 through 21.1.6

• Version 22.1.1 through 22.1.4

• Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory requirement for Service Engines is increased to 2GB. Before upgrading to any version in the 22.1.x release, ensure the Service Engines are configured to a capacity greater than 2 GB. The current considerations for memory sizing as listed here continue to apply.

• Before upgrading to version 22.1.2 or higher, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data .

• Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory recommended for an Essentials Controller is 24G. Ensure that the memory of an Essentials Controller is at least 24G before upgrade.

• The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

• As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

Issues Resolved in 22.1.4 Patch Releases

Issues Resolved in 22.1.4-2p6

Release Date: 08 December 2023

• AV-192083: Failure in Objsync connection over management interfaces between SEs might lead to memory exhaustion.
• AV-191642: A pki profile with CRL larger than 4 MB fails in replication across federation because of GRPC message size limitation
• AV-191615: When a WebSocket is utilized with front-end using HTTP/2 and backend using HTTP/1, then NSX Advanced Load Balancer does not terminate the v1 WebSocket on the backend if the “Upgrade” header sent by the server is not “ websocket” (all in lowercase), the upgrade header’s value being case sensitive.
• AV-191545: The source port range of the BFD control packets does not adhere to the RFC5881.
• AV-191387: When an incoming request carries an Avi-generated cookie for HTTP Cookie Persistence, that cookie is included in the request sent to the backend server. However, in certain transactions, if the server expires the AVI-specific cookie, clients will not present the cookie in subsequent transactions, resulting in persistence failure on AVI.
• AV-191149: Objsync could result in memory accumulation, causing out-of-memory issues on the SE. This is typically triggered by objsync peer connection failures, which may be due to port 9001 or 4001 not being accessible in DFW in NSX or a lack of management plane connectivity between SEs.
• AV-189818: Replication stalls with the error, “Sync Stalled, reason: replicating federationcheckpoint:< checkpoint_name>” while updating federation checkpoint object after making it active checkpoint object in Adaptive Replication mode.
• AV-188816: Some GSLB processes consume high memory in a long run causing overall node memory over growth.
• AV-158363: If a Virtual Service with an L4 application profile is configured through service application profile override, and the persistence profile is set to anything other than client IP persistence, it leads to Service Engine failure.

Issues Resolved in 22.1.4-2p5

Release Date: 30 October 2023

• AV-190461: Frequent updates to StringGroups attached to a DataScript, that also make repeated calls to avi.stringgroup functions may result in failures in StringGroup lookups.
• AV-186052: The Service Engine would crash if an LDAP response carries a referral to a server which is not reachable from the management interface.

Issues Resolved in 22.1.4-2p4

Release Date: 14 October 2023

• AV-180173: When HTTP Cookie Persistence is used, and there are longstanding connections, and if the virtual service configuration is changed, then for the subsequent requests over the connection, the persistent cookies are not honored, and a different backend server can get selected.
• AV-182813: For proper functioning of SE rebalancing, the Resource Monitor requires the metrics API to provide accurate and non-null values. However, due to read deadlocks on Postgres Metrics occurring around the hour boundary, the metrics are not returned, resulting in an empty response and causing SE Rebalance failures.
• AV-183885: If an HTTP/1.0 header arrives without a host header, which is NULL (permissible in HTTP/1.0), and this header is internally processed for comparison with GS domain names, it results in SE failure.
• AV-185506: If an NXDomain DoS attack is detected, the Service Engine may experience memory leakage.
• AV-186806: Service Engine crash occurs during a pool update followed by the deletion and reconfiguration of a child virtual service.
• AV-188464: Editing the Pool configuration via the GUI on an NSX-T cloud with Security Groups as Server definition will remove pool members until next discovery sync. This issue is visible even when the existing Pool configuration is not modified, but only saved via UI.

• AV-188904: Trailing RST on a closed L7 SSL VS connection may cause SE failure.

• AV-186671: Successive restarts of Service Engine results in creation of multiple events & event files leading to potential controller cluster instability.
• AV-189340: The se_log_agent fails with the error message, “SE crashed with fatal error” for external log streaming over TCP/TLS.
• AV-190126: Using Broadcom NIC as management with Mellanox Nic for datapath causes issues in bringing up the NIC.

Issues Resolved in 22.1.4-2P3

Release Date: 29 September 2023

• AV-188363: With combination of Broadcom and Mellanox in LSC hosts, because of the ring size computation logic configuring Mellanox results in initialisation error and stalls the SE connecting to the Controller.
• AV-186925: Service Engine might fail when it receives traffic in which the Ethernet header and the other headers arrive in separate packets.
• AV-188464:

Issues Resolved in 22.1.4-2P2

Release Date: 13 September 2023

• AV-179869: When a GSLB service is configured to return all the records if it is down, if this GSLB service has multiple CNAME records in it, only one of CNAME records will be included in the ‘down’ response.

• AV-180910: When a GSLB service member is configured as FQDN and GSLB service is using external health monitors, member FQDN will be passed as an environmental variable to external health monitor script.

• AV-181840: Security Manager crash when DNS servers are either not configured or not reachable.

• AV-182702: The Prometheus-metrics API endpoint provides empty responses randomly for a one-hour timeframe.

• AV-184154: SE data interface IP updated through the Controller UI or CLI is not replicating in the SE.

• AV-184189: The cloud name changes to “TRUSTED” or “UNTRUSTED” in the GUI instead of the actual name.

• AV-184284: Duplicated network names in the UI cause inability to uniquely identify a network.

• AV-184734: NSX Advanced Load Balancer AWS S3 backup failing when using only IAM roles with S3 bucket permissions.

• AV-184809: In the NSX Advanced Load Balancer UI, the message “No pools configured” is displayed although the pool has pool groups configured under it.

• AV-185279: The Routes field is missing, and the GCP cloud screen fails to load.

• AV-186355: DNS resolution for pool FQDN may result in a crash when the response is big enough to trigger the resolution over TCP.

Issues Resolved in 22.1.4-2P1

Release Date: 07 August 2023

• AV-171793: VS logs may not load intermittently or might exhibit delay in loading.

• AV-179893: A discrepancy between the timeline of federated queue and the timeline used during the Subscribe operation, triggers the repetitive cycle of Sync and Subscribe operations, resulting in high bandwidth utilization.

• AV-180535: Location information may not be available for the Logs/Events UI and DataScripts due to an internal socket issue.

• AV-180062: The IP Address/ FQDN field is not accepting hostname as valid input through the UI.

• AV-181781: Patch installation takes longer than the optimal duration.

• AV-181805: Controller memory management accounting issue for memory held in buffer and cached memory.

• AV-182830: L4 SSL DataScripts with collect API on request or response events may cause SE failure.

• AV-182827: Updating credentials in vCenter Cloud through the UI fails.

• AV-182499: The model of NIC used in the host is not supported in NSX Advanced Load Balancer’s DPDK version. Hence, the traffic for the VLAN interface configured with Mellanox interface fails to work.

• AV-183138: Long requests with SAML authentication can cause SE failure.

Issues Resolved in 22.1.4

Release Date: 23 June 2023
To refer to the upgrade checklist, click here.

• AV-134485: Bot Classification: Filtering log entries with blank fields and default values does not produce consistent results. Some fields can be incorrectly labelled/classified.

• AV-152096: On an AWS cloud, when moving virtual services from one SE group to another using an API call, some virtual services may go into a failed state with the error PrivateIPAddresslimitExceeded.

• AV-159518: Image upload fails for containerized Controller deployments while upgrading to version 22.1.3 from versions 20.1.1 through 20.1.7 and 21.1.1 through 21.1.4.

• AV-159552: Multiple event files are created owing to frequent Auth Manager restarts leading to scale and performance issues for log manager while indexing files.

• AV-165613: Service Time out during certificate creation through the UI or API in a Controller with more than 1,000 certificates.

• AV-166709: SE vNICs are receiving multicast traffic causing performance degradation.

• AV-166845: If a Pool’s name does not have the word “pool” in it, it cannot be used in the avi.requests DataScript functions.

• AV-166887: Due to a race condition, if an SE is rebooted from vCenter, the SE might end up with duplicate static subnets configured on multiple VNICs.

• AV-167281: Multiple failures in the metrics-mgr process due to invalid or non-UTF-8 characters when parsing metrics.

• AV-168153: Double syslog or syslog being sent to an old syslog server after changing the IP inside the alert syslog server configuration.

• AV-168340: When there are several pages on the events grid, an empty warning message is displayed on navigating from one page to another.

• AV-168432: ControlScripts do not work in a Controller installed over Baremetal or a Linux server cloud.

• AV-168433: Anonymous SMTP connection with Google fails because ‘[127.0.0.1]’ is sent as hostname in the ehlo message.

• AV-168482: In NSX Advanced Load Balancer version 22.1.3, when updating a WAF Policy via API, using a X-Avi-Version older than version 22.1.3, the update may fail with “Cannot add or remove elements in the list of internal field WafPolicy.required_data_files”. This can happen if a pre-CRS or post-CRS rule is added, which is using the @pmf operator on a data file from WAF Profile which is not already used in the current ruleset.

• AV-168659: Addition of a new string fails silently if it matches with a substring of one or more existing strings causing incorrect results in virtual service policies when the string group equals comparison is used in the match rules.

• AV-168742: NIC initialization fails in bare metal deployments because of DPDK memzone exhaustion.

• AV-168862: If a pool name has the character ‘/’ in its name, the part of the name after ‘/’ will be appended to the URI and sent to server.

• AV-168867: CC_IP_ATTACHED events get generated wrongly for virtual services.

• AV-168904: When a GSLB service in deactivated state is associated to a virtual service and later that virtual service is deleted, stale entries are created in GSLB service DNS virtual service list. When such a GSLB service is enabled, it may cause SE failure or memory corruption.

• AV-169410: Upgrading only the Controller cluster from NSX Advanced Load Balancer prior to version 22.1.3 to a version 22.1.3 or higher, SNI Child virtual services present on the SEs (on a version prior to 22.1.3), are stuck in OPER_INITIALIZING state with the reason “Vip is not active on ServiceEngine”.

• AV-169398: Let’s Encrypt renewal breaks in version 22.1.3 if renewal is executed by a different user other than that specified in the certificate management script parameters.

• AV-169440: Disabling virtual service traffic in NSX-T Cloud can take up to 5 minutes to take effect.

• AV-169464: Updated to the latest OpenSSL version to remediate potential exposure to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.

• AV-169505: In the ESSENTIALS edition, it is not possible to change the Default Cloud Type from No Orchestrator to VMware vCenter/vSphere ESX cloud in the edit mode through the UI.

• AV-169773: When a GSLB service is created with algorithm TOPOLOGY and the topology rules for member selection is based on AVI sites, then the DNS responses for the first few minutes of GSLB service creation can be fallback responses applicable to the TOPOLOGY algorithm.

• AV-169779: In an Analytics Profile, the following fields are not retaining updated values:
  • Log Severity
  • Non-Significant Log Severity
  • Filtered Log Severity
  • Hostname
  • Facility

• AV-169780: Virtual Service may stay in initializing state with error “Virtual service IP is not reachable in cloud” on NSX cloud.

• AV-170091: IPv6 VIP traffic fails and RST is seen in traffic capture. This is specific to PCAP deployments, where inet6 interface is present on both default (Linux) and Avi namespaces.

• AV-170116: When a DNS virtual service is bound to deactivated GSLB services, on disabling and re-enabling of the DNS virtual service, the virtual service may get stuck in the OPER_DOWN state.

• AV-170118: For DNS traffic over TCP in the case of pass-through, the TCP connection can linger for a longer duration because the load balancer expects either the client or the server to initiate the close.

• AV-170744: License remains in escrow for up to 2 hours.

• AV-170759: Some metrics are missing when the virtual service and SE pages are viewed through the UI or REST API.

• AV-170762: The SE DP start process is blocked due to stale processes in the control group from the previous invocation.

• AV-170811: Unable to switch from Enterprise with Cloud Services tier to Enterprise tier when licenses consumed is greater than what is available in Enterprise tier.

• AV-170900: The NSX Advanced Load Balancer UI becomes unresponsive when trying to open an application profile created using CLI, if it has an rl_profile object but does not have an optional rate_limiter object in it.

• AV-170926: CSR generation from the UI fails when certificate management profile associated with CSR has any sensitive fields. In case of Let’sEncrypt, CSR generation fails with the error Invalid Credentials.

• AV-171058: The error, “Failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \“python3\”: executable file not found in $PATH: unknown”, is displayed on ControlScript execution.

• AV-171114: Queries triggered by metrics manager fail due to the change in the query format, introduced by Postgres migration from version 10 to 14.

• AV-171116: Root/Intermediate EC CA cert is not loading because of errors due to missing algorithm values in specific certificates.

• AV-171173: Upgrade fails with error, “KeyError: ‘protocol’” if the config JSON has alerts with audit_compliance_event in it.

• AV-171222: When an SNI child Virtual Service with a DataScript in the CLIENT_SSL_PRE_CONNECT event is deleted while processing traffic. It results in Service Engine failure.

• AV-171581: Upgrade to 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IPs due to issues with updating DNS resolvers after upgrade.

• AV-171587: Due to a race condition, when an SE disconnects and reconnects with the Controller some time later, the virtual service placed on that SE can end up in a state where it is placed on more SEs than the number of SEs requested configured in the SE Group.
  This state persists. In the case of Parent-Child VS it can cause FQDN issues if the child VS is disabled/enabled in this state, causing the Parent and Child VS to be placed on a different set of SEs. This can lead to the child FQDN being handled sometimes by the Parent VS and sometimes by the Child VS.

• AV-171698: WAF requests may become slow if client_request_max_body_size in the WAF Profile is set to high values.

• AV-171954: If the criteria for match cases are in the negative (for example Does not equal, then there is no memory allocated in the Service Engine resulting in SE failure when trying to access the invalid memory.

• AV-172051: When a virtual service was attached to an SSL-enabled pool group, the “scheme” header was sent as an HTTP header to the backend server.

• AV-172209: Packet corruption when there is an MSS mismatch between client/server and interface MTU.

• AV-172220: Service Engine failure when using avi.http.response with two arguments in the RESP_FAILED DataScript event.

• AV-172510: Duplicate logging of warning messages during upgrade.

• AV-172541: The toggle button to enable/ disable new log views is not showing up if the user does not have specific permissions.

• AV-172551: SE failure when the NTLM server responded with a mismatched status code to the health monitor.

• AV-172563: Garbage Collect API “/api/cloud//gc" fails for OpenStack cloud.

• AV-172563: Config Backup to the AWS S3 bucket does not work with an Instance profile.

• AV-172793: When 40% of SE’s memory is less than 16 G, the effective packet buffers reduce significantly causing high packet buffer usage. This impact is noticed only on applying 21.1.4-2p14.

• AV-172832: SE creation fails in AWS us-east-1 region after upgrading.

• AV-172840: Log Manager stalled leading to spike in unbounded task queue affecting critical Controller services.

• AV-173206: SE may fail when rate-limiting requests that collect client insights.

• AV-173693: Issue with batch server enable/deactivate API for all tenants role without tenant specific role and superuser user without roles.

• AV-173748: When the GSLB service member config is changed to dissociate it from the Avi site or a third-party site, it may remain stuck in down state.

• AV-174065: If two or more virtual services are sharing an IPv6 VIP, disabling one of them renders other virtual services unreachable.

• AV-174223: Potential crash in Log Manager during Geo-IP translation.

• AV-174263: HSM initialization fails causing the traffic to VIP using the HSM certificate also to fail.

• AV-174338: Missing IP table rules in cis_mode resolved by including the IPv6 ports as well on the Controller.

• AV-174447: Network Profiles are not editable through the NSX Advanced Load Balancer UI in the Essentials license edition.

• AV-174523: On an AWS cloud, custom tags are not applied to AMI snapshots.

• AV-174847: Virtual service events stopped displaying due to an exception raised in the generation of ADD_NW_FAIL event.

• AV-174983: Certificate Import fails if it includes certificates without a Subject Common Name.

• AV-175121: The error Public key not matched:, displayed when uploading the certificate in PFX format to NSX Advanced Load Balancer.

• AV-175310: The NSX Advanced Load Balancer UI gets stuck if the user tries to open the application profile created using CLI which has a rate limiter action of type RL_ACTION_LOCAL_RSP and the file object is not populated.

• AV-175329: The DNS virtual service connection-log streaming rate is reduced to approximately 30k from NSX Advanced Load Balancer version 22.1.3 onwards.

• AV-175496: Service Engines failure when GSLB Services have multiple groups with the same name.

• AV-175544: NSX Advanced Load Balancer Service Engine operating as outbound NAT router drops incoming SYN packet from the back-end server if additional ECE & CWR bits are set in the TCP flags.

• AV-176147: Using the @IpMatch or @IpMatchFromFile operator in WAF and adding the same IP range twice may result in wrong results.

• AV-176510: Statecache manager failure on deleting a health monitor object in an adaptive replication enabled GSLB setup.

• AV-176511: Metrics-mgr crashes while processing metrics with Invalid or non-UTF-8 characters in the object ID or Entity ID.

• AV-176638: Sensitive Information of snmpv3 configs exposed in portal-webapp.log on failure of snmptrap send-attempt from the Controller.

• AV-176746: The EngineBoots value is always equal to 1 in SNMP GetResponse packets instead of reflecting the SNMP service/Controller restart count.

• AV-177218: When out-of-band DataScript APIs are used with a pool and the backend pool has IP Persistence enabled, the SE May fail if the connection strategy of the front end to backend connection is connection-switching if either connection multiplexing is disabled or if the NTLM detection with the backend server sends WWW-Authenticate : NTLM or WWW-Authenticate : Negotiate headers.

• AV-177015: Unable to create a DNS profile when the Use IAM role option is selected.

• AV-177222: Service engine deployed in Azure cloud with DPDK/ Dedicated Management enabled might fail to acquire the interface IP through DHCP.

• AV-178270: On changing the name of the GSLB service, it may transition to down state even when one or more pools belonging to the GSLB service are up.

• AV-178650: Parallel creation of VS VIPs using the same subnet can result in the allocation of the same IP from Infoblox leading to duplicate IPs.

• AV-178849: The virtual service oper_state is marked Down since the SE is unable to download config objects from the Controller due to hardcoding of Controller hostname.

• AV-179415: During a virtual service configuration update, which may or may not be triggered from the Controller, in the middle of TLS handshake, the SE might crash when accessing L7 configuration of the virtual service.

• AV-179787: SE does not connect to the Controller in the case of a co-located setup, where the SE and the Controller are deployed as containers on the same host.

• AV-180155: Alerts are being saved for user-defined action groups in the database even when the field External_only is enabled.

• AV-180629: Metrics Manager might crash during GRPC async server-handling.

• AV-181312: Patch upgrade to 22.1.3-2p6 gets cancelled at the installpatchimage task.

• AV-181830: Creating or updating a GslbService with site persistence enabled, fails with the error PKI profile is not configured for the request coming from a client with an API version lower than 22.1.3.

• AV-127214: SE failure due to incompatibility in hardware versions for LSC deployments on VMware ESXi VMs.

Key Changes in 22.1.4

• In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

• Support to modify the number of simultaneous SE Group upgrades.

• If the field ip6_autocfg_enabled is set false on a VNIC, the VNIC stops receiving multicast frames.

Known Issues in 22.1.4

• AV-127214: SE failure due to incompatibility in hardware versions for LSC deployments on VMware ESXi VMs.

• AV-182114: Symptoms: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.
  Workaround: From the CLI, manually disable the SE which exhibits this behavior.

• AV-182440: The client location information may be not unavailable in the Logs/Events and DataScripts, caused by a failure to clean up an internal socket when Logs or Events are queried.
  Workaround: Restart redis-service@6003 or LogMgr.

• AV-187931: When System-SCTP-Proxy TCP/UDP Profile is selected as network profile for virtual services, a port range cannot be specified under Service Ports. If a port range is configured, only the first port within the specified range handles traffic.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.4

Refer to this section before initiating upgrade.

• Upgrade to NSX Advanced Load Balancer to 22.1.4 is only supported from the following versions:
  

• Version 20.1.1 through 20.1.9

• Version 21.1.1 through 21.1.6

• Version 22.1.1 through 22.1.3

• Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory requirement for Service Engines is increased to 2GB. Before upgrading to any version in the 22.1.x release, ensure the Service Engines are configured to a capacity greater than 2 GB. The current considerations for memory sizing as listed here continue to apply.

• Before upgrading to version 22.1.2 or higher, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data .

• Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory recommended for an Essentials Controller is 24G. Ensure that the memory of an Essentials Controller is at least 24G before upgrade.

• The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

• As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

Patch Release Notes for 22.1.3

Issues Resolved in 22.1.3 Patch Releases

• AV-177767: IMPORTANT NOTE: An issue has been identified in patch releases 22.1.3-2p2, 22.1.3-2p3 and 22.1.3-2p4 that may result in some Controller services being impacted immediately after applying these patches. In some cases, there may be some service impact due to DNS resolution on the Controller being non-operational until the Controller can be rebooted. This issue is addressed in 22.1.3-2p5.

  • The issue will be encountered when installing or rolling-back to the patches 22.1.3-2p2, 22.1.3-2p3 or 22.1.3-2p4. It is therefore recommended not to apply these patches and instead to apply 22.1.3-2p5.

  • The issue will also be encountered when applying the 22.1.3-2p5 patch to a system that is already running the 22.1.3-2p2, 22.1.3-2p3 or 22.1.3-2p4 patch level. In this case, it is recommended to use Flexible Upgrade to apply the 22.1.3-2p5 patch to the Controller only first, then reboot the Controller/Cluster to remediate the issue, then apply the 22.1.3-2p5 patch to Service Engine Groups.

  • The issue will not be encountered when applying the 22.1.3-2p5 patch to a system running 22.1.3 (base release) or 22.1.3-2p1.

  • The issue will not be encountered when upgrading to 22.1.3-2p5 directly from an earlier release

Issues Resolved in 22.1.3-2p12

Release Date: 06 Dec 2023

• AV-182813: For proper functioning of SE re-balancing, the Resource Monitor requires the metrics API to provide accurate and non-null values. However, due to read deadlocks on Postgres Metrics occurring around the hour boundary, the metrics are not returned, resulting in an empty response and causing SE re-balance failures.
• AV-185506: If an NXDomain DoS attack is detected, the Service Engine may experience memory leakage.
• AV-187057: If a virtual service is in a fault state due to issues with a WAF policy, and if this WAF policy has Positive Security Model (PSM) groups configured, and if these groups were updated after the WAF policy entered the fault state, then deleting the WAF policy can cause SE failure.
• AV-188464: Editing the Pool configuration via the GUI on an NSX-T cloud with Security Groups as Server definition will remove pool members until next discovery sync. This issue is visible even when the existing Pool configuration is not modified, but only saved via UI.
• AV-192083: Failure in Objsync connection over management interfaces between SEs might lead to memory exhaustion.
• AV-191149: Objsync may cause memory build-up and might lead to Out-of-Memory eventually on the SE caused by objsync peer connection failures either due to port 9001 or 4001 not being open in DFW in NSX or no management plane connectivity between SEs.
• AV-191615: When a WebSocket is utilized with front-end using HTTP/2 and backend using HTTP/1, then NSX Advanced Load Balancer does not terminate the v1 WebSocket on the backend if the “Upgrade” header sent by the server is not “ websocket” (all in lowercase), the upgrade header’s value being case sensitive.

Issues Resolved in 22.1.3-2p11

Release Date: 22 August 2023

• AV-168340: When there are several pages on the events grid, an empty warning message is displayed on navigating from one page to another.
• AV-179916: Replication stalls when a file download fails even though a subsequent attempt is successful.
• AV-180535: In virtual service logs, the location of origin of the Client IP address is unavailable through the UI and DataScripts due to an internal socket issue.
• AV-180654: WAF PSM duplicate Rule ID generated owing to number of URI params restricted to 10000.
• AV-181723: Unable to assign an SNAT IP to an SNI parent virtual service that is attached to a content switching rule pool.
• AV-181805: Issue with accounting related to memory management in the Controller for memory held in buffers and caches.
• AV-183400: HTTP request header size greater than 4K with ICAP deployment enabled can cause Service Engine failure.
• AV-185279: Unable to edit a Cloud of type GCP in the UI if the optional Routes field is missing.

Issue Resolved in 22.1.3-2p10

Release Date: 22 July 2023

• AV-184284: Duplicated network names displayed in the UI causing inability to uniquely identify a network.

Issue Resolved in 22.1.3-2p9

Release Date: 19 July 2023

• AV-183885: An http1.0 header without a host header, which is NULL (allowed in http1.0) is processed internally during the comparison to GS-domain names, leads to SE crash.

Issues Resolved in 22.1.3-2p8

Release Date: 17 July 2023

• AV-180173: When HTTP Cookie Persistence is used, and there are longstanding connections, and if a config change happens for the virtual service, then for the subsequent requests over the connection, the persistent cookies are not honored, and a different backend server can get selected.
• AV-182827: Updating credentials in vCenter Cloud through the UI fails.
• AV-182499: The model of NIC used in the host is not supported in NSX Advanced Load Balancer’s DPDK version. The traffic for the VLAN interface configured with Mellanox interface fails to work.
• AV-181781: Patch installation takes longer than usual.
• AV-171793: Virtual service logs may not load intermittently or exhibit delay in loading.

Issues Resolved in 22.1.3-2p7

Release Date: 28 June 2023

• AV-180062: The syslog server field is not accepting hostname through the UI.
• AV-181312: Patch upgrade to 22.1.3-2p6 gets cancelled at the installpatchimage task.
• AV-181830: Creating or updating of a GSLB service with site persistence enabled,fails with the error PKI profile is not configured for the request coming from a lower version.

Issues Resolved in 22.1.3-2p6

Release Date: 08 June 2023

• AV-170900: The NSX Advanced Load Balancer UI becomes unresponsive when trying to open the application profile created using CLI, if it has an rl_profile object but does not have an optional rate_limiter object in it.

• AV-174447: Network Profiles are not editable through the NSX Advanced Load Balancer UI in the Essentials license edition.

• AV-177218: When out-of-band DataScript APIs are used with a pool and the backend pool has IP Persistence enabled, the SE May fail if the connection strategy of the front end to backend connection is connection-switching either due to Config (Connmux disabled) or due to NTLM detection with Backend server, sending WWW-Authenticate : NTLM or WWW-Authenticate : Negotiate headers.

• AV-178650: Parallel creation of VS VIPs using the same subnet can result in the allocation of the same IP from Infoblox leading to duplicate IPs.

• AV-179415: During a virtual service configuration update, which may or may not be triggered from the Controller, in the middle of TLS handshake, the SE might crash when accessing L7 configuration of the VS.

Issues Resolved in 22.1.3-2p5

Release Date: 31 May 2023

• AV-169410: Upgrading only the Controller cluster from NSX Advanced Load Balancer prior to version 22.1.3 to a version 22.1.3 or higher, SNI Child virtual services present on the SEs which are on version prior to 22.1.3 will be stuck in OPER_INITIALIZING state with the reason “Vip is not active on ServiceEngine”.

• AV-170054: In a scaled setup, state updates for some GSLB objects can be skipped due to an internal race condition.

• AV-172209: Packet corruption when there is an MSS mismatch between client/server and interface MTU.

• AV-177222: Service engine deployed in Azure cloud with DPDK/ Dedicated Management enabled might fail to acquire the interface IP through DHCP. With the fix, the DHCP will work.

• AV-176746: The EngineBoots value always equal to 1 in SNMP GetResponse packets. It should reflect the number of times of SNMP service/Controller restart count.

• AV-178849: The virtual service oper_state is marked Down since the SE is unable to download config objects from the Controller due to hardcoding of Controller hostname.

Issues Resolved in 22.1.3-2p4

Release Date: 28 April 2023

• AV-159552: Multiple event files created owing to frequent Auth Manager restarts.
• AV-168862: If a pool name has the character ‘/’ in its name, the part of the name after ‘/’ will be appended to the URI and sent to server.
• AV-177055: Bulk SE created under Patched SE Group takes longer to download the se.pkg.
• AV-171116: Root/Intermediate EC CA cert is not loading because of errors due to missing algorithm values in specific certificates.
• AV-172541: The toggle button to enable/ disable new log views is not showing up if the user does not have specific permissions.
• AV-173206: SE may fail when rate-limiting requests that collect client insights.
• AV-174338: Missing IPtable rules in cis_mode resolved by including the IPv6 ports as well on the Controller.
• AV-174847: Virtual service events stopped displaying due to an exception raised in the generation of ADD_NW_FAIL event.
• AV-175496: Service Engines were failing because the GSLB Service had multiple groups with the same name.
• AV-176511: Invalid or non-UTF-8 characters displayed when parsing metrics.
• AV-176638: Sensitive Information of snmpv3 configs exposed in portal-webapp.log on failure of snmptrap send-attempt from the Controller.
• AV-176939: Unable to save changes in the Edit My Account screen when the user does not have Controller-write permissions.

Issues Resolved in 22.1.3-2p3

Release Date: 07 April 2023

• AV-174523: Cloud custom tags do not get copied to AMI snapshots.

• AV-174263: HSM initialization fails, causing the traffic to VIP using the HSM certificate also to fail.

• AV-173693: Issue with batch server enable/disable API for all tenant’s roles without tenant-specific roles and superuser users without roles.

• AV- 173679: In an OpenStack cloud, the Avi Controller spins up all the SEs in parallel by executing APIs to OpenStack Controller. In a large scale deployment, SE creation can fail and hence a script is provided which can set the limit for the OpenStack cloud.

• AV-172832: SE creation fails in AWS us-east-1 region after upgrading to the 22.1.3 version.

• AV-172752: Support for modifying the number of parallel SE group upgrades.

• AV-172671: Virtual service failures due to insufficient memory and low packet buffer events on SE’s (causing connection drops).

• AV-172510: Duplicate logging of warning messages during an upgrade.

• AV-172051: When a virtual service was attached to an SSL-enabled pool group, the scheme header was sent as an HTTP header to the backend server.

• AV-171581: Upgrade to the 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IP addresses due to issues with updating DNS resolvers after an upgrade.

• AV-171222: When an SNI child Virtual Service with a DataScript in the CLIENT_SSL_PRE_CONNECT event is deleted while processing traffic, it results in Service Engine failure.

• AV-170903: Disabling of pool servers is blocked for NSX Group-based pool servers.

• AV-170762: SE DP start process is blocked due to stale processes in the control group from the previous invocation.

• AV-170091: IPv6 VIP traffic fails, and RST is seen in traffic capture. This is specific to PCAP deployments, where * inet6* interface is present on both default (Linux) and Avi namespaces.

• AV-166845: If a pool’s name does not have the word pool in it, it cannot be used in the avi.requests DataScript functions.

Issues Resolved in 22.1.3-2p2

Release Date: 29 March 2023

• AV-169773: If a GSLB service has topology rules for selecting GSLB site with AVI members, In cases of problems or delays in site status sync response, may not confirm to the configured rule action.

• AV-166709: SE vNICs are receiving multicast traffic causing performance degradation.

• AV-168433: For anonymous SMTP connection with Google fails because we are sending ‘[127.0.0.1]’ as hostname in ehlo message. This hostname works with local SMTP servers.

• AV-168867: CC_IP_ATTACHED events getting incorrectly generated for Virtual Services.

• AV-169464: OpenSSL-1.1.1f stack in the Controller and Service Engine are vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.

• AV-169473: OpenSSL-1.0.2g stack in the Controller and Service Engine are vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.

• AV-169779: In Analytics Profile, the following fields are not retaining updated values:

  • Log Severity
  • Non-Significant Log Severity
  • Filtered Log Severity
  • Hostname
  • Facility

* * AV-171058: The error, “failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \“python3\”: executable file not found in $PATH: unknown” displayed on ControlScript execution in FIPS mode.

• AV-171173: Upgrade fails with error: “KeyError: ‘protocol” if config JSON has alerts with audit_compliance_event in it.

• AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.

• AV-171819: Postgres replication file cleanup logic blocking leader election after leader power-off.

• AV-171954: Memory for negative headers match cases is not getting allocated on Service Engine, which may result in SE failure while accessing the invalid memory.

• AV-172563: Garbage Collect API “/api/cloud//gc" fails for OpenStack cloud.

• AV-173569: There may be a benign SSHD crash on the Controllers when upgrading to 22.1.3, applying patches 22.1.3-2p2 or higher. However, there is no other impact on the system’s functionality.

• AV-174223: Potential failure of Log Manager during Geo-IP translation.

Issues Resolved in 22.1.3-2p1

Release Date: 28 February 2023

• AV-147689: IP addresses allocated to a VS VIP are not released when the creation or update of VS VIP fails.
• AV-169086: Unable to log in to NSX Advanced Load Balancer using Safari.
• AV-168432: ControlScript does not work in a Docker Controller.
• AV-168904: When a DNS virtual service is attached to a deactivated GSLB service and later deleted, it causes stale entries in the GSLB service DNS virtual service list. When such a GSLB service is enabled, it may cause SE failure or memory corruption.
• AV-169398: Let’s Encrypt renewal breaks in version 22.1.3 if renewal is executed by a different user other than that specified in the certificate management script parameters.
• AV-170014: Federated Health Monitors are not displayed in the UI.
• AV-170116: When a DNS virtual service is bound to disabled GSLB services, on disabling and re-enabling of the DNS virtual service, the virtual service may get stuck in the OPER_DOWN state.
• AV-170759: Internal error updating and obtaining metrics for some tables seen in the UI and REST APIs.
• AV-170926: CSR generation from the UI fails when certificate management profile associated with CSR has any sensitive fields. In case of Let’sEncrypt, CSR generation fails with the error Invalid Credentials.
• AV-171114: Queries triggered by metrics manager fail owing to the difference in query format due to Postgres migration from version 10 to 14.

What’s New in 22.1.3

Release Date: 31 January 2023
To refer to the upgrade checklist, click here.

Cloud Connector

AWS

• Support for Instance Metadata v2 (IMDSv2) .

• The provision to use GP3 Volume type as default type for AWS Elastic Block Store (EBS) that backs the SE VMs .

• Support for Regional endpoints.

GCP

• Shared VPC multi-NIC support via the UI.

LSC

• DPDK support for Mellanox ConnectX-6 ethernet adapters.

NSX-T

• Support of ENS mode for VMware deployments (Tech Preview).

• General Availability of NSX Security-only (Distributed Firewall (DFW) to a Distributed Virtual Port Group (DVPG) mode deployments in the NSX-T Cloud). This requires VMware NSX version 3.2.2 or later.

• The scale of virtual services has been increased from 2000 to 5000 with NSX-T cloud in the NSX Advanced Load Balancer large Controller cluster.

Oracle

• NSX-ALB support for load Balancing of applications running in Oracle Cloud VMware Solution (OCVS) SDDC .

VMware vCenter/ vSphere ESX

• Support for VMware vCenter 8.0.

• Support for NSX-T segments spanning multiple VDS in vCenter cloud.

• Provision to modify the datastore name in the Service Engine group for a vCenter cloud in the write mode using the UI.

• The provision to assign user-defined vSphere tags to Service Engines during SE creation .

Core LB Features

• Username and ** Password** fields are treated as sensitive information in External Health Monitors and read as variables to prevent unauthorized exposure of authentication details .

• For HTTP request and response policies, under Matching Rules, new ** Criteria** introduced to match a Query with a string group or custom string in the URI .

• The maximum size of the HTTP request body for health monitors is increased from 1024 bytes to 5120 bytes .

• WebSockets support for different HTTP versions (HTTP/2 Client to HTTP/1 Server) .

• XFF Handling to retain one or more X-Forwarded-For headers coming with the request .

• The option to Enable HTTP2 Server Push using the UI.

• UI support to configure the maximum number of HTTP headers for requests and responses in Application Profile .

• The Enhanced Virtual Hosting (EVH) object supports multiple match criteria beyond host and path.

• Consistent Hash support in pools which are members of pool group .

• UI support for True Client IP in Layer 7 security features .

• Pool Configuration: Ability to select the Pool Type as Generic Application or OAuth .

DNS and IPAM

• Support to search for DNS records is added to the virtual service UI .

GSLB

• mTLS support for GSLB HTTPS health monitoring .

• The provision for multiple site persistence-enabled GSLB sites to use a single virtual service .

• UI support to define the priority for GSLB Service pools .

• The configuration to manually resume traffic for a GSLB pool member after it goes down .

Monitoring and Observability

• Inclusion of tenant_name to the Syslog formats when logs are streamed in an external server .

• Wildcard query parameter support in the GET API to fetch Prometheus metrics .

• API-support for Markers (previously known as labels) in log streaming body to be used as index for Splunk.

• The event Remote backup failed is generated when the copying backup files to remote server fails.

Networking

• IPv6 support for controller and SE communication using dedicated IPv6 secondary interfaces on the controller for IPv6 SE management to connect over (Tech Preview) .

• Dedicated secondary interfaces for secure channel connectivity to the SEs which can be configured for IPv6 IPs for SE management .

• Support for Stream Control Transmission Protocol (SCTP) (Tech Preview) .

• UI support for enabling SE packet capture tracking .

System

• Provision to evaluate the upgrade-readiness of the system prior to initiating the upgrade using the CLI. Enhanced upgrade experience with:
  • Visibility into the real time status of the pre-check evaluation
  • Clarity on errors and warnings displayed during the pre-check evaluation
  • On-demand detailed summary of all the pre-check evaluation steps that are performed to indicate upgrade readiness.

• UI support for upgrade of service engine groups belonging to non-admin tenants .

• Universal Client 10.4.1 support for Thales Luna HSM.

• The option to Import Private Key to HSM using the CLI and UI during certificate import .

Web Application Firewall (WAF) and Application Security

• The support to opt-in for CRS auto-updates for any WAF policy .

• When a CRS update is triggered, the option to choose if the new rules should be included into the DETECTION mode is provided .

• Bot Management: The support to detect, classify and manage bot traffic was introduced in version 21.1.1 and is now generally available.

• Support for @ipMatchFrFile operator.

• SSL/ TLS session persistence using session ID mapped to the server and port information using DataScripts .

• Support to dynamically schedule the next attempt to update CRL .

• Integration of Let’s Encrypt HTTP validation and certification automation for:
  • Resolving DNS-01 challenge
  • Automatic certificate renewal for imported certificates
  • Mechanism to force certificate renewal (on-demand)

• SSH service configured to ban/forbid Weak Host Key Algorithm and Key Exchange Algorithm.

• Caching of data received from introspection endpoint.

• Support for Client/Relying Party (RP)-initiated logout for OpenID Connect.

• Support to extract access token from the user’s OAuth session using DataScripts .

User Interface

• UI enhancements across configuration objects.

Issues Resolved in 22.1.3

• AV-144150: With connection multiplexing disabled, when the persistence cookie was sent in the second request, persistence was not honored and no persistence entry was made for the cookie.

• AV-145995: Possible configuration loss after a leader transitions to a follower which was not replicating the configuration from the leader. The configuration loss is seen if either one of the configuration’s and metrics’s database replication is working fine and the other is not.

• AV-148598: High CPU usage observed while streaming logs to external server using se_log_agent because of frequent connection resets.

• AV-148700: SSH service configured to ban/forbid Weak Host Key Algorithm and Key Exchange Algorithm.

• AV-150213: Frequent enabling and disabling of servers of a pool within a few seconds can cause loss of reporting of pool-server metrics on Service Engines with five or fewer virtual services placed on them.

• AV-150320: With SSL session resumption enabled, the pool’s SSL is using a TLS ticket from an SSL session with failed PKI validation.

• AV-150990: Unable to edit the VRF context in an NSX-T Cloud configured to use Overlay transport zones, through the UI.

• AV-151537: When an empty DataScript is configured from the CLI, the DataScript page fails to load in the UI.

• AV-154300: The server batch operations do not work when using markers on pools configured with granular RBAC.

• AV- 155317: DPDK driver failure with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.

• AV-154144: NSX-T cloud deployment fails to perform virtual service placement when the management segment name is changed on NSX-T.

• AV-154511: Controller versions 22.1.1 and 22.1.2 do not have the Service Engine metrics vm_stats.avg_cpu_usage , vm_stats.avg_mem_usage populated to the UI.

• AV-154738: The Avi Controller does not fetch all the services or groups from the NSX-T manager.

• AV-155145: Compliance mode by enabling common criteria is allowed without having to remove TLS 1.0 values from all the configured SSL/TLS profiles.

• AV-155512: Spaces in VIP Address Allocation Network between characters displays the error, Request field uuid contains bad character.

• AV-1555471: The NSX Advanced Load Balancer GCP cloud disables BFD on GCP Cloud routers while creating new VIP.

• AV-156737: Through the UI, the NSX-T cloud does not display all the segments available for selection for data or management segments when there are more than 1000 segments.

• AV-156741: Prometheus-metrics API doesn’t fetch all metrics for cases when output exceeds default dimension limit of 1000.

• AV-157154: In an NSX-T cloud in VLAN mode, the Server Network dropdown does not list placement server networks despite having configured subnets.

• AV-157333: Intermittent error in the IP reputation Sync.

• AV-157419: Infoblox IPAM/DNS fails to use the non-default DNS view.

• AV-157767: Over four billion packet transactions on a TCP connection can lead to a SE crash due to a counter overflow.

• AV-157962: Caching incomplete objects received from the backend server could lead to Service Engine failure if the connection to the server closes abnormally.

• AV-158056: WAF ignores the ignore_incomplete_request_body_error flag from the WAF profile when running in DETECTION mode.

• AV-158229: REST requests fail if the API version is set prior to 20.1.1.

• AV-158310: LDAP authentication fails when multiple LDAP auth profiles are configured.

• AV-158550: A no-access deployment of NSX Advanced Load Balancer in legacy HA mode in OpenStack environment sends a copy of the packet to standby SE, causing issues with virtual service traffic.

• AV-158634: On upgrading to 22.1.1-2p3, the NSX Advanced Load Balancer UI fails to load.

• AV-159182: During network downtime, packet buffers can get queued up causing packet buffer exhaustion leading to SE failure.

• AV-159203: Memory exhaustion on Service Engine causes Service Engine failure when attempting to establish a connection to the LDAP server.

• AV-159228: A virtual service with only an EC certificate with OCSP stapling enabled, can cause Service Engine failure.

• AV-159311: Under memory pressure , SE may fail due to connection memory allocation failures when processing buffered requests.

• AV-159031: Infoblox IPAM configuration cannot be completed using the UI if the IPv6 field is blank.

• AV-159527: The operational state of the GSLB pool Member operational state may be shown as OPER_DISABLED even when is it UP on one of the Service Engines.

• AV-159539: SNMPv3 traps and SNMPv3 GET responses use different engine ID.

• AV-160229: In NSX Advanced Load Balancer version 22.1.1, SE creation might fail in the NSX-T cloud setup if the management network is of Type Opaque Network.

• AV-160400: When working with a virtual service and an attached WAF policy in a non-admin tenant, log recommendations might give the error message WafCRS not found.

• AV-160418: Certificates with wildcard domains are not getting processed when using Enhanced Virtual Hosting.

• AV-160532: In a setup with large number of configuration objects, when flexible upgrade is initiated, there could be loss of configuration to the Service Engines running in the older version, leading to traffic loss.

• AV-160593: When Client Insights for a virtual service is set to Active, the virtual service is vulnerable to an HTTP desync attack on the /__avirum__ endpoint.

• AV-160771: SE fails to come up due to memory fragmentation in DPDK mode when the packet buffer’s memory exceeds 16G. With this update, the packet buffers are reduced, and the SE would be operational, but in a degraded mode. A host reboot is required for this update to take effect.

• AV-160898: Under some conditions, in virtual services referring to a WAF policy in which WAF CRS is selected at the SE boot up, some rules in the CRS section of WAF do not run all transformations before evaluating a request causing false negatives in rules 941160, 941170, 941210, 941220, 941310, 941350 and 942190.

• AV-160899: Switching Persistence profiles between App cookie and client IP Address can lead to SE failure.

• AV-160929: SYSLOG_JSON format-based syslog for config update event-based alerts can result in failure in sending syslog messages.

• AV-161155: Several failed tasks in vCenter displayed the status The request refers to an unexpected or unknown error type. However, the failed tasks do not interrupt services on NSX Advanced Load Balancer or vCenter.

• AV-161259: SE failure when updating HTTP Policy sets to stop using IP Reputation database and when the SE handles HTTP persistent connections during the update.

• AV-162794: If any Tier 1 gateway or segment configured in NSX-T cloud is deleted, it marks the cloud down, and may cause disruption on all the virtual services configured in the cloud.

• AV-162724: OpenStack Cloud: During VSVIP creation or update, IP address from a different subnet may be allocated when multiple subnets are available in the GUI.

• AV-162948: L3 encapsulation for a scaled-out UDP virtual service with the udp-per-pkt load balancing network profile may lead to SE failure.

• AV-163134: The show service engine command can cause SE failure if the command output is too long to fit in a page.

• AV-163620: Memory leak when the flag collect_client_fingerprints is enabled in the application profile.

• AV-164508: SE might fail while processing multiple EVH HTTP/2 requests.

• AV-164511: Log streaming stops working intermittently.

• AV-165093: Rules being removed from the IP table on deleting management access control on the Controller.

• AV-165161: Service Engine failure when a HTTP/2 server sends two RST_STREAM frames consecutively after a DATA frame with END_STREAM flag enabled in the same stream.

• AV-166183: Filename format of the Analytics Engine’s event mapping index leads to scale and performance issues for event handling.

• AV-165248: From NSX Advanced Load Balancer version 21.1.4 onwards, disabling one of the virtual services with shared VIP might display a warning if any of those virtual services have child virtual service.

• AV-166279: Service Engine failure seen with NTLM requests with Unicode characters in the username.

• AV-167068: If a network is moved to a custom VRF, the virtual services /pools attached to the network will go down in vCenter cloud.

• AV-166777: On upgrade from version 20.1.7 onwards to patch versions 22.1.2-2px, the SE creation on Azure cloud fails because certain set of instance types are not supported by Azure anymore.

• AV-168482: In version 22.1.3, when updating a WAF policy via API, using an X-Avi-Version lower than 22.1.3, the update may fail with Cannot add or remove elements in the list of internal field WafPolicy.required_data_files. This can happen if a pre-crs or post-crs rule is added, which is using the @pmf operator on a data file from the WAF profile which is not already used in the current ruleset.

• AV-177074: Resume of suspended SE groups that have virtual services scaled out may result in the exception, “ Se-Scale-in Operations for SE failed.Timedout in executing ResMonWorkerService.SeScaleInRpc request_pb : se_uuid: <> transaction_uuid: <>”.

Key Changes in 22.1.3

• To optimize performance, the options to generate logs based on time frame (Past Year, Past Quarter, and All Time) are removed from the NSX Advanced Load Balancer UI. These logs continue to be available through the API. See API Guide and How to view logs on Avi Vantage for a specific duration for more information.

• In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

• The length of the Linux counterpart of interfaces in DPDK mode is limited to 11 characters. The interfaces whose names exceed 11 characters will be placed in fault state.
  To recover from the fault state, remove the SE from the cloud configuration, change the Linux bond interface name to include less than 11 characters, and then add the SE back to the cloud configuration.

• The vCenter cloud no longer updates the host name of the pool servers with the VM name. Starting with version 22.1.3, the Hostname of the pool server VMs will not be updated by the cloud connector. User-defined names can be used as hostnames. The cloud connector will maintain the ‘external_uuid’ field in sync with the vCenter VM’s name.

• Starting with version 22.1.3, the standard Controller image (controller.ova) is signed using a SHA-256 hash. This image is compatible with the deployment and life-cycle management of Avi Controller from the NSX Manager UI from NSX version 4.1.0 onwards. For older versions of NSX, the image controller-sha1.ova should be used.
  Note: This only impacts the deployment of the Avi Controller from NSX Manager.

• The application profile of a DNS virtual service cannot be updated if it is attached to the system configuration. To change the virtual service’s application profile type, detach the virtual service from the system configuration and then modify the application profile.

• The names of management and data networks imported from NSX-T will be updated on the Avi Controller to match the respective name in NSX-T.

• The procedure to restore a Controller cluster after a failure, via the restore_config script, has changed. Cluster formation is now a two-step process:

  1. Restore the configuration on one of the nodes
  2. Reform the cluster by inviting the two new nodes to the cluster In addition, the following parameters used by the restore_config script are no longer supported:
  • DO_NOT_FORM_CLUSTER
  • VIP
  • FOLLOWER_IP [FOLLOWER_IP ...] For more information, see Backup and Restore .
    

System Limits

• Route limit per VRF is increased from 500 to 2000

• The maximum number of virtual services with real time metrics enabled in the Controller size LARGE has been increased from 200 to 1500.

• The minimum requirement of memory for the Controller in Essentials is increased from 16G to 24G.

Known Issues in 22.1.3

• AV-159518: Image upload fails for containerized Controller deployments while upgrading to version 22.1.3 from versions 20.1.1 through 20.1.7 and 21.1.1 through 21.1.4.
  
  • Workaround 1: Execute the following command under root user.
    

    
    $> sed -i "s/client_max_body_size 5000M;/client_max_body_size 10000M;/g" /etc/nginx/sites-enabled/default && service nginx restart && service maintenanceportal restart
    

  • Workaround 2: Manually update client_max_body_size to 10000M for /api/image in /etc/nginx/sites-enabled/default and restart maintenance_portal and Nginx.

• AV-157854: When configuring OAuth virtual services using the UI, at least one scope needs to be configured. If no scopes are required for the deployment, then configure a placeholder scope in the UI and remove it later using the CLI.

• AV-159426: Istio on AKO is not supported for NSX Advanced Load Balancer Controller version 22.1.3. AKO cannot push workload certificates from istiod to the Controller because of a change in certificate validation that requires common name to be present in certificates. The X.509 certificates generated by istiod, do not have common name.

• AV-163964: Possible connection failure when a Layer 4 virtual service is configured with TCP Fast Path profile, and the Service Engine MTU is lesser than both the client and server MTUs.
  Workaround: Disable the TSO feature for the relevant Service Engine groups:

  
  configure serviceenginegroup <SEGroup-Name>
  disable_tso
  save
  

• AV-173603: SAML authentication does not work with EVH if WAF is not enabled on the parent virtual service or if request body buffering is not enabled on the parent virtual service.
  Workaround: Enable request body buffering on the parent virtual service.
  Navigate to Templates > Profiles > Application. Select the required HTTP profile. Under the DDOS tab, click Enable Request Body Buffering to enable request body buffering for the parent virtual service.

• AV-174983: Importing a certificate without a subject Common Name fails with error { “error”: “‘common_name’” }.
  Workaround: Import the certificate with a Common Name. See the SSL Certificates article for more information on Common Name in SSL Certificates.

• AV-182114: Symptoms: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.
  Workaround: From the CLI, manually disable the SE which exhibits this behavior.

• AV-187931: When System-SCTP-Proxy TCP/UDP Profile is selected as network profile for virtual services, a port range cannot be specified under Service Ports. If a port range is configured, only the first port within the specified range handles traffic.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.3

Refer to this section before initiating upgrade.

• Upgrade to NSX Advanced Load Balancer to 22.1.3 is only supported from the following versions:
  

• Version 20.1.1 through 20.1.9

• Version 21.1.1 through 21.1.6

• Version 22.1.1 and 22.1.2

• Before upgrading to version 22.1.2 or higher, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data .

• Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory recommended for an Essentials Controller is 24G. Ensure that the memory of an Essentials Controller is at least 24G before upgrade.

• vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.x and higher.

• The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

• As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

For updates on Cloud Services in version 22.1.3, see NSX Advanced Load Balancer Cloud Services Version 22.1.3 is available on VMWare Docs .

Patch Release Notes for 22.1.2

Issues Resolved in 22.1.2-2p7

Release Date: 13 April 2023

• AV-166018: SE failure during boot-up due to race condition between SE-Agent and SE-log-agent.
• AV-157546: Connections may be dropped at the SE when GRO is enabled at the SE Group and when the TCP timestamp option is not present in TCP data.

Issues Resolved in 22.1.2-2p6

Release Date: 02 April 2023

• AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.
• AV-171581: Upgrade to 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IPs due to issues with updating DNS resolvers after upgrade.

Issues Resolved in 22.1.2-2p4

Release Date: 24 December 2022

• AV-162794: If any Tier 1 gateway or segment configured in NSX-T cloud is deleted, it brings down the cloud, and may cause disruption on all the virtual services configured in the cloud
• AV-161155: Several failed tasks in vCenter displayed the status ‘The request refers to an unexpected or unknown error type’. However, the failed tasks do not interrupt services on NSX Advanced Load Balancer or vCenter.
• AV-157154: Server network dropdown does not have options listed.

What’s New in 22.1.2-2p3

Release Date: 16 November 2022

• Support for SSL Session ID persistence using DataScripts.
  Changes made to Default-TLS DataScript template through the UI are overwritten by the latest Default-TLS with this upgrade.

Issues Resolved in 22.1.2-2p3

• AV-152343: Virtual service placement is stuck at OPER_RESOURCES due to an internal race condition which clears the discovered networks on the virtual services.
• AV-155512: Spaces in VIP Address Allocation Network between characters displays the error Request field uuid contains bad character.
• AV-157962: Caching-incomplete objects received from the backend server can lead to Service Engine failure if the connection to the server closes abnormally.
• AV-158310: LDAP auth fails when multiple LDAP auth profiles are configured.
• AV-159228: A virtual service with only an EC certificate with OCSP stapling enabled can cause Service Engine failure.

Issue Resolved in 22.1.2-2p2

Release Date: 26 October 2022

• AV-158634: On upgrading to 22.1.2-2p1, the NSX Advanced Load Balancer UI fails to load.\

Issues Resolved in 22.1.2-2p1

Release Date: 20 October 2022

Note: Patch version 22.1.2-2p1 has been withdrawn due to a known issue (AV-158634: On upgrading to 22.1.2-2p1, the NSX Advanced Load Balancer UI fails to load. The Controller is available and functional via CLI and APIs). If you have already applied 22.1.2-2p1, do either one of the following: Roll back the patch. Upgrade to patch version 22.1.2-2p2, which has the fix to this issue.

• AV-157159: Avi SE creation fails in NSX-T security-only mode
• AV-156899: On the UI, the Trusted IP configuration is not saved in a WAF Policy with Learning enabled.
• AV-155317: DPDK driver crash with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.
• AV-153369: Create and edit APIs are slow.
• AV-151537: When VsDataScriptSet is configured from the CLI and has no events attached, it causes failure in loading the DataScripts page via the UI.

Known Issue in 22.1.2-2p1

• AV-158634: On upgrading to 22.1.1-2p1, the NSX Advanced Load Balancer UI fails to load. The Controller is available and functional via CLI and APIs. See advisory note for more information.

Issues Resolved in 22.1.2 Patch Releases

Issues Resolved in 22.1.2-2p5

Release Date: 01 March 2023

• AV-152343: Virtual Service placement is stuck at OPER_RESOURCES due to an internal race condition which clears the discovered networks on the virtual services.
• AV-159539: SNMPv3 traps and SNMPv3 GET responses are using different engine IDs.
• AV-163620: Memory leak when flag ‘collect_client_fingerprints’ is enabled in the application profile.
• AV-164049: vCenter cloud creation fails to discover vCenter objects, if there are any distributed virtual port group with traffic filtering and marking feature enabled.
• AV-165161: Service Engine may fail while processing consecutive RST_STREAM frames from a HTTP/2 server that belong to the same stream.
• AV-166777: On upgrade from version 20.1.7 onwards to patch versions 22.1.2-2px, the SE creation on Azure cloud fails because certain set of instance types are not supported by Azure anymore.
• AV-168413: During continuous config operations, the agent memory usage might increase over a period of time.
• AV-168432: ControlScript does not work in a Docker Controller.
• AV-168862: If a pool name has the character ‘/’ in its name, the part of the name after ‘/’ will be appended to the URI and sent to server.
• AV-169464: OpenSSL-1.1.1f stack in the Controller and Service Engine is vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.
• AV-169473: OpenSSL-1.0.2g stack in the Controller and Service Engine is vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.
• AV-171114: Queries triggered by metrics manager fail owing to the difference in query format due to Postgres migration from version 10 to 14.

What’s New in 22.1.2

Release Date: 29 September 2022

Cloud Connector

• AWS: Support for shared VPC
• ** Azure**: Support for Dsv5-series flavour
• NSX-T: Support for 32 NSX-T clouds per NSX Advanced Load Balancer Controller cluster.

Core LB Features

• Support for load balancing in the round-robin mode at a per-SE level instead of the default per core.

System

• Increased system limit values for virtual services, servers, and Service Engines for NSX Advanced Load Balancer.

Issues Resolved in 22.1.2

• AV-142908: On failure of a macro-API containing a VSVIP change or a VSVIP post failure, the IP allocated is not released on failure.
• AV-144150: With connection multiplexing disabled, when the persistence cookie was sent in the second request, persistence was not honored and no persistence entry was made for the cookie.
• AV-146153: IPAM auto-allocate function does not work properly in NSX Advanced Load Balancer when connected to an Infoblox IPAM with a non-default network view and DNS view.
• AV-148238: Oracle client-based external health monitor may cause failure of other external health monitors due to excessive logging.
• AV-149146: Increased disk usage when application signature is enabled in the pulse connector configuration.
• AV-149858: External logs are not received on the external server when the whole pod/container got deleted or re-imaged.
• AV-150877: Connections are terminated if the application profile is set as System-SSL-Application, and the session is idle for 10 minutes.
• AV-150977: Unable to set auth mapping profile in VMware NSX ALB Basic.
• AV-150990: Unable to edit the VRF context in an NSX-T Cloud configured to use Overlay transport zones, through the UI.
• AV-151386: Log recommendations for CRS rules 920470, 920320, 920340, and 920341, which recommend to exclude REQUEST_HEADERS: Content-Type and REQUEST_HEADERS: User-Agent from the rule results in a broken rule which will always FLAG or REJECT the request.
  
• AV-151431: When connection multiplexing is disabled, persistence to a pool from prior requests can override the content-switching pool group selected by an HTTP request policy.
• AV-151469: SSL profile with only TLS1.3 protocol and TLS1.3 ciphers can cause a fault on the Service Engine.
• AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
• AV-151550: Upgrade fails for the FIPS-enabled setup for which configuration was imported after FIPS-mode was enabled.
• AV-151763: Service Engine failure when an HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag.
• AV-151942: Fetching Transport nodes API fails when the transport_zone_id filter is used.
  
• AV-152018: NSX Advanced Load Balancer does not display an error for duplicate VIP addresses.
• AV-152071: Controller service (security manager) fails as postgres database connection is not concurrency safe.
• AV-152250: When using the Certificate Management profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
• AV-152343: Virtual service gets stuck in the OPER_RESOURCES state due to an internal race condition that clears the virtual service’s discovered networks.
• AV-152444: Portal connector service logs can reveal user-sensitive information configured in the system configuration.
• AV-152581: Postfix package has a stale dependency on open SSL 1.1.1 in FIPS mode.
• AV-153196: When connection multiplexing is enabled, with HTTP cookie mode of persistence, the cookie with the first request does not get sent.
• AV-153348: In VMware cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in the edit mode.
• AV-153627: The service engine might crash when disabling and enabling sharing pool.
• AV-153725: False alert about IP reputation and App Signature sync failure when a registered Controller is disconnected from the pulse portal.
• AV-153739: vCenter discovery may get stuck when using a static IP address for SE data vNIC allocation.
• AV-154157:When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down dramatically. This is especially the case if these exclusions are on a group level.
• AV-154173: Disabling debugging for a virtual service does not stop debug logs from being written by the Service Engine.
• AV-155045: On an update to a virtual service with the bgp_peer_labels configured, the virtual service briefly goes down and comes up, resulting in connection drops.
• AV-127214: SE failure due to incompatibility in hardware versions for LSC deployments on VMware ESXi VMs.

Key Changes in 22.1.2

• In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

• Connections are terminated if the application profile is set as System-SSL-Application, and the session idle time is set as 10 minutes. Now the idle connection timeout for SSL connections is increased to 60 minutes.
• Postgres has been updated from version 10 to
  1. Prior to upgrading to version 22.1.2, it is recommended to export the metrics database as in case of rollback from NSX Advanced Load Balancer 22.1.2, Avi metrics data will not be restored.
• Network objects in NSX Advanced Load Balancer now sync with the name of the associated port group in vCenter. Previously, changing name of the port group and name of the network in NSX Advanced Load Balancer was independent of each other.

Known Issue in 22.1.2

• AV-182114: Symptoms: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.
  Workaround: From the CLI, manually disable the SE which exhibits this behavior.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.2

Refer to this section before initiating upgrade.

• Upgrade to NSX Advanced Load Balancer to 22.1.1 is only supported from the following versions:
  

• Version 18.2.6 through 18.2.13

• Version 20.1.1 through 20.1.9

• Version 21.1.1 through 21.1.5

• Version 22.1.1

Note: Upgrade from version 21.1.6 to 22.1.2 is not supported.

• Before upgrading to version 22.1.2 or higher, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data .

• Starting with NSX Advanced Load Balancer version 22.1.2, the minimum memory recommended for an Essentials Controller is 16 GB. Ensure that the memory of an Essentials Controller is at least 16 GB before upgrade.

• vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

• The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

• As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

Issues Resolved in 22.1.1 Patch Releases

Issues Resolved in 22.1.1-2p6

Release Date: 03 April 2023

• AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.
• AV-171581: Upgrade to 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IPs due to issues with updating DNS resolvers after upgrade.

Issues Resolved in 22.1.1-2p5

Release Date: 21 March 2023

• AV-168413: During continuous config operations, the agent memory usage might increase over a period of time.
• AV-166279: Service Engine failure seen with NTLM requests with unicode characters in the username.
• AV-165161: Service Engine may fail while processing consecutive RST_STREAM frames from a HTTP/2 server that belong to the same stream.
• AV-164049: vCenter cloud creation fails to discover vCenter objects, if there are any distributed virtual port group with traffic filtering and marking feature enabled.
• AV-163620: Memory leak when flag ‘collect_client_fingerprints’ is enabled in the application profile.
• AV-161259: SE failure when updating HTTP Policy sets to stop using IP Reputation database and when the SE handles HTTP persistent connections during the update.
• AV-160898: Under some conditions, in virtual services referring to a WAF policy in which WAF CRS is selected at the SE boot up, some rules in the CRS section of WAF do not run all transformations before evaluating a request causing false negatives in rules 941160, 941170, 941210, 941220, 941310, 941350 and 942190.
• AV-160593: When Client Insights for a virtual service is set to Active, the virtual service is vulnerable to an HTTP desync attack on the /__avirum__ endpoint.
• AV-160229: In NSX Advanced Load Balancer version 22.1.1, SE creation might fail in the NSX-T cloud setup if the management network is of a type opaque network.
• AV-158267: Service Engine failure seen with NTLM requests with unicode characters in the username.
• AV-157546: Connections may be dropped at the SE when the TCP timestamp option is not present in TCP data.
• AV-156765: Once Cloud Services get disconnected, it does not get connected without manual intervention.

Issue Resolved in 22.1.1-2p4

• AV-158634: On upgrading to 22.1.1-2p3, the NSX Advanced Load Balancer UI fails to load.

Issues Resolved in 22.1.1-2p3

Release Date: 13 October 2022

• AV-155512: Spaces in VIP Address Allocation Network between characters displays the error Request field uuid contains bad character.
• AV-153369: Create and edit APIs are slow.

Issues Resolved in 22.1.1-2p2

Release Date: 16 September 2022

• AV-154173: On enabling debugging for a virtual service, and disabling it, there are still debug logs written by the SE.
• AV-154157: When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down drastically, especially if the exclusions are on a group level.
• AV-153348: In VMware Cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in edit mode.

Issues Resolved in 22.1.1-2p1

Release Date: 01 September 2022

• AV-152250: When using the Certificate Management Profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
• AV-151763: Service Engine failure when a HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag set.
• AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
• AV-150990: Unable to edit VRF in NSX-T Cloud in Overlay mode through the UI.
• AV-150977: Unable to set auth mapping profile in the Basic edition.
• AV-148246: Parallel execution of the SSL certificate renewal scripts may fail.

What’s New in 22.1.1

Release Date: 15 July 2022

Cloud Connector

• AWS: UI support for schedule-based scale-out and scale-in of ASG servers.

• NSX-T: Support to create tenant-scoped clouds of type NSX-T.

• ** VMware**: Enhanced vCenter cloud for better performance and support for Content Library .

Core LB Features

• HTTP/2 support for virtual services with enhanced virtual hosting (EVH) enabled.

• HTTP/2: Support for dynamic HPACK table on the server side.

• Support for HTTP/2 server push which allows a server to pre-emptively send dependent resources based on the client’s request .

• Support to mark the Custom Value field in the HTTP Modify Header action of HTTP policies as sensitive, which when enabled, masks the value in UI, CLI, API and is encrypted in backup.

• Support to configure maximum number of HTTP headers in request and response in the HTTP Application Profile .

• Support to pass along X-Accel headers to the HTTP response sent to the client .

• Support to load balance FTP natively both for passive and active FTP.

• Provision to view cluster VIP runtime status including the status, time at which the last update was made, and status message.

• Support to show unique, user-defined error messages with reasons for failure in external health monitor logs using the tag nsx_alb_ext_hm_err_msg .

• Support to auto-detect NTLM apps with only negotiate headers present in the server and disable connection multiplexing .

GSLB

• Support to configure a topology policy to select the GSLB Service pool .
• Support for Adaptive Replication Mode.

Networking

• CLI support to configure GRO based on timer-mode using the field dpdk_gro_timeout_interval in SE group properties .

• Introducing timer mode of GRO operation to deliver better performance and reduce CPU utilization compared to the static mode GRO .

• Support to configure SE NTP servers for VM-based and LSC-based deployments .

• The ability to automatically configure multi-queue (RSS) and a number of dispatchers for SE deployments on public cloud (GCP, AWS, Azure)

Monitoring and Observability

• Support to exclude or include system events in All Events and Config Audit Trail pages.

• Support for RTM in prometheus-metrics API calls.

SDK and Integrations

• Multi-tenancy support for VMware ALB VRO Plugin.

• Swagger support for Basic, Essentials, and Enterprise licensing tiers.

System

• Support for remote file transfer protocol type SFTP (FTP over SSH) for configuration backup. .

• User-configurable parameter login_failure_count_expiry_window introduced to allow user to provide timeframe in which past login_failure_counts to be considered for account_lockout .

• Support for HSM integration in FIPS mode .

• UI support for configuring multiple remote authentication profiles . Ensure the existing automation scripts related to admin auth configuration(remote-auth) admin auth configuration ( remote-auth) are changed to version 22.1.1 to use the new auth model.
  Note: The API to configure remote auth is not backward compatible.

• Support for migration of virtual services from NSX DataCenter to NSX Advanced Load Balancer 22.1.1 using the stand-alone * NSX-T Convertor Tool*.

• Support to configure default tenant for remote users . .

• Support to apply and enforce label-based permissions on cloud objects .

• Support to set the from_name 0f the sender from the ** From** field in e-mail configuration via CLI.

User Interface

• UI enhancements across configuration objects.

• The SE UUID column is introduced to the Service Engine page as an optional column.

• Controller name and site name (if the site name is available), are displayed on the browser tab.

• Support to search NSX Advanced Load Balancer objects using markers from the NSX Advanced Load Balancer UI.

Web Application Firewall (WAF) and API Security

• WAF learning can be restricted to requests that come from a specific IP group or that are authenticated. If both an IP group and authentication are configured as condition for WAF learning, either condition is sufficient .

• Support to record the total memory consumed by WAF to process a request in virtual service application logs .

• Support to flag traffic via WAF when the TLS SNI and host header are different .

• Support for content type character sets in WAF Profile.

• Support to deactivate Bypass Static Extensions in WAF Policy.

• CLI support to import SSL private key to Thales Luna HSM .

• Support to:

  • Apply log recommendation to pre-CRS and post-CRS rules

  • Include recommendations for request time and regex complexity transgressions

  • Add exceptions on Request_Cookies_Names and Request_Headers_Names in WAF thereby allowing adding recommendations to the fields.

• TLS fingerprinting for Bot detection (Under Tech Preview).

• Support to send Assertion Consumer Service (ACS) URL and index to the IDP as part of the auto-authentication .

Issues Resolved in 22.1.1

• AV-132402: Setting non-default argument separator in the WAF Profile takes no effect.

• AV-137080: BFD echo mode does not work with NSX Advanced Load Balancer.

• AV-139518: On converting a No Orchestrator cloud to NSX-T cloud, some fields specific to the NSX-T cloud are read only and can not be configured. The option to and from the NSX-T Cloud type is not supported and the option to do so has been removed from Convert Cloud Type.

• AV-136469: When adding a GSLB pool member for a follower site through the NSX Advanced Load Balancer UI, clicking the Virtual Services drop down list displays an error VirtualService object not found!.

• AV-140199: For the TLS client, handshake API does not work as expected when connection is terminated after log server restart.

• AV-141435: Shell login hangs when the number of connections reaching WAIT_TIMEDOUT increase on the Shell server.

• AV-141493: When the Controller of version 21.1.3 or higher is configured with Cloud Services, rolling back Service Engines to version earlier than 21.1.3 results in failure of the corresponding SEs.

• AV-142030: Password reset link for admin account fails with the error message {error: “Invalid token”}.

• AV-142116: When incoming fragmented IPv4 packets (carrying TCP payload) post-reassembly get redirected to SE Linux interface in DPDK mode of operation, exhibit issue with IP checksum.

• AV-142174: Service Engine can fail if a virtual service is deleted while an ICAP request is being processed.

• AV-142218: False positives in Bot Management as requests are classified as Bad Bot based on the fact that the source IP is from public cloud providers range.

• AV-142620: Under VS VIP configuration, under Private IP, when the VIP Address Allocation Network is updated, the NSX Advanced Load Balancer UI was retaining the IP address associated with the network configured earlier.

• AV-143099: SSL certificate generation using control scripts for flows trying to connect to external SSL certificate authority (for example, LetsEncrypt, Venafi, Sectigo) may fail.

• AV-143121: With Infoblox IPAM, if an invalid domain is specified in the config, host record creation requests result in a timed-out error from Infoblox leading to the leader node UI and CLI becoming unresponsive.

• AV-143198: Service Engine may fail if the L7 virtual service listening service is configured with L4 app profile using override_application_profile and is followed by the virtual service’s network profile update.

• AV-146331: DNS section for virtual service VIPs were not loading for AWS and Azure Cloud types.

• AV-143699: When using WAF and CRS rules, a CRS rule which is part of a default deactivated CRS group (for example, group CRS_950_Data_Leakages) is executed.

• AV-143798:
  • Controller cluster goes down because the node appeared to run out of listening sockets
  • Intermittent 401 errors when trying to create or edit configuration via Terraform.
  • Internal goroutine API calls to the Controller display 401 errors

• AV-143988: POST API call made to Macro API /api/macro containing GSLB objects fails with the error message ” error”: “_perf() got multiple values for keyword argument ‘defer_octavius_request’“.

• AV-144016: SE might crash when updating a WAF policy that is referenced by a virtual service in fault state, with open connections.

• AV-144226: In a combination of virtual services with different network profiles, when Ignore Time Wait is enabled in some network profiles and disabled in the others, Ignore Time Wait enabled in a TCP proxy profile is not honored.

• AV-144235: Packet capture is not working on a virtual service when dedicated dispatcher is enabled on the SE.

• AV-144262: Creating/ updating IP address groups fails with the error {“error”: “Check checks.IpAddrGroupCheck Panicked!”} when UUID is present in the system configuration (ApiAccess and SshAccess).

• Upgrade fails in the WaitUntilClusterReadyLocally task due to timeout on waiting for the image_manager queue.

• AV-144544: When using write-access OpenStack cloud connector in large OpenStack environments, the NSX Advanced Load Balancer API can time out during bulk virtual service VIP operations.

• AV-144971: Updating large IpAddrGroups can fail with a service timeout.

• AV-145264: Creating a DNS-type Health monitor without any input in the dns_monitor field (keeping the dns_mmonitor field blank) results in a failure.

• AV-145662: NSX-T cloud creation is failing if there is no input in the Object Name Prefix, although this field is not mandatory in the UI.

• AV-145696: When the virtual service VIP is deleted from the Controller, the corresponding AWS Route 53 records are not removed.

• AV-145754: HTTP requests received with both Content-Length and Transfer-Encoding:Chunked headers, will be generating a significant application log with the message Client sent a request with both chunked Transfer-Encoding and Content-Length header.

• AV-146000: When sending RST packets, longstanding flows (for more than 30 sec) during upgrade leads to longer timeouts.

• AV-146188: Deleting an FQDN from virtual service VIP deletes all the FQDNs of a VIP on AWS Route 53.

• AV-146644: The error NUM_VIRTUALSERVICES: limit value 200, object count 200 is displayed when creating the 200th virtual service in UI of medium and large Controller sizes.

• AV-146648: se_agent segmentation fault when Controller cluster size changes while a user-agent cache request ( required for bot management) is ongoing from SE to Controller.

• AV-146774: When the albservicesconfig object is updated through the CLI or the API, there is a subsequent delay in syncing IP reputation and app signature, depending upon the configured time interval for service.

• AV-147689: IP addresses allocated to a VS VIP are not released when the creation or update of VS VIP fails.

• AV-148117: In case of an LSC cloud type with se_dp_isolation enabled, when the system is in stress, the show serviceengine cpu command might get stuck occasionally.

• AV-127214: SE failure due to incompatibility in hardware versions for LSC deployments on VMware ESXi VMs.

Key Changes in 22.1.1

• In version 22.1.1, the string length of the name field for all objects cannot exceed 256 characters. Creation and modification of an object fails if the name exceeds the maximum string length except for the following objects, for which the maximum string length is 280 characters:
  • DNSPolicy
  • HTTPPolicySet
  • NetworkSecurityPolicy
  • VsVip
  • Pool
  • PoolGroup
    If any name exceeds the maximum character threshold on upgrade, the upgrade will fail during the migration step and rollback. See Checklist for Upgrade for more information.

• In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

• Only single X-forward-proto will be sent to the server. If the client request contains an X-forward-proto header, then NSX Advanced Load Balancer rewrites it.

• Search of usable networks in IPAM is now insensitive to case.

• Jumbo frame support for NSX Advanced Load Balancer environments revised.

• If user-defined bot mapping is specified in the bot detection policy, no input is required in the system bot mapping reference.

• The name of individual BotMappingRule objects in a BotMapping object is mandatory. Hence, you will not be able to create any new objects without a name.

• ControlScripts that make API calls back to the Controller API using localhost must be updated to use the DOCKER_GATEWAY environment variable instead.

• It is recommended for a Service Engine to have at least 4 GB of memory when GeoDB is in use.

• Prior to NSX Advanced Load Balancer version 22.1.1, it was only possible to control the update (PUT) action on any resource field. Starting with NSX Advanced Load Balancer version 22.1.1, if the access is disallowed for any field, creation of objects is not permitted as well.
• Network objects in NSX Advanced Load Balancer now sync with the name of the associated port group in vCenter. Previously, changing name of the port group and name of the network in NSX Advanced Load Balancer was independent of each other.

Ecosystem Changes

• vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

Known Issue in 22.1.1

• AV-155317: DPDK driver crash with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.

• AV-182114: Symptoms: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.
  Workaround: From the CLI, manually disable the SE which exhibits this behavior.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.1

Refer to this section before initiating upgrade.

• Upgrade to NSX Advanced Load Balancer to 22.1.1 is only supported from the following versions:
  

• Version 18.2.6 through 18.2.13

• Version 20.1.1 through 20.1.9

• Version 21.1.1 through 21.1.4

• Version 21.1.5 through 22.1.1

• Starting with NSX Advanced Load Balancer version 22.1.1, the minimum memory recommended for an Essentials Controller is 16 GB. Ensure that the memory of an Essentials Controller is at least 16 GB before upgrade.

• vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

• The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

• As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

For updates on Cloud Services in version 21.1.1, see Cloud Services Release Notes .

Supported Platforms

Refer to System Requirements: Ecosystem

Product Documentation

For more information, please see the following documents, also available within this Knowledge Base.

Installation Guides

• NSX Advanced Load Balancer Installation Guides

Copyrights and Open Source Package Information

For copyright information and packages used, refer to open_source_licenses.pdf.

Avi Networks software, Copyright © 2015-2022 by Avi Networks, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php

Additional Reading

VMware Ports and Protocols