Upgrades in an Avi GSLB Environment

This article explains how infrastructure administrators and application owners may approach their respective upgrade scenarios in an Avi Vantage deployment in which GSLB has been configured. Some circumstances may mandate an immediate cessation of service; others may permit a graceful upgrade, with no interruption of service. Avi Vantage’s GSLB implementation supports graceful upgrades, even when challenged by actions such as:

  1. A new version of Avi Vantage — and potentially the GSLB code itself — needs to be installed on active and passive GSLB sites.
  2. A site-wide change must be made.
  3. An application server (or servers) must be replaced or its (their) configuration changed.
  4. A new version of a global application’s executable needs to be installed on all site servers currently running it.

Avi Vantage Upgrades

Notes:

  1. To avoid service disruption due to site upgrade, it is recommended that the DNS virtual service be scaled out to at least two Service Engines within the SE group.
  2. For general information on the Avi Vantage upgrade process, refer to How to Upgrade Avi Vantage Software.

Pre-upgrade Steps

Prior to Avi Vantage release 20.1.1, the GLSB upgrade process includes setting up the GLB sites in the maintenance mode. In the maintenance mode, none of the GSLB sites can participate in the GSLB process and configuration changes are not allowed on any of the GSLB sites.
Starting with Avi Vantage release 20.1.1, the followings are the two modes available for the upgrade of the active follower site. Enable one of these modes before starting the upgrade process:

  • Suspended Mode (available only for the active follower site)
  • Maintenance Mode

In the suspended mode, configuration changes are prohibited only on the follower sites, participating in the upgrade or maintenance activity. The other sites (the leader sites, and the remaining follower sites) still participate in the GSLB process. In the suspended mode, the GSLB leader sites accepts the configuration. The configuration changes are synchronized to all the follower sites except the one on which the suspended mode is enabled. If any error or wrong configuration occurs, it is localized to the specific site only. The error or crash issues do not get propagated to the other sites, and it does not affect the application hosted through GSLB. The application is accessible while the upgrade or maintenance work continues on one of the active follower sites. In suspended mode, configuration on the GSLB leader is not prohibited. It can have all the CRUD operations. After the upgrade, once the follower site is set to no suspended-mode, the follower will receive all the configuration from the leader site.
Notes:

  • It is applicable to the follower site only. It cannot be applied to the leader site.
  • All configuration sync is performed once the site exits the suspended mode.
  • After the upgrade, the one-time configuration synchronization is performed between the follower sites and the leader site.

Pre-upgrade Steps for GSLB Leader Site

For upgrading the leader site, only the maintenance mode is available. In maintenance mode, the configuration changes cannot be performed at any of the GSLB sites. For any new configuration, all the sites have to wait for the completion of the upgrade process. The maintenance mode is a global setting, it gets applied to all across the GSLB sites, and it is not local to only one of the sites.

Enabling Suspend Mode

Login to the Avi CLI and follow the step mentioned below:


[admin-controller]: configure gslb default
[admin-controller]: gslb> site index 1
[admin-controller]: gslb> suspended_mode

Post the upgrade of the follower site,the no suspended-mode is enabled on the follower, and it sync the information available on the leader site.


[admin-controller]: configure gslb gs1
[admin-controller]: gslb> site index 1
[admin-controller]: gslb> no suspended_mode

Note:

  • Disable the DNS virtual services on the active follower site when enabling the suspended mode on a specific site. Once the planned activity is over, re enable the DNS virtual services.

Maintenance

On the GSLB leader: Enable GSLB maintenance mode via the REST API or the following CLI command:

gslb maintenancemode enabled

As a result, Avi Vantage takes the following steps:

  • GSLB configuration changes are blocked. We do not want any GSLB configuration changes applied at the leader site while follower sites are getting upgraded.
  • Health-monitor probe frequency is reduced by changing the send_interval to 30 minutes.
  • The cached states of remote sites are not flushed for time T, according to the calculation:

    T = gslb_cfg.send_interval x gslb_cfg.clear_on_max_retries

    The remote site is not declared down during this interval.

Upgrading the Site

Upgrade the site to the new version of Avi Vantage. Typically, depending on configuration, the number of virtual services, etc., this may take anywhere from 15 - 45 minutes for the entire site (Controllers and SEs) to be upgraded. Followers are upgraded first; the leader is the last to be upgraded.

Upgrade status can be checked via the CLI command show upgrade status.

Post-upgrade

  • If before the upgrade of the follower site, the suspended mode is selected for the follower site, enable the no suspended-mode for that specific follower site.
  • If the maintenance mode is used as the pre-upgrade step, disable GSLB maintenance mode on the leader site via the CLI command gslb no maintenancemode. After performing this step, the newly upgraded remote site can build its runtime states from the rest of the GSLB ecosystem.
  • On the other hand, if ready to upgrade the next site, you may skip step 3.

Repeat steps 1 through 3 for all followers and finally the leader itself.

Note: After all sites have been upgraded, Ansible modules should be migrated to the latest version.

Error Scenarios

  • Follower Upgrade Failure: If the upgrade of a follower site fails, it will end in a GSLB disruption. The site will rollback to the previous version and come up. On the GSLB leader, disable maintenance mode. After determining the reason for failure and addressing it, resume the upgrade process.

  • Leader Upgrade Failure: On the leader, disable maintenance mode. Determine the reason and address it. In the meantime, no new features can be turned on.

Mixing Avi Vantage Versions

Avi recommends that all sites participating in GSLB run at the same Avi Vantage version and maintenance release.
During upgrade cycles, it may be not be possible or desirable to upgrade all sites to the same software version in a single maintenance window. In this situation, it is supported to mix sites running different versions of Avi Vantage for a period of time with the following important caveats:

  1. The GSLB leader site must be the last to be upgraded. It is explicitly not supported for the leader site to be running a later software version or maintenance release than any of the GSLB follower sites.
  2. GSLB configuration changes can be carried out on the leader during the period that sites are running mixed software versions with the caveat that new GSLB features in the later software version cannot be turned on until all sites have been upgraded.

Other Upgrades (not upgrading Avi Vantage itself)

In this section scenarios are illustrated via the Avi UI. The same operations are possible from the CLI and REST API.

CAVEAT: Upgrade actions described below are GSLB operation actions, not GSLB-configuration-changing options. Therefore, there is no guarantee (for a variety of reasons) that clients will “stop knocking at the door” of member virtual services previously known to them. In-flight connections to a VIP/VS at a site are not affected. This could be potentially addressed by changing the DNS TTL, but some intermediate DNS cache may not honor the TTL; traffic may continue arrive to the erstwhile VIP.

Upgrade a Global App on Specific Servers — The Member Disable-Enable Option

When it is time to upgrade a generally sound global application to fix bugs and/or add features, we use this option. It is graceful and non-disruptive. The fact that multiple sites are already delivering the service suggests that one site’s service can be turned off and upgraded while traditional service from N-1 remaining sites continues. This is the classical rolling upgrade scenario.

Refer to figure 1. On the GSLB leader, on a site-by-site basis, use the GSLB pool editor to uncheck the Enabled option of the relevant pool member (gs11 in the case of figure 1).

Note: We prefer that a member VS be upgraded after it has reached a quiescent state.

Next, make required changes to the application on the servers within that virtual service’s server pool(s), re-launch the application, and then check Enabled back on.

GSLB pool editor Figure 1. Pool member is enabled

Behind the scenes, disabling the GSLB pool member causes Avi Vantage to change the global DNS (the member’s IP address is withdrawn) such that traffic to the member virtual service (gs11 in this case) ceases. Traffic remains distributed across the remaining sites that host the application.

Upgrade a Global App Pervasively — The Global Service Disable-Enable Option

In some cases, a global application needs to be suspended immediately and pervasively, for example, until some bug or security breach has been addressed at all participating sites. In this case, rather than disable a member at a time as described above, we want all members to stop as a set. The global-service-disable option is the option.

Refer to figure 2. On the GSLB leader, navigate to Applications > GSLB Services. Click the box at the left of the application’s row. Then click on the Disable button. Do not click on Delete, as that would completely permanently remove the global service from the system configuration, which is not the goal.

GSLB services list Figure 2. GSLB services list

Behind the scenes, Avi Vantage changes the global DNS such that the FQDN is withdrawn.

Upgrade All Global Apps at a Specific Site — The Site Disable-Enable Option

Imagine a bank of servers housing all of the enterprise’s virtualized global applications. If data center space is tight, upgrading them to next-generation servers demands a fork-lift upgrade. Running these global apps simultaneously on the old bank while also launching them on the new bank is not feasible. Downtime at the site is unavoidable, and during it, client load must served by the remaining GSLB sites. The term “DC drain” is often used to describe the first phase of such an upgrade. The site-disable option is the answer.

Refer to figure 3. On the GSLB leader, navigate to Infrastructure > GSLB > Site Configuration. Check the site(s) on which you want all GSLB apps disabled. Then click the Disable button. Do not click Delete, as doing so would permanently remove the identified site(s) from the GSLB configuration.

GSLB sites list Figure 3. GSLB sites list

Behind the scenes, Avi Vantage changes the global DNS such that the IP addresses of all virtual services at the identified site(s) participating in all global applications are withdrawn. Local applications are not affected. Responses to requests for the FQDN of global applications will contain the IP addresses of virtual services running on N-1 remaining GSLB sites.

Notes:

  1. We do not allow the leader site to be disabled. To overcome this limitation, leadership should be passed over to a site that has already been upgraded. From its vantage point the former leader can then be disabled.
  2. Once a site is re-enabled, Avi Vantage resynchronizes all relevant information to the newly enabled site.