Performing a Product Upgrade to a PAC Environment

Introduction

A PAC environment is typically required to provide continuous uninterrupted operation. Performing a product upgrade, maintenance, or other system upgrades can present a challenge for PAC management.

This document describes best practice you should adopt when performing a product upgrade to your PAC environment.

Characteristics of a PAC environment

A Performance and Availability Cluster (PAC) environment, regardless of its use to support CICS or JES services, has some characteristics that must be understood before attempting any upgrade to the PAC.

A PAC environment can be described as two or more enterprise server regions configured in a scale-out architecture and potentially distributed over multiple machines working together as a single logical entity.

Each enterprise server region accesses information from shared resources that manage locking and synchronization of the information.

The shared PAC resources are:

A PAC region database accessed by the Micro Focus Database File Handler (MFDBFH). This database holds locks and shares information used by the enterprise server regions in the PAC.
A cross-region database accessed by the Micro Focus Database File Handler (MFDBFH). The cross-region database holds resource locks that can be used to control resource access across enterprise server regions in the PAC.
A PAC Scale-Out Repository (PSOR) stored in a NoSQL database, typically Redis.
Optional, one of more additional Scale-Out Repositories (SORs) for storing TSQs and TDQs.
Important: For performance reasons and to facilitate upgrades, TSQs and TDQs should not be stored in the PSOR.
One or more datastore databases. Datastores are used to store an application's VSAM data files, providing transactional support. Datastores are also used to store files that do not require recovery services, such as spool files, loadlibs, and configuration files.

It is when one or more of these shared components change or when record structures stored in some of these components change that risk is introduced when performing an update or patch process.

Upgrade types

Any update or patch applied to the system can potentially change its behavior. It is important to identify the changes being introduced by the upgrade or patch and determine the best way to apply the changes that mitigates risk and maintains PAC availability. Micro Focus strongly recommends that you review the Patch Update ReadMe prior to planning its installation.

There are three possible upgrade types, their suitability depends on the compatibility of the components being changed:

If the upgrade or patch changes shared PAC resources but are compatible. In this case, the coexistence of different versions in the same PAC is possible and you can follow the steps described in the Compatible Upgrade Process topic.
If the upgrade or patch impact or change one or more of the current shared PAC resources but they are incompatible. For example, changes to the PAC region database, changes to the PSOR layout, or changes to other PAC mechanisms that are incompatible with the existing PAC. In this case, you can follow the steps described in the Incompatible Upgrade Process topic.
If the upgrade requires you to change the fundamental infrastructure of the PAC. For example, the existing RDBMS or SOR vendor must be changed then you will need to create a new PAC. In this case, you can follow the steps described in Configuring a PAC chapter. You can then migrate the data from your existing PAC into the new PAC. However, the migration of data might require downtime or the loss of some data.

If there is ever any question as to which approach should be taken then the Incompatible Upgrade Process should be followed.

When an in-place upgrade is not possible

You can contact Micro Focus Customer Care for assistance with an in-place upgrade of a PAC. In the unlikely event that an in-place upgrade is not possible the Patch Update ReadMe will provide information on how best to proceed. In-place upgrades might not be possible when moving to a new major product version.

Pre-production testing

Micro Focus strongly recommends that you test the update and update process in a staging environment before deployment to production. The staging environment should mirror your production environment as closely as possible while remaining separate. In addition to being able to replicate the update process, a pre-production environment also enables application code to be recompiled and tested. Pre-production testing gives you the opportunity to encounter and resolve any potential issues before they can impact your production system.

Upgrading PAC machines in groups

Both the Compatible and Incompatible PAC upgrade processes aim to maintain PAC availability during the upgrade steps. However, each enterprise server region must be stopped at some point, to be upgraded. It is inevitable that some connected or connecting clients will be impacted. The extent of the impact depends on the nature of the client sessions and when each enterprise server is stopped.

Consider a client that connects to a TN3270 listener and expects to maintain a long-lived connection while it submits transactions over a long period of time. The client will be disconnected from the enterprise server region when it is stopped to be upgraded. The client will have to reconnect and will likely connect to a different enterprise server region in the PAC. If that new enterprise server region is next to be stopped and upgraded, the same client might be disconnected multiple times. This situation is undesirable for any connecting client.

The impact can be minimized by upgrading the machines in groups, rather than one at a time. Provided that client traffic can be handled temporarily by a proportion of the enterprise server regions, Micro Focus recommends that the other instances are shut down, upgraded, and restarted in groups. This approach can then be repeated until all of the regions have been upgraded.

Managing client connections during a PAC upgrade

Before a running enterprise server region is stopped to be upgraded, Micro Focus recommends you first prevent new client connections, and if possible, enable existing client connections to complete. In addition, any scheduled work, such as Micro Focus Batch Scheduler Integration (MFBSI), cassub, and MQ triggered transactions should be suspended or rescheduled to occur after the upgrade process has been completed.

When a user changes the state of an enterprise server listener to "Disabled" (or "Stopped"), the listener will stop listening on the designated port number. Most load balancers can be configured to recognize this and automatically update their pool of live connection endpoints, to prevent new client connections while maintaining any existing connections. Ideally, the load balancer also provides the ability to manually manage the pool of live connection endpoints. This has the added benefit that the listeners can be manually re-enabled for testing prior to enabling traffic again from the load balancer. Setting the listener state to "Disabled", rather than "Stopped", ensures that the listener will not start automatically after the enterprise server region is restarted. Changing the state from "Disabled" to "Started" will enable new client connections again.

Note: This will only work if there are no existing client connections.

Micro Focus recommends that preparations for a PAC upgrade include verifying the impact of disabling the enterprise server listeners on a load balancer sitting in front of the PAC. Any existing long-lived client connections (for example, to a TN3270 listener) will be unavoidably terminated when the enterprise server region is stopped and errors are reported in the communications log. The list of active client connections can be viewed using Enterprise Server Common Web Administration (ESCWA), in the navigation tab click the required enterprise server region and then click MONITOR > Client to view the region's CLIENT LIST table.

Backing up the Micro Focus Directory Server (MFDS) configurations

Micro Focus recommends that you backup each enterprise server instance that is running a PAC if it is on a version of Enterprise Developer earlier than 6.0.

Backing up your enterprise server instance is not essential for Enterprise Developer 6.0 and later. This is because the Patch Update installer will automatically preserve the MFDS configuration.

See To Export the Repository for more information on how to back up the MFDS configuration. Select the option to export all the server definitions and all the related security configuration information. See To Import a Repository for more information on how to restore the configuration backup.

Backing up the Enterprise Server Common Web Administration (ESCWA) configuration

Micro Focus recommends using ESCWA to configure and administer your PAC. An instance of ESCWA running on a single host is typically chosen to manage the PAC and store the PAC and SOR configuration. All of the enterprise server instances in the PAC are then managed from this single ESCWA instance. In order to do this, the remote Directory Server under which each enterprise server region runs must be added to the ESCWA configuration. See Enterprise Server Common Web Administration for more information.

Micro Focus recommends that your ESCWA instance runs on a machine that is not part of your PAC. This configuration simplifies the PAC product upgrade process and means that the version of the product running ESCWA does not need to be same version as the machines running the enterprise server regions that make up the PAC.

Note: For product versions prior to Enterprise Developer 6.0, the ESCWA configuration is stored in the application configuration file commonwebadmin.json. The application configuration file is located in $COBDIR/etc. This file must be manually backed up and restored.

The compatible and incompatible PAC upgrade processes

To illustrate both the compatible and incompatible PAC update processes, we can consider the following PAC configuration. A PAC with the named MYPAC, which has a PSOR named MYPSOR. The PAC consists of two enterprise server regions named Region 1 and Region 2. The enterprise server regions run on separate machines named Host 1 and Host 2:

In this example, the PAC consists of two enterprise server instances running on two separate machines. If the PAC consists of more than two enterprise server instance, Micro Focus strongly recommends that you consider upgrading the machines in groups rather than one at a time. Group updates minimizes the impact on connecting clients. See Upgrading PAC machines in groups for more information.

In this case, we can assume that the Enterprise Server Common Web Administration (ESCWA) service is running on a separate machine and does not need to be backed up and restored as part of this example.

We also assume that the Directory Server configuration for each enterprise server machine has already been backed up. This is not required for Enterprise Developer 6.0 and later, as the Patch Update installer will automatically preserve the MFDS configuration.