5.2 Upgrading CIS in Multi-Server Environment

5.2.1 Prerequisites on Source Server

  • Source Server: OES 2018 SP3 or OES 2023.

  • Ensure that the CIS server is upgraded to OES 23.4.

5.2.2 Recommendations for Upgrading CIS

Ensure to follow this sequence when upgrading CIS:

  1. Infrastructure servers

  2. CIS servers

  3. Data Scale servers

Type of Server

To find out whether the server is an Infrastructure or CIS or Data Scale, execute the following command on the server:

cat /etc/opt/novell/cis/configurationStatus.json

The configType parameter displays the following:

  • "configType":"infraHA" - This is an Infrastructure server.

  • "configType":"core" - This is a CIS server.

  • "configType":"scale" - This is a Data Scale server.

5.2.3 Upgrading Infrastructure Servers

Prerequisite

After upgrading to OES 23.4, ensure to check the data path port of each infrastructure server, and if the port number is missing in the firewall, add it.

  1. Check the data path port.

    docker info | grep "Data Path Port:"

  2. Add the data path port in UDP Ports section using yast2 firewall.

Procedure

  1. Stop the infrastructure services on any one of the node with the following command:

    sh /opt/novell/cis/bin/cis_ext_service.sh stop

  2. To verify that infrastructure services are stopped, run the following command:

    sh /opt/novell/cis/bin/cis_ext_service.sh status

    The output displays, “nothing found in stack: cis”.

  3. Ensure that all the infrastructure nodes are on OES 23.4. Run the following command on all the infrastructure nodes (3/ 5/ 7 - based on the configuration) to upgrade the service:

    sh /opt/novell/cis/bin/cis_ext_service.sh upgrade

    The output displays, “Upgrade completed successfully.

  4. Run the following command on any one of the node to start the infrastructure service:

    sh /opt/novell/cis/bin/cis_ext_service.sh start

  5. Run the following command to display the status of all the CIS services:

    docker ps

    NOTE:Until the data transfer from Elasticsearch to OpenSearch is completed, Elasticsearch service will be displayed in docker ps output.

5.2.4 Upgrading CIS Servers

  1. On node 1, upgrade the CIS server to OES 23.4.

    1. If CIS servers are configured with OES Cluster Services (NCS), then migrate the cluster resource to the upgraded node 1. Cluster resource will be up and running or might go to comatose because the CIS services are not updated with the latest changes that are done with OES 23.4.

    2. Make the CIS cluster resource offline and manually mount the volumes (where CIS is configured).

    3. Upgrade CIS by running the following script:

      sh /opt/novell/cis/bin/cis_upgrade.sh

  2. Modify the Load, Unload and Monitor script.

    1. Log in to iManager.

    2. Under Roles and Tasks, select Clusters > My Clusters, then select the cluster.

      If the cluster does not appear in your personalized list of clusters to manage, you can add it. Click Add, browse and select the cluster, then click OK. Wait for the cluster to appear in the list and report its status, then select the cluster.

    3. On the Cluster Manager page or Cluster Options page, select the CIS cluster resource to view its properties, then click the Scripts tab.

    4. Click the Load Script, Unload, or Monitor Script links to view or modify the scripts. If you modify a script, click Apply to save your changes before you leave the page.

      1. Edit the load script for the Cluster Pool as shown below:

        # update the links
        
        /bin/bash /opt/novell/cis/bin/update_cislinks.sh cis <New media path>
        
        # Beginning with OES 24.3, verify the cis subnet address
        /bin/bash /opt/novell/cis/bin/update_docker_subnet.sh
        
        # start the services
        
        exit_on_error /usr/bin/systemctl start oes-cis-fluentbit.service
        exit_on_error /usr/bin/systemctl start oes-cis-configuration.service
        exit_on_error /usr/bin/systemctl start oes-cis-auth.service
        exit_on_error /usr/bin/systemctl start oes-cis-data.service
        exit_on_error /usr/bin/systemctl start oes-cis-metadata.service
        exit_on_error /usr/bin/systemctl start oes-cis-policy.service
        exit_on_error /usr/bin/systemctl start oes-cis-mgmt.service
        exit_on_error /usr/bin/systemctl start oes-dashboard.service
        exit_on_error /usr/bin/systemctl start oes-cis-aggregator.service
        exit_on_error /usr/bin/systemctl start oes-cis-collector.service
        exit_on_error /usr/bin/systemctl start oes-cis-repaggregator.service
        exit_on_error /usr/bin/systemctl start oes-cis-repcollector.service
        exit_on_error /usr/bin/systemctl start oes-cis-gateway.service
         
        # wait before checking their status
        
        sleep 5
        
        # check the services
        
        exit_on_error /usr/bin/systemctl is-active oes-cis-fluentbit.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-configuration.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-auth.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-data.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-metadata.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-policy.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-mgmt.service
        exit_on_error /usr/bin/systemctl is-active oes-dashboard.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-aggregator.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-collector.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-repaggregator.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-repcollector.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-gateway.service
        
        # restart firewall if its running
        
        systemctl status firewalld.service
        
        if [ $? -eq 0 ]; then
        
            ignore_error systemctl restart firewalld.service
        
        fi
      2. Edit the unload script for the Cluster Pool as shown below:

        ignore_error /usr/bin/systemctl stop oes-cis-fluentbit.service
        ignore_error /usr/bin/systemctl stop oes-cis-auth.service
        ignore_error /usr/bin/systemctl stop oes-cis-data.service
        ignore_error /usr/bin/systemctl stop oes-cis-metadata.service
        ignore_error /usr/bin/systemctl stop oes-cis-policy.service
        ignore_error /usr/bin/systemctl stop oes-cis-mgmt.service
        ignore_error /usr/bin/systemctl stop oes-dashboard.service
        ignore_error /usr/bin/systemctl stop oes-cis-aggregator.service
        ignore_error /usr/bin/systemctl stop oes-cis-collector.service
        ignore_error /usr/bin/systemctl stop oes-cis-repaggregator.service
        ignore_error /usr/bin/systemctl stop oes-cis-repcollector.service
        ignore_error /usr/bin/systemctl stop oes-cis-gateway.service
        ignore_error /usr/bin/systemctl stop oes-cis-configuration.service
      3. Edit the monitor script for the Cluster Pool as shown below:

        exit_on_error /usr/bin/systemctl is-active oes-cis-fluentbit.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-auth.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-data.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-metadata.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-policy.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-mgmt.service
        exit_on_error /usr/bin/systemctl is-active oes-dashboard.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-aggregator.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-collector.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-repaggregator.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-repcollector.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-gateway.service
        exit_on_error /usr/bin/systemctl is-active oes-cis-configuration.service

      Changes do not take effect until you take the resource offline, and bring it online again.

  3. Verify the CIS services are successfully upgraded by using the following command:

    cishealth

    When the status displays “Healthy”, then the CIS server is successfully upgraded.

  4. Restart the agent services.

    systemctl restart oes-core-agent.service oes-dashboard-agent.service oes-cis-agent.service oes-cis-recall-agent.service oes-cis-scanner.service

    systemctl status oes-core-agent.service oes-dashboard-agent.service oes-cis-agent.service oes-cis-recall-agent.service oes-cis-scanner.service

5.2.5 Data Transfer from Elasticsearch to OpenSearch

The CIS Admin Console's Insights and Dashboard pages are not displaying any data since Elasticsearch stores the data of these pages.

To transfer the data from Elasticsearch to OpenSearch, complete the following steps:

  1. Ensure to have completed procedure mentioned in Section 5.2.3, Upgrading Infrastructure Servers and Section 5.2.4, Upgrading CIS Servers.

  2. Run the following command in any one of the infrastructure node to move data from the Elasticsearch to the OpenSearch:

    sh /opt/novell/cis/bin/cis_ext_service.sh migrate

    You can verify the logs at:

    /var/opt/novell/log/cis/cis_opensearch_move_<timestamp>.log

    NOTE:The duration of the migration can be prolonged depending on the data saved in Elasticsearch.

  3. Verify the data migration count.

    sh /opt/novell/cis/bin/cis_ext_service.sh verify

  4. On successfully transferring the data, the Insights and Dashboard pages of the CIS Admin Console will populate the existing data.

  5. Run the following command in all the infrastructure nodes to cleanup the Elasticsearch data.

    sh /opt/novell/cis/bin/cis_ext_service.sh completemigrate

  6. Stop the infrastructure services on any one of the nodes with the following command:

    sh /opt/novell/cis/bin/cis_ext_service.sh stop

  7. To verify that infrastructure services are stopped, run the following command:

    sh /opt/novell/cis/bin/cis_ext_service.sh status

    The output displays, “nothing found in stack: cis”.

  8. Run the following command on any one of the node to start the infrastructure service:

    sh /opt/novell/cis/bin/cis_ext_service.sh start

  9. Run the following command to display the status of all the CIS services:

    docker ps

5.2.6 Upgrading Data Scale Server

Prerequisite:

After upgrading to OES 23.4, ensure to check the gatewayforscale (8346) and dataatscale (8347) port on the configured server, and if the port number is missing in the firewall, add it.

Procedure:

  1. Restart the services.

    If both gatewayforscale and dataatscale services are available in the same server then you must start the gatewayforscale service first followed by dataatscale service.

    systemctl status oes-cis-gatewayforscale.service

    systemctl status oes-cis-dataatscale.service

  2. Verify the status of the service.

    systemctl restart oes-cis-gatewayforscale.service

    systemctl restart oes-cis-dataatscale.service

5.2.7 Updating to K3s Cluster

OpenText recommends moving to K3s as support for Docker Swarm will discontinue in OES 24.4.

After applying the OES 24.2 update or patch, perform the following steps to move the infrastructure servers from Docker Swarm to the K3s cluster:

  1. Login to the server as a root user.

  2. Verify if the CIS service on the CIS core server is in a Healthy state.

  3. Verify if all the available CIS infrastructure servers (3/5/7) are Active.

    docker node ls

  4. Run the following command on the CIS core server to move the infrastructure servers from Docker Swarm to the K3s cluster:

    cis_ext_service.sh upgradeinfra

  5. Verify if all the available CIS infrastructure servers (3/5/7) are moved to K3s.

    kubectl get nodes

    You must repeat Step 4 if the CIS infrastructure servers are not moved to K3s.