5.13.4 Troubleshooting Clustering

The clustering of MSS server installations requires secure communication among the server nodes and well as the configuration and initialization of sub-services provided by MSS.

If you encounter issues while setting up or using Clustering, try these troubleshooting tips.

  • Ports - Ensure that all ports for each system in the cluster are not being blocked by any firewall configuration.

  • Logging - First, consult the logs on each system in the cluster to help further identify the nature of the issue encountered.

  • Common issues - Then, check the common issues for specific errors or problems.

After you make a change, retry the Join Cluster process.

Ports

Check the table of Default port numbers used by MSS in the MSS Installation Guide to look for conflicts.

Logging

When trying to diagnose and troubleshoot clustering problems, refer to the logging output on each system involved in the MSS cluster — including systems that already exist in the cluster as well as a system being added or removed.

On each system, look in the <mss-install>/server/logs folder for the following log files.

  • container.log - contains logging out from the MSS server container itself, including the output for each step in the clustering process.

  • cassandra.log - contains logging output from the cassandra database node included with each MSS server.

  • cassandra-sidecar.log - contains logging output for the configuration and initialization of the cassandra service that occurs during the clustering process.

  • service-registry.log - contains logging output from the service-registry service that is included with each MSS server.

For diagnostic purposes, the logging output on each system can be increased. In a working production environment, however, we recommend that you restore the default logging output for performance and resource considerations.

To increase the logging output prior to making clustering configurations:

  1. Insert the following lines into <mss-install>/server/conf/log4j.xml.

    <Logger name="com.microfocus.mss.mgmt.console.viewcomponents.clustering"
level="debug"/>
    <Logger name="com.microfocus.centralmgmt.configuration.controller.clustering"
level="debug"/>
    <Logger name="com.microfocus.centralmgmt.configuration.services.clustering"
level="debug"/>

  2. Restart the system.

  3. After clustering is configured and working properly, remember to restore the default logging output.

Common issues

Try these troubleshooting tips for specific error messages or problems.

  • The database network adapter must be set before you can join the cluster

    1. Log into the Administrative Console of the specified server.

    2. Click Configure Settings - Clustering.

    3. Select the desired Database Network Adapter and click APPLY

  • Unable to retrieve the server certificate when running the join cluster wizard

    1. Ensure that the correct server name and secure port are entered in the wizard.

    2. If using a non-default configuration, be sure that the proper servlet context is entered in the wizard.

  • Invalid user name or password

    Ensure that the credentials entered for the specified server in the Join cluster wizard match the credentials of a user with admin rights on that server.

  • Failure encountered during the testing connection step

    1. Take note of the systems specified in the error message.

    2. Ensure the server address for all systems can be resolved from all other systems.

    3. Ensure that the HTTPS port (default of 443) and system port (default of 8003) is accessible from all systems.

  • Failure encountered when updating the cluster truststore

    1. Take note of the systems specified in the error message.

    2. Ensure that the <mss-install>/server/etc directory is writable.

    3. Ensure that <mss-install>/server/etc/system-trustcerts.bcfks can be opened with KeyStore Explorer.

    4. If a non-default configuration is being used, ensure that these properties are set correctly in <mss-install>/server/conf/container.properties:

      • servletengine.system.ssl.trustStoreFileName
      • servletengine.system.ssl.trustStorePassword

  • Failure encountered when initializing system ports

    1. Take note of the systems specified in the error message.

    2. Refer to servletengine.log and container.log for further diagnostics.

  • Failure encountered when configuring the replication role

    1. Ensure that the cassandra database is running properly on the system being added or removed from the cluster.

    2. Refer to cassandra.log and container.log for further diagnostics.

  • Failure encountered when updating the service registry

    1. Take note of the systems specified in the error message.

    2. Ensure that the file <mss-install>/server/microservices/service-registry/service.yml exists and is writable.

  • Failure encountered in any of the database-related steps

    1. Take note of the systems specified in the error message.

    2. Refer to these log files for further diagnostics:

      • <mss-install>/server/logs/cassandra.log
      • <mss-install>/server/logs/cassandra-sidecar.log
      • <mss-install>/server/microservices/cassandra/logs/debug.log

    3. Ensure that the <mss-install/server/microservices/cassandra/conf/cassandra.yaml file exists and is writable.

    4. In <mss-install>/server/cassandra/conf/cassandra.yaml, ensure that:

      • the listen_interface property is set to the correct interface name.

      • the storage_port (defaults to 7000) and ssl_storage_port (defaults to 7001) property files are accessible from all other systems in the cluster.

  • Cannot enter the Clustering view in the MSS Administrative Console because the local database node is not running

    Refer to these log files for further diagnostics:

    • <mss-install>/server/logs/cassandra.log
    • <mss-install>/server/microservices/cassandra/logs/debug.log

  • The cassandra database node will not start because an invalid listen_interface is configured

    1. In <mss-install>/server/microservices/cassandra/conf/cassandra.yaml, be sure that the listen_interface property is set to the correct interface name.

    2. If the correct interface name cannot be determined:

      a. Comment out the listen_interface property line and uncomment the listen_address property line, which should be set to localhost.

      b. Search for -seeds: and be sure the value is set to "127.0.0.1"

    3. Save the file and restart the MSS server. The Clustering view should now be accessible.

    4. In the Clustering view, select the Database Network Adapter with the IP address that is accessible to all systems in the cluster.

    5. Click APPLY.

  • The SERVER STATUS is DOWN for one or more nodes displayed in the Management Server Nodes table

    1. Take note of the server address.

    2. Refer to the appropriate log on that system for further diagnostics.

  • The DATABASE STATUS is displaying DOWN for a system where you verified that the database node is running.

    Ensure that the system port, configured via the servletengine.system.ports property in container.properties (default = 8003), is not being blocked by any firewall configurations on that system.

Related topics