18.3 OpenSearch in Cluster Mode

Complete the steps in the section Enabling Event Visualization.
Create certs folder in /etc/opensearch on the external OpenSearch nodes and assign the owner as opensearch by running the following command:
```
chown opensearch:opensearch /etc/opensearch/certs
```
Navigate to <sentinel_installation_path>/opt/novell/sentinel/3rdparty/opensearch/config/certs.
Copy and paste node.pem, node-key.pem and root-ca.pem certificate files to /etc/opensearch/certs in each external OpenSearch node.

Configure the /etc/opensearch/opensearch.yml file on each external OpenSearch nodes by updating or adding the following information:

Property and Value	Notes
discovery.seed_hosts: ["<IP of the master eligible OpenSearch node in the cluster>","<IP of the master eligible OpenSearch node in the cluster>", "<IP of the master eligible OpenSearch node in the cluster>", and so on]	Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. You can format non-scalar values as arrays. discovery.seed_hosts: ["IP1","IP2:port_number","seeds.mydomain.com"]
cluster.name: <opensearch>	The cluster name must be same for all the nodes. The cluster name specified in Sentinel OpenSearch node's opensearch.yml is "opensearch".
node.name: <node_name>	The node name must be unique for each node. The node name specified in Sentinel OpenSearch node's opensearch.yml is "sentinel-opensearch-node".
network.host: <IP address of the host>	If you are using hostname instead of IP address, make sure that the hostname is resolvable by all the nodes in the OpenSearch cluster and the Sentinel server.
node.roles: <[data, ingest]>	This property is used to restrict the roles that a node can perform, for example, data storing, ingestion and coordinating. If this property is not specified, the node is eligible to be master, which means it is eligible to perform all the above specified roles.
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.allow_unsafe_democertificates: true plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: certs/node.pem plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem plugins.security.allow_default_init_securityindex: true plugins.security.authcz.admin_dn: 'CN=ADMIN' plugins.security.nodes_dn: 'CN=opensearch-node' plugins.security.ssl.transport.enabled_protocols: ["TLSv1.2", "TLSv1.3"] plugins.security.ssl.http.enabled_protocols: ["TLSv1.2", "TLSv1.3"]	These properties enable secure communication with security plugin in OpenSearch that comes bundled by default. These TLS certificates are used to secure transport-layer traffic (node-to-node communication within your cluster) and REST-layer traffic (communication between a client and a node within your cluster).
path.data: ["/<es1>", "/<es2>"]	Spread data across multiple independent disks or locations to reduce the disk I/O latency.Configure multiple paths for storing Elasticsearch data. For example /es1, /es2, and so on. File permission needs to be set to Elasticsearch users for the above paths.For best performance and manageability, mount each path to a separate physical disk (JBOD).

Repeat all of the above steps on each external nodes of the OpenSearch cluster.
In the Sentinel server OpenSearch node, configure the <sentinel_installation_path>/opt/novell/sentinel/3rdparty/opensearch/config/opensearch.yml as follows:
1. Ensure that the values of cluster.name and discovery.seed_hosts in the opensearch.yml file are same as the OpenSearch.yml file in external OpenSearch node.
Copy and paste jdk folder located at <sentinel_installation_path>/opt/novell/sentinel to the external OpenSearch nodes.
Set the variable JAVA_HOME to jdk folder.
Add the external OpenSearch nodes IP addresses to the ServerList property in the <sentinel_installation_path>/etc/opt/novell/sentinel/config/opensearch-index.properties file.

For example: ServerList=<External_opensearch_Node1_IP>:<Port>,<External_opensearch_Node2_IP>:<Port>

Restart Sentinel:

systemctl restart sentinel.service

<sentinel_installation_path>/opt/novell/sentinel/bin/server.sh restart

Restart each external OpenSearch node:
```
systemctl restart opensearch.service
```

Verify that the OpenSearch cluster is formed, by running the following command:

<sentinel_installation_path>/opt/novell/sentinel/bin/opensearchRestClient.sh 10.204.104.50 9200 GET _cat/nodes?v

Ensure that all the existing alert data and event data (if available) are moved to the external OpenSearch nodes.
For optimal performance and stability of the Sentinel server, configure the OpenSearch node in the Sentinel server as a dedicated master-eligible node so that all the event visualization data is indexed in external OpenSearch nodes:
1. Stop the internal node (Sentinel server)
```
<sentinel_installation_path>/opt/novell/sentinel/bin/server.sh stopOSS
```
2. Set the following in the internal node’s opensearch.yml file:
```
node.roles: [master] 
```
  And set the following in the external node’s opensearch.yml file:
```
node.roles: [data, ingest] 
```
3. Run opensearch-node repurpose to clean all the shards:
```
<sentinel_installation_path>/opt/novell/sentinel/3rdparty/opensearch/bin/opensearch-node -v repurpose
```
4. Start the internal OpenSearch node:
```
<sentinel_installation_path>/opt/novell/sentinel/bin/server.sh startOSS
```
5. Restart each external OpenSearch node:
```
systemctl restart opensearch.service
```

IMPORTANT:Whenever an external OpenSearch node goes down, the OpenSearch cluster restarts automatically, due to which, there might be a temporary issue in launching dashboards through Opensearch Dashboards and alert search.

When the Sentinel server is restarted, ensure that you restart the external OpenSearch nodes as well.

IMPORTANT:If you change the value of eventvisualization.traditionalstorage.enabled to false, opensearch.yml file is modified. The changes are observed in discovery.seed_hosts, network.host fields, and a new field discovery.type is added.