18.3 OpenSearch in Cluster Mode

Complete the steps in the section Enabling Event Visualization.

Configure the /etc/opensearch/opensearch.yml file on each external OpenSearch nodes by updating or adding the following information:

Property and Value	Notes
discovery.seed_hosts: ["<IP of the master eligible OpenSearch node in the cluster>","<IP of the master eligible OpenSearch node in the cluster>", "<IP of the master eligible OpenSearch node in the cluster>", and so on]	Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. You can format non-scalar values as arrays. discovery.seed_hosts: ["IP1","IP2:port_number","seeds.mydomain.com"]
cluster.name: <opensearch _cluster_name>	The cluster name that you specify must be same for all the nodes.By default, the cluster name specified in Sentinel OpenSearch node's opensearch.yml is "cluster-name".
node.name: <node_name>	The node name must be unique for each node.
network.host: _<networkInterface>:ipv4_	If you are using hostname instead of IP address, make sure that the hostname is resolvable by all the nodes in the OpenSearch cluster and the Sentinel server.
thread_pool.write.queue_size: 300
thread_pool.search.queue_size: 10000	Once the search queue size reaches its limit, OpenSearch discards any pending search requests in queue. You can increase the search queue size based on the below calculation:threadpool.search.queue_size = Average number of widget queries per user for a dashboard x number of shards (per day index) x number of days (search duration)
index.codec: best_compression
path.data: ["/<es1>", "/<es2>"]	Spread data across multiple independent disks or locations to reduce the disk I/O latency. Configure multiple paths for storing OpenSearch data. For example /es1, /es2, and so on. File permission needs to be set to OpenSearch users for the above paths. For best performance and manageability, mount each path to a separate physical disk (JBOD).

Repeat all of the above steps on each external nodes of the OpenSearch cluster.
In the Sentinel server OpenSearch node, configure the <sentinel_installation_path>/opt/novell/sentinel/3rdparty/opensearch/config/opensearch.yml as follows:
1. Ensure that the values of cluster.name and discovery.seed_hosts in the opensearch.yml file are same as the OpenSearch.yml file in external OpenSearch node.
Add the external OpenSearch nodes IP addresses to the ServerList property in the <sentinel_installation_path>/etc/opt/novell/sentinel/config/opensearch-index.properties file.

For example: ServerList=<External_opensearch_Node1_IP>:<Port>,<External_opensearch_Node2_IP>:<Port>
Restart Sentinel:
```
rcsentinel restart
```
Restart each external OpenSearch node:
```
systemctl restart opensearch.service
```

Verify that the OpenSearch cluster is formed, by running the following command:

<sentinel_installation_path>/opt/novell/sentinel/bin/opensearchRestClient.sh 10.204.104.50 9200 GET _cat/nodes?v

Ensure that all the existing alert data and event data (if available) are moved to the external OpenSearch nodes.
For optimal performance and stability of the Sentinel server, configure the OpenSearch node in the Sentinel server as a dedicated master-eligible node so that all the event visualization data is indexed in external OpenSearch nodes:
1. Stop the internal node (Sentinel server)
```
rcsentinel stopOSS
```
2. Set the following in the internal node’s opensearch.yml file:
```
node.roles: [master] 
```
  And set the following in the external node’s opensearch.yml file:
```
node.roles: [data, ingest] 
```
3. Run opensearch-node repurpose to clean all the shards:
```
<sentinel_installation_path>/opt/novell/sentinel/3rdparty/opensearch/bin/opensearch-node -v repurpose
```
4. Start the internal OpenSearch node:
```
rcsentinel startOSS
```
5. Restart each external OpenSearch node:
```
systemctl restart opensearch.service
```

IMPORTANT:Whenever an external OpenSearch node goes down, the OpenSearch cluster restarts automatically, due to which, there might be a temporary issue in launching dashboards through Opensearch Dashboards and alert search.

When the Sentinel server is restarted, ensure that you restart the external OpenSearch nodes as well.

IMPORTANT:If you change the value of eventvisualization.traditionalstorage.enabled to false, opensearch.yml file is modified. The changes are observed in discovery.seed_hosts, network.host fields, and a new field discovery.type is added.