18.3 Elasticsearch in Cluster Mode

Complete the steps in the section Enabling Event Visualization.

Configure the /etc/elasticsearch/elasticsearch.yml file on each external Elasticsearch nodes by updating or adding the following information:

Property and Value	Notes
discovery.seed_hosts: ["<IP of the master eligible elasticsearch node in the cluster>","<IP of the master eligible elasticsearch node in the cluster>", "<IP of the master eligible elasticsearch node in the cluster>", and so on]	Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. You can format non-scalar values as arrays. discovery.seed_hosts: ["IP1","IP2:port_number","seeds.mydomain.com"]
cluster.name: <Elasticsearch _cluster_name>	The cluster name that you specify must be same for all the nodes.
node.name: <node_name>	The node name must be unique for each node.
network.host: _<networkInterface>:ipv4_	If you are using hostname instead of IP address, make sure that the hostname is resolvable by all the nodes in the Elasticsearch cluster and the Sentinel server.
thread_pool.write.queue_size: 300
thread_pool.search.queue_size: 10000	Once the search queue size reaches its limit, Elasticsearch discards any pending search requests in queue. You can increase the search queue size based on the below calculation:threadpool.search.queue_size = Average number of widget queries per user for a dashboard x number of shards (per day index) x number of days (search duration)
index.codec: best_compression
path.data: ["/<es1>", "/<es2>"]	Spread data across multiple independent disks or locations to reduce the disk I/O latency. Configure multiple paths for storing Elasticsearch data. For example /es1, /es2, and so on. File permission needs to be set to Elasticsearch users for the above paths. For best performance and manageability, mount each path to a separate physical disk (JBOD).

Repeat all of the above steps on each external Elasticsearch nodes of the Elasticsearch cluster.
In the Sentinel server Elasticsearch node, configure the <sentinel_installation_path>/opt/novell/sentinel/3rdparty/elasticsearch/config/elasticsearch.yml as follows:
1. Ensure that the values of cluster.name and discovery.seed_hosts in the elasticsearch.yml file are same as the elasticsearch.yml file in external Elasticsearch node.
Add the external Elasticsearch nodes IP addresses to the ServerList property in the <sentinel_installation_path>/etc/opt/novell/sentinel/config/elasticsearch-index.properties file.

For example: ServerList=<External_Elasticsearch_Node1_IP>:<Port>,<External_Elasticsearch_Node2_IP>:<Port>
Restart Sentinel:
```
rcsentinel restart
```
Restart each external Elasticsearch node:
```
/etc/init.d/elasticsearch restart
```
Verify that the Elasticsearch cluster is formed, by running the following command:
```
curl -X GET http://<Elasticsearch_IP_of_the_Sentinel_server>:<Port>/_cat/nodes?v
```
Ensure that all the existing alert data and event data (if available) are moved to the external Elasticsearch nodes.
For optimal performance and stability of the Sentinel server, configure the Elasticsearch node in the Sentinel server as a dedicated master-eligible node so that all the event visualization data is indexed in external Elasticsearch nodes:
1. Stop the internal node (Sentinel server)
```
rcsentinel stopES
```
2. Set the following internal nodes in the elasticsearch.yml file:
```
node.master: true 
node.data: false 
node.ingest: false 
```
3. Run elasticsearch-node repurpose to clean all the shards
```
<sentinel_installation_path>/opt/novell/sentinel/3rdparty/elasticsearch/bin/elasticsearch-node -v repurpose
```
4. Start the internal Elasticsearch node
```
rcsentinel startES
```
5. Restart each external Elasticsearch node:
```
/etc/init.d/elasticsearch restart
```

IMPORTANT:Whenever an external Elasticsearch node goes down, the Elasticsearch cluster restarts automatically, due to which, there might be a temporary issue in launching dashboards through Kibana and alert search.

When the Sentinel server is restarted ensure that you restart the external Elasticsearch nodes as well.

IMPORTANT:If you change the value of eventvisualization.traditionalstorage.enabled to false, elasticsearch.yml file is modified. The changes observed in the fields are discovery.seed_hosts, network.host, and a new field discovery.type is added.