18.2 Elasticsearch in Cluster Mode

Complete the steps in the section Enabling Event Visualization in Sentinel.

Configure the /etc/elasticsearch/elasticsearch.yml file on each external Elasticsearch nodes by updating or adding the following information:

Property and Value	Notes
discovery.seed_hosts: ["<IP of the master eligible elasticsearch node in the cluster>","<IP of the master eligible elasticsearch node in the cluster>", "<IP of the master eligible elasticsearch node in the cluster>", and so on]	Provides a list of the addresses of the master-eligible nodes in the cluster. May also be a single string containing the addresses separated by commas. You can format non-scalar values as arrays. discovery.seed_hosts: ["IP1","IP2:port_number","seeds.mydomain.com"]
cluster.name: <Elasticsearch _cluster_name>	The cluster name that you specify must be same for all the nodes.
node.name: <node_name>	The node name must be unique for each node.
network.host: _<networkInterface>:ipv4_	If you are using hostname instead of IP address, make sure that the hostname is resolvable by all the nodes in the Elasticsearch cluster and the Sentinel server.
thread_pool.write.queue_size: 300
thread_pool.search.queue_size: 10000	Once the search queue size reaches its limit, Elasticsearch discards any pending search requests in queue. You can increase the search queue size based on the below calculation:threadpool.search.queue_size = Average number of widget queries per user for a dashboard x number of shards (per day index) x number of days (search duration)
index.codec: best_compression
path.data: ["/<es1>", "/<es2>"]	Spread data across multiple independent disks or locations to reduce the disk I/O latency. Configure multiple paths for storing Elasticsearch data. For example /es1, /es2, and so on. For best performance and manageability, mount each path to a separate physical disk (JBOD).

Repeat all of the above steps on each external Elasticsearch nodes of the Elasticsearch cluster.
In the Sentinel server Elasticsearch node, configure the <sentinel_installation_path>/opt/novell/sentinel/3drparty/elasticsearch/config/elasticsearch.yml as follows:
1. Ensure that the values of cluster.name and discovery.seed_hosts in the elasticsearch.yml file are same as the elasticsearch.yml file in external Elasticsearch node.
Add the external Elasticsearch nodes IP addresses to the ServerList property in the <sentinel_installation_path>/etc/opt/novell/sentinel/config/elasticsearch-index.properties file.

For example: ServerList=<External_Elasticsearch_Node1_IP>:<Port>,<External_Elasticsearch_Node2_IP>:<Port>
Enabling Secure Communication between External Elasticsearch Nodes as well as between Sentinel and Elasticsearch Cluster if there is an External Elasticsearch Cluster Setup

The latest release of Sentinel enables secure communication between the Sentinel server and external Elasticsearch cluster as well as between different nodes of the Elasticsearch cluster. This section explains the steps on how to enable these secure settings for cases where you have an external Elasticsearch cluster connected to the Sentinel server.

Steps to be followed for securing intra cluster communication between Elasticsearch nodes:
1. Generate the certificates for all the external Elasticsearch nodes in the cluster. You can first create all the external Elasticsearch certificates in the Sentinel node itself and can then copy them to the respective Elasticsearch nodes. For this, first run the following command in the Elasticsearch home directory <sentinel_installation_path>/opt/novell/sentinel/3rdparty/elasticsearch of the Sentinel:
```
./bin/elasticsearch-certutil cert --ca <CA certificate filename>.p12 --out config/certs/newNode.p12
```
  You are prompted to enter the password for your CA certificate. You are also prompted to create a password for the generated certificate.
2. Copy the certificates to the respective external Elasticsearch nodes. For example, copy the newNode.p12 file into the /etc/elasticsearch/certs/ directory of the newNode of the external Elasticsearch cluster. Provide read-write permissions to the certificates on the new machines using the chmod command.
  
  NOTE:If the certs directory is not present, you need to create the same.
3. After generating and copying the certificates to all the external Elasticsearch nodes, add the following settings in the /etc/elasticsearch/elasticsearch.yml file of all the external Elasticsearch nodes:
  - xpack.security.enabled: true
  - xpack.security.transport.ssl.enabled: true
  - xpack.security.transport.ssl.keystore.path: certs/newNode.p12
  - xpack.security.transport.ssl.truststore.path: certs/newNode.p12
  - xpack.security.transport.ssl.verification_mode: certificate
4. On each of the external Elasticsearch nodes, store the password for the generated keystore and truststore certificate file in the Elasticsearch keystore. For this, run the following commands in the Elasticsearch home directory /usr/share/elasticsearch of all the external Elasticsearch nodes:
```
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
```
```
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
```
Steps to be followed for securing Sentinel to Elasticsearch cluster communications:
1. Switch to the novell user:
```
su novell
```
2. Run the following command, to generate a http certificate for an external Elasticsearch node from the Sentinel machine:
```
<sentinel_installation_path>/opt/novell/sentinel/bin/javacert.sh --generateES <provide path where the http certificate should be generated, example /home/novell/http.pks> <http certificate password> <keyalias>
```
3. Copy the http certificate to the Elasticsearch node. For example, copy the http.pks file into the ES_PATH_CONF/certs/ directory on the Elasticsearch node. Provide read-write permissions to the certificates on the new machines.
  
  NOTE:If the certs directory is not present, you need to create the same.
4. Add the following settings in the ES_PATH_CONF/elasticsearch.yml file in all the external Elasticsearch nodes:
  - xpack.security.http.ssl.enabled: true
  - xpack.security.http.ssl.keystore.path: certs/http.pks
5. Run the following command in the Elasticsearch home directory /usr/share/elasticsearch of all the external Elasticsearch nodes to save the password of the http certificate to the Elasticsearch keystore:
```
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
```
Restart Sentinel:
```
rcsentinel restart
```
Restart each external Elasticsearch node:
```
/etc/init.d/elasticsearch restart
```

Verify that the Elasticsearch cluster is formed, by running the following commands:

cd <sentinel_installation_path>/opt/novell/sentinel/bin

./elasticsearchRestClient.sh <sentinel_ip> <Port used for the Elasticsearch> GET _cat/nodes

Ensure that all the existing alert data and event data (if available) are moved to the external Elasticsearch nodes.
For optimal performance and stability of the Sentinel server, configure the Elasticsearch node in the Sentinel server as a dedicated master-eligible node so that all the event visualization data is indexed in external Elasticsearch nodes:
1. Stop the internal node (Sentinel server)
```
rcsentinel stopES
```
2. Set the following internal nodes in the elasticsearch.yml file:
```
node.master: true 
node.data: false 
node.ingest: false 
```
3. Run elasticsearch-node repurpose to clean all the shards
```
<sentinel_installation_path>/opt/novell/sentinel/3rdparty/elasticsearch/bin/elasticsearch-node -v repurpose
```
4. Start the internal Elasticsearch node
```
rcsentinel startES
```
5. Restart each external Elasticsearch node:
```
/etc/init.d/elasticsearch restart
```

IMPORTANT:Whenever an external Elasticsearch node goes down, the Elasticsearch cluster restarts automatically, due to which, there might be a temporary issue in launching dashboards through Kibana and alert search.

When the Sentinel server is restarted ensure that you restart the external Elasticsearch nodes as well.

IMPORTANT:If you change the value of eventvisualization.traditionalstorage.enabled to false, elasticsearch.yml file is modified. The changes observed in the fields are discovery.seed_hosts, network.host, and a new field discovery.type is added.