Configuring ESM as a Transformation Hub Producer in Distributed Correlation Mode

Distributed event forwarding is available when ESM is installed in distributed correlation mode. The feature allows you to forward events from ESM to Transformation Hub at a high rate. Distributed event forwarding leverages the distributed infrastructure of ESM to allow ESM to spread the work of event forwarding across the cluster, similar to how ESM distributes event correlation. This allows event forwarding to scale horizontally.

Events that ESM forwards to Transformation Hub can subsequently be read by another ESM instance or multiple ESM instances. Those ESM instances do not have to be installed in distributed correlation mode in order to read events from Transformation Hub.

If you need to forward events from ESM to Transformation Hub at a high rate (generally higher than 10K events per second) OpenText recommends that you use ESM in distributed correlation mode and use distributed event forwarding.

Note: Distributed Forwarding does not support SSL client-side authentication with Transformation Hub.

Distributed event forwarding requires the following:

CA certificate of the Transformation Hub cluster
ESM filter that determines which events to forward

You can create a filter or use one that is installed with ESM.
List of broker socket addresses (in the form host:port) of the Transformation Hub cluster to which you want to forward events

The Transformation Hub documentation refers to these as "worker nodes."
Transformation Hub topic name to which you want to publish events

Note: Perform the configuration steps that are described below on the persistor node of the ESM cluster.

Obtaining and Importing the Transformation Hub CA Certificate

Transformation Hub maintains its own certificate authority (CA) to issue certificates for individual nodes in the Transformation Hub cluster. ESM needs that CA certificate in its truststore so that it will trust connections to Transformation Hub. For information about obtaining the certificate, see the information about viewing and changing the certificate authority. You might need to contact the Transformation Hub administrator to obtain the CA certificate if you do not have sufficient privileges to access the Transformation Hub cluster.

If you have root access to the Transformation Hub cluster (version 3.x or later), you can obtain the CA certificate as follows:

master=<master node host name or IP address>

ssh root@$master env K8S_HOME=/opt/arcsight/kubernetes /opt/arcsight/kubernetes/scripts/cdf-updateRE.sh > /opt/arcsight/manager/th.ca.crt

The command copies the CA certificate to the file /opt/arcsight/manager/th.ca.crt. You must then import that certificate into the ESM cluster using the certadmin tool in the ESM Administrator's Guide.

To import the Transformation Hub CA certificate to ESM:

Run the following command to ensure that the ESM cluster is running:
```
/etc/init.d/arcsight_services status
```
If it is not, run the following command:
```
/etc/init.d/arcsight_services start
```
Wait for the cluster to completely start.
Import the Transformation Hub CA certificate:
```
bin/arcsight certadmin -importcert /opt/arcsight/manager/th.ca.crt
```
Note the alias that is reported in the output.
Run the following command and verify that the alias that was reported in the output from the previous command is listed as an approved certificate:
```
bin/arcsight certadmin -list approved
```
If the alias is not listed, re-import the Transformation Hub CA certificate.

Configuring the Filter and Destination

After you import the Transformation Hub CA certificate to ESM, configure a filter and the destination properties. You specify the destination properties in a file that is used as input to the configure-event-forwarding command, as seen in the ESM Administrator's Guide.

To configure the filter and destination:

Create a filter to select the events that you want to forward or use a filter that is installed with ESM.

For more information about creating event filters, see the information about filtering events in the ArcSight Console User's Guide.
Edit forwarding.properties in /opt/arcsight/manager/config/ with the following information:
- Name of the filter that you want to use to select the events to be forwarded.
- Comma separated list of socket addresses of the worker nodes of the Transformation Hub cluster. Each socket address should be in the form host:port.
Note: Optionally, you can copy forwarding.properties to a different file and edit that file. If so, you must specify the file name when running the configure-event-forwarding utility. Otherwise, configure-event-forwarding will read the default forwarding.properties file.

The properties file is in the following format:

# Provide the filter that determines which events to forward.

# The filter may be specified by ID or by Name.

# If name is used then it can be the simple name or the fully qualified URI.

#

# e.g. filterID=2jq5Og-sAABCAFyopCdLChw==

# or

# filterName=Non-ArcSight Internal Events

# or

# filterName=/All Filters/ArcSight System/Event Types/Non-ArcSight Internal Events

filterName=Non-ArcSight Internal Events

# List the socket addresses of the destination worker nodes

hostPortCSV=host1.example.com:9093,host2.example.com:9093,host3.example.com:9093

# Specify the destination topic name

topicName=esm-forwarded-events
Run the following command to validate the settings in the properties file:
```
bin/arcsight configure-event-forwarding -validate <file>
```
Note: If you do not specify a file name, <ARCSIGHT_HOME>/config/forwarding.properties is the default file.

The command output indicates whether the filter and connections to the forwarding destination are valid.
If the settings in forwarding.properties are valid, run the following command to set the configuration and save it in the information repository:
```
bin/arcsight configure-event-forwarding -commit <file>
```
Note: If you do not specify a file name, <ARCSIGHT_HOME>/config/forwarding.properties is the default file.
When you are ready to actively filter and forward events, run the following command to enable event forwarding:
```
bin/arcsight configure-event-forwarding -enable
```
To disable event forwarding, run the following command:
```
bin/arcsight configure-event-forwarding -disable
```
Note: ESM does not cache events when you disable distributed event forwarding. Events that ESM ingests while forwarding is disabled will not be forwarded, even when you subsequently enable forwarding. Events that ESM ingests after you enable forwarding will be forwarded.
To view the current settings for distributed event forwarding, including whether it is enabled or disabled, run the following command:
```
bin/arcsight configure-event-forwarding -print
```

Modifying the Filter and Configuration

The properties file is only used to validate and commit the configuration. The information repository stores the configuration itself. If you need to modify the configuration, it is not sufficient to simply edit the file. You must edit the file and then run bin/arcsight configure-event-forwarding -validate <file> and bin/arcsight configure-event-forwarding -commit <file> again to overwrite the old configuration. It is not necessary to stop and start the ESM cluster to modify the configuration. Distributed event forwarding will automatically detect the updated configuration. You can run bin/arcsight configure-event-forwarding -print to view the configuration that is currently in use.

If you modify the filter that distributed event forwarding uses, event forwarding automatically detects the updated filter and begins using it to select events for forwarding as soon as you save the filter update. Therefore, be cautious when modifying the filter.

Instead of modifying the filter, you can create a new filter and test it before you commit to using it. When you are sure that the filter is correct, specify the new filter in the properties file, then run bin/arcsight configure-event-forwarding -validate <file> and bin/arcsight configure-event-forwarding -commit <file> to apply the change to the current configuration.

Troubleshooting Event Forwarding Throughput

The maximum rate at which events can be forwarded depends on many factors, including the following:

Amount of other work that distributed correlation must do to support all of the rules, data monitors, and other content that you have defined
File input/output contention
CPU contention
Memory contention
Network contention, especially the network between distributed correlation nodes and the Transformation Hub worker nodes
Maximum rate at which Transformation Hub can accept messages

It might be that the maximum throughput of event forwarding is not enough to support the number of events that need to be forwarded in a given period of time. When that happens, events yet to be forwarded will build up inside message bus. This buildup of unforwarded events is called lag. If the lag is too high, the ESM cluster will stop ingesting events to reduce the lag. When this happens it is called backpressure.

The Acceptable Lag value in the Cluster View dashboard of Command Center defines the amount of lag that can occur before backpressure is applied. For more information, see the information about using the Cluster View dashboard in the ArcSight Command Center User's Guide.

Distributed event forwarding leverages the ESM distributed infrastructure to gain horizontal scalability. The correlator does the work of event forwarding. You might be able to add event forwarding throughput by adding correlator(s), either on an existing node or on a new distributed correlation node. However, this will only increase throughput if it is the correlators that are causing the bottleneck (for example, because of CPU or memory limitations). If the network or Transformation Hub is causing the bottleneck, adding correlators might not have any effect. For information about adding correlators, see Adding Correlators and Aggregators.

Optionally, you can disable backpressure that might occur as a result of unforwarded events, but you should only use this option if you accept that some events might never be forwarded. To disable backpressure, run the following command:

bin/arcsight configure-event-forwarding -backpressure disable

Note: When you disable backpressure, ESM will not slow the ingestion of events. If event forwarding cannot keep up with the rate at which events are being filtered for forwarding, the lag might build up inside the message bus to the point where the oldest yet-to-be-forwarded events are dropped and therefore not forwarded at all.

To enable backpressure, run the following command:

bin/arcsight configure-event-forwarding -backpressure enable