Overriding Application Properties

Each Transformation Hub module (Kafka, Zookeeper, and so on) has many additional properties available, and there may be a need for system administrators to override the default values for some of these properties. This section covers how to override these property values.

Property values (for properties that support overrides) are set by injecting environment variables in the respective container's start-up environment. These variables are read from a user-supplied properties file, in a specific location on the Network File Server (NFS). To see the available properties for override, consult the respective module's published documentation.

Note that in most cases, this feature is not required for normal operation of Transformation Hub, and most likely will be used at the direction of technical support. Not all properties support overrides; please check with technical support before making any changes to your configuration.

For Kafka, ZooKeeper, and Schema Registry properties, consult the appropriate Confluent documentation.
The properties for routing processor and stream processor modules are detailed below.

Note: Legacy properties prefixed with arcsight.eventbroker will continue to function as they did in previous versions, but as explained below, newly added properties must be prefixed with arcsight.th. If two properties of the same name are set with different prefixes, the property with arcsight.th will supersede the other one.

Configuring the Values
Routing Processor and Stream Processor Properties
Changing Value Examples

Configuring the Values

Create a file named arcsight-env-override.properties under the folder <NFS_root_DIRECTORY>/transformationhub/config.

The <NFS_root_DIRECTORY> path is described in this guide as the NFS root folder (usually /opt/arcsight/nfs/volumes). For more information, refer to the section Creating the NFS Shares.

Add properties to the file. To each property, add the module prefix from the table below.

Module	Prefix
Kafka	`arcsight.th.kafka.`
Schema Registry	`arcsight.th.schema-registry.`
ZooKeeper	`arcsight.th.zookeeper.`
Routing/C2AV/Enrichment Processor	`arcsight.th.sp.`

Delete the pods for which properties were defined, or, alternatively, redeploy Transformation Hub.
To verify the changes, search the log file (after the container's status is back to Running) for matching properties.

Routing Processor and Stream Processor Properties

As explained above, prefix these properties with arcsight.th.sp. to create an override.

Property Name	Default Value	Description
`RETRIES`	2147483647	The number of retries for broker requests that return a retry-able error.
`RETRY_BACKOFF_MS`	100	The amount of time (milliseconds), before a request is retried. This applies if the retries parameter is configured to be greater than 0.
`RECEIVE_BUFFER_BYTES`	65536	The size of the TCP receive buffer to use when reading data. If the value is -1, the OS default will be used.
`MAX_PARTITION_FETCH_BYTES`	1048576	The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer.
`MAX_REQUEST_SIZE`	1048576	The maximum size of a request in bytes.
`BUFFER_MEMORY`	33554432	The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
`BATCH_SIZE`	16384	the default batch size in bytes when batching multiple records sent to a partition
`LINGER_MS`	100	the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together
`HEARTBEAT_INTERVAL_MS`	1000	The expected time (milliseconds) between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group.
`MAX_POLL_INTERVAL_MS`	3600000	The maximum delay (milliseconds) between invocations of poll() when using consumer group management
`MAX_POLL_RECORDS`	100	The maximum number of records returned in a single call to poll().
`SESSION_TIMEOUT_MS`	180000	The timeout (milliseconds) used to detect client failures when using Kafka's group management facility
`REQUEST_TIMEOUT_MS`	305000	The configuration controls the maximum amount of time (milliseconds) the client will wait for the response of a request.
`CONNECTIONS_MAX_IDLE_MS`	540000	The maximum amount of time (milliseconds) before idle connections are closed.
`TH_NUM_THREADS`	6	The number of threads to execute stream processing.

Changing Value Examples

To change the value of ZOOKEEPER_MAX_CLIENT_CNXNS to 65, in ZooKeeper, and to change the value of SCHEMA_REGISTRY_KAFKASTORE_TIMEOUT_MS in the Schema Registry, create a file, <NFS Volume mount>/transformationhub/config/arcsight-env-override.properties, and add the following lines:

arcsight.th.zookeeper.ZOOKEEPER_MAX_CLIENT_CNXNS=65

arcsight.th.schema-registry.SCHEMA_REGISTRY_KAFKASTORE_TIMEOUT_MS=20000

Example of verifying the change by searching the log:

kubectl -n transformationhub1 logs th-zookeeper-0 | grep ZOOKEEPER_MAX_CLIENT_CNXNS
Environment override script set: ZOOKEEPER_MAX_CLIENT_CNXNS=65 ZOOKEEPER_MAX_CLIENT_CNXNS=65