Overriding Application Properties

Each Transformation Hub module (Kafka, Zookeeper, and so on) has many additional properties available, and a system administrator might be required to override the default values for some of these properties. This section covers how to override these property values.

Property values (for properties that support overrides) are set by injecting environment variables in the respective container's start-up environment. These variables are read from a user-supplied properties file, in a specific location on the Network File Server (NFS). To see the available properties for override, consult the respective module's published documentation.

Note that in most cases, this feature is not required for normal operation of Transformation Hub, and most likely will be used at the direction of technical support. Not all properties support overrides; please check with technical support before making any changes to your configuration.

• For Kafka, ZooKeeper, and Schema Registry properties, consult the appropriate Confluent documentation.
• The properties for routing processor, stream processor and web service modules are detailed below.

• Routing Processor and Stream Processor Properties
• Web Service Properties
• Configuring the Values
• Changing Value Examples

Routing Processor and Stream Processor Properties

As explained in Configuring the Values, prefix these properties with arcsight.th.sp. to create an override.

Property Name	Default Value	Description
RETRIES	2147483647	The number of retries for broker requests that return a retry-able error.
RETRY_BACKOFF_MS	100	The amount of time (milliseconds), before a request is retried. This applies if the retries parameter is configured to be greater than 0.
RECEIVE_BUFFER_BYTES	65536	The size of the TCP receive buffer to use when reading data. If the value is -1, the OS default will be used.
MAX_PARTITION_FETCH_BYTES	1048576	The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer.
MAX_REQUEST_SIZE	1048576	The maximum size of a request in bytes.
BUFFER_MEMORY	33554432	The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
BATCH_SIZE	16384	The default batch size in bytes when batching multiple records sent to a partition
LINGER_MS	100	The producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together
HEARTBEAT_INTERVAL_MS	1000	The expected time (milliseconds) between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group.
MAX_POLL_INTERVAL_MS	3600000	The maximum delay (milliseconds) between invocations of poll() when using consumer group management
MAX_POLL_RECORDS	100	The maximum number of records returned in a single call to poll().
SESSION_TIMEOUT_MS	180000	The timeout (milliseconds) used to detect client failures when using Kafka's group management facility
REQUEST_TIMEOUT_MS	305000	The configuration controls the maximum amount of time (milliseconds)  the client will wait for the response of a request.
CONNECTIONS_MAX_IDLE_MS	540000	The maximum amount of time (milliseconds) before idle connections are closed.
TH_NUM_THREADS	6	The number of threads to execute stream processing.
MAX_BLOCK_MS	60000 (milliseconds)	Controls how long the KafkaProducer's send(), partitionsFor(), initTransactions(), sendOffsetsToTransaction(), commitTransaction() and abortTransaction() methods will block.
DELIVERY_TIMEOUT_MS	120000 (milliseconds)	An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker (if expected), and the time allowed for retriable send failures.

Web Service Properties

As explained in Configuring the Values, prefix the names of these properties with arcsight.th.web-service. to create an override.

Property Name	Default Value	Description
WS_AUTH_ARCMC_CONNECTION_TIMEOUT	30000	The amount of time (milliseconds) before a request to connect to ArcMC is retried due to ArcMC timeout.
WS_AUTH_ARCMC_CONNECTION_NUM_RETRIES	2	The number of times that TH retries to connect ArcMC due to ArcMC timeout.

Configuring the Values

1. Create a file named arcsight-env-override.properties under the folder <NFS_root_DIRECTORY>/transformationhub/config.
The <NFS_root_DIRECTORY> path is described in this guide as the NFS root folder (usually /opt/arcsight/nfs/volumes). For more information, refer to the section Creating the NFS Shares.
2. Add properties to the file. To each property, add the module prefix from the table below.

Module	Prefix
Kafka	arcsight.th.kafka.
Schema Registry	arcsight.th.schema-registry.
ZooKeeper	arcsight.th.zookeeper.
Routing/C2AV/Enrichment Processor	arcsight.th.sp.
Web Services	arcsight.th.web-service.

3. Delete the pods for which properties were defined, or, alternatively, redeploy Transformation Hub.
4. To verify the changes, search the log file (after the container's status is back to Running) for matching properties.

Changing Value Examples

To change the value of ZOOKEEPER_MAX_CLIENT_CNXNS to 65, in ZooKeeper, and to change the value of SCHEMA_REGISTRY_KAFKASTORE_TIMEOUT_MS in the Schema Registry, create a file, <NFS Volume mount>/transformationhub/config/arcsight-env-override.properties, and add the following lines:

arcsight.th.zookeeper.ZOOKEEPER_MAX_CLIENT_CNXNS=65

arcsight.th.schema-registry.SCHEMA_REGISTRY_KAFKASTORE_TIMEOUT_MS=20000

Example of verifying the change by searching the log:

kubectl -n transformationhub1 logs th-zookeeper-0 | grep ZOOKEEPER_MAX_CLIENT_CNXNS
Environment override script set: ZOOKEEPER_MAX_CLIENT_CNXNS=65
ZOOKEEPER_MAX_CLIENT_CNXNS=65