Completing the Database and Kafka Scheduler Setups - Google Cloud

This section details the process for completing the database and Kafka Scheduler setups for both On-premises and cloud deployments.

Gathering Certificates for the Kafka Scheduler Setup

The database and deployed capabilities need to establish a trusted connection. To do so, generate the key pair for the Kafka Scheduler.

This step is required even if you use non-SSL communication between the Kafka Scheduler and Transformation Hub, because the schema registry is always SSL-enabled.

  1. Run these commands on your database node1 to generate the Kafka Scheduler private key file kafkascheduler.key.pem and the certificate signing request file kafkascheduler.csr.pem:

    cd <yourOwnCertPath>/
    If you installed using the ArcSight Platform Installer, the default location is /opt/arcsight-db-tools/cert/
    openssl req -nodes -newkey rsa:2048 -keyout kafkascheduler.key.pem -out kafkascheduler.csr.pem -subj "/C=US/ST=State/L=City/O=Company Inc./OU=IT/CN=kafkascheduler"

  2. Copy the certificate signing request kafkascheduler.csr.pem to your cluster, bastion host, or jump host.

  3. Run the following commands on your cluster or your bastion host to sign the certificate signing request using your cluster RE certificate:

    export VAULT_POD=$(kubectl get pods -n core -o custom-columns=":metadata.name"| grep itom-vault)
export PASSPHRASE=$(kubectl get secret vault-passphrase -n core -o json 2>/dev/null | jq -r '.data.passphrase')
export ENCRYPTED_ROOT_TOKEN=$(kubectl get secret vault-credential -n core -o json 2>/dev/null | jq -r '.data."root.token"')
export VAULT_TOKEN=$(echo ${ENCRYPTED_ROOT_TOKEN} | openssl aes-256-cbc -md sha256 -a -d -pass pass:"${PASSPHRASE}")
export COMMON_NAME=kafkascheduler
export CSR=$(cat ${COMMON_NAME}.csr.pem)
    WRITE_RESPONSE=$(kubectl exec -it -n core ${VAULT_POD} -c vault -- bash -c "VAULT_TOKEN=$VAULT_TOKEN vault write -tls-skip-verify -format=json RE/sign/coretech csr=\"${CSR}\"") && \
echo "${WRITE_RESPONSE}" | jq -r ".data | .certificate" > ${COMMON_NAME}.crt.pem && \
echo "${WRITE_RESPONSE}" | jq -r ".data | if .ca_chain then .ca_chain[] else .issuing_ca end" > issue_ca.crt

  4. Copy the RE signed certificate file kafkascheduler.crt.pem to database node1 <yourOwnCertPath>.

  5. Copy the issue_ca.crt to database node1 <yourOwnCertPath>.

Enabling the Database to Receive SSL Connections

The following procedures are required for data privacy. Perform the first two procedures below on database node1.

Creating the Database Server Key and Certificate

Follow these steps to generate database CAs and certificates:

  1. Log in to database node1.

  2. Change to your own certificates directory path:

    cd <yourOwnCertPath>
    For deployment with arcsight-platform-instsaller, the default location is /opt/arcsight-db-tools/cert/

  3. Run this command to create a certificate authority (CA) for the database:

    openssl req -newkey rsa:4096 -sha256 -keyform PEM -keyout generated-db-ca.key -x509 -days 3650 -outform PEM -out generated-db-ca.crt -subj "/C=US/ST=State/L=City/O=Company Inc./OU=IT/CN=Database/emailAddress=admin@microfocus.com" -nodes

  4. Run this command to create the database server key:

    openssl genrsa -out generated-db-server.key 4096

  5. Create the database server certificate signing request by running the following command:

    openssl req -new -key generated-db-server.key -out generated-db-server.csr -subj "/C=US/ST=State/L=City/O=Company Inc./OU=IT/CN=DatabaseServer/emailAddress=admin@microfocus.com" -nodes -sha256

  6. Sign the Certificate Signing Request with self-signed CA by running the following command:

    openssl x509 -req -in generated-db-server.csr -CA generated-db-ca.crt -CAkey generated-db-ca.key -CAcreateserial -extensions server -days 3650 -outform PEM -out generated-db-server.crt -sha256

Setting up the Database SSL Configuration

These steps will update the SSL configuration in the database.

  1. Move the following files to database node1 <yourOwnCertPath> as root by running these commands:

    This step is only required for a new database installation. You can skip this step if this is an upgrade and the files are already there.
    cd <yourOwnCertPath>/
ls <yourOwnCertPath>/

    • The output should have the following files:

      • generated-db-ca.crt

      • generated-db-server.crt

      • generated-db-server.key

      • generated-db-ca.key

      • generated-db-ca.srl

      • generated-db-server.csr

      • issue_ca.crt

      • kafkascheduler.crt.pem

      • kafkascheduler.key.pem

  2. For chained CAs, run the commands to split the CAs into individual files:

    cat issue_ca.crt | awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "issue_ca_part." c ".crt"}'
    chown -R <dbadmin_username>:<dbadmin_username> <yourOwnCertPath>

  3. Run the following commands on database node1 to update the database SSL configuration:

    cd /opt/arcsight-db-tools
    ./db_ssl_setup --disable-ssl

    If the attempt fails, drop the certificate manually by running the three commands below:

    sudo su - dbadmin

    vsql -U <dbadminuser> -w <dbadminpassword> -c "ALTER TLS CONFIGURATION server CERTIFICATE NULL;"
    vsql -U <dbadminuser> -w <dbadminpassword> -c "DROP CERTIFICATE IF EXISTS server CASCADE;"

  4. Enable database SSL for a single issue CA or chained issue CAs:

    • For a single issue CA, run this command:

      ./db_ssl_setup --enable-ssl --vertica-cert-path <yourOwnCertPath>/generated-db-server.crt --vertica-key-path <yourOwnCertPath>/generated-db-server.key --vertica-ca-path <yourOwnCertPath>/generated-db-ca.crt --client-ca-path <yourOwnCertPath>/issue_ca.crt

      -or-

    • For chained issue CAs, run this command, specifying each CA certificate in the chain one by one, separated by a comma in the client-ca-path parameter:

      ./db_ssl_setup --enable-ssl --vertica-cert-path <yourOwnCertPath>/generated-db-server.crt --vertica-key-path <yourOwnCertPath>/generated-db-server.key --vertica-ca-path <yourOwnCertPath>/generated-db-ca.crt --client-ca-path <yourOwnCertPath>/issue_ca_part.1.crt,<yourOwnCertPath>/issue_ca_part.2.crt[,...]

Configuring Deployed Capabilities to Use SSL for Database Connection

  1. Log in to the OMT Management Portal.
  2. Navigate to Fusion > Database Configuration > Database Certificate(s).
  3. Enable the Use SSL for Database Connection option.
  4. Copy the complete contents of the file generated-db-ca.crt, created from the steps earlier, into the Database Certificate(s) text area.
  5. Click Save to activate the configuration changes.

Enabling the Database to Ingest Events from Transformation Hub

The database uses an event consumer, the Kafka scheduler, to ingest events from Transformation Hub's Kafka component. Follow these steps when configuring the Kafka Scheduler for a new installation of the ArcSight Database:

Before you perform these steps, ensure that you have enabled SSL for the database. For information, see Enabling the Database to Receive SSL Connections.

  1. Log in to the database node1 as root.

  2. Change to the database tools directory:

    cd /opt/arcsight-db-tools/

  3. Run the following command on database node1 to configure the schema registry server setting:

    ./schema_registry_setup <FQDN of ArcSight Platform Virtual IP for HA, single master node or cloud DNS name for your cluster> <yourOwnCertPath>/issue_ca.crt <yourOwnCertPath>/kafkascheduler.crt.pem  <yourOwnCertPath>/kafkascheduler.key.pem
    You must provide the absolute path to the certificate.

  4. Configure the SSL setup:

    On database node1, configure the SSL setting for the Kafka Scheduler by using one of the following methods, plain text or SSL:

    Plain Text (non-SSL)

    This method requires that you first enable Allow plain text (non-TLS) connections to Kafka. For more information, see Configuring the Deployed Capabilities.

    Run this command to disable SSL for the Kafka scheduler:

    ./sched_ssl_setup --disable-ssl

    SSL

    This method uses the crt and key files gathered or generated in earlier steps. The issue_ca.crt file should contain all chained CAs. For the Kafka scheduler to use SSL, run the following command:

    ./sched_ssl_setup --enable-ssl --sched-cert-path <yourOwnCertPath>/kafkascheduler.crt.pem --sched-key-path <yourOwnCertPath>/kafkascheduler.key.pem --vertica-ca-key <yourOwnCertPath>/generated-db-ca.key --vertica-ca-path <yourOwnCertPath>/generated-db-ca.crt --kafka-ca-path <yourOwnCertPath>/issue_ca.crt

  5. Run this command on database node1 to create the Kafka Scheduler:

    • If the Kafka Scheduler was configured to use plain-text in the previous step, use port 9092:

      ./kafka_scheduler create <th_kafka_nodename1>:9092,<th_kafka_nodename2>:9092[,...]

    • If SSL was enabled for the Kafka Scheduler in the previous step, use port 9093:

      ./kafka_scheduler create <th_kafka_nodename1>:9093,<th_kafka_nodename2>:9093[,...]

  6. Start the Kafka Scheduler and checker on database node1:

    ./kafka_scheduler start
./kafka_scheduler messages
./kafka_scheduler events
The dbadmin user has access to all the certificate/keys files.