Installing the Database in AWS

This section provides information about installing the ArcSight Database in Amazon Web Services Deployment (AWS).

Understanding Methods for Connecting to AWS S3 Buckets

The database uses a single communal storage location for all data and for the catalog (metadata). Communal storage is the database's centralized storage location, shared among the database nodes. This mode supports communal storage in Amazon S3, which must be set up by your cloud administrator before you can install the database.

Other prerequisites or considerations for the S3 bucket:

Launching Database Instances in AWS

The ArcSight Database compute operations are performed on a cluster of AWS EC2 (Elastic Compute Cloud) nodes. These nodes must be deployed in AWS before the database can be installed on them.

  1. Sign in to the Amazon EC2 console and select a region.

  2. Select Launch Instances.

  3. Enter a name for your instance and add required tags.

  4. Search for AMIs. The database supports Red Hat 8.4 AMIs. For information about finding an AMI, see AWS documentation, Find a Linux AMI.

    For example. If you are in the US East (N. Virginia) region, you can use the following AMIs:

  5. After selecting the AMI, and reviewing the details, select an Instance Type. Choose the AWS EC2 instance type that best matches your requirements.

    We recommend using an m5d instance type, because it has the best balance of cost, performance, and ease of setup for ArcSight workloads. Take into account that m5d is an instance store that provides temporary storage. For example, data files stored on the instance are lost when the instance is stopped. The durable EBS instance is also supported with Elastic Block Store (EBS). This instance type provides durable storage. For example, data files stored on the instance persist after the instance is stopped.
    For more information on which size of m5d (instance store ) and /m4 (EBS) to use for your workload, see the Technical Requirements for ArcSight Platform.

  6. For Key pair, select an existing one or create new.

  7. Configure your instance with the VPC, private subnets and security groups.

  8. In the Configure storage section, change the size for the Root volume type to 30 GiB, the minimum size for root, or higher.

    If you use the EBS Instance type, m4, add an EBS volume that follows the System Performance Benchmarks for Sizing and Tuning in the Technical Requirements for ArcSight Platform.
  9. In the Advanced details section, go to the IAM instance profile (IAM role) and select the IAM profile you configured to access the S3 bucket for database communal storage. For more information, see Understanding Methods for Connecting to AWS S3 Buckets.
  10. In the Summary section, configure the number of AWS EC2 instances to deploy. A minimum of three is required for a highly available database cluster, but more might be necessary to handle your workload. For more information on how many instances you need, see the Technical Requirements for ArcSight Platform.

  11. Configure the number of AWS EC2 instances to deploy. A minimum of three is needed for a highly available database cluster. For more information, on how many instances you need, see the Technical Requirements for ArcSight Platform.

  12. Click Launch Instances.

  13. Navigate to EC2, and verify your instances are available.

Creating the Swap File

A partition with swap space configured is required to enable virtual memory access. The database requires a minimum of 2 GB of swap space on all database nodes. Perform the following procedure on all database nodes to setup the swap space.

  1. From the bastion, login to the database node using SSH and the key pair selected during instance launching. For example:

    ssh -i <FilePath to the ec2 keypair/IdentityFile> root@<node1..n>

  2. Create the file that will be used for swap: 

    sudo dd if=/dev/zero of=/swapfile bs=1024 count=2621440
    The count 261440 is the minimum value required for installation, but it is also an example. You can set the value higher.

  3. Enable read and write to the swap file for the root user (only): 

    sudo chmod 600 /swapfile

  4. Set up the file as a Linux swap area: 

    sudo mkswap /swapfile

  5. Enable the swap with the following command: 

    sudo swapon /swapfile

  6. Enable the swap permanently as root user by editing the /etc/fstab file and appending the following line:

    echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab

  7. Verify that swap is enabled: 

    sudo swapon --show
    Output example:

NAME      TYPE SIZE USED PRIO
/swapfile file 2.5G   0B   -2

Formatting Devices for the Installation

The database requires a storage device formatted in ext4 for all nodes. Perform the following procedure on all database nodes to setup ext4 in devices.

The steps below are required to be performed on all AWS EC2 instances that are created. For more information about creating instances, see Launching Database Instances in AWS

Format the device type applicable to the AWS EC2 type you are using in the instance, either an instance Store type or EBS type:

Formatting Instance Store Type Devices

The steps below are required to be performed on all AWS EC2 instances that are created.

ONLY  perform this procedure if you are using Instance Store type devices.

  1. From the bastion, login to the database node using SSH and the key pair selected during launching.
    For example:

    ssh -i <FilePath to the ec2 keypair/IdentityFile> root@<node1..n>

  2. Install the nvme tool:

    sudo dnf install -y nvme-cli

  3. Create an /opt mount point. If /opt already exists, skip this step.

    sudo mkdir /opt/vertica

  4. Use the nvme tool to locate the nvme device and format it:

    EPHEMERAL_DISK=$(sudo nvme list | grep 'Amazon EC2 NVMe Instance Storage' | awk 'NR==1{ print $1 }')
    echo $EPHEMERAL_DISK
    sudo mkfs.ext4 $EPHEMERAL_DISK
    sudo mount -t ext4 $EPHEMERAL_DISK /opt

  5. Add the ephemeral disk mount to /etc/fstab:

    echo "$EPHEMERAL_DISK /opt ext4 defaults,nofail 0 0" | sudo tee -a /etc/fstab

  6. To list the mount points for verification, run the lsblk command:

    lsblk

Formatting EBS Type Devices

The steps below are required to be performed on all AWS EC2 instances that are created. These steps for EBS also assume you are formatting a single device. For more information about creating instances, see Launching Database Instances in AWS.

ONLY perform this procedure if you are using EBS type devices.

  1. Run the following command to locate the EBS device attached to the instance, and to select the device that is not mounted:

    lsblk -f
    #Output from lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT 
xvda
└─xvda1 xfs 35761952-413c-42e8-b047-a5deb7510f29 / 
xvdb

  2. Run the following command to format the device to ext4:

    sudo mkfs.ext4 /dev/xvdb
    NOTE: xvdb is an example and will be <yourDeviceName> from the output in the previous step.

  3. Format the device:

    sudo mount -t ext4 /dev/xvdb /opt

  4. Create the mount point. If /opt already exists, skip this step.

    sudo mkdir /opt/vertica

  5. Modify the /etc/fstab file, and add an entry to mount the device on OS boot:

    echo "/dev/xvdb /opt ext4 defaults 0 0" | sudo tee -a /etc/fstab

Persisting Operating System Settings

The database requires that you manually configure several general operating system settings. Perform the following procedures on all database nodes to setup ext4 in devices.

  1. Run this command to set the limit for open files so that it meets database requirements. This will add the parameters to the /etc/sysctl.conf file.

    cat << EOF | sudo tee -a  /etc/sysctl.conf
net.core.somaxconn = 1024
net.core.wmem_max = 16777216	
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.rmem_default = 262144
net.core.netdev_max_backlog = 100000
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_wmem = 8192 262144 8388608
net.ipv4.tcp_rmem = 8192 262144 8388608
net.ipv4.udp_mem = 16777216 16777216 16777216
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384
vm.swappiness = 0
EOF

  2. Update the/etc/rc.local file by running the commands below. This file contains scripts and commands that run each time the system is booted, and the database requires that I/O Scheduling be set to deadline or noop. The command will add the applicable lines to the file, based on the devices type in your AWS instance, either instance Store type or EBS type:

    Instance Store type rc.local settings (only) 

    cat << EOF | sudo tee -a /etc/rc.local
echo deadline > /sys/block/nvme0n1/queue/scheduler
echo deadline > /sys/block/nvme1n1/queue/scheduler
echo deadline > /sys/block/nvme2n1/queue/scheduler
echo deadline > /sys/block/md0p1/queue/scheduler

/sbin/blockdev --setra 2048 /dev/nvme0n1
/sbin/blockdev --setra 2048 /dev/nvme1n1
/sbin/blockdev --setra 2048 /dev/nvme2n1
/sbin/blockdev --setra 2048 /dev/md0p1

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

    Note: Replace the device in the example command above with your own instance device on which the /opt is mounted. For example, the following device must be replaced with your own device: sys/block/nvmen0n1

    EBS type rc.local settings (only) 

    cat << EOF | sudo tee -a /etc/rc.local
echo deadline > /sys/block/xvda/queue/scheduler
echo deadline > /sys/block/xvdb/queue/scheduler
echo deadline > /sys/block/xvdc/queue/scheduler

/sbin/blockdev --setra 2048 /dev/xvda
/sbin/blockdev --setra 2048 /dev/xvdb
/sbin/blockdev --setra 2048 /dev/xvdc

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

    Note: Replace the device in the example command above with your own instance device on which the /opt is mounted. For example, the following device must be replaced with your own instance device:

    sys/block/dev/xvdb

  3. Modify the file permissions:

    sudo chmod +x /etc/rc.d/rc.local

  4. Run the following commands to disable the system firewall:

    sudo systemctl mask firewalld
sudo systemctl disable firewalld
sudo systemctl stop firewalld
    During installation, the database requires that host-based firewalls are disabled on database nodes. After installation, the host-based firewalls can be enabled and the database requires several ports to be open on the local network. We recommend for optimal performance using host-based firewalls between database nodes and a network-based firewall to protect the segment that database cluster is within. However, there is no restriction against using a network-based firewall between database nodes. When using any kind of firewall, ensure that all the database ports are available (see the Technical Requirements for ArcSight Platform). For more information, see Firewall Considerations in the ArcSight Database Guide.

  5. Set SELinux to permissive mode in /etc/selinux/config.

    SELINUX=permissive

    For more information, see SELinux Configuration.

  6. Run this command to ensure that rng-tools packages are installed in all cluster nodes:

    sudo dnf install rng-tools -y

  7. Set the UTC time for all cluster nodes:

    sudo timedatectl set-timezone UTC

  8. Reboot the system for your changes to take effect.

    This step is required to update the device mounts and other settings.

Enabling Root Login for AWS Passwordless Communication

All commands require root privileges which can be obtained through the sudo command.

  1. Connect to one of your nodes and edit the /etc/ssh/sshd_config configuration file.

    sudo vi /etc/ssh/sshd_config

  2. Change the following parameter to yes if the value is not already set to that.

    PermitRootLogin yes

  3. Proceed as follows:

    • If you had to change the PermitRootLogin parameter to yes, run this command:

      sudo service sshd reload

    • If the PermitRootLogin parameter was already set to yes, proceed to the next step.

  4. Patch the authorized keys file for the root user by copying the RHEL file to the root user. This enables logging in with the same key which is available for the ec2-user (rhel-8.4).

    sudo cp ~<ec2-user>/.ssh/authorized_keys ~root/.ssh/authorized_keys
  5. Update the AWS cloud configuration file by editing /etc/cloud/cloud.cfg and changing the disable_root value to 0.
    sudo vi /etc/cloud/cloud.cfg

  6. Repeat all the above steps for all nodes in the cluster.

  7. Generate SSH keys for the database nodes:

    1. Log in to one of the database nodes. This becomes the initiator node (node1) for your cluster.

    2. Run the following command as root:

      ssh-keygen -q -t rsa -f ~/.ssh/id_rsa

    3. Run the following command as root, for each of the database nodes (initiator node inclusive):

      ssh-copy-id -f "-o IdentityFile <FilePath to the ec2 keypair>" root@<node1..n>
      Enter : Yes

    4. To verify the passwordless communication, run the following command:

      ssh root@<node1,node2..>

      Output example:

Activate the web console with: systemctl enable --now cockpit.socket
Last login: Thu Jan 20 19:26:50 2022 from <node1,node2..>

Next Step: Setting Up and Installing the Database for AWS