Installing the Database in Azure

This section provides information about installing the ArcSight Database in Azure.

Understanding Azure Blob Storage

The database uses a single communal storage location for all data and for the catalog (metadata). Communal storage is the database's centralized storage location, shared among the database nodes. This mode supports communal storage in Azure Blob Storage, which must be set up by your cloud administrator before you can install the database. To install the database with communal storage as Azure Blob Storage, you need the following:

  1. An Azure storage account name with a valid Azure subscription.

  2. An Azure storage container.

  3. An Azure account key.

For more information see, Azure Portal.

Preparing the Azure Virtual Machine

This section describes how to prepare the Azure virtual machine for database installation.

  1. Log in to the Azure portal.
  2. Select an active Azure subscription.
  3. Click Create a Resource or select an existing resource.
  4. In the search box (case-insensitive), enter RHEL-8.6 minimal.
  5. Click create.
  6. In the Virtual machine name field, enter a name for the virtual machine.
  7. In the Image drop-down list, select supported image.
  8. In the Size drop-down list, select the VM that will be accomplishing the database requirements. For example, D8s_v3.
  9. Select the appropriate Authentication type, and enter the details.
  10. In the Select inbound ports drop-down list, be sure SSH (22) is selected.
  11. In the Public inbound ports area, select the Allow selected ports option.
  12. Click Next: Disks.
  13. From the Disks tab and create a new disk of minimum size 256 GB.
  14. For Data disks and Host caching select Read/write.
  15. Select the Networking and your Virtual network.
  16. Create a subnet for the database and select it in the subnet drop down.
  17. Click Review + create to review and create a VM ; wait to complete deployment.
  18. Click Go to your resource and launch the new VM.
  19. In the left navigation panel, click Overview, find DNS name, and click the associated link.
  20. Set Assignment to static and enter the DNS name label.
  21. Click Save.
  22. In the left navigation pane, click Networking.
  23. Select your Network Interface.
  24. From the left menu, select IP configuration.
  25. Navigate to the IP Forwarding Field, and select Enabled.
  26. Click Save.
  27. SSH to your VM with it's public IP address.
  28. Log in with your VM user, then become root.
  29. Change your root password.
  30. Create a folder for Vertica by running the command:
    31. mkdir /opt/vertica
  31. Create partitions on the datadisk.
    Vertica requires a minimum 2 GB swap partition irrespective of the amount of RAM installed. The remaining part of the disk, you can disk format ext4 and mount to /opt/vertica.
    32. [root@vertica1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2)

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x2959fe99.
 
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
 
Command (m for help): n
Partition type:
p   primary (0 primary, 0 extended, 4 free)
e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-536870911, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-536870911, default 536870911): ${sizePartitionForSwap}
Partition 1 of type Linux and of size 4 GiB is set
 
Command (m for help): n
Partition type:
p   primary (1 primary, 0 extended, 3 free)
e   extended
Select (default p):
Using default response p
Partition number (2-4, default 2):
First sector (4196352-536870911, default 4196352):
Using default value 4196352
Last sector, +sectors or +size{K,M,G} (4196352-536870911, default 536870911):
Using default value 536870911
Partition 2 of type Linux and of size 254 GiB is set
 
Command (m for help): p
 
Disk /dev/sdc: 274.9 GB, 274877906944 bytes, 536870912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0xcca9a285
 
Device Boot      Start         End      Blocks   Id  System
/dev/sdc1         2048     2097152     1047552+  83  Linux
/dev/sdc2      2099200   536870911   267385856   83  Linux
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.
  32. Create a swap by running the command:
    33. mkswap /dev/sdc1
  33. Activate the swap by running the command:
    34. swapon /dev/sdc1
  34. To format rest of disk to ext4, run the command:
    35. mkfs.ext4 /dev/sdc2
  35. Get the last created UUID of disks from the output by running the command:
    36. blkid
  36. For swap (/dev/sdc1) and for rest disk (/dev/sdc2) take note of the given UUID values, modify the /etc/fstab file, and add the following lines by replacing the UUIDs:
    37. UUID=<UUID sdc1>    none	  swap	sw	  0 0
UUID=<UUID sdc2>    /opt/vertica  ext4	 defaults  0 0
  37. Mount all by running the command:
    38. mount -a
  38. Check if /opt/vertica is assigned by running the command:
    39. df -h
  39. Enable the swap by running the command:
    40. swapon -a
  40. Check the swap size by running the command:
    41. free -h
  41. Repeat these steps for each expected database node.

 

Installing Prerequisites

This section describes how to install the prerequisites necessary to install the Azure database.

  1. Configure passwordless communication from the node1 server to all of the node servers in the cluster.

    1. On the node1 server, run the ssh-keygen command:

      ssh-keygen -t rsa

    2. Copy the key from node1 to all of the nodes, including node1, using the node IP address:

      ssh-copy-id -i ~/.ssh/id_rsa.pub root@$node1,root@$node2,root@$node3
  2. Set up and activate /etc/rc.local by running the following command:
    3. #!/bin/sh
function drive {
block_device=`realpath $(df $1 | grep '^/' | cut -d' ' -f1)`
partition=$(echo $block_device | sed -e "s#/dev/##")
if [[ $partition == dm-* ]]; then
echo $partition
else
echo $partition | cut -c1-3
fi
}
cat > /etc/rc.local << EOF
#!/bin/sh
touch /var/lock/subsys/local
/sbin/blockdev --setra 2048 /dev/$(drive /)
/sbin/blockdev --setra 2048 /dev/$(drive /opt)
echo deadline > /sys/block/$(drive /)/queue/scheduler
echo deadline > /sys/block/$(drive /opt)/queue/scheduler
echo never > /sys/kernel/mm/transparent_hugepage/enabled
tuned-adm profile throughput-performance
EOF
chmod 755 /etc/rc.local
/etc/rc.local

  3. Add the following parameters to /etc/sysctl.conf.

    4. Parameter Description
    net.core.somaxconn = 1024 Increases the number of incoming connections
    net.core.wmem_max = 16777216 Sets the send socket buffer maximum size in bytes
    net.core.rmem_max = 16777216 Sets the receive socket buffer maximum size in bytes
    net.core.wmem_default = 262144 Sets the receive socket buffer default size in bytes
    net.core.rmem_default = 262144 Controls the default size of receive buffers used by sockets
    net.core.netdev_max_backlog = 100000 Increase the length of the network interface input queue
    net.ipv4.tcp_mem = 16777216 16777216 16777216
    net.ipv4.tcp_wmem = 8192 262144 8388608
    net.ipv4.tcp_rmem = 8192 262144 8388608
    net.ipv4.udp_mem = 16777216 16777216 16777216
    net.ipv4.udp_rmem_min = 16384
    net.ipv4.udp_wmem_min = 16384
    vm.swappiness = 0

    Defines the amount and frequency at which the kernel copies RAM contents to a swap space

    For more information, see Check for Swappiness in the ArcSight Database Guide.

  4. Next, run the following command:
    5. sysctl -p

  5. To disable the firewall WARN (N0010), use iptables: 

    iptables -F

    iptables -t nat -F

    iptables -t mangle -F

    iptables -X

    systemctl mask firewalld

    systemctl disable firewalld

    systemctl stop firewalld

    The database requires several ports to be open on the local network. Micro Focus does not recommend that you place a firewall between nodes (all nodes should be behind a firewall), but if you must use a firewall between nodes, ensure that all the database ports are available (see Firewall ports in Technical Requirements for ArcSight Platform). For more information, see Firewall Considerations in ArcSight Database Guide.
  6. Set SELinux to permissive mode in /etc/selinux/config. 
    SELINUX=permissive

    For more information, see SELinux Configuration in ArcSight Database Guide.

  7. In /etc/default/grub, append line GRUB_CMDLINE_LINUX with intel_idle.max_cstate=0 processor.max_cstate=1.

    For example:

    GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable"

    Execute the following command:

    grub2-mkconfig -o /boot/grub2/grub.cfg

  8. If you have a high concurrency workload and if the database is CPU bound, reboot the virtual machine by running the following command:
    9. sudo sysctl -w net.core.netdev_max_backlog=2000
  9. Reboot for your changes to take effect.
  10. For RHEL, you must run RHEL using the following command:
    11. dnf install libnsl
  11. Install the packages by running the following command:
    12. yum install -y java-1.8.0-openjdk gdb mcelog sysstat dialog chrony tzdata wget
  12. Modify the /etc/bashrc by running the following command:
    13. export VERTICA_FAILURE_THRESHOLD=FAIL
  13. Apply the changes by running the following command:
    14. source ~/.bashrc
  14. Repeat these steps for each expected database node.

 

Configuring and Installing the Database Server

This section describes how to configure and install the Azure database.

Before installing the database, ensure that you estimate the storage needed for the incoming EPS (event per second) and event size, and also evaluate the retention policy accordingly.
  1. On the Database cluster node1 server, create a folder for the database installer.

    For example:

    mkdir /opt/arcsight-db-tools
    2. /opt/arcsight-db-tools should not be under /root or /opt/vertica.

  2. From the master node where you performed the Downloading Installation Packages steps, copy the following file on the Database cluster node1 server:

    db-installer_x.x.x-x.tar.gz

    to the /opt/arcsight-db-tools directory

  3. To extract the installer file and place it in the correct directory, run the following commands:

    cd /opt/arcsight-db-tools
tar xvfz db-installer_x.x.x.x.tar.gz

  4. Edit the config/db_user.properties file and add all database node IPs to the hosts property.

    Property

    Description

    hosts

    A comma separated list of the database servers in IPv4 format (for example, 1.1.1.1,1.1.1.2,1.1.1.3).

    If it is necessary to construct the cluster, avoid using local loopback (localhost, 127.0.0.1, etc.).

  5. Install the database.

    ./db_installer install

  6. When prompted, create the database administrator user.

    The database administrator user account is used during database deployment, configuration, upgrade, and debugging. For security reasons, the platform deployed capabilities will not ask you for the credentials for this user.

    ---------------------------------------
Please specify a username for [ DB Admin ] user:
dbadmin

---------------------------------------
Please specify a password for [ DB Admin ] user:
********
Re-enter password:
********
    For a list of options that you can specify when installing the database, see Understanding the Database Installer Options.

  7. Specify the shard count. The default shard count is 3 for single-node or 18 for multi-node to allow for scalability.

    • Single-node:

      # ========================================
# STEP 1: Specify Database Shard Count for Eon Mode
Do you plan to keep the database cluster to a single node in the future?
If yes, the database will be optimized for performance on a single node by setting the default shard count to 3.
Shard Count [3]:
Confirm shard count [3]?(y/n):y
Check memory size, 48GB required for single node installation with shard count > 3.
PASS: Single node installation for shard count: 3

    • Multi-node:

      # ========================================
# STEP 1: Specify Database Shard Count for Eon Mode
Recommended shard count for multi node database deployment is 18. 
Shard Count [18]:
Confirm shard count [18]?(y/n):y

  8. Set up the communal storage type for Azure Blob Storage when prompted. For example:

    # ========================================
# STEP 2: Specify communal storage details
Supported communal storage types - 
1) S3
2) Azure Blob Storage
Choose a communal storage type from the above (1/2):2
Specify account name: <your Azure account name>
Do you want to use managed identity to authenticate with Azure storage container(y/n):n
Specify account key:
Is TLS enabled(y/n):y
Specify Azure container for communal storage: <your Azure container name>
Specify the folder under container for communal storage if applicable:
Communal storage url is: https://<your account name>.blob.core.windows.net/mycontainer/myblob

  9. Create the schema.

    ./db_installer create-schema

  10. When prompted, create the following users:

    • App admin user: A regular database user granted elevated permissions for performing operations on the database to manage the database, schema, and resource pools. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

    • Search user: A regular database user with permissions restricted to event search operations. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

  11. Monitor your database cluster status constantly. For more information, see Database Cluster Node Status and Monitoring the Database.

    • Database nodes status: Ensures all nodes are up

    • Database nodes storage status: Ensures storage is sufficient

Next Step - If your deployment includes Intelligence: (Conditional – Intelligence) Configuring Settings for Elasticsearch in Azure

Next Step - If your deployment does not include Intelligence: Installing the OMT Infrastructure