Understanding Data Ingestion from Your Environment

Your environment has the capabilities to ingest both live data (from SmartConnectors) or archived data (from Logger).

Ingesting Data from SmartConnectors

To collect data, your local environment uses SmartConnectors. These SmartConnectors intelligently collect a large amount of heterogeneous raw event data from devices in an enterprise network, process the data into ArcSight events, then compress and transport data to destination devices. SmartConnectors also automate the process of ingesting and managing logs from any device and in any format through normalization and categorization of logs into a unified format. They can parse individual events and normalize event values into the common event schema for log consumers.

You as the customer must host SmartConnectors to ensure that data sources within your environment send data to ArcSight. You can install or run these SmartConnectors from:

Depending on how you configure the installed SmartConnectors, they will send data in batches to an Amazon S3 bucket destination as a temporary storage for collected events) or to an ArcSight SaaS destination as a live stream. ArcSight then consumes the data. When you receive the ArcSight SIEM as a Service account, OpenText provisions the destination for you to use and store the collected data. If you use just the Log Management & Compliance service, you will configure the Amazon S3 bucket and create an AWS IAM user account to properly configure the SmartConnectors. The SmartConnectors will connect to the S3 bucket as the AWS IAM user. However, if you use the Real-time Threat Detection service with or without Log Management service, you will configure the SmartConnector to send data to the ArcSight SaaS destination. You will need the host name of the MSK cluster and the user name and password that OpenText provides for SmartConnector configuration. The SmartConnectors will connect to the ArcSight SaaS destination using the provided credentials. For more information about configuring SmartConnectors to connect to MSK, see the Installation Guide for SmartConnectors.

The SmartConnectors must have internet connectivity directly or through a proxy. Configuring the connectors to connect directly or through a proxy to the configured destination avoids the need to open specific firewall ports or establish a VPN connection for each connector.

Note: Although you might configure SmartConnectors to use the ArcSight SaaS destination, you will still need to create the AWS IAM user account. You need the account to access the Amazon S3 bucket to download the files for installing and upgrading the ArcSight components that your environment needs.

To help you effectively monitor and manage a large deployment of SmartConnectors, use the centralized management interface in ArcSight Management Center (ArcMC).

You can install a standalone instance of ArcMC to manage multiple SmartConnectors. To use the full capabilities of ArcSight, review SmartConnector Installation Overview in the Installation Guide for ArcSight SmartConnectors.

Ingesting Data from ArcSight Logger

ArcSight SaaS can import archived data from all available Loggers, thus eliminating the need to continue managing them in your environment. The system stores the imported data in the ArcSight Database, making it available for Search and Reporting activities once the migration has completed successfully.

To perform the migration process, see Setting Up Data Ingestion and Importing Logger Data to the ArcSight Database (SaaS).