You can use the data_uploader.sh script to migrate data to one of the following data storage components:
Forwarding Older Sentinel Events to Transformation Hub
Once Arcsight Intelligence Integration is configured, older Sentinel events in the existing event data partitions can be forwarded using the data uploader tool.
Run the data uploader tool with the required parameters to forward data present in existing partitions to Transformation Hub:
-from: From date range in the format YYYYMMDD
-to: To date range in the format YYYYMMDD
-masterurl: The hostname and port number of the Master Transformation Hub node.
NOTE:Transformation Hub runs a confluent Schema Registry on the default port 32081
-workerurl: The hostname and port number of the Worker Transformation Hub node.
-certdir: The path to the custom directory on the Sentinel Server that contains the Transformation Hub certificate, key, and Certification Authority (CA).
-target: The <INTERSET> destination where you want to upload the data (destination is case sensitive).
For more information see the Sentinel Administration Guide.
OpenSearch: You can forward only event data to OpenSearch. Before you forward the data, ensure that you have enabled event visualization. For more information, see Section 18.0, Configuring OpenSearch for Event Visualization.
Run the data uploader tool with the required parameters to forward data present in existing partitions to OpenSearch:
-from: From date range in the format YYYYMMDD
-to: To date range in the format YYYYMMDD
-url: IP address or hostname and port number (https://<host:port>) of the OpenSearch node to which you want to upload data.
-target: The <OPENSEARCH> destination where you want to upload the data (destination is case sensitive).
Optional Parameters:
-path: The location of the data folder. For event data, the default location is <sentinel_installation_path>/var/opt/novell/sentinel/data/eventdata/events. For raw data, the default location is <sentinel_installation_path> /var/opt/novell/sentinel/data/rawdata/online.
-eps: The maximum rate at which data should be uploaded to OpenSearch. By default, it is unlimited.
-statusfile: Location of the file that contains the data upload status. The default location is /var/opt/novell/sentinel/log/data_upload_status.json.
-retryinterval: Retry interval (in seconds) for checking if any new events have arrived.
-filter: Only those events, which match this Lucene filter will be eligible for upload.
The script transfers data for the date range (from and to) you specify. When you run the script, it displays the mandatory and optional parameters you should specify to initiate the data migration and also the information about the relevant properties to use for the desired data storage component.
The script must be run as novell user. Therefore, ensure that the data directories and any files you specify have appropriate permissions for novell user. By default, the script migrates data from primary storage. If you want to migrate data from secondary storage, specify the appropriate path for secondary storage when running the script.
To migrate data:
Log in to the Sentinel server as the novell user.
Run the following script:
<sentinel_installation_path>/opt/novell/sentinel/bin/data_uploader.sh
Follow the on-screen instructions and run the script again with the required parameters.
The migrated data will have the retention period as set in the target server.
Once the data migration is done, the script records the status such as partitions migrated successfully, partitions failed to migrate, number of events migrated, and so on. For partitions with previous day and current day’s date, the data transfer status will show IN_PROGRESS considering events that may come in late.
Run the script again in scenarios where the data migration did not complete successfully or where the data migration status for partitions still indicate IN_PROGRESS. When you re-run the script, it first checks the status file to understand the partitions that were already migrated and then continues to migrate only the remaining ones. The script maintains the logs in the /var/opt/novell/sentinel/log/data_uploader.log directory for troubleshooting purposes.