13.6 Configuring Data Synchronization

Sentinel provides the ability to synchronize data to an external database, so that you can use third-party or custom reporting systems to search the data in the external database with more advanced tools than what are provided in Sentinel.

13.6.1 Overview

Sentinel can store the data in an external database by synchronizing a subset of the data that Sentinel gathers.

Sentinel uses the following process to synchronize data:

  1. Sentinel gathers the events from the Event Sources through the Connectors.

  2. Sentinel uses Collectors to normalize the event data.

  3. The normalized event data is then sent to the Sentinel message bus.

  4. The event data is then stored and indexed in the file system in the primary storage.

  5. The data synchronization policies allow events in the primary storage to be copied and stored in PostgreSQL and external SQL databases.

    1. User-defined data synchronization policies synchronize the filtered event data to an external SQL database. For information about the certified databases, see Sentinel System Requirements.

    2. Report Data Definitions (RDD) generate system data synchronization policies that are used to copy event data into tables in the internal PostgreSQL database. These data synchronization policies cannot be edited or deleted. Reports that rely on an RDD will search internal database tables for events instead of the primary storage. These kinds of reports search internal tables instead of the event store because they utilize more complex SQL SELECT statements that need to join event data to the data in other tables in the internal database.

Figure 13-1 Data Synchronization

Sentinel allows you to partition tables if they are in the internal PostgreSQL database. When you choose to partition a table in the internal PostgreSQL database, a new table partition is created for each days worth of data.

NOTE:The supported version of PostgreSQL is 12.1

Partitions are only used with RDD data sync policies. Partitioning has advantages and disadvantages:

Advantages

  • If a retention period is in force, old data can be deleted quickly. When data has aged, it is much quicker to drop a partition than it is to delete individual table records.

  • Reports that query on the event time field might be quicker, because it is only necessary to search the partitions that have the specified event times.

Disadvantages

  • Reports that do not query on event time might be slower where there are multiple of partitions, because every partition must be searched.

  • Each partition causes one or more schema items to be created and managed by the database system. If there is no retention period, the number of partitions just keeps growing.

13.6.2 Creating Data Synchronization Policies

When Sentinel syncs data to an external database, it is not as fast when it writes to the file system. Therefore, you need to ensure that you use filters on the data sync policies to synchronize only the most important data. Consider the following factors based on your business needs for data sync policies:

  • The CPU, RAM, and disk capacity of the Sentinel system

  • Number of EPS scaled per system

  • Number of searches and reports running on a Sentinel system

  • Filters added to the data sync policies

Populating IP addresses in Human Readable Format

By default, Sentinel populates IP address fields in hexadecimal format for efficiency reasons. You can choose to populate the IP address fields in human readable format automatically, by performing the following steps:

  1. Log on to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file and set the datasync.saveIPinDottedNotation property to true.

  3. Restart the Sentinel server.

Enabling SSL Communication for Data Synchronization

You can establish an SSL connection to synchronize data with external databases. Sentinel does not perform certificate validation or authentication.

To enable SSL communication, performing the following steps:

  1. Log in to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file.

  3. If the jsse.enableCBCProtection property is not listed, add this property and set it to false as follows:

    jsse.enableCBCProtection=false

  4. Open the /etc/opt/novell/sentinel/config/databasePlatforms.xml file.

  5. Identify the database platform for which you need to enable SSL connection.

  6. Set the JDBC property as follows:

    For MSSQL: Set the SSL property to require as follows:

    <JDBCProperties>

  <Property name="ssl" value="require"/>

</JDBCProperties>

    For PostgreSQL: Set the SSLOFF property to false as follows

    <JDBCProperties>

      <Property name="ssloff" value="false"/>

</JDBCProperties>

    For Oracle: Set the SSLOFF property to false as follows:

    <JDBCProperties>

      <Property name="ssloff" value="false"/>

</JDBCProperties>

  7. Restart the Sentinel server.

Creating a Data Synchronization Policy

  • Prerequisite for IBM DB2: By default, Sentinel includes the JDBC driver for all the certified databases except for IBM Db2. For Db2, you must manually copy the JDBC driver from the Db2 server to the Sentinel server.

    1. Copy the db2jcc4.jar file from the Db2 server to the following location on the Sentinel sever:

      #/opt/novell/sentinel/lib/

    2. Change the owner of the file to novell, using the following command:

      chown novell:novell <filename>

    3. Configure the server.conf file.

      1. In the /etc/opt/novell/sentinel/config directory, open the server.conf file.

      2. Add the following java classpath property:

        wrapper.java.classpath.7=/opt/novell/sentinel/lib/db2jar file

        For example, wrapper.java.classpath.7=/opt/novell/sentinel/lib/db2jcc4-9.7.0.0.jar

    4. Restart the Sentinel server:

      rcsentinel restart

To create a data synchronization policy:

  1. Log in to Sentinel as a user in the administrator role.

  2. From Sentinel Main, click Storage > Data Synchronization.

  3. Click Create to create a new data synchronization policy.

  4. Use the following information to create the data synchronization policy:

    Filter query: Select a saved filter to use in the data synchronization policy.

    This filter determines which events are stored in the external database. For more information, see Configuring Filters in the Sentinel User Guide.

    Policy name: Specify a name for the data synchronization policy.

    Retention period: Specify how many days to retain the events in the external database.

    Start data synchronization time: Specify when to start synchronizing events to the external database.

    Batch size: Specify how many events are sent to the external database at once.

    Sleep period: Specify the length of time that the data synchronization process sleeps before checking to see if there are more events to process.

    Schedule: Select when the data is synchronized to the external database.

    • All the time: Synchronizes events to the external database constantly.

    • Custom: Allows you to configure specific time periods to perform data synchronization so that it does not occur when the system is busy.

      If you select Custom, specify the following information to set the custom synchronization time:

      • Day of the Week: Select the day of the week, or select Everyday.

      • Start time: Specify the time to start the synchronization process. You can enter 24:00 hour time and it is converted to 12:00 hour time.

      • Duration: Specify the synchronization period in minutes.

    If you do not see the data in the database tables immediately, you need to wait for the next synchronization cycle.

  5. Use the following information to define the connection to the external database:

    Database type: Select the type of external database.

    Host name: Specify the host name of the server where the external database is installed.

    Port: Specify the port used to connect to the external database.

    User name: Specify the name of the user that authenticates to the external database.

    Password: Specify the password of the database user.

    Database: Specify a unique name for the external database. For Oracle database, you can either use the Database name or the Service name.

    Field Mapping: Allows you to map fields in the event to fields in the external database.

  6. Click Save to create the data synchronization policy.

Creating a Table for Event Data Synchronization

  1. Complete Step 2 through Step 4 in Creating Data Synchronization Policies.

  2. Click Field Mapping.

  3. Select Create table.

  4. Use the following information to create the table:

    Table name: Specify a name for the table.

    Table Space (Optional): Specify a tablespace for the table.

    Index Space (Optional): Specify a tablespace for the index.

    Summarize Events: Select this option if you want a summary of events during a specific period.

    Summary Period (Minutes): If you selected Summarize Events, you must specify the amount of time in minutes to summarize events.

  5. Map the fields in the table to the desired fields.

  6. Click Create Table.

  7. Click Save.

Using an Existing Table for Event Data Synchronization

  1. Complete Step 2 through Step 4 in Creating Data Synchronization Policies.

  2. Click Field Mapping.

  3. Select Select existing table.

    Starting from Sentinel 8.x, the size of the Message (msg) event field has been increased from 4000 to 8000 characters to accommodate more information in the field.

    If you are creating a data synchronization policy that synchronizes the Message (msg) event field to an external database, you must increase the size of the Message (msg) field’s mapped column in the external table accordingly.

    NOTE:The above update is applicable only if you are upgrading previous versions of Sentinel to 8.x.

  4. Browse to a select an existing table you want to use, then click OK.

  5. (Optional) Select the Summarize Events option if you want a summary of events during a specific period.

  6. (Optional) If you selected Summarize Events, specify the amount of time in minutes to summarize events.

  7. Change the field mappings for the desired fields.

  8. Click Save.

Setting Retention Period in Default RDD Policies

By default, the retention period value is set to 30 days for all RDD policies that do not have retention period specified. However, you can change the retention period value.

To change the default retention period value:

  1. Log on to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file.

  3. Add the default.global.datasync.retentionperiod property and set it to the required value.

    NOTE:If you set the value of this property to zero, the RDD table entries are never deleted.

  4. Restart the Sentinel server.

13.6.3 Managing Data Synchronization Policies

You can edit, delete, and view the status of each data synchronization policy you create on the Data Synchronization page. If your policy is a custom synchronization policy and you perform a resynchronization, the data synchronizes during the next synchronization cycle.

Event Views use data synchronization policies to display data dynamically and more accurately. Data synchronization policies related to Event Views remain enabled and active while being used by an Event View. If there are no data requests from an Event View for a given data synchronization policy within a specified time period, the data synchronization policy will be automatically deleted.

To specify the time period for which an Event View related data synchronization policy should remain active:

  1. Log in to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file.

  3. Set the desired time in hours for the sentinel.realtime.datasynctimeout property.

  4. Save the modified configuration.properties file.

  5. Restart Sentinel.