Using the Regex Tool

The FlexConnector Development Kit includes the FlexConnector Regex Tester (Regex Tool) that analyzes .log (event data) files using configuration files (parsers, or .properties files), and can also generate regular expressions to use as properties in configuration files that you create.

Use the Regex Tool only with Regex (regular expression) parsers.

See Developing Custom Parsers for Syslog SmartConnectorsfor general instructions on using the Regex Tool to create a custom parser for a syslog SmartConnector.

To analyze log files using a parser in the Regex Tool:

1. Copy the parser file and log file you wish to analyze into this location:

   ARCSIGHT_HOME\current\user\agent\flexagent

2. Run the Regex Tool by executing:

   ARCSIGHT_HOME\current\bin\arcsight regex

3. Select File > Load FlexAgent Regex File and browse to ARCSIGHT_HOME\current\user\agent\flexagent to select and load the parser file (the .properties file).

4. Select File > Load Log File and browse to ARCSIGHT_HOME\current\user\agent\flexagent to select and load the corresponding .log file. The first line of the file appears in the Message field, and the number of lines in the file displays on the window title bar.

   

   Also, you can load .csv files instead of a .log file for analysis. In this case, choose File > Load CSV Export with Raw Event rather than File > Load Log File. The .csv file you load must contain a header as well as the raw event data. Use this feature to parse and test raw events that did not initially parse correctly, and that you have exported to a .csv file.

5. If you are working with a syslog connector, select Options > Treat as Syslog Subagent. Click the check box to select.

6. Click Generate to produce a regular expression that will parse the line shown in the Message field, as shown below:

   

   Notice that literals, such as the square brackets around the date and time, are preserved in the generated regular expression.

   Use the navigation buttons to view different lines in the log file.

   

7. Analyze the log file line by line using the navigation buttons.

8. Select File > Exit when data analysis is compete.

   When you use the Regex Tool to analyze data, two files are generated:

   • regextester.properties
   • registrycache.properties

   Delete these generated files when you are done with your data analysis. If you do not delete these files, data will persist in the Regex Tool interface.

To create lines for use in configuration files (parsers):

1. Run the Regex Tool by executing:

   ARCSIGHT_HOME\current\bin\arcsight regex

2. Select File > New FlexAgent Regex File.

3. Enter a name for the new .properties file. This file is generated in the location:

   ARCSIGHT_HOME\current\user\agent\flexagent

   The new Regex .properties file is generated containing generic Regex you can use to begin creating a configuration file. This Regex is generated one line at a time, and does not generate an entire parser. The Regex tool lists recommended fields to tokenize and map that are associated with the generated Regex. For example:

   

4. When you are done, select FlexConnector File > Save FlexConnector Regex File.

   The Regex tool can also be used to edit existing configuration files by choosing File > Load FlexConnector Regex File.

   If changes do not work as expected, revert to the previously saved version of the file by clicking ReloadParser.

   Caution: The Regex tool is designed for single-line use only. You can load the entire log file into the tool, but can only process one event at a time.

5. Select File > Exit when data analysis is compete.