Event operations evaluate, compare, and count events. Each operation works on a set of events, receiving a set of events as input and returning a set of events as output. The current event processed by a rule often has a special meaning within the language semantics. The current event is always part of the set of events output by the operation unless the set is empty. If an input set of an operation is empty, then the operation is not evaluated.
The individual correlation operations evaluate sets of events and, if matches are produced, generates sets of corresponding events. You can chain together multiple correlation operations by using the flow operator. The most common use of a chain construct is to begin a chain with a filter to select a subset of events. The subset is then grouped by operations, such as Trigger, or by cross-event comparisons, such as Window. For more information on the flow operator, see Flow Operator.
<operation 1> [flow <operation N> ...]
<operation 1> is a fully-specified operation.
In the above example, the output from one operation is treated as input to the next operation, much like the UNIX pipe operator. The last operation in a chain's output causes the whole rule to fire, generating a correlated event.
A filter consists of evaluation expressions that evaluate the current event from the real-time event stream. It compares the event field values with user-specified values by using a wide set of comparison and match operators. If a match is found, the output set is the matched event, and then the filter resets.
The syntax for the filter operation is:
<filter expression> ::= "filter("<evaluation expression 1> [NOT|AND|OR <evaluation expression 2>] […] [NOT|AND|OR <evaluation expression n>]")"
<evaluation expression 1..N> is an expression constructed of one event field reference (name or ID) and a comparison or match operator. <evaluation expression> ::= "e." <event_name | event_ID> <comparison operator | match operator> <user_specified_value>
Multiple evaluation expressions can be combined by using the standard Boolean operators (AND, OR, and NOT) and grouping parentheses.
For example, the following rule detects whether the current event has a severity of 4 and the event name contains either “FW” or “Comm.”
filter(e.sev = 4 and (e.evt match regex ("FW") or e.evt match regex ("Comm")))
The filter operation supports the following operators:
Filter expressions can be combined by using the Boolean operators AND, OR, and NOT. The filter Boolean operator precedence (from the highest to the lowest precedence) is described in the following table:
Table B-1 Boolean Operators
|
Operator |
Meaning |
Operator Type |
Associativity |
|---|---|---|---|
|
NOT |
logical NOT |
unary |
None |
|
AND |
logical AND |
binary |
left to right |
|
OR |
logical OR |
binary |
left to right |
Standard arithmetic operators can be used to build a condition that compares the value of an event ID and a user-specified value, which is either a numeric value or a string field. The standard arithmetic operators in Sentinel are =, <, >, !=, <=, and >=.
Examples:
filter(e.sev > 3) filter(e.BeginTime < 1179217665) filter(e.iufname != Administrator
)
The match regex operator is used to match an event field value by using standard regular expressions. This operator is used only for string type tags.
Examples:
Match the exact phrase (case sensitive):
filter(e.evt match regex ("LoginUser"))
Match the exact phrase (not case sensitive):
filter(e.evt match regex ("(?i)^TeSt EvenT$"))
The value includes the phrase (not case sensitive):
filter(e.evt match regex ("(?i)EsT EVen"))
Match the exact phrase that contains quotation marks, escape quotation marks with \x22 or \u0022:
filter(e.evt match regex (\x22test\x22))
For more information about the match regex operator, see Regular Expressions.
The match subnet operator is used to match event field IP addresses to a subnet in standard CIDR notation. This operator is used only for IP address fields.
Example:
filter(e.dip match subnet (10.0.0.1/22)) filter(e.sip = 10.0.0.1 or e.sip=10.0.0.2) and (e.dpint=80)
For more information on CIDR notation, see CIDR.
The inlist operator is used to perform a lookup on an existing dynamic list of string values. This operator returns TRUE if the event field value is present in the list. The dynamic list name and the list elements names are case sensitive. When you specify the dynamic list name and list elements names, ensure that you match the case for the lookup to work properly. For more information on Dynamic Lists, see Section 10.0, Configuring Dynamic Lists.
For example, the following expression is used to evaluate whether the source IP address of the current event is present in a Dynamic List named MailServerList. If the source IP address is present in this list, the expression evaluates to TRUE.
filter(e.sip inlist MailServerList)
As another example, this filter expression combines the NOT operator and the inlist operator. This expression evaluates to TRUE if the source IP address is not present in the Dynamic List named MailServerList.
filter(not (e.sip inlist MailServerList))
The following expression is used to evaluate whether the event name of the current event equals “File Access” and the InitiatorUserName is not present in a Dynamic List named AuthorizedUsers. If both conditions are true for the current event, the expression evaluates to TRUE.
filter(e.evt="File Access" and not(e.sun inlist AuthorizedUsers))
The isnull operator returns TRUE if the event field value is equal to null. For example:
Filter(isnull(e.sip))
The output of this expression is either the empty set (if the evaluation expression evaluates to FALSE) or a set containing the current event (if the evaluation expression evaluates to TRUE).
If the filter is the last or only operation of a correlation rule, the output set of the filter is used to construct a correlated event.The input event that matched the filter is associated with the correlated event.
If the filter is not the last operation of a correlation rule (that is, if the filter is followed by a flow operator), the output set of a filter is used as the input set to other operations through the flow operator.
The filter operator can be used to compare event field values with other event field values within the same event. For example:
filter(e.sip=e.dip)
The trigger operation counts a number of events for a specified duration. The trigger command defines a threshold count of events within a time condition and applies an optional discriminator that splits events into unique buckets on which to apply the threshold count and time window conditions. Trigger itself does not apply any filtering. Every event that enters the trigger function is put in a bucket and is counted. If you want to use trigger on certain type of events, you must prefilter the events and then flow that output to trigger.
<trigger expression> ::= "trigger("<threshold count>","<evaluation period>","discriminator ("<list of event fields>")")"
<threshold count> is an integer value specifying the number of events in a single bucket that are necessary for the rule to fire.
<evaluation period> indicates the duration for which old events are kept in the bucket and counted towards the threshold. The duration can be seconds (s), minutes (m), hours (h), and days (d.).
discriminator (<list of event fields>) specifies the set of fields to use to segregate each event into its unique bucket. All fields are logically combined to define the bucket.
For example, the following rule detects if 5 events with the same source IP address happen within 10 seconds.
trigger(5,10,discriminator(e.sip))
Note that the discriminator is used to split the input stream of events into distinct buckets where all events in a bucket have identical data for the specified fields (SourceIP in the example above.) Multiple fields can be used to create buckets where all events in a single bucket have, for example, the same SourceIP and TargetUserName. However, you cannot count across multiple buckets. For example, you cannot use the Trigger operation to detect conditions where a source system contacts N distinct target systems.
If the specified count is reached within the specified duration, a set of events containing all of the events maintained by the trigger is output.
When receiving a new input set of events, a trigger first discards the outdated events (events that have been maintained for more than the duration) and then inserts the current event. If the number of resulting events is greater than or equal to the specified count, the trigger outputs a set containing all of the events.
If a trigger is the last operation (or the only operation) of a correlation rule, then the output set of the trigger is used to construct a correlated event. The raw events associated with the correlated events are the trigger operation output set of events with the current event listed first.
The Window operation compares the current event to a set of past events that are stored in a window that is defined as part of the Window operation. Window allows you to temporarily store events of interest and then use the data within those stored events to filter the incoming raw events.
For example, suppose you want to detect if someone attempts to log in to an account several times and fails, but then guesses that password and succeeds in logging in. You would collect the failed login events and store them in the window, then compare new successful login events against those stored events and match the username. If a match is found, then the user first failed to log in and then did so successfully. Any event field can be matched against any other event field for more sophisticated correlations.
The incoming raw stream of events is split before it arrives at the window operation:
Storage filter (w.event tag): One stream of raw events is matched against the storage filter and, if any events match, they are placed in the window. Note that the current event is not part of this stream (that is, this stream is delayed by one event.) Events in the window are referred to with the “w.” prefix, which can be interpreted as "the set of events in the window."
Evaluation filter (e.event tag):
The other stream of raw events is filtered and then passed, one at a time, into the Window operation. Each event (the current event) is matched against the set of events stored in the window by using the defined comparison expression. The current event passing into the Window operation is referred to with the e.
prefix, which can be interpreted as "the current event."
You can define two filters for the Window operation to constrain resource usage. You define the first filter within the Window operation itself to set the storage filter. You define the second filter as a prefix to the Window operation using a standard in-line filter that sets the evaluation filter. These filters are optional, but recommended.
To ensure that the current event does not match itself in cases where the comparison operator is matching on the same field and a single event would match both the evaluation and storage filters, the set of events in the window does not include the current event.
NOTE:It is critical to design your storage and evaluation filters carefully to minimize resource usage by the Window operation. The window stores the event UUID and any fields necessary for comparison for all events that match the storage filter. Therefore, constraining the set of matched events is important to reduce memory use.
The syntax is as follows:
<window expression> ::= "window(" <comparison expression> "," <storage filter> "," <storage_time_period> ")"
<comparison expression> matches a current event field against a field in the set of past events stored in the window. The standard comparison operators are supported.
<comparison expression> ::= "w." <event_ID | event_name> <comparison operator> "e." <event_ID | event_name>
Multiple comparison expressions can be combined by using the standard Boolean AND, OR, and NOT operators. Note that it is also possible to compare stored event fields against literals, in which case the Window operation ignores the contents of the current event and always copies it to the output as long as an event that matches the literal is in the window.
<storage filter> is an embedded filter expression that defines the storage filter. All events that match this filter are stored in the window for later comparison until they expire.
<storage filter> ::= <filter_rule>
<filter_ rule> is a simple expression using the filter operator. For more information, see Filter Operation.
<storage_time_period> is the total time for which a single event is stored in the window. The storage time period can be seconds (s), minutes (m), hours (h), and days (d.)
<storage_time_period> ::= <1 or more digits> "s" | "m" | "h" | "d"
The following rule detects whether the current event has a source IP address that matches the source IP address of an event that happened within the past 60 seconds, with the past events limited to those whose source IP address is within the specified subnet. This Window would typically be preceded by an evaluation filter to restrict the set of events evaluated by the Window.
window(w.sip = e.sip, filter(e.sip match subnet (10.0.0.10/22),60)
As another example, the following rule is a domino type of rule. An attacker exploits a vulnerable system and uses it as an attack platform. This Window would typically be preceded by an evaluation filter to restrict the set of events evaluated by the Window.
filter(e.XDASTaxonomyName = "XDAS_AE_IDS_PROBE" OR e.XDASTaxonomyName = "XDAS_AE_IDS_PENETRATE") flow window((e.sip = w.dip AND e.dp = w.dp AND e.evt = w.evt), filter(e.XDASTaxonomyName = "XDAS_AE_IDS_PROBE" OR e.XDASTaxonomyName = "XDAS_AE_IDS_PENETRATE"), 1h)
The following rule identifies a potential security breach after a denial of service attack. The rule fires if the destination of a denial of service attack has a service stopped within 60 seconds of the attack.
filter(e.rv51="Service" and e.rv52="Stop") flow window (e.sip = w.dip, filter(e.XDASTaxonomyName = "XDAS_AE_DOS"), 60)
If the Window operation matches a current event against the window based on the comparison expression, the output set is the incoming event plus all matching past events
If no events in the window match the current event, the output set is empty.
If a window is the last or only operation of a correlation rule, the output set of the window is used to construct a correlated event. The raw events associated with the correlated events are the window operation output set of events with the current event first.
All window simple evaluation expressions must include an event ID in the form w.[event_ID].
Every event coming in to the Correlation Engine that passes the storage filter is put into the window of past events except for the most recent, or current, event.
If no storage filter expression is defined, all events coming into the Correlation Engine are stored by the window. With extremely high event rates or long duration storage time period, this might require a large amount of memory.
To minimize memory usage, only the relevant parts of the past events, not all event ID values, are maintained in memory.
The Gate operation is used to combine multiple subrules together to detect conditions where several distinct activities happen within a given time period. The order in which each activity occurred is not considered.
The gate operation is made up of one or more nested subrules and can be configured to fire if some, any, or all of the subrules fire within a specified time. The subrules can be a simple rule or another Composite rule. For more information on Composite rules, see Composite Rule.
<gate expression> ::= "gate("<subrule 1>","<subrule 2>","<subrule n>","<mode>","<evaluation period>","discriminator ("<list of event fields>")")"
<subrule 1...N> rules are the rule definitions for 1 to n subrules. Each subrule is an independent, valid, correlation rule.
<mode> can be one of the subrules that must be triggered for the Gate operation to trigger.
<evaluation period> indicates the time period over which the specified number of subrules must fire in order for the whole Gate rule to fire.
discriminator (<list of fields>) acts similar to the discriminator in the Trigger operation. The output from each subrule is placed into unique buckets based on the data in the fields specified in this discriminator. The <mode> count and <evaluation period> is then applied to each bucket separately.
For example, the following rule is a typical perimeter security IDS inside/outside rule:
filter(e.sev > 3) flow gate(filter(e.sn = "in"), filter(e.sn = "out"), all, 60s, discriminator(e.dip, e.evt))
Sequence rules are similar to gate rules, except that all subrules must fire in sequence for the overall Sequence operation to fire.
The subrules can be a simple rule or another Composite rule.
Syntax:
<sequence expression> ::= "sequence("<subrule 1>","<subrule 2>","<subrule n>","<evaluation period>","discriminator("<list of event fields>")")"
<subrule 1..N> are the rule definitions for 1 to n subrules. Each subrule is an independent, valid, correlation rule.
<evaluation period> is a time period expressed in seconds (s), minutes (m), or hours (h).
discriminator (<list of fields>) acts similar to the discriminator in the Trigger operation; the output from each subrule is placed into unique buckets based on the data in the fields specified in this discriminator. The <evaluation period> is then applied to each bucket separately.
For example, this rule detects three failed logins by a particular user in 10 minutes followed by a successful login by the same user.
sequence (filter(e.evt="failed logins") flow trigger(3, 600, discriminator(e.sun, e.dip)), filter(e.evt="goodlogin"), 600, discriminator(e.sun, e.dip))
A Sequence Timeout rule fires only when an event that matches the first subrule is not followed by an event that matches the second subrule in a specified time frame. A Sequence Timeout rule has two subrules.
Each subrule can be a simple rule or complex rule such as Sequence rule, Composite rule or a Sequence Timeout rule itself.
<sequence_timeout expression> ::= "sequence_timeout("<subrule 1>","<subrule 2>","<evaluation period>","discriminator("<list of event fields>")")"
<subrule 1> and <subrule 2> are the two subrule definitions. Each subrule is an independent and valid correlation rule.
<evaluation period> is a time period expressed in seconds (s), minutes (m), or hours (h).
discriminator (<list of fields>) acts similar to the discriminator in the Trigger operation. The output from each subrule is placed into unique buckets based on the data in the fields specified in this discriminator. The <evaluation period> is then applied to each bucket separately.
For example, the rule detects a scenario where the server is stopped but did not start again within an interval of 5 minutes.
sequence_timeout(filter(e.evt="service stopped"), filter(e.evt="service started"), 300)
Similarly, the following rule detects three failed logins with the same user ID from three different source IPs in 60 seconds, but not followed by locking the user account in the specified time frame.
sequence_timeout(filter(e.evt="LoginFailed") flow trigger(3, 60, discriminator(e.sun, distinct e.sip)), filter(e.evt="AccountLocked"), 70)
NOTE:It is critical to design the first subrule carefully to minimize resource usage by the Sequence Timeout operation. To fire a Sequence Timeout rule, events matching the first subrule are stored in memory from the time they arrive till the rule duration expires. Extremely high event rates or longer rule duration may require a large amount of memory. So constraining the set of matched events for the first subrule is important to reduce memory use.
The Distinct operation considers only the unique values in events. You can use this operation in scenarios where you need to differentiate scans versus flood type attacks. In flood type attacks the overall quantity is important, with scans the overall unique information disseminated is important.
The following are some examples where you can use the Distinct operation:
Example B-1 You want a rule to trigger if five events have unique source IP addresses but the same destination IP address within a 10 second period. You can create a free-form rule as follows:
trigger(5,10, discriminator(e.dip, distinct e.sip))
Example B-2 You want a rule to trigger if three severity 5 events occur from distinct combination of initiator IP addresses and initiator users within a 60 second period. You can create a free-form rule as follows:
filter(e.sev = 5) flow trigger(3, 60, discriminator(distinct e.sip, distinct e.sun))
Example B-3 You want a rule to trigger if three severity 5 events occur from the same initiator IP address but distinct initiator users within a 60 second period. You can create a free-form rule as follows:
filter(e.sev = 5) flow trigger(3, 60, discriminator(e.sip, distinct e.sun))
Example B-4 You want a rule to trigger if three severity 5 events occur from the same initiator user but distinct initiator IP addresses within a 60 second period. You can create a free-from rule as follows:
filter(e.sev = 5) flow trigger(3, 60, discriminator(distinct e.sip, e.sun))