Chronicle SIEM: Outcomes & Functions
GSP1100
Overview
Chronicle Security Operations empowers cloud-first, modern SecOps teams to protect their organizations confidently, with cloud-native architecture, petabyte scale, sub-second queries, and automated responses.
In this lab you will learn how to perform basic administrative tasks within the Chronicle platform.
Objectives
In this lab, you learn how to perform the following tasks:
- Outcomes
- Functions
Adding Outcomes to Rule
Now that you have built single and multi event rules and added entity data to your rules, you will take a look at the outcome section of the rules.
Outcomes
- Outcomes provide an opportunity to add context to the detections that the analyst (or other systems) can consume. While the rule itself has certain metadata to describe it, you may want additional information in your detection to provide to the analyst about the victims, users, or systems.
- A special field in the outcome section of the YARA-L rule is $risk_score (integer). This value, when added to the outcome section, will also be displayed within Enterprise Insights. The Enterprise Insights view displays the domains and assets most in need of investigation within your enterprise. You can also utilize functions to aggregate or compute values in outcomes but more on that in the next section.
- The outcome section that can accommodate both single and multi-event rules though the syntax is slightly different.
- The outcome section cannot reference field variables that are not already defined in the events section. For example, in your password spray attack example, you were just looking at the failed authentication events. You can’t ask for outcome fields associated with the successful login unless you had already specified event variables for those fields in the event section too.
Supported Operations
-
Supported aggregate functions are:
- max
- min
- sum
- array
- array_distinct
- count
- count_distinct
-
"If" statements and mathematical operations can be used to calculate risk score and work with other outcome fields within the context of the aggregates
-
Example
- The outcome section supports a set of aggregations, listed above. You can also use "if" statements to modify the value of fields like risk_score. In this example, you are saying that if the hostname of the server is ad-server, you will add a risk score of 90. If the hostname is not our ad-server, you will add 50.
- From there you will further increment based on the severity of the event. If you use risk scores for your rules, it is wise to define your range upfront prior to defining risk values so that your risk value is consistent across all rules. In this case, you should propose a risk range of 0-100
Adding Outcome to a Rule - Brute Force Example
Let’s look at the brute force login example and overlay an outcome section to it. Notice that you did not make changes to any of the other sections. Remember that you cannot just add an arbitrary event field that isn’t defined in the events section, but you can add any fields associated with the $fail and $success events in this example. You could define them in the events section with placeholder variables but because you are just outputting it straight into the outcome section, that isn’t needed.
- The result of this will be individual outcome fields with this additional metadata to better describe the detection. You might be thinking that if I have a single value in my output, I should just be able to say outcome field name = placeholder variable or constant. However, for multi-event rules, all outcomes require some sort of aggregate function, so the use or array or array_distinct would be considered mandatory.
Exercise #5: Outcomes
This exercise will build on top of the password spray rule that you created in the previous labs and will add the outcome section to the existing rule. Find your password spray rule and add an outcome section and use the following detail in the bullet points below. Once you are done, go ahead and test your rule, view the output, work with the detection and add some additional columns to your view as you see fit.
-
Find the password spray rule we created earlier - win_password_spray_
-
Enhance the existing rule to add an Outcome section with the following details:
- Risk score of 50
- Distinct count of the number of unique user ids attempted
- Count of the total number of user ids attempted
- An array of the unique user ids that were attempted
- A field called tlp (traffic light protocol) with a value of amber
-
Once you are happy with your syntax, test your rule for the past three days
-
Remember, supported aggregates are:
- max
- min
- sum
- array
- array_distinct
- count
- count_distinct
-
"If" statements and mathematical operations can be used to calculate risk score and work with other outcome fields
Exercise #5 Answer
Review Exercise #5: Outcomes
Let’s start in the rules editor and find your rule. Below you can see the rule that you created earlier.
Outcome Section
When looking at the outcome section that was added, notice the different aggregate functions that were added to each field. The field names in the outcome section are arbitrary, yours could be different. The only one that would be considered reserved would be risk_score which would also show up in the Enterprise Insights screen when creating an alert.
-
A best practice when using outcomes would be to define a common taxonomy that can be used across multiple rules. The reason for this is that analysts can then recognize these values for what they are with consistency across the rules. Taking it a step further, if an integration is in place for tools such as Chronicle SOAR, having a common mapping of fields in outcome can help promote a seamless integration between key values from rules to case management.
-
Above you can see the additional columns available to you from the outcome section. Notice that the detection has the outcome fields but the underlying events do not. This is because these values are written to the outcome section of the schema which is part of the detection that was created.
Working with Functions & Lists
You have reached the last section of our training where you will add some additional capabilities to work with the rules engine. This section will focus on the use of functions and lists. You may already have worked with functions when working with regular expressions in the past. That is one way functions can be used but here you are going to look at a number of additional functions and go through a few examples and exercises.
Introduction to Functions
- Functions are used to extract values from events and if they match the criteria, they can be incorporated into the logic of the rule.
- Functions are used to work with fields and their associated values when matching on an entire field isn’t always possible and fold them into the criteria for the rules.
- Functions live within the event section of the YARA-L rule but can also be used as a boolean in the outcome section.
- Below is a high-level list of the types of functions
- String
- Regular Expression
- Date
- Math
- Net
String function
There are three string functions: concatenation, upper/lower case conversion, and base64 decoding.
Concatenation
-
Concatenation provides the ability to put two strings, or integers together and can be a combination of fields and constants. In the example below, perhaps one log stream provides the hostname without the domain name whereas another hostname contains the fully qualified domain name. In this case, you could concatenate a constant like .google.com to the hostname from the first event and compare it to the second event which has the full FQDN.
-
One event has the FQDN in the event and the other only has the hostname but a comparison is desired
Example:
strings.concat($selection1.principal.hostname,".google.com") = $selection2.principal.hostname
Lower and Upper-Case Conversion
-
Good when doing comparisons between two events and text formatting is not the same case
-
strings.to_lower() & strings.to_upper()
Example:
strings.to_upper($selection1.principal.hostname) = strings.to_upper($selection2.principal.hostname) -
Lower and upper case can be used to compare disparate events together. As previously discussed, "no case" is also available to be used as well but this serves as another mechanism that can provide greater precision in your criteria as well as greater efficiency.
Base64 Decoding
- The base64 decoding function can decode base64 which can then be compared to a string value
- Good for identifying base64 strings of a specific type for additional analysis
- Will generally be used in conjunction with Regex functions to extract the Base64 from the rest of the command line
Regular Expression Functions
The regular expression functions are likely the most important for users.
-
re.regex - Match
-
Used in rules all the time to find strings within a field in a UDM event
-
Simple Example
re.regex($selection.target.process.file.full_path, `powershell\.exe`) re.regex($selection.target.process.file.full_path, /powershell\.exe/) -
More Elaborate Example below
$selection.target.process.command_line = /vssadmin.*create.*shadow.*\/for=c:/ nocase re.regex($selection.target.process.command_line, `vssadmin.*create.*shadow.*/for=c:`) nocase -
The first function re.regex is used to find strings within a field in a UDM event. This can be done in a couple of ways. In the first example, you can see the function invoked followed by the field itself, and then after a comma the pattern we are matching against. You can see this being used with forward quotes (`) or with forward slashes (/). We saw the use of a forward slash before by just using an equal sign.
-
If you want to take a more elaborate example, you can see equivalent functions. Because there are spaces between strings we use .* to handle spaces or other arbitrary switches mixed into the pattern you are looking for. Also, notice the use of nocase to handle the chance that an adversary might use different capitalization in the command. The first example with the equal sign is what we have used earlier and it can also be used in UDM search.
-
Double quoted strings can also be used with regular expression functions but is generally not recommended. Double quotes will generally be used for normal strings and escape characters are required making the regex a bit more cumbersome and more difficult to read.
-
The other two regex functions are capture and replace. Capture is really good for extracting portions of fields to use in event criteria. In the below example, you are looking to capture the string after -enc. If you wanted to isolate encoded Powershell, this is what you could use. The capture is show inside the parenthesis.
-
re.capture - Capture
-
Good to extract a substring for additional operations
-
Example
re.capture($selection.target.process.command_line, "-enc (.*)") -
re.replace - Replace
-
Used to swap out values that can then be compared
-
Example
re.replace($selection.principal.hostname, "dev", "prod") -
The replace function is used to swap out one value for another, including a null string. In this example, you are swapping out the string dev for prod. Another good example of the use of replace would be to take a process such as powershell.exe but perhaps compare it against a list of processes that didn’t have .exe appended to them and use the replace to shorten the process name.
Combining String(Base64) & RegEx Functions
- You want to find all Base64 encoded PowerShell events but only alert on the ones that contain WebClient or WebRequest within the encoded PowerShell
- Below is one more example of how you can leverage functions in your rules. In this example, you are looking for all Process Launches that utilize encoded Powershell. You are using the regex function to find both the string powershell.exe as well as flags that denote encoding. These are not all of the flags but it shows how Chronicle can take more than one option and work through different options in a single line of criteria.
-
The next line is color-coded because there is a lot going on. You are capturing the encoded command within the target.process.command_line after the various encoded commands and you are using the (?i) to make sure you don’t miss capturing the encoded strings by saying that your specification is case insensitive. Once you have your encoded string, you are decoding it using the strings.base64_decode function. Commonly you may end up with null bytes separating characters, which could lead to gaps in detection. To combat that, you can use the re.replace function to look for \0 (null bytes) and replace them with null basically nothing. This has all been written to the placeholder variable $dps.
-
You can then take that placeholder variable and perform a regex on it to determine if you see the strings WebClient or WebRequest or whatever strings you care about.
-
Finally in the outcome section you are outputting the placeholder value to a field so that an analyst who is triaging this alert can see the decoded PowerShell without having to decode it themselves.
Detection Results
-
Below is the output against your data set. Notice how you have the encoded PowerShell in the far right column and the decoded detections in the red box.
Date Functions
- get_minute, get_hour, get_day_of_week, get_week, current_seconds
- Could be used when looking for outliers of standard operations
Example
-
If a PowerShell administrative process kicks off on a specific schedule with a consistent day of the week or hour of the day, you can detect all other activity outside of that window
-
events:
- The date functions provide a way to isolate on specific parts of time fields that can then be used in rules. For example, if an encoded PowerShell is being monitored but a specific encoded PowerShell process is scheduled to be run at a specific time, an exception can be made by using the time function in the event logic.
Math & Net Functions
-
The math and net functions are the final two groups of functions. Currently, absolute value (math.abs) is the only math function Good for multi-event rules that look for the duration of an event and utilize with greater than or less than operators
-
net.ip_in_range_cidr works with both IPv4 and IPv6 Is useful to bind rules to specific parts of the network
events: $selection.metadata.product_event_type = "3" and net.ip_in_range_cidr($selection.principal.ip, "10.1.0.0/24")
List Manager
-
The list manager capability provides analysts the ability to build lists that can then be compared to events in rules. The syntax is pretty straightforward; the field or variable is followed by the operator "in", in lowercase, followed by a percentage sign and the list name. Here are two examples, the first is very straightforward and the second is a little more elaborate where you'll have the regex function to capture the process that can then be compared against a list called processes_of_interest.
-
Create lists that can be used in conjunction with rules
- Duplicate or add to existing lists
-
Rule Synta
-
field/variables in %list_name
-
Examples
$selection1.principal.hostname in %key_servers re.capture($selection1.target.process.file.full_path, /.*\\(.*)/) in %processes_of_interest
Exercise #6: Functions and Lists
Now that you have reviewed the functions, you'll build a new rule. The ntds.dit file contains all of the contents of an active directory domain. Gaining access to this essentially would be considered the complete compromise of an AD domain. To mitigate this threat, you are going to build a rule that detects when specific processes that might allow access to ntds.dit are invoked.
- You will want to find all process creation events in Windows and Sysmon and the event codes of interest are 4688 and 1. You have already created a list called ntds_suspicious_processes that contains processes of interest. You will want to compare those processes to the field target.process.file.full_path.
- You should have an additional list called key_servers that contains a list of servers to apply this rule. You will use the principal.hostname for your comparison against the list. However, the hostname field contains a fully qualified domain name but the list just has the hostname, so the domain name needs to be stripped off to make matching possible.
- Finally, you don’t want an alert on every single event seen. For example, it is highly likely that each process will have both a Sysmon/1 and Windows/4688 event occurring at the same time, so you should group by the event_type to fire every 5 minutes.
- As you build the rule, don’t forget to test your rule and adjust as needed.
- You'll start by navigating to the list manager section of the Chronicle instance. Click in the upper right hand corner of the screen where the square with the dots are. When you click on this, you will see a menu of options. Click on List Manager.
Background: The ntds.dit file contains all contents of the active directory domain. Being able to gain access to this effectively is the end of our domain without needing to rebuild.
Create a new rule to detect when specific processes are executed that may indicate attempts to gain access to ntds.dit
- Find all events process creation events (metadata.event_type of PROCESS_LAUNCH)
- The list ntds_suspicious_processes contains the processes of interest Compare them to the target.process.file.full_path field You will need to use the regex capture command to isolate the string to compare
- The list key_servers contains a list of servers that this rule applies to
- However the principal.hostname values are all in the form of servername.stackedpads.local
- Use the regex replace to trim the domain from the hostname
- You don't need an alert for every event; set your rule to fire against a 5 minute window, grouped by metadata.event_type
Once you are happy with your syntax, test your rule over the specified time range
Rule Syntax with List Manager
Examples
- $selection1.principal.hostname in %key_servers
- re.capture($selection1.target.process.file.full_path, /.*\(.*)/) in %processes_of_interest
Regular Expression Functions for Capture and Replace
- re.capture - Capture
- Good to extract a substring for additional operations
- Example
- re.capture($selection.target.process.command_line, "-enc (.*)")
- re.replace - Replace
- Used to swap out values that can then be compared
- Example
- re.replace($selection.principal.hostname, "dev", "prod"
- The below two lists with their associated values must be loaded into List Manager for this rule to work
| Title | key_servers |
| Description | Key server names |
| Details | activedir sql exchange sap |
| Title | ntds_suspicious_processes |
| Description | process names often associated with accessing ntds.dit |
| Details | ntdsutil vssadmin diskshadow esentutl |
-
You'll start by navigating to the list manager section of the Chronicle instance. Click in the upper right hand corner of the screen where the square with the dots are. When you click on this, you will see a menu of options. Click on List Manager.
-
Above are the two lists that you will be using for your comparisons. It is a good idea before crafting your event criteria to see what you are going to be comparing against. Above you can see the suspicious processes are the names of the processes but without the .exe. You can also see the key_servers have the hostname but they are not fully qualified domain names so you need to ensure that the hostname in the event stream does not have this either.
Event Criteria
Let’s take a look at the event section and its criteria.
- You have parenthesis with the "or" statement around the two product_event types.
- To work with the suspicious process list, you need to look at the field target.process.file.full_path but you want to capture just the command itself and do not want the path before it or the .exe. Putting a parenthesis with .* will capture up to but not including the .exe listed. This regex capture can then be compared using the "in" operator against all values in the list.
- When you look in the principal.hostname field, you can see that the hostnames have .stackedpads.local appended to them. Because your key_server list does not include the full domain, you can use the regex replace statement to match on the string .stackedpads.local and then replace it with null, represented by “”. You can then match that shortened hostname to your list using the in operator.
- Finally you'll have a placeholder variable against the event variable metadata.event_type. You will use this variable in your match to limit the number of detections that you will get by grouping like events together.
Match and Condition
With the event criteria in place, you can move to our match and condition sections
- The match will be the $event_type variable you created in events so that events can be grouped together. Now you may be wondering why we didn’t group by product_event_type and used event_type instead. You don’t want the product_event_type because this would make each Windows/4688 and each Sysmon/1 event unique. However, the metadata.event_type for both of these events is process_launch which makes this a nice way to group both events together.
- The condition doesn’t take into account entities or multiple event thresholds so the condition in this case, is just the event variable $selection. Your condition might be different depending on the variable you assigned to your events and that is fine.
- Below is the complete rule.
Below are the results of your test. You can see four detections and if you expand any of them, you can see multiple events underneath it. Because the Win/4688 and the Sysmon/1 events fired within that same match window, you benefit by seeing both events but are not encumbered with duplicate detections.
Tips on Rule Building from Engineering
Examine the behavior and content of the Rule
- Does the rule make sense?
- Accuracy?
- Multiple approaches to achieving the same detection result
- How frequently does the rule create Detections?
- Are 10k Detections a day useful?
- How frequently does the rule create Errors?
- All the time? Not helpful for customers.
- Some of the time? Issues with data skew means the rule may fail when it’s most needed.
Attempting Optimization - Overview
Conceptually, think of the backend as an engine that performs queries in a series of steps:
- Get data - Get data in a given time range
- Filter data - Filter out some of that data
- Correlate data - for multiple event rules; find related events in the same window
- Optimizations target one of these steps
- Get less data
- Filter that data in a cheaper way
- Add filters so there’s less to correlate
- Change the way you correlate
- The above are all examples of alternate aproaches to rule creation
Attempting Optimization - Get less data
-
Would the rule be just as effective with a smaller window size?
-
Why? Every time you execute the rule, with a larger window size we have to look back further to see if the condition applies to the aggregate of the previous 24 hours, instead of the previous 1 hour.
Attempting Optimization - Filter data cheaper
-
Prefer filtering on non-repeated fields instead of repeated fields
-
Why? We need to “unwrap” repeated fields, whereas we don’t need to unwrap non-repeated fields
-
Note: this also applies to the upcoming UDM Additionals feature, since that queries a repeated field
-
Consider populating fields with a parser instead of inefficient filters
-
Why? Perhaps the value you want to search on isn’t populated in the UDM, so you’re parsing the value out from a more complex field that contains that value as part of a longer string. Consider if it is possible to populate that value elsewhere in the UDM, to allow for more efficient, direct filtering.
-
Prefer direct comparison (=, <, >) to RegEx comparison.
-
People sometimes instinctively use RegEx when a literal comparison is possible
-
Why? The backend does not have to do a RegEx comparison for every row; literal comparison is easier
-
Prefer ip_range_in_cidr to RegEx functions
-
People sometimes use RegEx when using net.ip_range_in_cidr function is possible
-
Why? Backend does not have to do a RegEx comparison for every row; ip_range_in_cidr is more efficient
-
Combine RegExs on the same field into the same RegEx call.
-
Why? RegEx’s are expensive. If you can compile and execute many RegEx calls it can add up
-
Case in point: converting 50 RegEx calls into a single RegEx resulted in a 10x resource usage decrease
-
Some filters can discard large amounts of data earlier, making subsequent filtering cheaper. Add filters on the fields we order on for the given customer, when possible
-
Why? Data on the backend isn’t ordered randomly; it’s ordered in a certain well-defined order. When we filter on fields first in this order, we filter more efficiently (IFNULL optimization)
Attempting Optimization - Correlate less data
- What is correlation? A multiple event rule where we have to see if two events have matching fields
- Imagine we have two massive sets of events where we need to match everything on one side with everything on the other side
- Anything we can do to make either side smaller is helpful!
- Can you filter the size of one of the sides of the join more, so that less correlation needs to happen?
-
The more specific the filter, the better. Try to add more filters to $e1 without changing the behavior of the rule.
-
The idea above is that if you are looking for 3 kinds of events and you don’t want events that don’t match event2, then there may not be a reason to add more criteria around not matching event3. That may not always be the case but again this is about optimization, not setting hard and fast rules.
-
Can we remove event variables?
-
Why? More event variables mean more queries to the backend, gathering duplicate data. You should always try to make the rule work with fewer queries.
Hop Window
-
Below is a good visualization of hop windows.
match: $hostname over 5m -
Generally, 10 windows per event, since hop period = MIN(Window Size/10, 1 Hour)
Sliding Window
-
What if there are a lot of $a?
-
The sliding windows can be resource intensive if the starting event of the window occurs very frequently due to the number of windows that must be kept open until the timer runs out.
-
Sliding Windows: Filter events on the “before” / “after” event variable to reduce the number of windows
-
Guideline: 5 events per window size may produce up to 25 windows. Orders of magnitude over (say, hundreds) of windows are probably too much
-
To mitigate this, adding additional criteria to make that window start much more specific is ideal.
-
Sliding Windows: switch to Hop Windows.
-
If you can’t filter a large number of $e1, you can reduce the number of windows by switching to hop windows. You can still get ordering.
-
Alternatively, using a time comparison in the events between event1 and event2, as you did in the brute force login example allows you to stick with the hop window but not have to compromise the idea of a sliding window.
-
Non-existence:
- Non-existence rules were not covered in the examples in this workshop, but the idea is that you can look for activities where you would expect to see multiple events occur but perhaps one of them does not and that is significant unto itself. The caveat to this is that in case you have late arriving data, you don’t want to trigger a non-existence rule because you have some latency in your collection framework due to any number of reasons. Running non-existence rules at a frequency less than 1 hour is considered a best practice because Chronicle is trying to make sure that we have the data collected to provide the best information around these kinds of rules.
- Consider running with ~1hr Run Frequency since Non-existence rules are automatically delayed by 1 hour anyway to give data time to come in
Summary
What Else Could You Do aka Things To Try On Your Own
- Here are some additional capabilities that you should try on your own
- Correlating threat intelligence IOCs with UDM events
- Using prevalence from domains, IP addresses and files to drive detections
- Setting a threshold for a rule based on a variable in the outcome section, like risk_score
- Non-existence - something happens, but the absence of a second thing is noteworthy
Parting Thoughts
- Rules use UDM fields - UDM searches can serve as a good start for your rules
- Start simple and test rules on existing data and add additional criteria as you go
- Meta, Events and Condition sections are required for all rules
- Multi-event rules require the Match section as well
- nocase is useful for handling command line variances (caps will evade without it)
- Beware of extra spaces in commands when looking for command line
Handy links
- YARA-L Whitepaper
- UDM Field List
- YARA-L Language Syntax
- Additional Example Rules
- New To Chronicle Blog Series
- UDM
- Single Event Rules
- Multi-Event Rules
- Outcomes
Congratulations!
You now have now a solid understanding of several basic administrative actions when using Chronicle.
Finish your Quest
This self-paced lab is part of the Chronicle SIEM: Rules Quest. A Quest is a series of related labs that form a learning path. Completing this Quest earns you a badge to recognize your achievement. You can make your badge (or badges) public and link to them in your online resume or social media account. Enroll in this quest and get immediate completion credit if you have successfully completed this lab. Search for other available quests in the Cloud Skills Boost catalog.
Manual Last Updated October 27, 2023
Lab Last Tested October 27, 2023
Copyright 2023 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.
