Bumping up against Splunk quotas can be frustrating. One reason why you might be hitting your quotas is because of the verbosity of Windows logs. The inspiration for taming Windows logs came from Mark Runal’s Blog. My Window’s server firewalls log dropped packets. These dropped packets are logged in EventID 5156. Through some regex magic in props.conf and transforms.conf the EvenetID can be trimmed down to succinct log entry.

EventID 5156

transforms.com

[replace4688]
REGEX = (?ms)EventCode=(4688).*TaskCategory=(Process Creation).*Keywords=(Audit Success|Audit Failure)\s+Message=A new process has been created.*Security ID:\s+([^\r\n]+).*Account Name:\s+([^\r\n]+).*Account Domain:\s+([^\r\n]+).*Logon ID:\s+(\w+).*Process Information:\s+(?:.*)New Process ID:\W+(\w+)\W+New Process Name:\s+([^\r\n]+).*Token Elevation Type:\W\w+\W+(?:\()([0-9])(?:\))\W+Creator Process ID:\W+(\w+)\W+Process Command Line:(\w+|[^\r\n]+)
DEST_KEY = _raw
FORMAT = Trimmed Event EventCode=$1 $2 $3 Security_ID=$4 Account_Name=$5 Domain=$6 Logon_ID=$7 New_Process_ID=$8 Process_Name=$9 TokenElevationType=$10 Creator_Process_ID=$11 Process_Command=$12

props.conf

[WinEventLog:Security]
TRANSFORMS-winsec_events_manipulation = replace5152_5156
EXTRACT-winsec_5152_5156_custom_fields = ^Trimmed Event EventCode=(?<EventCode>5152|5156) (?<Keywords>Audit Success|Audit Failure) (?<Process_ID>\S+) (?<Application_Name>.+) (?<Direction>Outbound|Inbound) (?<Source_Address>\S+) (?<Source_Port>\S+) (?<Destination_Address>\S+) (?<Destination_Port>\S+) (?<Protocol>\S+) (?<Filter_Run_Time_ID>\S+) (?<Layer_Name>\S+) (?<Layer_Run_Time_ID>\S+) (?<TaskCategory>blocked a packet|permitted a connection)

EventID 4688

This log entry captures each program that is executed, who the program ran as and the process that started this process. On a large number of servers the number of these log entries can be prolific. The log also contains unnecessary information describing the event. This descriptive information is logged again and again with each passing event. The relevant information can be extracted adding the following lines to props.conf and transforms.conf in Splunk’s configuration directory.

transforms.conf

[replace4688]
REGEX = (?ms)EventCode=(4688).*TaskCategory=(Process Creation).*Keywords=(Audit Success|Audit Failure)\s+Message=A new process has been created.*Security ID:\s+([^\r\n]+).*Account Name:\s+([^\r\n]+).*Account Domain:\s+([^\r\n]+).*Logon ID:\s+(\w+).*Process Information:\s+(?:.*)New Process ID:\W+(\w+)\W+New Process Name:\s+([^\r\n]+).*Token Elevation Type:\W\w+\W+(?:\()([0-9])(?:\))\W+Creator Process ID:\W+(\w+)\W+Process Command Line:(\w+|[^\r\n]+)
DEST_KEY = _raw
FORMAT = Trimmed Event EventCode=$1 $2 $3 Security_ID=$4 Account_Name=$5 Domain=$6 Logon_ID=$7 New_Process_ID=$8 Process_Name=$9 TokenElevationType=$10 Creator_Process_ID=$11 Process_Command=$12

props.conf

[WinEventLog:Security]
TRANSFORMS-winsec_events_manipulation2 = replace4688
EXTRACT-winsec_4688_custom_fields = ^Trimmed Event EventCode=(?<EventCode>4688) (?<TaskCategory>Process Creation) (?<Keywords>Audit Success|Audit Failure) Security_ID=(?<Security_ID>.*(?=Account_Name))Account_Name=(?<Account_Name>.*(?=Domain))Domain=(?<Domain>\w+) Logon_ID=(?<Logon_ID>\w+) New_Process_ID=(?<New_Process_ID>\w+) Process_Name=(?<Process_Name>.*(?=TokenElevationType=))TokenElevationType=(?<TokenElevationType>[1-3](?:\W))Creator_Process_ID=(?<Creator_Process_ID>\w+) Process_Command=(?<Process_Command>.*)

Custom Events

Since changes to the transforms.conf and props.conf require a restart of the splunk service it is best to do all the work outside of these config files. I use the following steps to test my regex before editing the config files.

  1. Develop a search query in splunk that captures the events you want.
    Example: earliest=-5@ 4688
  2. In Event Actions select show source and copy the result.
  3. Navigate to RegEx101 and paste the contents of the log entry into the “Test String” and develop the necessary RegEx using PCRE regex and use the global flags ms
  4. Use the regex command in the Splunk web interface to test the regex. The result of this search should be the events you are looking for.
    Example: earliest=-5m@ | regex _raw=”(?ms)EventCode=(4688).*TaskCategory=(Process Creation).*Keywords=(Audit Success|Audit Failure)\s+Message=A new process has been created.*Security ID:\s+([^\r\n]+).*Account Name:\s+([^\r\n]+).*Account Domain:\s+([^\r\n]+).*Logon ID:\s+(\w+).*Process Information:\s+(?:.*)New Process ID:\W+(\w+)\W+New Process Name:\s+([^\r\n]+).*Token Elevation Type:\W\w+\W+(?:\()([0-9])(?:\))\W+Creator Process ID:\W+(\w+)\W+Process Command Line:(\w+|[^\r\n]+)”
  5. Take the working regex from step 4 and create a new stanza in transforms.conf. the FORMAT line defines how the event will be outputed. If your not dealing with values with spaces then you may not have to add identifies to each value and you can just use $1 $2 $3 etc.
    [replace4688]
    REGEX = (?ms)EventCode=(4688).*TaskCategory=(Process Creation).*Keywords=(Audit Success|Audit Failure)\s+Message=A new process has been created.*Security ID:\s+([^\r\n]+).*Account Name:\s+([^\r\n]+).*Account Domain:\s+([^\r\n]+).*Logon ID:\s+(\w+).*Process Information:\s+(?:.*)New Process ID:\W+(\w+)\W+New Process Name:\s+([^\r\n]+).*Token Elevation Type:\W\w+\W+(?:\()([0-9])(?:\))\W+Creator Process ID:\W+(\w+)\W+Process Command Line:(\w+|[^\r\n]+)
    DEST_KEY = _raw
    FORMAT = Trimmed Event EventCode=$1 $2 $3 Security_ID=$4 Account_Name=$5 Domain=$6 Logon_ID=$7 New_Process_ID=$8 Process_Name=$9 TokenElevationType=$10 Creator_Process_ID=$11 Process_Command=$12
  6. Restart the splunk service. Verify your events are being formatted corectly by searching for them.
    Example: earliest=-5@ 4688
  7. Use the rex command in the splunk web interface to develop field extraction regex. To define the extracted field name and what text will be extracted use the format (?<FieldName>regex). When you execute the search you should see the extracted fields populating in the left hand column
    Example: earliest=-5@ 4688 | rex field=_raw “^Trimmed Event EventCode=(?<EventCode>4688) (?<TaskCategory>Process Creation) (?<Keywords>Audit Success|Audit Failure) Security_ID=(?<Security_ID>.*(?=Account_Name))Account_Name=(?<Account_Name>.*(?=Domain))Domain=(?<Domain>\w+) Logon_ID=(?<Logon_ID>\w+) New_Process_ID=(?<New_Process_ID>\w+) Process_Name=(?<Process_Name>.*(?=TokenElevationType=))TokenElevationType=(?<TokenElevationType>[1-3](?:\W))Creator_Process_ID=(?<Creator_Process_ID>\w+) Process_Command=(?<Process_Command>.*)”
  8. Create a new stanza in props.conf. The formatting of the props.conf file can be found in documentation
    Example:
    [WinEventLog:Security]
    TRANSFORMS-winsec_events_manipulation2 = replace4688
    EXTRACT-winsec_4688_custom_fields = ^Trimmed Event EventCode=(?<EventCode>4688) (?<TaskCategory>Process Creation) (?<Keywords>Audit Success|Audit Failure) Security_ID=(?<Security_ID>.*(?=Account_Name))Account_Name=(?<Account_Name>.*(?=Domain))Domain=(?<Domain>\w+) Logon_ID=(?<Logon_ID>\w+) New_Process_ID=(?<New_Process_ID>\w+) Process_Name=(?<Process_Name>.*(?=TokenElevationType=))TokenElevationType=(?<TokenElevationType>[1-3](?:\W))Creator_Process_ID=(?<Creator_Process_ID>\w+) Process_Command=(?<Process_Command>.*)
  9. Restart splunk service and use search to verify that the events are being properly transformed and fields extracted.
    Example: earliest=-5@ 4688