Parser Editor

This topic has instructions for using the Sumo Logic parser editor. You can use the editor to customize system parsers, and to create your own custom parsers. We provide parser templates that you can use as a starting point for creating custom parsers.

For a complete list of standard parsers, see Parsers in the Cloud SIEM Content Catalog.

See additional articles for more information about the Sumo Logic Cloud SIEM parsers:

• Parsing Language Reference Guide
• Parsing Patterns
• Parser Troubleshooting

note

The instructions that follow assume that you have already written your parser code.

Watch the following micro lesson to learn how to apply parsers to Cloud SIEM data sets.

Check parser code for mapping hints

Your parser code must contain statements that tell Cloud SIEM what log mapping to use when creating records from the field dictionary the parser creates for log messages. 

Make sure your parser code includes MAPPER statements that specify the vendor, product, and the event ID that the log messages to be parsed contain, and a FORMAT statement that defines the message format.

• MAPPER:vendor—Use this statement to identify the vendor that supplies the product. For example: MAPPER:vendor = AWS.  
• MAPPER:product—Use this statement to identify the product whose logs will be parsed by your parser. For example: MAPPER:product = CloudTrail.  
• MAPPER:event_id—Use this statement to specify the event ID to assign to parsed events. For some log messages this is a constant, for example, for a Windows Event: MAPPER:event_id = Security-4624. In other cases, you may need to form the event ID from fields contained in log messages. In that case, you can define an event ID pattern. For example: MAPPER:event_id = {{eventType}}-{{eventName}}.
• FORMAT—Use this statement to specify the format of the log messages to be parsed. For example: FORMAT = JSON.

Configure and test a custom parser

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Logs > Parsers.
   New UI. In the top menu select Configuration, and then under Logs select Parsers. You can also click the Go To... menu at the top of the screen and select Parsers.
2. Navigate to the folder where you’d like to create the parser. If you want to create a new folder, click Add and select New Folder.  You don’t have to organize your parsers in folders, but it's easier to manage them if you do.
   
3. Click Add and select New Parser to display the Create Parser page.
   
4. Name. Enter a distinctive name for the parser. Typically the parser name indicates the product or service whose messages it will parse.
5. Description. (Optional) Describe the parser.
6. Parser Configuration. Paste your parser code in this area.
7. Import Messages from. In this step, you enter or fetch messages that you’ll use to test whether the parser parses the messages correctly. There are three options:
   • Sumo Log Search. You can enter a log search query to obtain a selected number of log messages. Follow the instructions in Using Sumo Logic log search below.
   • Saved Logs. You can select a set of messages that you saved when previously using the Paste Logs option. Follow the instructions in Using saved logs below.
   • Paste Logs. You can paste logs directly into the Log Messages area. Follow the instructions in Using paste logs below. 

Parse Logs

1. After you’ve obtained sample messages using one of the methods above, click Parse Logs.
2. If all of the sample messages are parsed successfully, you’ll see results like those shown below in the Parsed Messages section of the editor. The Event Details section shows the key-value pairs that were parsed from the raw message. If your results indicate that there were warnings, unparsed, or dropped messages, see Understanding parsing failures and warnings.
   
   note

   One of the two messages that was parsed was cut out of the screenshot to make the image shorter. 

3. Once your new parser is working, and you want to start using it, follow the instructions in Configuring a source to use a parser.

Get sample messages

This section describes the three methods of obtaining messages for use in testing your parser.

Using Sumo Logic log search

To import messages by running a Sumo Logic search:

1. Choose the Sumo Log Search option to display this popup.
   
2. Enter a log query, time range, the number of messages you want returned, and click OK.
3. The popup now displays the results of your search.
   
4. Click OK to close the popup.
5. The Sample logs portion of the parser editor now contains the sample messages. 
6. Proceed to Parse logs.

Using paste logs

To import messages by pasting them in the editor:

1. Choose the Paste Logs option to display this popup.
   
2. Raw Logs. Paste your log messages into this area.
3. Breaker. Use this option to tell the parser editor how to split the text you entered into messages. The options are:
   • Line \n. Choose this option to break the text at line breaks.
   • JSON. Choose this option for JSON messages.
   • Custom Regex. Choose this if you want to use a regex to define the split. The popup will refresh and prompt you for the regex.
4. Click Break Messages.
5. The popup refreshes and shows how the pasted text was broken into individual messages. Review the messages to verify they were split correctly.
   
6. Click OK to close the popup. The Sample logs portion of the parser editor now contains the sample messages. Note the Save Messages As option. You can save the messages you just broke up for use in any additional testing of the parser that you may need to do.
   
7. To save the message, click the Save Messages As option.
8. On the Save Messages popup, enter a name for the saved messages, and click Save.
9. Proceed to Parse logs.

Using saved logs

To import previously saved messages:

1. Click Saved Logs in the Sample Logs section of the editor. This popup appears:
   
2. Select a saved file of sample messages from the list in the File Name section of the popup.
3. The messages from the selected file appear in the Preview Logs section of the page.
4. Click Get Logs.
5. The popup closes and the logs that you retrieved now appear in the Sample Logs section of the editor.
   
6. Proceed to Parse logs.

Understanding parsing failures and warnings

When you test your parser, the editor presents a count of how many messages were successfully parsed, and the counts of messages in the following categories:

• Parsed messages with warnings—A warning or error occurred but the message was was partially parsed, depending on where the warning or error occurred. The most common cause of a warning is applying a RENAME_FIELD statement to a field that isn't present in the message.

• Unparsed messages—An error occurred that caused parsing to fail. Potential causes of parsing failures include:

  • An unmatched regex.
  • Invalid XML, when using XML parsing.
  • Invalid JSON parsing, when using JSON parsing.
  • Fewer CSV fields in the message than expected. 
  • Attempting a transform on a field that doesn't exist unless you use TRANSFORM_FIELD_IF_PRESENT.

• Dropped messages—The message was dropped due to a DROP statement in the parser. 

Create a local configuration for a system parser

You can customize any of the system parsers that are built into Cloud SIEM. When you open an system parser for editing, you'll see its code in the System Configuration section. For a system parser, the UI also provides an area for entering your customizations — that's the part of the page labeled Local Configuration. The parsing language statements you enter there will be executed in addition to the those in the system configuration. If a statement you add to the system configuration already exists in the system configuration, the local statement will override the system statement. For example, if the system configuration has:

START_TIME_FIELD = eventTime

and the local configuration has:

START_TIME_FIELD = _messagetime

the local statement overrides the system statement.

Here is an example of a local configuration that overrides the START_TIME_FIELD and TIME_PARSE statements.

The system configuration and local configuration are separate, so your customizations are preserved when Sumo Logic updates the parser.

Use cases for local configuration

You can use a local configuration to override any statement in a system parser, and add additional logic to the parser using any of the statements supported by the parsing language.

One use case for a local configuration to override one or more of a parser’s time handling statements. For example, if the logs to be parsed don’t have a timestamp, you could set START_TIME_FIELD = _messagetime. This causes the Sumo Logic core platform message time to be used as the _starttime in the field dictionary your parser creates from a message. Or, if the time formats in the logs to be parsed do not exactly match the format that a system parser assumes, you use a local configuration to specify a different TIME_PARSER setting.

Another common reason to set up a local configuration is to pre-parse the content of a JSON object. If your parser is going to process an encapsulated JSON object, you can use a local configuration to pre-parse the original log message from the object.

To create a local configuration:

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Logs > Parsers.
   New UI. In the top menu select Configuration, and then under Logs select Parsers. You can also click the Go To... menu at the top of the screen and select Parsers.
2. In the System folder, navigate to the parser you want to modify and choose Edit from the three-dot kebab menu.
   
3. The parser editor opens. The parser code is shown in the System Configuration area.
   
4. Paste your custom parser code in the Local Configuration area.
5. Use one of the methods in Get test messages above, and then click Parse Logs.

Move a parser

You can move a parser from one location to another within the parser editor’s folder structure. To do so, navigate to the parser you want to move, and select Move from the three-dot kebab menu. 

Export and import a parser

You can export a parser as JSON, and import it to another Sumo Logic org.

1. Navigate to the parser you want to export and choose Export from the three-dot kebab menu.
2. On the Export popup, click Copy to Clipboard and then click Done.
   
3. Access the Sumo Logic org where you want to import the parser.
4. Go to Manage Data > Logs > Parsers.
5. Navigate to the folder where you want to store the parser.
6. Choose Import from the three-dot kebab menu.
7. Enter a name for the parser, paste the code you exported into the popup, and click Import.
   

Setting Cloud SIEM log mapping information

In this step you configure one or more Log Mappings. If all of the messages your parser will process contain the same fields, and you want to create records of the same type, a single Log Mapping will suffice. For some data sources, you will likely need to create more than one Log Mapping. For example:

With some CloudTrail logs messages, you might want to create a different record type, depending on the event ID in a message. In some cases, an authorization record is appropriate, while in others, an audit or audit change record would be a better fit. 

In some CloudTrail messages, the field mapping (the mapping between a key in the field dictionary and a Cloud SIEM record) will vary, depending on the Event ID in the message. For example, you may want to map data into the Cloud SIEM schema field action, but the data you want to map is located in different keys of the original CloudTrail JSON messages depending on the CloudTrail event type.

To create your mapping, see Creating a Structured Log Mapping. After setting up the mapping or mappings, complete the steps in Configuring a source to use a parser, below.

Configuring a source to use a parser

This section explains how to configure a Sumo Logic core platform source to send the messages it collects to a parser. This involves configuring a field for the source: you'll create a _parser field that defines the path to the parser. 

1. Navigate to your custom parser in the editor.
2. Hover over the row that contains the parser.
3. Click the three-dot kebab icon, and select Copy Path and save the path.
   
4. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > Collection.
   New UI. In the Sumo Logic top menu select Configuration, and then under Data Collection select Collection. You can also click the Go To... menu at the top of the screen and select Collection.
5. Navigate to the source that produces the messages your custom parser will process.
   
6. Click +Add Field. 
7. Two blank fields appear, below any fields that have already been defined for the source. Enter _parser as the field name and the path to your parser as the value. 
   
   An orange icon indicates that the _parser field has not been created in your Sumo Logic core platform org yet.

Parser templates

We provide a number of parsers to extract data for normalization (see Parsers in the Cloud SIEM Content Catalog). However, you might need to create custom parsers for data sources not included in the parsers we offer. For this reason, we provide parser templates to aid in creating your own custom parsers.

Access parser templates

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Logs > Parsers.
   New UI. In the top menu select Configuration, and then under Logs select Parsers. You can also click the Go To... menu at the top of the screen and select Parsers.
2. Open the System folder.
3. Scroll down to the Parser Templates folder and open it.
4. Browse the templates. Available formats include:
   • CEF
   • CSV
   • JSON
   • Key value
   • LEEF
   • Unstructured (regular expression)
   • Windows XML
   • XML

What's inside the templates

The parser templates cover common log formats and scenarios. Each template has two versions, one with verbose commentary on each component of the parser, and another without commentary that you can duplicate and use to quickly start creating a custom parser.

Watch the following video for a walkthrough of the parser templates.