Collect Amazon CloudWatch Logs using a Lambda Function

note

We strongly recommend the alternative collection process described on AWS Kinesis Firehose for Logs Source, which is more robust and reliable, where you do not need to manage resources. Lambda based collection methods are limited by time out, concurrency, and memory limits.

warning

This method is highly not recommended as it does not handle failures and may lead to data loss. Additionally, the data format generated by this method might be incompatible with some applications in our App Catalog and thus requires customization of those apps.

The following instructions describe how to download and configure an AWS Lambda function for Amazon CloudWatch Logs and send them to Sumo Logic.

Step 1: Add a Hosted Collector and HTTP Source

1. Configure a Hosted Collector.
2. Configure an HTTP Source.

note

When you configure the HTTP Source, make sure to save the HTTP Source Address URL. You will need this to configure the Lambda Function.  

Step 2: Create Lambda function

Sumo provides a Lambda function for use with Amazon Web Services (AWS). It collects AWS Lambda logs using CloudWatch Logs and it extracts and adds a RequestId field to each log line to make correlation easier. 

To add an Amazon Lambda function:

1. Sign in to the AWS Lambda Console.
2. On the Functions page, click Create a Function.
3. On the Create Functions page, select Use a blueprint option.
4. In the Basic information section, enter the following.
   
   • Blueprint name. Enter sumologic and select Send Cloudwatch logs to Sumologic.
   • Function name. Enter a name for the function. Default is sumologic-cwl-lambda.
   • Role. Choose one of the following options.
     • Use an existing role. If you have any appropriate roles, you can select one.
     • Create a new role with basic Lambda permissions. If you select this option, you can continue without choosing any policy templates—it will create a role with basic Lambda execution privileges by default.
     • Create a new role from AWS policy templates.
       • Role Name. Enter a name for the role.
       • Policy templates. Choose the template and make sure that the policy has following permissions logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents so that Lambda can push the logs to Cloudwatch.
5. In the CloudWatch Logs trigger section, you can create a trigger, or click Remove if you prefer to create it later in Step 3. To create the trigger:
   
   • Log Group. Select the log group that serves as the event source. Events sent to the log source will trigger your Lambda function. 
   • Filter Name. Enter a filter name.
   • Filter Pattern (Optional). May be left blank. For information about AWS filter patterns, see Filter and Pattern Syntax in AWS Help.
6. In the Environment Variables section, set the value of the variable to the URL of the HTTP source from Step 1 to which your logs will be sent. In addition, you can set any of the following optional variables:
   
   • ENCODING (Optional). Use when decoding CloudWatch log events, default is utf-1.
   • SOURCE_CATEGORY_OVERRIDE (Optional). Override _sourceCategory value configured for the HTTP source.
   • SOURCE_HOST_OVERRIDE (Optional). Override _sourceHost value configured for the HTTP source.
   • SOURCE_NAME_OVERRIDE (Optional). Override _sourceName value configured for the HTTP source.
7. Click Create Function. A success message will be displayed.
   

note

This solution only ingests log messages to Sumo Logic. If you want to include log levels (such as INFO, DEBUG, etc.) to Sumo Logic, find and update the regex in Lambda code deployed in your AWS account with
var consoleFormatRegex = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z\s(\w+?-\w+?-\w+?-\w+?-\w+)\s/;

Step 3: Assign CloudWatch Log Groups to Your Lambda Function

info

You will need at least one CloudWatch Log Group to assign to your Lambda function. For details on how to create a CloudWatch Log Group, see Create a CloudWatch Log Group.

note

If you have more than one CloudWatch Log Group to assign to your Lambda function, repeat the below steps for each Log Group. Otherwise, skip this step.

1. Sign in to the AWS Lambda Console.
2. On the Functions page, select the Lambda function created in Step 2.
3. In the Function overview section, click the + Add trigger button.
   
4. In the Add Trigger prompt, select CloudWatch Logs from the Select a source dropdown.
   
5. Select a CloudWatch Log Group to add to your function. You need at least one CloudWatch Log Group to see this option.
6. Add a Filter Name to your trigger.
7. (Optional) Add a Filter Pattern to your trigger. For information about AWS filter patterns, see Filter and Pattern Syntax in the AWS documentation.
8. Click Add to add the trigger to your Lambda function.

note

Sample CloudWatch log ingested for Amazon RDS PostgreSQL using Lambda log forwarder.

{
    "timestamp":1705908605000,
    "message":"2024-01-22 07:30:05 UTC::@:[561]:LOG:  checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.002 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB"
}