Amazon VPC Flow Cloud Security Monitoring and Analytics

The Amazon VPC (Virtual Private Cloud) Flow - Cloud Security Monitoring and Analytics app thoroughly assess Amazon VPC Flow logs to gain a better understanding of your environment and associated traffic patterns. Dig deep into the data, broken down by access levels, group creation, etc.

The Amazon VPC Flow Logs show the IP network traffic of your VPC, allowing you to troubleshoot traffic and security issues. The Amazon VPC Flow Logs App leverages this data to provide real-time visibility and analysis of your environment. It consists of predefined searches and Dashboards.

See Amazon VPC Flow Logs more information.

The VPC Flow Logs can be published to Amazon CloudWatch Logs and Amazon S3. You can use either of the below methods to collect Amazon VPC Flow Logs.

Each method has advantages. Using an Amazon S3 source is more reliable, while using a CloudWatch Logs source with the CloudFormation template allows you to optimize your logs. With the CloudWatch Logs source and CloudFormation template, you can customize logs by adding more information and filtering out unwanted data. The Security Groups dashboard utilizes customized logs that are generated from the Lambda function and created with the CloudFormation template from logs sent to CloudWatch Logs.

Field Extraction Rule for VPC Flow logs

Here is an example Field Extraction Rule for VPC Flow logs.

Rule Name: VPCFlowLogFER
Applied at: Ingest Time
Scope (Specific Data):
_sourceCategory=<Source category for respective VPC flow log source>
Parse Expression:
json "logStream", "logGroup", "message", "direction" as logStream, logGroup, msg, direction nodrop
| if (_raw matches "{*", msg, _raw) as msg
| parse field=msg "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status nodrop

Collect Amazon VPC Flow Logs from CloudWatch using CloudFormation

This section has instructions for collecting VPC Flow Logs using a CloudFormation template. Alternatively, you can Collect Amazon VPC Flow Logs using Amazon S3 Source.

This section has instructions for collecting logs for the Amazon VPC Flow Logs app.

The diagram below illustrates the collection process for Amazon VPC Flow Logs. VPC is enabled to send logs to Amazon CloudWatch. A Lambda function subscribes to a CloudWatch Log Group to obtain the flow logs, and then sends the data on to a Sumo Logic HTTP Source on a hosted collector. The AWS resources are created by a Sumo-provided CloudFormation template.

Step 1: Enable Amazon VPC Flow Logs

You can enable Amazon Virtual Private Cloud (VPC) Flow Logs from the Amazon Web Services (AWS) Management Console, the AWS Command Line Interface (CLI), or by making calls to the Elastic Compute Cloud (EC2) API.

To enable Amazon Virtual Private Cloud (VPC) Flow Logs from the AWS console:

1. Go to VPC management, and go to the VPC list.
2. Select the VPC.
3. Click Actions > Create Flow Log.
4. On the Create Flow Log page, select a Role to use Flow logs.
   • If you haven't set up IAM permissions, click Set Up Permissions.
   • From the new tab, VPC Flow Logs is requesting permissions to use resources in your account:
   • From the IAM Role, select Create a new IAM Role.
   • Add a Role Name that describes your logs, for example, VPC-Flow-Logs.
   • Click Allow.
5. Back in Create Flow Log, enter the new role you created in Role.
6. In Destination Log Group enter a descriptive name such as VPCFlowLogs.
7. Click Create Flow Log. It can take up to an hour for the log group to show up in CloudWatch Logs.

Step 2: Configure hosted collector and HTTP source

1. Create a Hosted Collectorin Sumo Logic.
2. Configure an HTTP Source in Sumo Logic. When configuring the source:
3. Under Advanced Options for Logs, for Timestamp Format, click Specify a format.
4. Under Format, enter: epoch.
5. Under Timestamp locator, enter: \s(\d{10,13})\s\d{10,13}.
6. Click Save.

Step 3: Create AWS functions and resources

Follow the steps on Amazon CloudWatch Logs, starting with the Download the CloudFormation template step and ending with the Dealing with alarms step. As you perform the procedure note the additional instructions below, regarding log format and optional environment variables.

Configure LogFormat correctly (Required)

When you Create a stack on the AWS CloudFormation console, in Step 5, make sure you select either VPC-JSON or VPC-RAW in the LogFormat field in the Specify Details window.

Environment Variables for VPC Flow log collection (Optional)

When you Configure environment variables for Lambda functions, in addition to the variables listed, you can optionally also define the following environment variables.

If you define the environment variables below, do it for both of the Lambda functions created by the CloudFormation template.

---

Environment variable: VPC_CIDR_PREFIX
Description: Comma-separated list of IP prefixes for filtering out internal traffic. For example, vpcCIDRprefix= 10.8.0.0,10.9.0.0 filters out logs whose destinationIP and sourceIP matches any of the two prefixes 10.8.0.0 and 10.9.0.0.

For example, if VPC_CIDR_PREFIX = “10.0.”, then all the IPs with 10.0.*.* will match the prefix.

---

Environment variable: INCLUDE_SECURITY_GROUP_INFO
Description: This option is supported only if you set LogFormat to VPC-JSON. Set to true to include the following fields in logs:

• vpc-id
• subnet-id
• aws-region
• security-group-ids
• direction

If you set the value to true, follow the instructions in Grant Lambda permissions (Optional).

Grant Lambda permissions (Optional)

This step is supported only if INCLUDE_SECURITY_GROUP_INFO is set to true.

The Lambda function fetches list of Elastic Network Interfaces using the describeNetworkInterfaces API. You need to grant permission to Lambda by adding the following inline policy in the SumoCWLambdaExecutionRole role. See the instructions on Creating Policies on the JSON Tab in AWS help.

Paste the JSON below, after adding the ARN of the Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DescribeENILambdaPerms",
            "Effect": "Allow",
            "Action": "ec2:DescribeNetworkInterfaces",
            "Resource": "*"
        }
    ]
}

Step 4: Subscribe the Lambda function to the VPC Flow Log group

1. Select the VPC Flow Log group in the CloudWatch Logs management panel. This is the Log Group created in the first part (VPCFlowLogs was used).
2. Click Actions and select Stream to Lambda Function.
3. Select the Lambda function created by the CloudFormation template. Its name starts with "SumoCWLogsLambda".
4. Click Next.
5. Select JSON for Log Format.
6. Click Next.
7. Click Start Streaming. Wait a few minutes, and check to make sure your logs are flowing into Sumo.

Collect Amazon VPC Flow Logs Using an Amazon S3 Source

This section has instructions for collecting Amazon VPC Flow Logs using an Amazon S3 source. If you prefer to collect VPC logs using a CloudFormation template, see Collect Amazon VPC Flow Logs using a CloudFormation Template.

Step 1: Enable Amazon VPC Flow Logs

1. You can use an existing S3 bucket, or create a new one, as described in Create a S3 bucket in AWS help.
2. Create flow logs for your VPCs, subnets, or network interfaces. For instructions, see Creating a Flow Log that Publishes to Amazon S3 in AWS help.
3. Confirm that logs are being delivered to the S3 bucket. Log files are saved to the bucket using following folder structure: bucket_ARN/optional_folder/AWSLogs/aws_account_id/vpcflowlogs/region/year/month/day/log_file_name.log.gz.

Step 2: Configure Amazon S3 Source

1. Grant Access to an Amazon S3 Bucket.
2. Enable logging using the AWS Management Console.
3. When you create an AWS Source, you associate it with a Hosted Collector. Before creating the Source, identify the Hosted Collector you want to use, or create a new Hosted Collector. For instructions, see Create a Hosted Collector.
4. Add an AWS Source for the S3 Source to Sumo Logic. When you configure the S3 source:
   1. In the Advanced Options for Logs section, uncheck the Detect messages spanning multiple lines option.
   2. In the Processing Rules for Logs section, add an Exclude messages that match processing rule to ignore the following file header lines: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status.

Installing the AWS VPC Security App

Now that you have set up collection, install the Sumo Logic App for Amazon VPC Flow to use the preconfigured searches and dashboards that provide insight into your data.

To install the app:

1. From the Sumo Logic navigation, select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. Optionally, you can scroll down to preview the dashboards included with the app. Then, click Install App (sometimes this button says Add Integration).
   note

   If your app has multiple versions, you'll need to select the version of the service you're using before installation.

4. On the next configuration page, under Select Data Source for your App, complete the following fields:
   • Data Source. Select one of the following options:
     • Choose Source Category and select a source category from the list; or
     • Choose Enter a Custom Data Filter, and enter a custom source category beginning with an underscore. For example, _sourceCategory=MyCategory.
   • Folder Name. You can retain the existing name or enter a custom name of your choice for the app.
   • All Folders (optional). Default location is the Personal folder in your Library. If desired, you can choose a different location and/or click New Folder to add it to a new folder.
5. Click Next.
6. Look for the dialog confirming that your app was installed successfully.
   

Once an app is installed, it will appear in your Personal folder or the folder that you specified. From here, you can share it with other users in your organization. Dashboard panels will automatically start to fill with data matching the time range query received since you created the panel. Results won't be available immediately, but within about 20 minutes, you'll see completed graphs and maps.

Viewing AWS VPC Security Dashboards

Analytics and Monitoring dashboards to provide operational security for AWS VPC flow data sources.

VPC Flow Logs - Security Monitoring - Overview

Description: See the details of security group activities and all AWS activities divided by read only and non read only.

Use Case: Provides analysis of group activity events including revoking and authorizing access, creating and deleting groups, and other events.

VPC Flow Logs - Security Analytics - Accepts & Rejects

Description: See the details of security group activities and all AWS activities divided by read only and non read only.

Use Case: Provides analysis of group activity events including revoking and authorizing access, creating and deleting groups, and other events.

VPC Flow Logs - Security Analytics - Traffic Direction Monitoring

Description: See the details of security group activities and all AWS activities divided by read only and non read only.

Use Case: Provides analysis of group activity events including revoking and authorizing access, creating and deleting groups, and other events.

Use Case: Provides analysis of group activity events including revoking and authorizing access, creating and deleting groups, and other events.