Amazon CloudFront

Amazon CloudFront is a content delivery network (CDN) that provides an easy way for companies to distribute content to end-users with low latency and high data transfer speeds. The Sumo Logic app for Amazon CloudFront provides analytics on visitor information, rates and statistics, content being served, and other metrics. The app uses predefined searches and Dashboards that provide visibility into your environment for real time analysis of overall usage.

Log types

The Sumo Logic app for Amazon CloudFront uses logs from an Amazon S3 bucket. These logs will be generated by the Amazon CloudFront service. For setup details, refer to Collecting Logs for Amazon CloudFront.

Sample log messages

2017-09-27 00:21:12 ORD51-M1 335 65.30.1.138 GET domain.cloudfront.net /content/FDW/HLS/HLS360p/FDW_s01e002_tv_hv_or_en_xx_HLS360p_16x9_00_v00154.ts 403 https://www.company.com/stream/food-wars/s01e002 Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010.12;%2520rv:54.0)%2520Gecko/20100101%2520Firefox/54.0 Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHBzOi8vZDM5dG5yZzRlbnBrNncuY2xvdWRmcm9udC5uZXQvY29udGVudC9GRFcvSExTL0hMUzM2MHAvRkRXX3MwMWUwMDJfdHZfaHZfb3JfZW5feHhfSExTMzYwcF8xNng5XzAwX3YwMDE1NC50cyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTUwMzk4MTc1OX0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiI2NS4zMC4xLjEzOC8zMiJ9LCJEYXRlR3JlYXRlclRoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTUwMzk3NDI1OX19fV19&Signature=NCDlKRp6Nv9triqA1r-RBulrMXlvQCRxH16c3dP4RGdmwx8yQO0d75%7EdN94-wwaQ2x7NDlzNUrn7IXkUyHJN3S9kdx7RfVt-gQw9E3hMc4rYYe5NVR0wAeye%7E3gMKuFY%7EhshJqMrbE96HmzzhgQ5qS9gW797PDiwddCmtjYxqgndfF7jO2JJ9QwSpHfqcn5Ceo89Ra0mxwjo4JYu8JfiDhbOAkTU7mpy1ql%7EmYOuwc4zntjMK%7ERKOtcrV3sP9uIunpdh6Ur0-pOmPYTJt13VgUfoYmFB0CJnc8TMosN8ouqMIlSnLXfeKiIdDiP%7EGQKtYeZ54NVx6PqrmOQBSVhikg__&Key-Pair-Id=APKAIWFUV66JZQCBHYXA - Error h5BKcPRKo5oIEz0KZ06V6bRCTJttiW_WUJQmT71jjTnYGE8pA1kfQA== domain.cloudfront.net https 722 0.001 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Error HTTP/2.0

Sample queries

Count of HTTP Status Codes

_sourceCategory= aws/cf | parse "*\t*\t*\t*\t*\t*\t*\t*\t*\t*\t*\t*\t*\t*\t*" as _filedate,_ftime,edgeloc, scbytes, c_ip,method,cs_host,uri_stem,status,referer,user_agent,uri_query,cookie,edgeresult,requestid
| count as count by status
| sort by count

Collecting logs for the Amazon CloudFront app

Prerequisites

Before you install the Amazon CloudFront app, complete the following tasks:

1. Enable CloudFront logging.
2. Confirm that access logs are being delivered to the Amazon S3 bucket.
3. Grant Sumo Logic Access to the Amazon S3 Bucket.

Once you begin uploading data, your daily data usage will increase. It's a good idea to check the Account page to make sure that you have enough quota to accommodate additional data in your account. If you need additional quota, you can upgrade your account at any time.

Add an AWS Source

When you create an AWS Source, you'll need to identify the Hosted Collector you want to use or create a new Hosted Collector. Once you create an AWS Source, associate it with a Hosted Collector. For instructions, see Configure a Hosted Collector.

Rules

• If you're editing the Collection should begin date on a Source, the new date must be after the current Collection should begin date. (Note that if you set this property to a collection time that overlaps with data that was previously ingested on a source, it may result in duplicated data to be ingested into Sumo Logic.)
• Sumo Logic supports log files (S3 objects) that do NOT change after they are uploaded to S3. Support is not provided if your logging approach relies on updating files stored in an S3 bucket. S3 does not have a concept of updating existing files, you can only overwrite an existing file. When this overwrite happens, S3 considers it as a new file object, or a new version of the file, and that file object gets its own unique version ID.
• Sumo Logic scans an S3 bucket based on the path expression supplied, or receives an SNS notification when a new file object is created. As part of this, we receive a file name (key) and the object's ID. It's compared against a list of file objects already ingested. If a matching file ID is not found the contents of the file are ingested in full.
• When you overwrite a file in S3, the file object gets a new version ID and as a result, Sumo Logic sees it as a new file and ingests all of it. If with each version you post to S3 you are simply adding to the end of the file, then this will lead to duplicate messages ingested, one message for each version of the file you created in S3.
• Glacier objects will not be collected and are ignored.
• If you're using SNS you need to create a separate topic and subscription for each Source.

S3 Event Notifications Integration

The Sumo Logic S3 integration combines scan-based discovery and event-based discovery into a unified integration that gives you the ability to maintain a low-latency integration for new content and provide assurances that no data was missed or dropped.

When you enable event-based notifications, S3 will automatically publish new files to Amazon Simple Notification Service (SNS) topics, which Sumo Logic can be subscribed. This notifies Sumo Logic immediately when new files are added to your S3 bucket so we can collect it. For more information about SNS, see the Amazon SNS product docs.

Enabling event-based notifications is an S3 bucket-level operation that subscribes to an SNS topic. An SNS topic is an access point that Sumo Logic can dynamically subscribe to in order to receive event notifications. When creating a Source that collects from an S3 bucket, Sumo Logic assigns an endpoint URL to the Source. The URL is for you to use in the AWS subscription to the SNS topic so AWS notifies Sumo when there are new files. See Configuring Amazon S3 Event Notifications for more information.

You can adjust the configuration of when and how AWS handles communication attempts with Sumo Logic. See Setting Amazon SNS Delivery Retry Policies for details.

Micro Lesson

Watch this micro lesson video to learn more about our S3 event notifications integration, which combines scan based discovery and event-based discovery into a unified integration that gives you the ability to maintain a low latency integration for new content and provide assurances that no data was missed or dropped.

Create an AWS Source

These configuration instructions apply to log collection from all AWS Source types.

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > Collection.
   New UI. In the Sumo Logic top menu select Configuration, and then under Data Collection select Collection. You can also click the Go To... menu at the top of the screen and select Collection.
2. On the Collectors page, click Add Source next to a Hosted Collector, either an existing Hosted Collector, or one you have created for this purpose.
3. Select your AWS Source type.
4. Enter a name for the new Source. A description is optional.
5. Select an S3 region or keep the default value of Others. The S3 region must match the appropriate S3 bucket created in your Amazon account.

info

Selecting an AWS GovCloud region means your data will be leaving a FedRAMP-high environment. Use responsibly to avoid information spillage. See Collection from AWS GovCloud for details.

6. For Bucket Name, enter the exact name of your organization's S3 bucket. Be sure to double-check the name as it appears in AWS, for example:
7. For Path Expression, enter the wildcard pattern that matches the S3 objects you'd like to collect. You can use one wildcard (*) in this string. Recursive path expressions use a single wildcard and do NOT use a leading forward slash. See About Amazon Path Expressions for details.
8. Collection should begin. Choose or enter how far back you'd like to begin collecting historical logs. You can either:
   • Choose a predefined value from dropdown list, ranging from "Now" to “72 hours ago” to “All Time”, or
   • Enter a relative value. To enter a relative value, click the Collection should begin field and press the delete key on your keyboard to clear the field. Then, enter a relative time expression, for example -1w. You can define when you want collection to begin in terms of months (M), weeks (w), days (d), hours (h), and minutes (m). If you paused the Source and want to skip some data when you resume, update the Collection should begin setting to a time after it was paused.
     note

     If you set Collection should begin to a collection time that overlaps with data that was previously ingested on a source, it may result in duplicated data to be ingested into Sumo Logic.

9. For Source Category, enter any string to tag the output collected from this Source. Category metadata is stored in a searchable field called _sourceCategory. Some examples: _sourceCategory: aws/observability/alb/logs or _sourceCategory: aws/observability/clb/logs.
10. Fields. Click the +Add Field link to add custom log metadata Fields. Define the fields you want to associate, each field needs a name (key) and value. The following Fields are to be added in the source:
    • Add an account field and assign it a value which is a friendly name / alias to your AWS account from which you are collecting logs. Logs can be queried via the “account field”.
    • Add a region field and assign it the value of respective AWS region where the Classic Load Balancer exists.
    • Add an accountId field and assign it the value of the respective AWS account id which is being used.
    • A green circle with a check mark is shown when the field exists and is enabled in the Fields table schema.
    • An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema or is disabled it is ignored, known as dropped.
11. For AWS Access, choose between the two Access Method options below, based on the AWS authentication you are providing.
    • For Role-based access, enter the Role ARN that was provided by AWS after creating the role. Role-based access is recommended (this was completed in the prerequisite step Grant Sumo Logic access to an AWS Product).
    • For Key access, enter the Access Key ID and Secret Access Key. See AWS Access Key ID and AWS Secret Access Key for details.
12. Log File Discovery. You have the option to set up Amazon Simple Notification Service (SNS) to notify Sumo Logic of new items in your S3 bucket. A scan interval is required and automatically applied to detect log files.
    • Scan Interval. Sumo Logic will periodically scan your S3 bucket for new items in addition to SNS notifications. Automatic is recommended to not incur additional AWS charges. This sets the scan interval based on if subscribed to an SNS topic endpoint and how often new files are detected over time. If the Source is not subscribed to an SNS topic and set to Automatic the scan interval is 5 minutes. You may enter a set frequency to scan your S3 bucket for new data. To learn more about Scan Interval considerations, see About setting the S3 Scan Interval.
    • SNS Subscription Endpoint (recommended option). New files will be collected by Sumo Logic as soon as the notification is received. This will provide faster collection versus having to wait for the next scan to detect the new file. We highly recommend using an SNS Subscription Endpoint for its ability to maintain low-latency collection. This is essential to support up-to-date Alerts. The following steps use the AWS SNS Console. (Alternatively, you can use AWS CloudFormation; see Using CloudFormation to Set Up an SNS Subscription Endpoint).
      1. To set up the subscription, you need to get an endpoint URL from Sumo to provide to AWS. This process will save your Source and begin scanning your S3 bucket when the endpoint URL is generated. Click Create URL and use the provided endpoint URL when creating your subscription in step B.
      2. Go to Services > Simple Notification Service and click Create Topic. Enter a Topic name and click Create topic. Copy the provided Topic ARN, which you’ll need for the next step. Make sure that the topic and the bucket are in the same region.
      3. Again, go to Services > Simple Notification Service and click Create Subscription. Paste the Topic ARN from step B above. Select HTTPS as the protocol and enter the Endpoint URL provided while creating the S3 source in Sumo Logic. Click Create subscription and a confirmation request will be sent to Sumo Logic. The request will be automatically confirmed by Sumo Logic.
      4. Select the Topic created in step B and navigate to Actions > Edit Topic Policy. Use the following policy template, replace the SNS-topic-ARN and bucket-name placeholders in the Resource section of the JSON policy with your actual SNS Topic ARN and S3 Bucket name:

         {
           "Version":"2008-10-17",
           "Statement":[
           {
             "Effect":"Allow",
             "Principal":{
               "AWS":"*"
               },
               "Action":[
               "SNS:Publish"
               ],
               "Resource":"SNS-topic-ARN",
               "Condition":{
                 "ArnLike":{
                   "aws:SourceArn":"arn:aws:s3:*:*:bucket-name"
                 }
               }
             }
           ]
         }
      5. Go to Services > S3 and select the bucket to which you want to attach the notifications. Navigate to Properties > Events > Add Notification. Enter a Name for the event notification. In the Events section select All object create events. In the Send to section (notification destination) select SNS Topic. An SNS section becomes available, select the name of the topic you created in step B from the dropdown. Click Save.
13. Set any of the following under Advanced:
    • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
    • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs cannot be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
    • Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
    • Enable Multiline Processing. See Collecting Multiline Logs for details on multiline processing and its options. This is enabled by default. Use this option if you're working with multiline messages (for example, log4J or exception stack traces). Deselect this option if you want to avoid unnecessary processing when collecting single-message-per-line files (for example, Linux system.log). Choose one of the following:
      • Infer Boundaries. Enable when you want Sumo Logic to automatically attempt to determine which lines belong to the same message. If you deselect the Infer Boundaries option, you will need to enter a regular expression in the Boundary Regex field to use for detecting the entire first line of multiline messages.
      • Boundary Regex. You can specify the boundary between messages using a regular expression. Enter a regular expression that matches the entire first line of every multiline message in your log files.
14. Create any Processing Rules you'd like for the AWS Source.
15. When you're finished configuring the Source, click Save.

SNS with one bucket and multiple Sources

When collecting from one Amazon S3 bucket with multiple Sumo Sources, you need to create a separate topic and subscription for each Source. Subscriptions and Sumo Sources should both map to only one endpoint. If you were to have multiple subscriptions Sumo would collect your objects multiple times.

Each topic needs a separate filter (prefix/suffix) so that collection does not overlap. For example, the following image shows a bucket configured with two notifications that have filters (prefix/suffix) set to notify Sumo separately about new objects in different folders.

Update Source to use S3 Event Notifications

You can use this community-supported script to configure event-based object discovery on existing AWS Sources.

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > Collection.
   New UI. In the Sumo Logic top menu select Configuration, and then under Data Collection select Collection. You can also click the Go To... menu at the top of the screen and select Collection.
2. On the Collection page, navigate to your Source and click Edit. Scroll down to Log File Discovery and note the Endpoint URL provided, you will use this in step 12.C when creating your subscription.
3. Complete steps 12.B through 12.E under Create an AWS Source > 12. Log File Discovery.

Troubleshoot S3 Event Notifications

In the Sumo Logic UI, under 'Log File Discovery', there is a red exclamation mark with the message 'Sumo Logic has not received a validation request from AWS'.

Steps to troubleshoot:

1. Refresh the Source’s page to view the latest status of the subscription in the SNS Subscription section by clicking Cancel then Edit on the Source in the Collection tab.
2. Verify you have enabled sending Notifications from your S3 bucket to the appropriate SNS topic. This is done in Create an AWS Source > 12. Log File Discovery > Step E.
3. If you didn’t use CloudFormation, check that the SNS topic has a confirmed subscription to the URL in AWS console. A "Pending Confirmation" state likely means that you entered the wrong URL while creating the subscription.

In the Sumo Logic UI, under 'Log File Discovery', there is a green check with the message 'Sumo Logic has received an AWS validation request at this endpoint', but still high latencies.

The green check confirms that the endpoint was used correctly, but it does not mean Sumo Logic is receiving notifications successfully.

Steps to troubleshoot:

1. AWS writes CloudTrail and S3 Audit Logs to S3 with a latency of a few minutes. If you’re seeing latencies of around 10 minutes for these Sources it is likely because AWS is writing them to S3 later than expected.
2. Verify you have enabled sending Notifications from your S3 bucket to the appropriate SNS topic. This is done in the Fields step of Create an AWS Source.

Multiline Processing Boundary Regex Example

If your CloudFront log message is of this format:

2017-06-13    22:02:13    SYD1 ..............

You could use this Boundary Regex:

^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}.*

Installing the Amazon CloudFront app

Now that you have set up collection for CloudFront, install the Sumo Logic app for Amazon CloudFront to use the pre-configured searches and dashboards that provide visibility into your environment for real-time analysis of overall usage.

To install the app, do the following:

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Install App.
   note

   Sometimes this button says Add Integration.

4. Click Next in the Setup Data section.
5. In the Configure section of your respective app, complete the following fields.
   1. Key. Select either of these options for the data source.
      • Choose Source Category and select a source category from the list for Default Value.
      • Choose Custom, and enter a custom metadata field. Insert its value in Default Value.
6. Click Next. You will be redirected to the Preview & Done section.

Post-installation

Once your app is installed, it will appear in your Installed Apps folder, and dashboard panels will start to fill automatically.

Each panel slowly fills with data matching the time range query and received since the panel was created. Results will not immediately be available, but will update with full graphs and charts over time.

Viewing Amazon CloudFront dashboards

All dashboards have a set of filters that you can apply to the entire dashboard. Use these filters to drill down and examine the data to a granular level.

• You can change the time range for a dashboard or panel by selecting a predefined interval from a drop-down list, choosing a recently used time range, or specifying custom dates and times. Learn more.
• You can use template variables to drill down and examine the data on a granular level. For more information, see Filtering Dashboards with Template Variables.
• Most Next-Gen apps allow you to provide the scope at the installation time and are comprised of a key (_sourceCategory by default) and a default value for this key. Based on your input, the app dashboards will be parameterized with a dashboard variable, allowing you to change the dataset queried by all panels. This eliminates the need to create multiple copies of the same dashboard with different queries.

Overview

• Client Geo Distribution. Performs a geo lookup search and displays visitor's client distribution for the last 24 hours on a map of the world.
• Cache Hit and Miss. Displays the cache's hits and misses over time in timeslices of five minutes for the last three hours in a pie chart.
• HTTP Status Codes Over Time. Shows HTTP status codes over time in timeslices of five minutes for the last three hours in a bar chart.
• Visitor Access Platforms. Provides information on the platforms that visitors use to access the site for the last three hours in a pie chart.
• Requests Served by Edge Location. Displays visitor requests served by edge location sorted by count for the last three hours in a pie chart.
• Number of Unique Visitors. Shows unique visitors to the site based on IP address over the last three hours in a single value chart.

Latency Monitoring

• Longest Latency by GeoLocation. See the locations with long latency in the last hour on a world map.
• 90th 95th 99th Pct Time_taken Trend. See the trend of 90th, 95th, and 99th percentiles of time taken in the last 24 hours on a line chart.
• Outlier - Average Latency Time. See the outlier of the average latency time in the last 24 hours on a line chart.
• Outlier - Average Latency Time by Edge Location. See the details of the outlier of average latency time in the last 24 hours such as the time, edge location, average time taken displayed in a table.
• Average Latency Time by CloudFront Edge. See the average latency time by Cloud Front Edge in the last 24 hours on a line chart.
• Average Latency Time in Seconds by Region. See the average latency time in seconds by region across the world in the last 24 hours on a bar chart.
• Global Latency Time in Seconds. See the global latency times in the last 24 hours on a line chart.

Visitor Statistics

• Client Geo Distribution. Performs a geo lookup search and displays visitor's client distribution for the last 24 hours on a map of the world.
• Requests Served by Edge Location. Displays visitor requests served by edge location sorted by count for the last three hours in a pie chart.
• Visitor Access Platforms. Provides information on the platforms that visitors use to access the site for the last three hours in a pie chart.
• Visitor Session Duration Distribution Histogram. Displays the duration of visitor sessions distributed by count and bucket size in a histogram.
• Unique Visitors Over Time. Shows unique visitors to the site by based on IP address in timeslices of five minutes over the last three hours in a column chart.
• Visitor Browsers and Devices. Displays the devices and browsers, counted by platform, used by visitors to access the site over the last three hours in a stacked column chart.

Web Operations

• Edge Result. Displays edge results by count and sorted by type for the last three hours in a pie chart.
• Client and Server Errors Over Time. Shows client and server errors over time in timeslices of five minutes for the last three hours in a column chart.
• HTTP Response Classes. Provides HTTP response classes by count in timeslices of five minutes for the last three hours in a timeline.
• Cache Hit and Miss Over Time. Displays the cache's hits and misses over time in timeslices of five minutes for the last three hours in a stacked column chart.
• HTTP Status Codes Over Time. Shows HTTP status codes over time in timeslices of five minutes for the last three hours in a timeline.
• Traffic and Megabytes Served. Provides information on site traffic hits and Megabytes served in timeslices of one hour over the last 24 hours in a combination column and line chart.

Upgrade/Downgrade the Amazon CloudFront app (Optional)

To update the app, do the following:

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
   Optionally, you can identify apps that can be upgraded in the Upgrade available section.
3. To upgrade the app, select Upgrade from the Manage dropdown.
   1. If the upgrade does not have any configuration or property changes, you will be redirected to the Preview & Done section.
   2. If the upgrade has any configuration or property changes, you will be redirected to Setup Data page.
      1. In the Configure section of your respective app, complete the following fields.
         • Key. Select either of these options for the data source.
           • Choose Source Category and select a source category from the list for Default Value.
           • Choose Custom and enter a custom metadata field. Insert its value in Default Value.
      2. Click Next. You will be redirected to the Preview & Done section.

Post-update

Your upgraded app will be installed in the Installed Apps folder, and dashboard panels will start to fill automatically.

note

See our Release Notes changelog for new updates in the app.

To revert the app to a previous version, do the following:

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. To version down the app, select Revert to < previous version of your app > from the Manage dropdown.

Uninstalling the Amazon CloudFront app (Optional)

To uninstall the app, do the following:

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Uninstall.

Additional logs and metrics collection (Optional)

Log and metric types

• CloudWatch Metrics
• Real-time logs
• CloudWatch Logs
  • Edge Function Logs
• CloudTrail Logs

Configure metrics collection

• Collect CloudWatch Metrics with namespace AWS/CloudFront using AWS Kinesis Firehose for Metrics source. For AWS/CloudFront metrics and dimensions, refer to Amazon CloudFront CloudWatch metrics.

Configure logs collection

• Collect real-time Logs using the Amazon S3 source if you have real-time logs destination set to S3 bucket. With CloudFront real-time logs, you can get information about requests made to distribution in real-time (logs are delivered within seconds of receiving the requests). You can use real-time logs to monitor, analyze, and take action based on content delivery performance.
• Collect Amazon CloudWatch Logs using the AWS Kinesis Firehose for Logs source. You can use Amazon CloudWatch Logs to get logs for your edge functions, both Lambda@Edge and CloudFront Functions.
• Collect AWS CloudTrail Logs using the AWS CloudTrail source. Amazon CloudFront is integrated with CloudTrail, an AWS service that captures information about every request sent to the CloudFront API by your AWS account, including your IAM users. CloudTrail periodically saves log files of these requests to an Amazon S3 bucket that you specify. CloudTrail captures information about all requests, whether they were made using the CloudFront console, the CloudFront API, the AWS SDKs, the CloudFront CLI, or another service, for example, AWS CloudFormation.