Sumo Logic Infrequent Data Tier App

The Infrequent Data Tier App provides visibility into the On-demand Search usage and costs associated with Infrequent Data Tier by providing intuitive pre-configured dashboard and searches. Infrequent Data Tiers are an economical, fully managed log analytics solution for high volume, infrequently accessed data. With Infrequent Data Tiers, organizations have a solution that can aggregate, store and analyze verbose sources such as App Debug, CDN, Load Balancer, and other infrequently accessed logs at a dramatically lower price point.

With growing adoption of modern application stacks that leverage micro-services and distributed architectures, organizations are generating more and more machine data. A large part of this data is not required for every day mission-critical operations. However, organizations still need to retain and analyze this data, as it can provide value across multiple groups within a digital enterprise. A cost efficient solution that manages infrequently used data is currently lacking in the market place, forcing customers to either not retain the data or build home-grown solutions that are difficult to maintain, secure, and don’t provide easy access to insights when enterprise needs it.

Log Types

The Search Audit Index for Infrequent Data Tier App has the following fields:

• analytics_tier
• capacity_used
• daily_capacity
• data_retrieved_bytes
• data_scanned_bytes
• execution_duration_ms
• is_aggregate
• query
• query_end_time
• query_start_time
• query_type
• remote_ip
• retrieved_message_count
• scanned_message_count
• scanned_partition_count
• session_id
• status_me

You can start using the index by using following query:

_index=sumologic_search_usage_per_query \
analytics_tier="Infrequent"

Sample Query

The following query is from the Data Scanned Over Time panel of the Infrequent Cost dashboard.

_index=sumologic_search_usage_per_query analytics_tier = "Infrequent" !(user_name=*sumologic.com) !(status_message="Query Failed")
| fields data_scanned_bytes, query, is_aggregate, query_type, remote_ip, retrieved_message_count, scanned_message_count, scanned_partition_count, session_id, status_message, user_name
|data_scanned_bytes / 1Gi as sizeInGB
| timeslice 1d
| sum (sizeInGB) as total_data_scanned_GB, count as query_count by _timeslice
| fillmissing timeslice (1d)

Prerequisites

availability

Sumo Logic Infrequent Data Tier App is only available for Enterprise Suite customers.

Account Type	Account Level
Credits	Enterprise Suite

Collecting Data for the Infrequent Data Tier App

Infrequent Data Tiers enable you to ingest log or machine data for a minimal cost per GB. Your data is then securely stored by Sumo Logic and is instantly available on-demand for interactive analysis without any additional preparation, re-ingestion, or rehydration. This service is ideal when you need to quickly and/or periodically investigate issues, troubleshoot code, configuration problems, or address customer support cases which can rely on searching high volumes of data for insights. This allows you to only pay for the specific data sets that you analyze at a given time.

Infrequent Data Tiers allow you to effectively analyze high volumes of log data that is accessed on a minimal basis. In this data tier, you pay for the amount of data scanned in each search. This on-demand payment model allows you to minimize costs by only paying the data accessed.

You can verify if you already have the Search Audit Index enabled by querying:

_index=sumologic_search_usage_per_query

If no results are returned even for longer time ranges, it means you don't have Search Audit index enabled.

Installing the Infrequent Data Tier App

Complete the following instructions to install the Sumo Logic Infrequent Data Tier App.

required step

Before installing the Infrequent Data Tier App, you need to enable the Search Audit Index, which is currently in beta. Please reach out to your account executive or Sumo Logic support to get the Search Audit index enabled.

To install the app:

1. From the left nav, select App Catalog.
   
2. Search for your app name and select it.
3. When you get to your app page:
   • If you want to see a preview of the dashboards included with the app before installing, scroll down to Dashboard Preview.
   • If your Sumo Logic app has multiple versions (not all apps do), select the version of the service you're using.
     
   • Click Add Integration.
     
4. On the next configuration page, Select Data Source for your App, complete the following fields:
   • Data Source. Select one of the following options:
     • Choose Source Category, and select a source category from the list; or
     • Choose Enter a Custom Data Filter, and enter a custom source category beginning with an underscore. For example, _sourceCategory=MyCategory.
   • Folder Name. You can retain the existing name or enter a custom name of your choice for the app.
   • All Folders (optional). Default location is the Personal folder in your Library. If desired, you can choose a different location and/or click New Folder to add it to a new folder.
5. Click Next.
   
6. You'll see a dialog confirming that the app was installed successfully.
   

Once an app is installed, it will appear in your Personal folder, or other folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't be available immediately, but within 20 minutes, you'll see full graphs and maps.

Viewing Infrequent Data Tier Dashboards

Each dashboard has a set of filters that you can apply to the entire dashboard, as shown in the following example. Click the funnel icon in the top dashboard menu bar to display a scrollable list of filters that are applied across the entire dashboard.

You can use filters to drill down and examine the data on a granular level.

Each panel has a set of filters that are applied to the results for that panel only, as shown in the following example. Click the funnel icon in the top panel menu bar to display a list of panel-specific filters.

Data Scanned Overview

The Infrequent Data Tier - Data Scanned Overview dashboard provides visibility into the total amount of data scanned by infrequent queries. This allows you to monitor and control on-demand search costs for the Infrequent Tier.

Use this dashboard to:

• Monitor data scan information per day.
• Analyze monthly comparisons of scanned data.
• Understand data scanned by caller module, query type, and query status.

Data Scanned by Users

The Infrequent Data Tier - Data Scanned by Users dashboard provides insights into the amount of infrequent data scanned by specific users, user trends, and the geographic locations of users.

Use this dashboard to:

• Monitor data scan information by user.
• To analyze trends in data scanned by users over a 30 day period.
• Monitor the geographical locations for data scan information.

Query Analysis

The Infrequent Data Tier - Query Analysis dashboard provides visibility into the Infrequent Data Tier queries that are most expensive, as well as detailed information on the queries.

Use this dashboard to:

• Identify the most expensive queries on the Infrequent Tier.
• Analyze queries executed against Infrequent Tier.

Credits Consumed

The Infrequent Data Tier - Credits Consumed dashboard provides visibility into the total amount of Cloud Flex Credits your organization has consumed with infrequent queries. This allows you to monitor and control on-demand search costs for the Infrequent Tier.

Use this dashboard to:

• Monitor credits consumed per day.
• Monitor credits consumed per user.
• Monitor the geographical locations of credit consumption.
• Identify the most expensive queries on the Infrequent Tier.