Linux Source Template

The Linux source template generates an OpenTelemetry configuration that can be pushed to a remotely managed OpenTelemetry collector (abbreviated as otelcol). By creating this source template and pushing the configuration to the appropriate OpenTelemetry agent, you can ensure the collection of Linux logs and host metrics for Sumo Logic.

Fields created by the source template

When you create a source template, the following fields are automatically added (if they don’t already exist):

• sumo.datasource. Fixed value of linux.
• deployment.environment. This is a user-configured field set at the time of collector installation. It identifies the environment where the Linux system resides, such as dev, prod, or qa.
• host.group. This is a collector-level field that is user-configured at the time of collector installation. It identifies the Linux host group.
• host.name. This is tagged through the resourcedetection processor. It holds the value of the host name where the OTel collector is installed.

Prerequisites

For logs collection

This app is based on the following log files from the Ubuntu Linux machine.

• auth.log
• syslog
• daemon.log
• dpkg.log
• kern.log
• CentOS, Amazon Linux, and Red Hat
• audit/audit.log
• secure
• messages
• yum.log

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

Configuring the Linux source template

Follow the below steps to set a remotely managed OpenTelemetry collector and push the source template to it.

Step 1: Set up remotely managed OpenTelemetry collector

In this step, we'll install the collector and add a uniquely identifiable tag to these collectors.

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > OpenTelemetry Collection.
   New UI. In the Sumo Logic top menu select Configuration, and then under Data Collection select OpenTelemetry Collection. You can also click the Go To... menu at the top of the screen and select OpenTelemetry Collection.
2. On the OpenTelemetry Collection page, click + Add Collector.
3. In the Set up Collector step:
   1. Choose your platform (for example, Linux).
   2. Enter your Installation Token.
   3. Under Tag data on Collector level, add a new tag to identify these collectors as having Apache running on them (for example, application = Apache).
   4. Leave the Collector Settings at their default values to configure collectors as remotely managed.
   5. Under Generate and run the command to install the collector, copy and run the installation command in your system terminal where the collector needs to be installed.
4. After installation is complete, click Next to proceed.
5. Select a source template (for example, Apache source template) to start collecting logs from all linked collectors, then proceed with the data configuration.
   

To revisit this screen later: From the Classic UI, select Manage Data > Collection > Source Template. From the New UI, select Configuration > Source Template.

Step 2: Configure the source template

In this step, you will configure the yaml required for Linux Collection. Below are the inputs required for configuration:

• Name. Name of the source template.
• Description. Description for the source template.

Logs collection

• Fields/Metadata. You can provide any customer fields to be tagged with the data collected. By default, sumo tags _sourceCategory with the value otel/linux.
• Logs. The following fields are pre-populated with default paths, for common log files that are used in different Linux distributions. Not all paths might be relevant for your operating system. Modify the list of files as required or leave the default values.

Metrics collection

• Metrics. Select the metric scrappers you want to enable. By default, metric collection for CPU, memory, disk, load, file system, network, and paging are enabled and process metric collection is disabled.

Enable process metric collection (optional)

By default, the collector will not send process metrics to Sumo Logic. This is because the number of processes running on a host can be very large, which would result in a significant increase in Data Points per Minute (DPM).

Click the Enable process metric collection checkbox to collect process-level metrics.

• Name of process. Add the list of process names.
• Include/Exclude the above pattern. Signifies if you want to exclude or include the metrics for the processes listed previously.
• Match type for process name. Select if the process name given should be considered for a strict match with the host machine processes or if it should be considered as regex when matching.
  

note

If you need to edit the process list in the future, you can do this manually in the OTEL config yaml by adding or removing in the names list under process scrapper.

process:
  include:
    names: [ <process name1>, <process name2> ... ]
    match_type: <strict|regexp>

• Scan Interval. The frequency at which the source is scanned.
• Processing Rules. You can add processing rules for logs/metrics collected. To learn more, refer to Processing Rules.

Step 3: Push the source template to the desired remotely managed collectors

info

A new source template will always be created with the latest version of the source template.

Follow the below steps to create a data collection configuration to gather the required logs and link them to all the collectors with the help of collector tags.

1. Complete the source template form with the name and file path for your logs (for example, error logs or access logs), then click Next.
2. Under Link Collectors, you will have the option to link the collectors using the collector name or by adding tags to find the group of collectors (for example, application = Apache).
   
3. Preview and confirm the collectors that will be linked (fetched automatically) to the newly created source template.
   
   
4. Click Next to complete the source template creation. In the background, the system will apply the configuration to all the linked collectors and will start collecting the respective telemetry data from the remote host (in the example, it would start collecting Apache error logs).
5. Click the Log Search or Metrics Search icons to search for and analyze your data collected for this source template.

info

Refer to the changelog for information on periodic updates to this source template.