Redis - OpenTelemetry Collector

Redis is an in-memory data structure that implements a distributed, in-memory key-value database with durability options.

The Sumo Logic app for Redis helps you monitor the Redis database cluster. The preconfigured dashboards provide detailed analysis based on logs and metrics. The panels provide details such as RDBMemory Usage, events, RDB, AOF events, resource usage, commands running, and cache hit rate.

Redis logs are sent to Sumo Logic through OpenTelemetry filelog receiver. The OpenTelemetry collector runs on the same host as Redis and uses the Redis Receiver and the Sumo Logic OpenTelemetry Exporter to send the metrics to Sumo Logic.

Log and metrics types

This app supports metrics and logs for Redis in Cluster mode or Standalone mode. The Redis logs are generated in files as configured in the cluster node configuration files in /etc/redis/redis.conf. For more details on Redis logs, see Redis documentation.

The app supports metrics generated by the Redis Receiver.

Creating fields in Sumo Logic for Redis

The following are fields that will be created as part of the Redis App install if not already present.

• db.cluster.name. User configured. Enter a name to identify this Redis cluster. This cluster name will be shown in the Sumo Logic dashboards.
• db.system. Has fixed value of redis.
• deployment.environment. User configured. This is the deployment environment where the Redis cluster resides. For example: dev, prod or qa.
• db.node.name. Has the value of host name of the machine which is being monitored.
• sumo.datasource. Has fixed value of redis.

Prerequisites

For metrics collection

The Redis INFO command returns information and statistics about a Redis server (see Redis INFO for details). The Redis receiver extracts values from the result and converts them to OpenTelemetry metrics.

For logs collection

This section provides instructions for configuring log collection for Redis running on a non-Kubernetes environment. By default, Redis logs are stored in a log file.

Follow the instructions to set up log collection:

1. To configure the Redis log file, locate your local redis.conf configuration file in the database directory. By default, Redis logs are stored in /var/log/redis/redis-server.log.
2. After determining the location of conf file, open your redis.conf configuration file in a text editor to modify its logging parameters as such:
   1. Specify the server verbosity level. The value loglevel in conf file can be set to one of the following, in ascending order of severity: debug, verbose, notice, warning.
   2. Specify the log file name. Also the empty string can be used to force Redis to log on the standard output. If you use the standard output for logging but daemonize, logs will be sent to /dev/null logfile.
3. Save the redis.conf file and restart the Redis server using the command: sudo service redis-server restart.

Once the logs are configured to write to a local file, follow the steps below to configure the collection in Sumo Logic.

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

Collection configuration and app installation

As part of data collection setup and app installation, you can select the App from App Catalog and click on Install App. Follow the steps below.

Step 1: Set up Collector

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

1. Select the Add a new Collector option.
2. Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Step 2: Configure integration

OpenTelemetry works with a configuration YAML file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform.

In this step, you will configure the YAML file required for Redis Collection.

Below are the inputs required:

• endpoint (no default). The hostname and port of the Redis instance, separated by a colon. (For example: localhost:6379.)
• Redis logs Path. Enter the path to the log file for your Redis instance.

The log file path configured to capture redis logs must be given here. The files are typically located in /var/log/redis/redis-server.log. If you are using a customized path, check the redis.conf file for this information.

You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the Download YAML File button to get the YAML file.

Click on the Download YAML File button to get the YAML file.

For Linux platform, click Download Environment Variables File to get the file with the password which is supposed to be set as environment variable.

note

Each Redis replica requires an independent Sumo Logic OpenTelemetry collector to be installed for effective logs and metrics monitoring.

Step 3: Send logs to Sumo Logic

Once you have downloaded the YAML file as described in the previous step, follow the below steps based on your platform.

Linux

1. Copy the YAML at /etc/otelcol-sumo/conf.d/ folder in the Redis instance that needs to be monitored.
2. Move the env file in the following directory:

    /etc/otelcol-sumo/env/
3. Restart the otelcol-sumo process using:

   sudo systemctl restart otelcol-sumo

macOS

1. Copy the YAML at /etc/otelcol-sumo/conf.d/ folder in the Redis instance that needs to be monitored.
2. Restart the otelcol-sumo process using:

   otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --conf "glob:/etc/otelcol-sumo/conf.d/*.yaml"

Chef

1. Copy the yaml file into your Chef cookbook files directory files/<downloaded_yaml_file>.
2. Use a Chef file resource in a recipe to manage it. cookbook_file '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>' do mode 0644 notifies :restart, 'service[otelcol-sumo]', :delayed end
3. Add the recipe to your collector setup to start collecting the data. Every team typically has their established way of applying the Chef recipe. The resulting Chef recipe should look something like:

   cookbook_file '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>' do
       mode 0644
       notifies :restart, 'service[otelcol-sumo]', :delayed
   end

Ansible

1. Place the file into your Ansible playbook files directory.
2. Run the Ansible playbook.

   ansible-playbook -i inventory install_sumologic_otel_collector.yaml
   -e '{"installation_token": "<YOUR_TOKEN>", "collector_tags": {<YOUR_TAGS>}, "src_config_path": "files/conf.d"}'

Puppet

1. Place the file into your Puppet module files directory modules/install_otel_collector/files/<downloaded_yaml>.
2. Use a Puppet file resource to manage it.

   file { '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>':
       ensure => present,
       source => 'puppet:///modules/install_otel_collector/<downloaded_yaml_file>',
       mode => '0644',
       notify => Service[otelcol-sumo],
     }
3. Apply the Puppet manifest. Every team typically has their established way of applying the Puppet manifest. The resulting Puppet manifest should look something like:

   node 'default' {
       class { 'install_otel_collector'
         installation_token => '<YOUR_TOKEN>',
         collector_tags => { <YOUR_TAGS> },
       }
   
       service { 'otelcol-sumo':
         provider => 'systemd',
         ensure => running,
         enable => true,
         require => Class['install_otel_collector'],
       }
   
       file { '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>':
         ensure => present,
         source => 'puppet:///modules/install_otel_collector/<downloaded_yaml_file>',
         mode => '0644',
         notify => Service[otelcol-sumo],
       }
     }

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample log messages

Here's a sample log message in a non-Kubernetes environment.

5275:S 17 Mar 2021 19:13:38.138 * MASTER <-> REPLICA sync: Finished with success

Sample queries

Logs

This sample query is from the Redis - Logs dashboard > Logs panel.

Query string

db.cluster.name=* sumo.datasource="redis"
| json auto maxdepth 1 nodrop
| if  (isEmpty(log), _raw, log)  as message
| count by message
| limit  100
| fields message

Metrics

This sample query is from the Redis - Cluster Operations dashboard Metrics panel.

(cmd=set* or cmd=get* or cmd=incr* or cmd=decr* or cmd=mget or cmd=mset* or cmd=strlen or cmd=psetex or cmd=append) 
deployment.environment=*  db.cluster.name=* db.node.name=*  
| delta 
| sum

Viewing Redis dashboards

Logs

The Redis - Logs dashboard provides a detailed analysis based on logs. The panels provide details such as RDBMemory Usage, events, RDB, and AOF events.

Use this dashboard to:

• Review errors and warnings generated by the server.
• Review the RDBMemory Usage, events, RDB, and AOF events.

Overview

The Redis - Overview dashboard provides an at-a-glance view of the Redis server status and error logs, along with database metrics.

Use this dashboard to review keyspace hit, connected clients, Uptime, Evicted vs Expired Keys, and Commands Per Seconds.

Cluster Operations

The Redis - Cluster Operations Operations dashboard provides a high-level view of the activity and health of Master and Slaves on your network. Dashboard panels display visual graphs and detailed information on cache rate and most importantly, resource usage.

Use this dashboard to review keyspace hit, Cache miss rate by host, CPU - System vs User usage by host, and Network Usage - Inbound/Outbound by Host.

Cluster Overview

The Redis - Cluster Overview dashboard provides a high-level view of cluster infrastructure, comparisons, and trends.

Use this dashboard to review Average Master Uptime, Connected Slaves, Redis Uptime by Host, Commands by Host, and Instantaneous OPS per sec by Host.

Command Stats Delta

The Redis - Command Stats Delta dashboard provides an at-a-glance view of commands running on your Redis server.

Use this dashboard to review String Commands, Key Commands, Hash Commands, Sets Commands, Sorted Sets Commands, List Commands, Transaction Commands, and Scripting Commands.