Skip to main content

Cassandra - OpenTelemetry Collector

Thumbnail icon Thumbnail icon

The Cassandra app is a log and metrics-based app that helps you monitor the availability, performance, health, and resource utilization of your Cassandra clusters. The pre-configured dashboards provide insight into resource utilization, cache/Gossip/Memtable statistics, error and warnings, request served and latency, storage, and compaction.

Cassandra logs are sent to Sumo Logic through OpenTelemetry filelog receiver and cassandra metrics are sent to Sumo Logic using JMX opentelemetry receiver with the target_system set as cassandra.

The app supports Logs from the open-source version of Cassandra. The App is tested on the 4.0.0 version of Cassandra.

Schematics

Fields creation in Sumo Logic for Cassandra​

Following are the Fields which will be created as part of Cassandra App install if not already present:

  • deployment.environment. User configured. Through this Cassandra cluster is identified by the environment where it resides. For example: dev, prod, or qa.
  • db.cluster.name. User configured. Enter a name to identify this Cassandra cluster. This cluster name will be shown in the Sumo Logic dashboards.
  • db.system. Has fixed value of cassandra.
  • db.node.name. Holds the value of the Fully Qualified Domain Name (FQDN) of the machine from which the OpenTelemetry collector is collecting logs and metrics.
  • sumo.datasource. Has fixed value of cassandra.

Prerequisites​

For metric collection​

JMX receiver collects Cassandra metrics from Cassandra server as part of the OpenTelemetry Collector (OTC).

  1. Follow the instructions in JMX - OpenTelemetry's prerequisites section to download the JMX Metric Gatherer. This gatherer is used by the JMX Receiver.

  2. Set the JMX port as part of JAVA_OPTS for Tomcat startup. Usually, it is set in the /etc/systemd/system/cassandra.service or C:\Program Files\apache-tomcat\bin\tomcat.bat file.

    JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=11099 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${CASSANDRA_CONF_DIR}/jmx.password -Dcom.sun.management.jmxremote.access.file=${CASSANDRA_CONF_DIR}/jmx.access"

For log collection​

Cassandra has three main logs: system.log, debug.log, and gc.log which hold general logging messages, debugging logging messages, and java garbage collection logs respectively.

These logs by default live in ${CASSANDRA_HOME}/logs, but most Linux distributions relocate logs to /var/log/cassandra. Operators can tune this location as well as what levels are logged using the provided logback.xml file. For more details on Cassandra logs, see this link.

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

Collection configuration and app installation​

As part of data collection setup and app installation, you can select the App from App Catalog and click on Install App. Follow the steps below.

Step 1: Set up Collector​

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

  1. Select the Add a new Collector option.
  2. Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Collector

Step 2: Configure integration​

In this step, you will configure the yaml required for Cassandra Collection. Path of the log file configured to capture Cassandra logs needs to be given here.

Below are the inputs required:

  • The path to system.log is required here. This file is typically located in /var/log/cassandra. If you're using a customized path, check the respective conf file for this information.
  • For metrics, you're required to provide the JMX endpoint (default is localhost:7199) along with the collection_interval (default is 1 minute).

You can add any custom fields which you want to be tagged with the data ingested in Sumo. Click on the Download YAML File button to get the yaml file.

YAML

Step 3: Send logs to Sumo​

Once you have downloaded the yaml file as described in the previous step, follow the below steps based on your platform.

  1. Copy the yaml to /etc/otelcol-sumo/conf.d/ folder for the Cassandra instance which needs to be monitored.
  2. Restart the collector using:
 sudo systemctl restart otelcol-sumo

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample log messages​

  INFO [ScheduledTasks:1] 2023-01-08 09:18:47,347 StatusLogger.java:101 - system.schema_aggregates

Sample metrics message​

{
    "queryId": "A",
    "_source": "cassandra-otel",
    "_metricId": "dasIFS6RvCVxEcsljWGpnQ",
    "_sourceName": "Http Input",
    "host.id": "i-071ade75d03650657",
    "host.group": "cassandraHostGrp",
    "telemetry.sdk.version": "1.38.0",
    "os.type": "linux",
    "sumo.datasource": "cassandra",
    "db.system": "cassandra",
    "db.node.name": "ip-54-163-607-30.ec2.internal",
    "_sourceCategory": "Labs/cassandra-otel",
    "deployment.environment": "dev",
    "_contentType": "Carbon2",
    "host.name": "ip-54-163-607-30.ec2.internal",
    "metric": "cassandra.client.request.write.latency.50p",
    "_collectorId": "000000000FB3B451",
    "_sourceId": "000000005A60F624",
    "unit": "us",
    "db.cluster.name": "cassandra_cluster",
    "telemetry.sdk.language": "java",
    "telemetry.sdk.name": "opentelemetry",
    "service.name": "unknown_service:java",
    "_collector": "Labs - cassandra-otel",
    "max": 229,
    "min": 0,
    "avg": 131.25,
    "sum": 525,
    "latest": 67,
    "count": 4
}

Sample log queries ​

Following is the query from Cassandra App's overview Dashboard's Nodes Up Panel:

%"sumo.datasource"=cassandra %"deployment.environment"=* %"db.cluster.name"=*   "INFO" | json "log" as _rawlog nodrop 
| if (isEmpty(_rawlog), _raw, _rawlog) as _raw
| parse regex field=_raw "(?<level>[A-Z]*) *\[(?<thread_name>[^\]]*?)[:_-]?(?<thread_id>[0-9]*)\] (?<Date>.{10} .{12}) *(?<source_file>[^:]*):(?<source_line>[0-9]*) - (?<message>.*)"
| if (message matches "InetAddress * is now UP",1,0) as UP
| timeslice 1d
| sum(UP) as UP by _timeslice
| sort by _timeslice asc

Sample metrics query​

Following is the query from Cassandra App's overview Dashboard's Number of Requests Panel:

sumo.datasource=cassandra deployment.environment=* db.cluster.name=* db.node.name=* metric=cassandra.client.request.count | sum

Viewing Cassandra dashboards​

Overview​

The Cassandra - Overview dashboard provides an at-a-glance view of Cassandra backend and frontend HTTP error codes percentage, visitor location, URLs, and clients causing errors.

Use this dashboard to:

  • Identify number of nodes which are up and down
  • Gain insights into Memory - Init, used, Max and committed
  • Gain insights into the error and warning logs by thread and Node activity
Collector

Cache Stats​

The Cassandra - Cache Stats dashboard provides insight into the database cache status, schedule, and items.

Use this dashboard to:

  • Monitor Cache performance.
  • Identify Cache usage statistics.
Cache Stats

Errors and Warnings​

The Cassandra - Errors and Warnings dashboard provides details of the database errors and warnings.

Use this dashboard to:

  • Review errors and warnings generated by the server.
  • Review the Threads errors and warning events.
Errors and Warnings

Gossip​

The Cassandra - Gossip dashboard provides details about communication between various cassandra nodes.

Use this dashboard to:

  • Determine nodes with errors resulting in failures.
  • Review the node activity and pending tasks.
Gossip

Memtable​

The Cassandra - Memtable dashboard provides insights into memtable statistics.

Use this dashboard to:

  • Review flush activity and memtable status.
Memtable

Resource Usage​

The Cassandra - Resource Usage dashboard provides details of resource utilization across Cassandra clusters.

Use this dashboard to:

  • Identify resource utilization. This can help you to determine whether resources are over-allocated or under-allocated.
Resource Usage

Compaction​

The Cassandra - Compactions dashboard provides insight into the completed and pending compaction tasks.

Compaction

Requests​

The Cassandra - Requests dashboard provides insight into the number of request served, number of error request, and their distribution by status and operation. Also you can monitor the read and write latency of the cluster instance using this dashboard.

Requests

Storage​

The Cassandra - Storage dashboard provides insight into the current value of total hints of your Cassandra cluster along with storage managed by the cluster.

Storage
Status
Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.