Monitoring Your K8s Environment

This page provides insights for effectively monitoring your Kubernetes environment with Sumo Logic, and is organized according to the individual areas of the Kubernetes architecture.

Navigating your Kubernetes environment

Sumo Logic allows you to monitor and troubleshoot your applications in Kubernetes using an intuitive mental model of Kubernetes hierarchies, instead of the server-based focus.

We currently provide four hierarchical Views of the Kubernetes system: Node, Deployment, Service, and Namespace. These views make it easy to traverse your Kubernetes hierarchy to monitor specific components, identify problems, discover root causes, and take progressive action.

• Node: Observe the infrastructure topology of resources (e.g., physical nodes, VMs) on your private cloud, public cloud or bare metal.
• Deployment: Observe how your Kubernetes deployment(s) perform against your specified criteria and manage changes.
• Service: Observe how your Kubernetes Service(s) interacts with your other services within and outside your cluster.
• Namespace: Track environments with many users spread across multiple teams, or projects like dev, lab, and prod.

These intuitive hierarchies can be accessed from the Kubernetes views.

Open Kubernetes views

prerequisites

To start monitoring your Kubernetes environment, you'll need to set up data collection and core dashboards installation (learn more).

Kubernetes views provide a visual representation of your Kubernetes stack.

To navigate and analyze your Kubernetes environment, do the following:

1. Classic UI. Go to the Home screen and select Explore.
   New UI. In the main Sumo Logic menu, select Observability, and then under Infrastructure Monitoring, select Kubernetes. You can also click the Go To... menu at the top of the screen and select Kubernetes.
2. In the upper-left corner of the screen, select the Kubernetes view you want from the dropdown menu.
3. Click the arrow to the left of a content name to expand and view its contents.
   
4. Dead entities are shown faded:
   
5. Drill down into the clusters to view the pods and containers. The data for your selection is displayed in the panels of the dashboard on the right.
   
6. Optional: Select another type of dashboard display from the dropdown menu at the top of the dashboard, and select another time interval.
   

Drilling into related content 

Sumo Logic provides relevant log searches and dashboards to consider investigating, as well as other locations with relevant content. This facilitates quickly discovering the root cause and devising a plan of action.

To discover and view related content, select the graph data point you are interested in. In the following screenshot, we selected a cell in a honeycomb chart. A panel appears on the right of the window with details and a list of related content links.

Summary tab

Select links from the Summary tab, to go directly to:

• Linked dashboards
• Recommended dashboards - Currently, only dashboards from the Kubernetes App are supported.

Custom dashboards

You can make your own custom dashboards to show up in the view by using the stack linking capability within Dashboards.

In order to link a dashboard to an existing Kubernetes hierarchy, your entity key in Stack Linking must be the same as the explored entity on the view.

For example, if you want you custom Kubernetes deployment dashboard to show up in explorer, you will have to add the following entities in the stack linking (as shown below), since those entities are used as filters on the view:

• Cluster
• Namespace
• Deployment

Once you have created the stack linking, your custom dashboard should appear in the list of dashboard associated with a specific entity view in explorer.

Using metadata to power your search

You can create fields with key-value pairs that label logs with custom metadata. Referencing log data with fields based on meaningful associations makes searches easier and more intuitive. Sumo Logic allows you to add custom fields to collectors that define key-value pairs at the source level. The custom fields in the metadata streams are then automatically extracted for searching, querying, and graphing. This allows you to view results for intuitively referenced subsets not traditionally tagged as source categories. 

This page shows you how to define a custom field on a collector, and then how to effectively use the custom metadata to search log data.

tip

To learn more about adding meaningful information to your data so you have more control and an easier time referencing data in searches, see the Data Enrichment and Fields pages.

Adding custom fields to collectors

You can add custom fields to collectors for more intuitive searches, partitions, and Role-Based Access Control (RBAC) queries. After which, the log data that passes through the collector automatically inherits the custom metadata. You can create a custom field label for anything that is "collected" and adapt your logs to familiar naming conventions. 

note

Custom metadata is usually set up by your administrator. Check with your site administrator before adding a custom key-value pair to a collector.

The following task shows you how to create a custom field for a collector. In this process, you assign a custom key-value pair in the field to tag the metadata. In this example, we create two fields with a key-value pairs - one for a cluster and one for a pod. 

To add a custom field to a collector, do the following:

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > Collection.
   New UI. In the top menu select Configuration, and then under Data Collection select Collection. You can also click the Go To... menu at the top of the screen and select Collection.
2. Select the collector to which you want to add a custom key-value pair. In this example, we've selected the Falco collector.
   
   The Edit Collector dialog appears.
3. Click Add Field.
   
4. Enter a Field Name and Value in the respective text fields. In this example, we created a field for a cluster with the label k8s.dev and a pod with the name pod_test and label k8s.test. This allows you to easily search for log data for that cluster or pod.
   •  A green circle with a check mark appears when the field exists and is enabled in the Fields table schema.
   •  An orange triangle with an exclamation point appears when the field doesn't exist yet, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema or is disabled it is ignored, known as dropped.
     
5. Click Save.

Now, any logs sent to this Collector will have these key-value pairs associated with it. With this association, you can search for cluster=k8s.dev or pod_test=k8s.test to return your logs.

Leveraging metadata for quicker results

In this section, you'll learn how to use metadata to search by components of the Kubernetes environment, such as containers, pods, and namespaces for localized investigation and analysis. You will also use metadata set with key-value pairs to effectively find the log data, and display Kubernetes labels and view the respective data in your query results.

To use metadata to view Kubernetes components and display Kubernetes label results, do the following:

1. Classic UI. Go to the Home screen and select Log Search.
   New UI. In the main Sumo Logic menu, select Logs > Log Search. You can also click the Go To... menu at the top of the screen and select Log Search.
2. Indicate the metadata namespace. In this example, we entered namespace=sumologic.
   
3. Click Start to run the query, then under Hidden Fields on the Messages tab, click namespace to display the metadata for that Kubernetes component. Notice that the namespace field moves Hidden Fields to Display Fields.
   
4. To view metadata for a key-value pair, enter the key-value pair in the query text field. In our example, we wanted to view the metadata for the prometheus container and entered container=prometheus.
5. Then we expanded the search range by changing the time interval from the last 15 minutes to the Last 60 minutes.
6. To further investigate the container, we clicked Logreduce, which groups common log messages into signature groupings. 
7. To examine the details of the smaller set of signatures that appear, under Select Count we selected. Oftentimes when troubleshooting a problem, our lesser quantity contains the root cause.
   
   The warning Endpoints ended with: too old resource version may be something to investigate or just the indication of an ongoing upgrade.
8. To view data for other Kubernetes components, we can enable them one by one by selecting the checkbox next to namespace, cluster, container, pod, service, and/or Source Host.

note

If this was an ongoing upgrade, the Source Host ip address would help pinpoint the servers that may still need upgrading.