Google Compute Engine is an Infrastructure as a Service (IaaS) component of Google Cloud Platform that delivers virtual machines running in Google's data centres and worldwide fibre network. The Sumo Logic app for Google Compute Engine helps you to monitor your infrastructure by providing preconfigured dashboards that allow you to view the activities, users, and message severity along with performance metrics monitoring CPU, network, disk, and quotas for your Google Compute Engine infrastructure.
Log and metrics types
The Sumo Logic app for Google Cloud Functions uses the following logs and metrics:
- Google Cloud Audit Logs for Compute Engine - These logs provide information about Compute Engine API calls and operations.
- Compute Engine Platform Logs
- Google Compute Engine Metrics
_collector="HTTP Source for GCP Pub/Sub" logName resource timestamp
| json "message.data.resource.type" as type
| parse regex "\s+\"logName\":\"(?<log_name>\S+)\""
| where type = "gce_instance" and log_name matches "projects/*/logs/cloudaudit.googleapis.com%2Factivity"
| parse regex "\s+\"resourceName\":\"projects/\S+/zones/(?<zone>\S+)/instances/(?<instance>\S+)\""
| json "message.data.resource.labels" as labels
| json field=labels "project_id" as project
| json "message.data.protoPayload.authenticationInfo.principalEmail" as user
| count as requests by user
| sort by requests
| limit 10
Sample metric query
project_id=* region=* cloud.platform=gcp_cloudsql database_id=* metric=database/cpu/utilization statistic=average | eval _value*100 | avg
Collecting logs for the Google Compute Engine
This page describes the Sumo pipeline for ingesting logs from Google Cloud Platform (GCP) services, and provides instructions for collecting logs from Google Compute Engine.
Collection process for GCP services
The key components in the collection process for GCP services are Google Logs Export, Google Cloud Pub/Sub, and Sumo’s Google Cloud Platform (GCP) source running on a hosted collector.
The GCP service generates logs which are exported and published to a Google Pub/Sub topic through Log Router. You will then set up a Sumo Logic Google Cloud Platform source that subscribes to this topic and receives the exported log data.
Configuring collection for GCP uses the following process:
- Configure a GCP source on a hosted collector. You'll obtain the HTTP URL for the source, and then use Google Cloud Console to register the URL as a validated domain.
- Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.
- Create an export of GCP logs from Google Log Router. Exporting involves writing a filter that selects the log entries you want to export, and choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink.
See the following sections for configuration instructions.
Configure a Google Cloud Platform source
The Google Cloud Platform (GCP) Source receives log data from Google Pub/Sub.
You can use the same GCP Source to receive log data from multiple GCP services. For example, you can send logs collected from Google Cloud Application Engine, Google Cloud IAM, and Google Cloud Audit.
However, this is not recommended since you cannot define specific Source Category values to each GCP service. If you create a GCP Source for each service you can define a specific Source Category to each service.
This Source will be a Google Pub/Sub-only Source, which means that it will only be usable for log data formatted as data coming from Google Pub/Sub.
- In Sumo Logic, select Manage Data > Collection > Collection.
- Select an existing Hosted Collector upon which to add the Source. If you don't already have a Collector you'd like to use, create one, using the instructions on Configure a Hosted Collector.
- Click Add Source next to the Hosted Collector and click Google Cloud Platform.
- Enter a Name to display for the Source. A Description is optional.
- Source Host (Optional). The Source Host value is tagged to each log and stored in a searchable metadata field called
_sourceHost. Avoid using spaces so you do not have to quote them in keyword search expressions. This can be a maximum of 128 characters.
- Source Category (Optional). The Source Category value is tagged to each log and stored in a searchable metadata field called
_sourceCategory. See our Best Practices: Good Source Category, Bad Source Category. Avoid using spaces so you do not have to quote them in keyword search expressions. This can be a maximum of 1,024 characters.
- Fields. Click the +Add Field link to add custom log metadata Fields, then define the fields you want to associate. Each field needs a name (key) and value. Look for one of the following icons and act accordingly:
- An orange triangle with an exclamation point is shown when the field doesn't exist in the Fields table schema. In this case, an option to automatically add the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema it is ignored, known as dropped.
- A green circle with a check mark is shown when the field exists in the Fields table schema.
- Advanced Options for Logs.
- Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
- Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
- Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
- Processing Rules. Configure any desired filters, such as allowlist, denylist, hash, or mask, as described in Create a Processing Rule.
- When you are finished configuring the Source, click Save.