Google Cloud Storage

The Sumo Logic app for Google Cloud Storage helps you monitor both logs and metrics in Google Cloud Storage. The preconfigured dashboards provide insight into bucket storage, request operations, data transfer, authentication patterns, performance metrics, and optimization opportunities for your Cloud Storage resources.

Log and metric types

The App uses:

• Google Cloud Audit Logs - Logs events on multiple GCP services, including Cloud Storage.
• Google Cloud Metrics for Cloud Storage - Provides performance and usage metrics for Google Cloud Storage.

Sample log message

{
  "message": {
    "data": {
      "logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity",
      "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {
          "principalEmail": "user@example.com"
        },
        "methodName": "storage.objects.create",
        "resourceName": "projects/_/buckets/my-bucket/objects/my-file.txt",
        "serviceName": "storage.googleapis.com"
      },
      "resource": {
        "labels": {
          "bucket_name": "my-bucket",
          "project_id": "my-project"
        },
        "type": "gcs_bucket"
      }
    }
  }
}

Sample metric message

{
  "_source":"google-cloud-storage-metrics",
  "cloud.platform":"gcp_storage",
  "_metricId":"4F7HG9wrLEJvzydQF-DlQQ",
  "location":"us-central1",
  "raw_metric":"storage.googleapis.com/storage/object_count",
  "Statistic":"Average",
  "project_id":"my-project", 
  "metric":"storage/object_count",
  "bucket_name":"example-bucket",
  "cloud.provider":"gcp",
  "max":15420,
  "min":15400,
  "avg":15410.5,
  "sum":30821,
  "latest":15420,
  "count":2
}

Sample log queries

_sourceCategory=*gcp* data logName resource "\"type\":\"gcs_bucket\""
| parse regex "\"logName\":\"(?<log_name>[^\"]+)\""
| where log_name matches "projects/*/logs/cloudaudit.googleapis.com%2*"
| json "message.data.resource.labels", "message.data.protoPayload.methodName" as labels, method
| where method matches "*create" or method matches "*delete"
| json field=labels "project_id", "bucket_name", "location" as project, bucket_name, location
| timeslice 1h
| count as operations by _timeslice, method
| transpose row _timeslice column method
| fillmissing timeslice(1h)

Sample metric queries

cloud.provider=gcp project_id=* metric=storage/object_count statistic=average 
| quantize using sum
| sum by project_id, bucket_name

Collect logs for Google Cloud Storage

This section describes the Sumo Logic pipeline for ingesting logs from Google Cloud Platform (GCP) services, and explains how to collect logs from Google Cloud Storage.

Collection Process for GCP Services

The key components in the collection process for GCP services are Google Logs Export, Google Cloud Pub/Sub, and Sumo's Google Cloud Platform (GCP) source running on a hosted collector.

The GCP service generates logs that are exported and published to a Google Pub/Sub topic via the Google Cloud Logging Log Router. You will then set up a Sumo Logic Google Cloud Platform source to subscribe to this topic and receive the exported log data.

Configuring collection for GCP

Follow the steps below to configure the collection for GCP:

1. Configure a GCP source on a hosted collector. You'll obtain the HTTP URL for the source.
2. Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.
3. Create an export of GCP logs from Google Log Router. Exporting involves writing a filter that selects the log entries you want to export and choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink.

Refer to the following sections for configuration instructions.

note

Logs from GCP services can be exported to any destination. Any GCP logs can be excluded from the Logs router.

Configure a Google Cloud Platform Source

The Google Cloud Platform (GCP) Source receives log data from Google Pub/Sub.

note

You can use the same GCP Source to receive log data from multiple GCP services. For example, you can send logs collected from Google Cloud Application Engine, Google Cloud IAM, and Google Cloud Audit.

However, this is not recommended since you cannot define specific Source Category values for each GCP service. If you create a GCP Source for each service you can define a specific Source Category for each service.

This Source will be a Google Pub/Sub-only Source, indicating that it will only be usable for log data formatted as data coming from Google Pub/Sub.

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Collection > Collection.
   New UI. In the Sumo Logic top menu select Configuration, and then under Data Collection select Collection. You can also click the Go To... menu at the top of the screen and select Collection.

2. Select an existing Hosted Collector upon which to add the Source. If you do not already have a Collector you'd like to use, create one, using the instructions on Configure a Hosted Collector.

3. Click Add Source next to the Hosted Collector and click Google Cloud Platform.

4. Enter a Name to display for the Source. A Description is optional.
   

5. Source Host (Optional). The Source Host value is tagged to each log and stored in a searchable metadata field called _sourceHost. Avoid using spaces so you do not have to quote them in keyword search expressions. This can be a maximum of 128 characters.

6. Source Category (Optional). The Source Category value is tagged to each log and stored in a searchable metadata field called _sourceCategory. See our Best Practices: Good Source Category, Bad Source Category. Avoid using spaces so you do not have to quote them in keyword search expressions. This can be a maximum of 1,024 characters.

7. Fields. Click the +Add Field link to add custom log metadata Fields, then define the fields you want to associate. Each field needs a name (key) and value. Look for one of the following icons and act accordingly:

• If an orange triangle with an exclamation point is shown, use the option to automatically add or enable the nonexistent fields before proceeding to the next step. The orange icon indicates that the field doesn't exist, or is disabled, in the Fields table schema. If a field is sent to Sumo that does not exist in the Fields schema or is disabled it is ignored, known as dropped.
• If a green circle with a checkmark is shown, the field exists and is already enabled in the Fields table schema. Proceed to the next step.

8. Advanced Options for Logs.
   

• Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
• Time Zone. There are two options for the Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs cannot be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
• Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.

9. Processing Rules. Configure any desired filters, such as allowlist, denylist, hash, or mask, as described in Create a Processing Rule.

10. When you are finished configuring the Source, click Save.

Configure a Pub/Sub Topic for GCP

You need to configure a Pub/Sub Topic in GCP and add a subscription to the Source URL that belongs to the Sumo Logic Google Cloud Platform Source you created. Once you configure the Pub/Sub, you can export data from Google Logging to the Pub/Sub. For example, you can export Google App Engine logs, as described on Collect Logs for Google App Engine.

1. Create a Pub/Sub Topic in GCP. Refer to the Google Cloud documentation for the latest configuration steps.

2. Create a Pub/Sub subscription to the Source URL that belongs to the Sumo Logic Google Cloud Platform Source you created. See Google Cloud documentation for the latest configuration steps.

   • Use a Push Delivery Method to the Sumo Logic Source URL. To determine the URL, navigate to the Source on the Collection page in Sumo Logic and click Show URL.

Limitations

Google limits the volume of data sent from a Topic. Our testing resulted in the following data limits:

Topics	Megabytes per second	Payload size
One	18 MBps (1.5 TB/day)	100 KB
One	6 MBps (0.5 TB/day)	2.5 KB

note

These limits may vary based on your setup and are based on our previous tests.

We recommend the following:

• Shard messages across topics within the above data limits.
• Ask GCP to increase the allowable capacity for the topic.

Create export of Google Cloud Storage logs from Google Logging

In this step, you export logs to the Pub/Sub topic you created in the previous step.

1. Go to Logging and click Logs Router.
   

2. Click Create Sink.
   

3. As part of Create logs routing sink, add the following information.

4. Enter a Sink Name. For example, "gce-vm-instance".

5. Select "Cloud Pub/Sub" as the Sink Service.

6. Set Sink Destination to the Pub/Sub topic you created in the Google Cloud Platform Source procedure. For example, "pub-sub-logs".

7. In Choose logs to include in sink section for resource_type, replace "<resource_variable>" with "gcs_bucket".

8. Click Create Sync.

note

By default, GCP logs are stored within Cloud Logging, but you can configure Log Router to exclude them as detailed here without affecting the export to Sumo Logic as outlined above.

Collecting metrics for the Google Cloud Storage app

For metrics collection in Sumo Logic, use the GCP Metric source.

1. Set up the Google Service Account.

2. Set up a GCP Metric source in Sumo Logic. While setting up the source, select Cloud Storage as the service from the dropdown to get the Google Cloud Storage metrics.

Installing the Google Cloud Storage App

Now that you have set up the collection for Google Cloud Storage, install the Sumo Logic App to use the pre-configured searches and dashboards that provide visibility into your environment for real-time analysis of overall usage.

To install the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, Manage Collectors capability depending upon the different content types part of the app.

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Install App.
   note

   Sometimes this button says Add Integration.

4. Click Next in the Setup Data section.
5. In the Configure section of your respective app, complete the following fields.
   1. Field Name. If you already have collectors and sources setup, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value .
6. Click Next. You will be redirected to the Preview & Done section.

Post-installation

Once your app is installed, it will appear in your Installed Apps folder, and dashboard panels will start to fill automatically.

Each panel slowly fills with data matching the time range query and received since the panel was created. Results will not immediately be available, but will update with full graphs and charts over time.

Viewing Google Cloud Storage dashboards

All dashboards have a set of filters that you can apply to the entire dashboard. Use these filters to drill down and examine the data to a granular level.

• You can change the time range for a dashboard or panel by selecting a predefined interval from a drop-down list, choosing a recently used time range, or specifying custom dates and times. Learn more.
• You can use template variables to drill down and examine the data on a granular level. For more information, see Filtering Dashboards with Template Variables.
• Most Next-Gen apps allow you to provide the scope at the installation time and are comprised of a key (_sourceCategory by default) and a default value for this key. Based on your input, the app dashboards will be parameterized with a dashboard variable, allowing you to change the dataset queried by all panels. This eliminates the need to create multiple copies of the same dashboard with different queries.

Google Cloud Storage - Operational Overview

Provide a high-level view of Google Cloud Storage activity, focusing on operation volumes, types, and key contributors across buckets and users/service accounts.

Google Cloud Storage - Access and Security

See information about access patterns, security configurations (IAM changes), and potential security events related to Google Cloud Storage.

Google Cloud Storage - Object Activity

See information about object-level interactions within Google Cloud Storage. Requires Data Access Logs to be enabled.

Google Cloud Storage - Error and Access Denied Analysis

Analyze errors and specifically permission denied events within Google Cloud Storage to identify failure patterns, problematic operations, and potential security issues.

Google Cloud Storage - Usage and Activity Overview

See information about high-level summary of storage consumption, data transfer volumes, and overall API activity for quick health checks and usage monitoring.

Create monitors for Google Cloud Storage app

Google Cloud Storage - Operations and Error Analysis

See information about key Google Cloud Storage usage metrics, operational activity, and performance indicators, including storage size, egress, API requests, latency, and errors, with breakdowns by various dimensions like storage class, location, object type, and operation method.

Google Cloud Storage - Storage Optimization and Authorization

See information about key Google Cloud Storage bucket optimization opportunities, including the storage class distribution, object lifecycle management, and authorization patterns to identify potential cost savings and security improvements.

From your App Catalog:

1. From the Sumo Logic navigation, select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. Make sure the app is installed.
4. Navigate to What's Included tab and scroll down to the Monitors section.
5. Click Create next to the pre-configured monitors. In the create monitors window, adjust the trigger conditions and notifications settings based on your requirements.
6. Scroll down to Monitor Details.
7. Under Location click on New Folder.
   note

   By default, monitor will be saved in the root folder. So to make the maintenance easier, create a new folder in the location of your choice.

8. Enter Folder Name. Folder Description is optional.
   tip

   Using app version in the folder name will be helpful to determine the versioning for future updates.

9. Click Create. Once the folder is created, click on Save.

Google Cloud Storage alerts

Name	Description	Alert Condition	Recover Condition
Google Cloud Storage - Bucket Deleted	This alert is triggered when a GCS bucket deletion event is detected (Threshold: > 0 events over 5 minutes), indicating a potentially critical data loss scenario requiring immediate investigation.	Count > 0	Count < = 0
Google Cloud Storage - Bucket Lifecycle Configuration Changed	This alert is triggered when a bucket's Object Lifecycle Management configuration is created or modified (Threshold: > 0 events over 1 hour), which could impact data retention or storage costs.	Count > 0	Count < = 0
Google Cloud Storage - Bucket Retention Policy Locked	This alert is triggered when a bucket's retention policy is locked (Threshold: > 0 events over 15 minutes), an irreversible action critical for compliance.	Count > 0	Count < = 0
Google Cloud Storage - High Class A Operation Rate	This alert is triggered when the count of Class A operations (e.g. create, delete, or modify etc.) for any bucket exceeds the defined threshold (Default: 10,000 operations over 1 hour), potentially indicating high API costs.	Count > 10000	Count < = 10000
Google Cloud Storage - High Client Error Count	This alert is triggered when the number of client-side errors (e.g., Not Found, Forbidden) for any bucket exceeds the defined threshold (Default: 20 errors over 5 minutes), indicating potential application or permission issues.	Count > 5	Count < = 5
Google Cloud Storage - High Egress Volume	This alert is triggered when the total data downloaded (egress) from any bucket exceeds the defined threshold (Default: 10 GB over 15 minutes), potentially indicating high costs or unusual activity.	Count > 10737418240	Count < = 10737418240
Google Cloud Storage - High Object ACL Mutation Count	This alert is triggered when modifications to specific Object ACLs occur frequently and exceed the defined threshold (Default: 5 mutations over 15 minutes), which might indicate non-standard permission management.	Count > 5	Count < = 5
Google Cloud Storage - High Server Error Count	This alert is triggered when the number of server-side errors (e.g., Internal Server Error) occurs and exceeds the defined threshold (Default: 5 errors over 5 minutes), potentially indicating GCS platform issues or throttling.	Count > 5	Count < = 5

Upgrade/Downgrade the Google Cloud Storage app (Optional)

To update the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, Manage Collectors capability depending upon the different content types part of the app.

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
   Optionally, you can identify apps that can be upgraded in the Upgrade available section.
3. To upgrade the app, select Upgrade from the Manage dropdown.
   1. If the upgrade does not have any configuration or property changes, you will be redirected to the Preview & Done section.
   2. If the upgrade has any configuration or property changes, you will be redirected to Setup Data page.
      1. In the Configure section of your respective app, complete the following fields.
         1. Field Name. If you already have collectors and sources setup, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value .
      2. Click Next. You will be redirected to the Preview & Done section.

Post-update

Your upgraded app will be installed in the Installed Apps folder, and dashboard panels will start to fill automatically.

note

See our Release Notes changelog for new updates in the app.

To revert the app to a previous version, do the following:

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. To version down the app, select Revert to < previous version of your app > from the Manage dropdown.

Uninstalling the Google Cloud Storage app (Optional)

To uninstall the app, do the following:

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Uninstall.