Google Cloud

The Google Cloud integration connects an organization, a folder or a single project to Maple. You choose what each connection collects:

There is no OAuth step and no service account key. You run a generated gcloud script in Cloud Shell, and Maple gets no write access to Google Cloud.

If your workloads also send telemetry over OpenTelemetry, read Google Cloud with OpenTelemetry: it covers which telemetry to send over OpenTelemetry and which to collect here, and why GKE container logs are left out by default.

Prerequisites

• You are an admin of the Maple organization.
• You can sign in to Cloud Shell with the roles for what you connect:

If you can’t get Owner, see Roles for running the script.

The host project holds Maple’s Pub/Sub topic, subscription and read-only service account. A project is its own host project. For a folder or organization, name a shared operations project inside it that won’t be deleted. Google bills the Pub/Sub and Cloud Monitoring usage to the host project.

Metrics and resources needs billing enabled on the host project: Cloud Monitoring only answers for projects that have a billing account. The script checks this before it changes anything.

Find your IDs

Connect

1. Open Integrations → Google Cloud in Maple.
2. Under What to connect, choose Organization, Folder or Project and enter its ID. An organization or folder covers every project in it, including new ones.
3. For an organization or folder, enter the Host project ID.
4. Under What to collect, leave Log forwarding and Metrics and resources ticked, or untick one. Click Get setup script.
5. Click Open Cloud Shell and sign in with the roles above. Click Authorize if Cloud Shell asks.
6. Click Copy script, paste it into Cloud Shell and press Enter. The script takes about a minute. To change which logs are forwarded, choose a log filter before you copy.
7. Return to Maple. Maple confirms the connection shows a check mark within a minute of the script ending, usually in seconds.

A first run for a project prints:

Maple setup for project acme-prod
  Log forwarding          on
  Metrics and resources   on

Checking access
  ✓ Signed in as jane@acme.com
  ✓ Project acme-prod found (Acme Production)
  ✓ Billing is enabled
  ✓ jane@acme.com has the permissions this script needs
  ✓ Log filter accepted

Log forwarding
  ✓ APIs enabled (Pub/Sub, Cloud Logging, Cloud Resource Manager)
  ✓ Topic created
  ✓ Push subscription created
  ✓ Log sink created
  ✓ Sink allowed to write to the project
  ✓ Sink allowed to publish to the topic
  ✓ Check message sent to Maple through the topic

Metrics and resources
  ✓ APIs enabled (Cloud Monitoring, Cloud Asset, IAM, IAM Credentials, Cloud Resource Manager)
  ✓ Read-only service account created
  ✓ Read-only roles granted (Monitoring Viewer, Cloud Asset Viewer, Service Usage Consumer)
  ✓ Maple allowed to read as that account
  ✓ Maple notified

Done. Google Cloud is set up for Maple.
  Maple confirms it within a minute: https://app.maple.dev/integrations?integration=gcp
  Logs:    a new sink can take about 10 minutes to start forwarding. What is logged before
           it does is not forwarded later.
  Metrics: the first read lands within about 10 minutes.

If a step fails, the script stops and prints what to do. Fix it and paste the script again: it continues where it stopped. Running the script again is safe, and its last line says what the run did:

After the check mark, each switch waits for its first data:

The script contains a secret that lets anyone send logs to your Maple organization. Don’t share it or commit it, and run it in Cloud Shell: there neither the paste nor the secret ends up in shell history. zsh keeps the whole paste, secret included. If you paste the script into zsh on your own machine, remove the entry from your shell history afterwards.

The push endpoint with the secret is also visible in the subscription’s configuration and in the host project’s Admin Activity audit log, to anyone who can read those. If it leaks, disconnect and connect again: the new connection gets a new secret.

The first two lines and the last two lines of the pasted text run the script in a bash process of its own, so a failed step can’t close your Cloud Shell session.

A project, folder or organization can be connected once per Maple organization. A project that sits inside a connected organization or folder is collected twice if you also connect it on its own.

Change what a connection collects

1. Flip Log forwarding or Metrics and resources on the connection. Maple saves the switch at once.
2. The connection reads Changes pending and lists what the script will create or remove in Google Cloud. A connection whose setup script has not run yet reads Setup pending instead.
3. The setup panel opens with the updated script. Copy it and run it in Cloud Shell again. If the panel is closed, click Show setup script. The notice goes away when the run reports to Maple.

A run that switches Log forwarding off takes one to two minutes: after it deletes the sink it waits a minute for Google to stop routing to the topic, so your project logs no sink error.

Until the script runs, Google Cloud keeps what the last run set up:

A connection keeps one switch on. To stop collecting, disconnect.

Log filter

The setup panel’s Log filter decides which filter the script writes onto the sink:

Include GKE container logs only for workloads that don’t send their logs to Maple over OpenTelemetry. Otherwise each line is stored twice: see GKE container logs.

For any other filter, paste the script into an editor first, edit LOG_FILTER near its top and set LOG_FILTER_MODE to set. The filter uses the Logging query language. The script checks the filter with Google before it creates anything. Later runs with Keep the sink’s current filter leave your filter in place, while the other two options replace it.

Verify

Each connection shows a status per switch. The page updates on its own.

The page header and the Integrations list show the connection’s worst status: Needs attention, Setup pending, Changes pending, Waiting for data or Healthy.

Then check the data:

1. Open Logs and look for a service named after one of your Cloud Run services, functions or instances.
2. In the metrics explorer, search for the prefix gcp.
3. Under Dashboards → Browse templates, select Google Cloud and click Create dashboard. To chart one project only, first fill in Project ID under Parameters, below the preview.
4. Open Infrastructure → Google Cloud for a table of your workloads per service.

Permissions

The script enables these APIs in the host project:

• pubsub.googleapis.com and logging.googleapis.com, for log forwarding: carrying the log entries and routing them through the sink.
• monitoring.googleapis.com, cloudasset.googleapis.com, iam.googleapis.com and iamcredentials.googleapis.com, for metrics and resources: reading metrics, listing resources, creating the read-only service account and minting its short-lived tokens. No key is ever created.
• cloudresourcemanager.googleapis.com, for both: granting the roles below.

For log forwarding it creates a Pub/Sub topic and push subscription in the host project and a log sink on the project, folder or organization. For metrics and resources it creates a read-only service account in the host project. Each is named maple- followed by 24 hexadecimal characters unique to the connection.

It grants these roles:

• Pub/Sub Publisher (roles/pubsub.publisher) to the sink’s Google-managed writer identity, on Maple’s topic: publishing log entries to the topic.
• Logs Writer (roles/logging.logWriter) to the same writer identity, on the host project: Google requires it on the project that holds a sink’s destination.
• Monitoring Viewer (roles/monitoring.viewer) to the read-only service account, on the project, folder or organization: reading metrics (monitoring.timeSeries.list).
• Cloud Asset Viewer (roles/cloudasset.viewer) to the read-only service account, on the project, folder or organization: listing resources (cloudasset.assets.searchAllResources). The role can also read resource metadata and IAM policies.
• Service Usage Consumer (roles/serviceusage.serviceUsageConsumer) to the read-only service account, on the host project: calling both read APIs through the host project.
• Service Account Token Creator (roles/iam.serviceAccountTokenCreator) to Maple’s service account, on the read-only service account: minting short-lived tokens for that one account.

roles/monitoring.viewer and roles/cloudasset.viewer are the narrowest predefined roles for these calls, and both are read-only. On a folder or organization, every project under it inherits them.

Roles for running the script

Owner on the host project is the simplest. Without it, these predefined roles together cover what the script does:

Before it changes anything, the script asks Google which permissions the signed-in account holds and stops with the missing ones.

Collected data

Logs

Maple receives every entry the sink’s filter lets through. The recommended filter excludes:

The filter, one clause per line in the same order:

NOT log_id("cloudaudit.googleapis.com/data_access")
AND NOT httpRequest.userAgent:"GoogleHC"
AND NOT protoPayload.methodName="io.k8s.coordination.v1.leases.update"
AND NOT logName:"serialconsole.googleapis.com"
AND NOT resource.type="k8s_container"

Include GKE container logs sets the filter without the last clause. To forward different logs, see Log filter.

The script reports to Maple through a log named maple-setup. Maple reads that entry as the report and does not store it, so don’t write your own logs under that name.

Each entry gets its service.name from the resource that wrote it, so workload logs join the traced service of the same name:

Every entry also carries cloud.provider (gcp), gcp.resource.type and the resource’s labels as gcp.resource.labels.*, plus cloud.account.id (the project ID) and cloud.region when the resource has them. GKE container entries add k8s.cluster.name, k8s.namespace.name, k8s.pod.name and k8s.container.name.

The log body is textPayload, or the message or msg field of jsonPayload, or the whole jsonPayload as JSON when it has neither. The other jsonPayload fields become log attributes. Audit logs use the method name as the body, and request logs the method, URL and status. An entry’s trace and spanId become the log’s trace and span IDs.

Metrics

Maple stores these Cloud Monitoring metrics at one-minute resolution. The name is the Cloud Monitoring type under a gcp. prefix: run.googleapis.com/request_count becomes gcp.run.request_count.

Value conventions:

• Counters are delta sums per minute. Chart them with sum, not rate.
• Latency and other distributions are gauges with a quantile attribute (0.5, 0.95, 0.99).
• Utilization is a fraction from 0 to 1.
• Latencies and execution times are in milliseconds.

Service names follow the same rule as logs: Cloud Run services, functions, GKE containers and Compute Engine instances use the workload name, and everything else is gcp/<resource type>. Narrow further by resource attribute:

Every series also carries cloud.account.id (the project ID) and, for resources with a location, cloud.region. Google aggregates away every other label, such as revision, pod, response code and device, before the data reaches Maple. These metric labels are kept as attributes:

Resources

Every hour Maple lists these resources from Cloud Asset Inventory, with their project, location, state and labels: projects, Cloud Run services and jobs, Cloud Functions, GKE clusters, Compute Engine instances, Cloud SQL instances, Pub/Sub topics and subscriptions, and load balancer URL maps, backend services and forwarding rules.

Infrastructure → Google Cloud

Infrastructure → Google Cloud is in the sidebar and on the Infrastructure overview while a connection has Metrics and resources switched on. The overview row names the services that reported in the time range. The page has one tab per service that reported metrics in the selected time range, with one row per workload. The tab you are on stays when you change the range. Counts and byte totals cover the time range; everything else is the average over it.

Search a tab by name, project or location, and click a column to sort by it. A dash means the workload did not report that metric. CPU below one core is written in millicores: 250m is a quarter of a core. Cloud Functions (2nd gen) run on Cloud Run and appear on the Cloud Run tab. GKE node and Pub/Sub topic metrics are collected but have no tab: chart them in a dashboard.

The Resources tab shows the first 500 resources that match its filters, and how many projects Maple found. It does not depend on the time range.

Until a connection’s setup script has run, the page reads Finish setting up Google Cloud. After the run it reads Collecting your first Google Cloud metrics until the first read lands, within about 10 minutes.

With Metrics and resources switched off on every connection, the page leaves the sidebar and the overview. Its address still opens it: for a time range with data it shows the service tabs with what Maple collected before, under the notice Google Cloud metrics are switched off, and otherwise it reads Turn on metrics for Google Cloud. The Resources tab is back once a connection collects again.

Google Cloud costs

Google bills Pub/Sub and Cloud Monitoring usage to your account, separately from your Maple plan.

Google bills Pub/Sub for the log entries that pass through the topic and the subscription. A narrower LOG_FILTER lowers it. See Pub/Sub pricing.

Google bills Cloud Monitoring API reads to the host project. Each connection runs 46 timeSeries.list queries every 5 minutes, and the cost grows with the number of time series in the project, folder or organization. See Google Cloud Observability pricing.

Cloud Asset Inventory searches are free of charge. See Cloud Asset Inventory pricing.

Limits

• Pub/Sub delivers one log entry per HTTP request, at least once. A redelivered entry is stored again. Its Cloud Logging insertId is in the log.record.uid attribute.
• The subscription keeps undelivered entries for one day. Entries Maple could not accept within that day are lost.
• Connecting an organization and a folder or project inside it collects that part twice. Each connection has its own sink and its own metric reads.
• Metrics arrive every 5 minutes, 5 minutes behind. The first read, and the first read after a pause, covers at most the last hour.
• One read takes up to 20,000 data points per metric, about 4,000 series, and up to 2 minutes in total. Past that, the connection says how many metric queries were cut short. For a folder or organization, connect its folders or projects separately to collect all of it.
• Maple reads up to 10 connections of one Maple organization every 5 minutes. With more, they take turns.
• The resource list holds up to 10,000 resources per connection.
• Load balancer metrics cover global external Application Load Balancers only.

Troubleshooting

In Cloud Shell

A step that fails stops the script. It prints what went wrong, Google’s answer, and a line that starts with What to do:. Nothing needs undoing: fix it and paste the script again. A connection’s ID can’t be changed, so a wrong ID is fixed by removing the connection in Maple and connecting the right ID. In the lines below, account stands for the account you are signed in as.

When you create a connection in Maple, “The Google Cloud project acme-prod is already connected.” means this Maple organization already has a connection for it. Change that connection’s switches instead.

Log forwarding

Metrics and resources

Can’t read metrics and Receiving metrics, incomplete come with one of the messages below. The first means no read has succeeded in the last 10 minutes. The second means the last read is under 10 minutes old and something is missing. The messages about the resource list show under a green Receiving metrics. Where a project’s message says “the project”, a folder’s or organization’s says “the host project”.

A message that ends in parentheses, such as “(Cloud Monitoring returned 403, BILLING_DISABLED)”, quotes Google’s answer. Include it when you write to support.

On a self-hosted Maple, the Metrics and resources switch reads Not available on this Maple deployment until MAPLE_GCP_SERVICE_ACCOUNT_EMAIL names a Google service account the deployment owns. Reading metrics also needs MAPLE_GCP_SERVICE_ACCOUNT_KEY, that account’s key file, base64-encoded.

Domain restricted sharing

An organization policy can restrict which identities may hold IAM roles in your organization (constraints/iam.allowedPolicyMemberDomains, or a custom constraint on member domains). Maple’s service account lives outside your organization, so such a policy can reject a grant the script makes, most often the one that lets Maple read as your read-only service account. The script then stops with “An organization policy (domain restricted sharing) blocks this grant.”

An Organization Policy Administrator can lift the policy for the host project while the script runs:

1. In the Google Cloud console, open IAM & Admin → Organization Policies with the host project selected.
2. Open Domain restricted sharing, click Manage policy, choose Override parent’s policy and add a rule that allows all. This applies to the host project only.
3. Paste the setup script again. It continues where it stopped.

Google checks the policy when a grant is made, so the grant stays in place if you restore the policy afterwards.

Disconnect

1. On Integrations → Google Cloud, click Disconnect on the connection.
2. Click Copy cleanup script and run it in Cloud Shell. It deletes the log sink, topic, subscription and read-only service account, and ends with “Done. Everything the setup script created is gone.”
3. Wait for the check mark in the dialog, then click Disconnect. Maple stops reading the connection’s metrics, and stops accepting its logs within about a minute. Data already in Maple is kept.

The cleanup takes one to two minutes when the connection forwards logs: after it deletes the sink it waits a minute for Google to stop routing to the topic, so your project logs no sink error. Switching Log forwarding off and running the setup script takes as long, for the same reason.

Disconnect anyway disconnects without the cleanup. Google Cloud then keeps publishing logs to Pub/Sub, billed by Google, until the cleanup script runs. Maple keeps a panel with Copy cleanup script on the page until you click Done.

The cleanup script leaves the APIs it switched on enabled, and the Logs Writer role of Google’s logging service account on the host project, which other sinks share.

A connection whose setup script Maple never saw run has a Remove button instead and asks once. Maple still offers the cleanup script afterwards, in case the setup script ran part of the way.

Next steps

• Google Cloud with OpenTelemetry: what to instrument on GKE, Cloud Run and Compute Engine.
• Logs: search Google Cloud logs next to your application logs.
• Dashboards: start from the Google Cloud template.
• Infrastructure → Google Cloud: scan every workload of a service in one table.
• Alert rules: alert on any gcp.* metric.
• API reference: manage connections with the /v2/integrations/gcp endpoints.