On-call routing and incident coordination

Metrics → Incidents → On-call combines the response queue, routing plan, personal channel consent and temporary coverage. It attaches one coordination workspace to an original threshold, SLO, cron or browser incident. It does not create a second alert or replace the source's recovery rules.

Set up a rotation

1. Configure operational email, Slack or signed-webhook channels in Metrics → Channels. Review their recipients and destinations.
2. Each responder signs in with Google and a passkey, opens My channels, selects up to three channels that reach them, and records their own consent. Administrators cannot enroll another responder on their behalf. Channel names do not prove delivery; confirm recipients with the channel administrator.
3. Create one routing plan in the environment. Choose 1–12 distinct current workspace members in rotation order, an explicit ISO timestamp anchor, an IANA display timezone, and 1–168 elapsed hours per shift. For example, a 24-hour rotation anchored at 2026-03-07T14:00:00Z changes the following day at 14:00Z, even when local daylight saving time changes.
4. Select sources and severities. Threshold rules use their configured severity; fast SLO burn incidents are critical and slow burn incidents are warning; cron and browser incidents are warning. Cron routing also requires Errors read/write authority for the plan editor and every responder. Exactly one environment may own the application-wide fallback for threshold rules without an environment.
5. Review the optional private runbook, delivery allowance, and routing revision, then enable the plan. Routing begins with incidents opened after this revision was saved; there is no historical paging backfill.

Current workspace membership, active user status and Metrics read/write permissions are checked for the plan editor, responders and delivery claims. Workspace-managed suspension or stale directory authority denies participation. Browser monitor creator/owner authority is also checked. The console needs apps.read; Cron readers additionally need errors.read and acknowledgement needs errors.write.

Escalation and coverage

The worker evaluates up to 20 plans and 20 new incidents per plan per pass, with a 30-second schedule and fair plan cursors. At first observation it copies the plan revision and selects the primary responder for the source opening timestamp, followed by the next two distinct rotation members. One- or two-person rotations have correspondingly fewer stages. The second and third stage delays are increasing values from 1 to 240 minutes after source opening. No repeat paging cycles are created.

Each target captures their consent revision and the fingerprint of each destination, including the exact active vault credential revision. Unenrolled or unauthorized responders have no eligible destinations. Their missing consent is visible in the incident workspace; a later enrollment does not silently rewrite an existing incident's schedule.

An override replaces the primary for source incidents opened inside its interval. Overrides must be nonoverlapping, start now or later, last at most seven days, and end within the next 90 days. There are at most 20 active/upcoming overrides. They affect newly captured coordination; removing or adding an override does not re-page existing incidents.

Stages expire when the next stage becomes due; the final stage expires after one hour. Late workers skip obsolete stages instead of sending a burst. Provider calls are bounded to ten seconds; dispatch, including vault unwrap and its guards, stops waiting after fifteen seconds. A delayed unwrap cannot send after that deadline. A destination is attempted at most once per stage. An uncertain provider response, process interruption or expired claim is recorded as unknown, without automatic retry. These semantics favor avoiding duplicate pages; they do not provide exactly-once external delivery.

Reservations are shared by all environments in an application and reset at midnight UTC. Each plan may claim only while shared usage is below its configured 1–1,000 daily limit; 1,000 is the application ceiling. Reservations are not refunded on failure. This allowance is separate from original source notifications.

Saving a plan revision, pausing it, revoking consent, disabling/changing a destination, losing workspace authority, acknowledging/resolving the source, or deleting source evidence prevents pending delivery. Guards run again after vault unwrap and immediately before dispatch. A provider request already in flight cannot be recalled. sent means the provider accepted the call, not that a human received it. With SPM_EMAIL_PROVIDER=log, the outcome explicitly says log_only; no email leaves the process.

Respond and learn

Open a source-linked workspace from the response queue or the Coordinate action. Acknowledge and own changes the original incident to acknowledged, records the owner, and cancels pending escalation. A different responder must deliberately Take ownership, with a reason and a current revision. Concurrent edits are rejected instead of overwriting ownership.

Resolve through the original threshold, SLO, Cron or Browser controls. Coordination observes external status transitions without inventing a recovery condition or an exact acknowledgement actor/time that the source did not retain. Acknowledgement alone does not establish recovery.

Runbooks are plain-text snapshots. Private timeline notes and structured report fields require people.pii.read for both reading and writing; mutations also require source write authority. Up to 200 normal timeline entries and 20 immutable report revisions are supported. Reports contain a summary, impact, contributing causes, and follow-up actions with owners. Drafts can be saved during an incident; publishing requires the original source to be resolved. This is a private report, not an external publication or message.

Notes, runbooks, source messages and report bodies are never interpolated into provider messages. Notifications contain static operational text, an escalation stage and an authenticated private console link. Audit events contain IDs and content hashes, not those bodies. Operational reasons are stored in the security audit, so keep sensitive incident content in private notes instead.

Coordination content is hidden immediately after its 90-day maximum or the application's shorter retention, source logical deletion, module disablement or tenant deletion. Source retention may shorten availability further (resolved Browser incidents: seven days; resolved Cron: 30 days). Physical source deletion cascades the copied coordination. A fair hourly cleanup processes up to 50 active scopes and 200 expired/source-hidden records per scope. Application retention also applies while the incident is still open; at most 10,000 coordination records are retained per application. Configuration such as a saved runbook remains until edited or its application is deleted.

HTTP and provider boundaries

Read-only Management API:

GET /api/v1/apps/{org}/{app}/metrics/oncall?env=production
GET /api/v1/apps/{org}/{app}/metrics/oncall?env=production&source=alert&id={incidentId}

The token needs the existing read scope and Metrics permission. PII and Cron permissions are checked independently. Ingest keys cannot manage schedules or acknowledge incidents. Console mutations require same-origin, CSRF, current Google/passkey membership, an explicit reason, and a passkey verified within ten minutes.

Existing email/Slack/webhook provider contracts are reused. Slack incoming webhooks are one-way notifications; clicking the private link and authenticating is required to acknowledge. There are no Slack interaction callbacks or new PagerDuty/Grafana integrations. On-call webhook destinations must use public HTTPS without URL credentials; redirects are refused. All resolved IPv4/IPv6 answers must be public, with at most 16 addresses; private, mixed, mapped, transition and special-purpose answers are refused conservatively. DNS has a three-second deadline. The request pins one validated address while TLS verifies the original hostname, and rechecks delivery authority after DNS/vault I/O. Slack is restricted to hooks.slack.com. Only HTTPS port 443 is supported, redirects are refused, and response bodies are discarded without decompression. See the IANA IPv4 and IPv6 special-purpose registries for the underlying address classifications.

The workflow follows the response coordination and post-incident learning principles in the Google SRE workbook and Grafana IRM introduction. Slack delivery uses the existing incoming-webhook protocol.