Service objectives and error budgets

title: Service objectives and error budgets description: Measure observed request outcomes, available error budget and paired burn windows.

Metrics → SLOs measures availability or latency for an exact service and, optionally, an exact operation. Objectives use accepted OpenTelemetry SERVER spans in the selected application and environment. They measure observed request outcomes; they cannot establish total uptime, detect uninstrumented traffic or infer requests missing from an upstream sampler.

Set up an objective

Follow OpenTelemetry traces to connect your server or Collector with a secret ingest key. Keep collection at 100% and size its budgets for your traffic. Public ingest keys and Management API tokens cannot export spans.
Use a stable, non-sensitive service.name and operation name. Emit one SERVER span per request, with an explicit HTTP response status or OpenTelemetry OK/ERROR status. Internal and client spans do not count.
Create an objective under Metrics → SLOs. Choose availability or a maximum duration, target, rolling 7-, 28- or 30-day window, and minimum eligible events per evaluation window. Application retention must cover the period.
Select existing notification channels if you want delivery. With none selected, incidents remain available in the console. Run the worker for evaluation, history, notifications and physical cleanup.

The existing Node and Python OpenTelemetry integrations work without an additional SaaS Pro SDK entry point. The exporter settings are:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://YOUR_SPM_HOST/api/v1/otlp/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer YOUR_SECRET_INGEST_KEY
OTEL_SERVICE_NAME=checkout-api
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512

For Python, install opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http and your framework's official instrumentation. Configure a TracerProvider with a BatchSpanProcessor and opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter. For Node, use the official SDK/exporter setup in the trace guide. An existing Collector can forward the same HTTP endpoint using the trace guide's authenticated, environment-specific pipeline.

For a manually instrumented request, set SpanKind.SERVER and record http.response.status_code as a number. Alternatively, explicitly set OK or ERROR. End the span at request completion; its elapsed duration is the latency measurement. Do not emit an extra manual SERVER span when framework instrumentation already supplies one. Never attach customer text, bodies, SQL or credentials to operation names or attributes.

Read the evidence

Availability treats explicit OpenTelemetry ERROR or HTTP 500–599 as bad; HTTP 100–499 or explicit OK otherwise establishes a good outcome. An explicit error takes precedence over HTTP 200. A latency objective additionally counts a known request as bad when its duration exceeds the chosen threshold. A duration exactly equal to the limit qualifies. A span without a known outcome is excluded and counted separately, even if its duration is known.

For an observed window:

SLI = (eligible - bad) / eligible
allowed bad events = eligible × (1 - target)
budget remaining = (allowed bad events - bad) / allowed bad events
burn rate = bad / allowed bad events

The allowed count can be fractional. A negative remaining budget means the observed allowance was exceeded. It is not clamped to zero.

Every window ends at the last closed UTC minute. The rolling periods use elapsed UTC days and do not change at local midnight or daylight-saving transitions. Counts arrive atomically with accepted, deduplicated spans; retries cannot increase the numerator or denominator. Late accepted spans update their completion minute in the current objective generation. Automatic raw-trace expiry does not remove their aggregate contribution.

A new objective starts with no backfill. Changing its service, operation, target, period, kind, effective latency threshold or paused state starts a new generation. Renaming, selecting channels, changing minimum volume or toggling burn alerts preserves the generation. Pausing and resuming each start a new generation, preventing traffic across a pause from being presented as continuous tracking.

Budget and burn values are unavailable for windows with no eligible data, insufficient volume, unknown outcomes, sampled or rejected source evidence, incomplete tracking duration, insufficient application retention, or invalidated contributions. Known-outcome SLI may still be shown separately. The source settings at each aggregation epoch are retained; restoring 100% sampling cannot relabel an earlier sampled bucket as complete. Upstream sampling remains outside the collector's knowledge, so even a fully observed window is not a traffic census.

Burn incidents and history

The worker evaluates at most 30 objectives per pass, in oldest-attempted order, every minute. The fast alert requires both the 1-hour and 5-minute burn rates to reach period hours / 50. The slow alert requires both the 6-hour and 30-minute rates to reach period hours / 120. These correspond to consuming 2% of the period's budget in one hour or 5% in six hours. For a 30-day objective, the thresholds are 14.4× and 6×.

Both windows need sufficient observed evidence. No data, paused collection or incomplete evidence neither opens nor recovers an incident. Fast and slow incidents have separate identities. Recovery requires valid evidence that the paired condition has cleared. Acknowledge and manual resolve remain explicit operator actions; manual resolution does not claim recovery. A continuing breach can reopen after the five-minute cooldown measured from the previous opening.

The detail page shows paired windows, recent incidents, notification receipt states and up to 90 daily history rows. Each row is the latest recorded hourly evaluation for that UTC day, with its original definition and generation. Missing days stay missing. Delivery uses the existing vault-backed channels, with one durable receipt per incident, channel and phase. An uncertain delivery is marked unknown and is not automatically resent.

The multiwindow approach follows the Google SRE guidance on alerting on SLOs. The minimum volume setting is essential for sparse traffic; it does not make a sparse sample representative.

Privacy, access and retention

SLO buckets store only counts, scope, time, generation and collection policy. They do not copy request payloads, person identifiers or trace IDs. The audit ledger remains separate and append-only. Console changes require Metrics write, membership, same-origin CSRF validation, a reason and a passkey verified within ten minutes. Private reads require Metrics read. Management clients may GET /api/v1/apps/{app}/metrics/slos?env=production, adding id for details. Management writes are not exposed.

Deleting an individual trace conservatively invalidates every SLO window in that environment that overlaps the removed span completion range, including historical snapshots. This bounded invalidation may affect unrelated services; it prevents retained counters from claiming complete evidence after deliberate source deletion. Detail history carries sourceInvalidated; its original recorded result remains historical evidence and must not be treated as a current valid budget. Deleting an objective hides it and its incident actions immediately. Environment/application deletion cascades to its SLO data.

An environment permits 20 objectives, including those awaiting physical deletion, and one million minute/epoch buckets. At that bucket cap, trace acceptance continues but lost SLI coverage makes affected budgets unavailable. Buckets retain up to 30 days, and history and resolved incidents up to 90 days, each capped by application retention. Reads apply effective retention immediately. Hourly maintenance visits at most 50 environments, deleting at most 10,000 buckets, 10,000 history rows and 1,000 resolved incidents per environment. Pending deletion still consumes quota until cleanup. The detail view lists at most 50 source epochs and 40 recent incidents.