Metrics, web vitals, uptime and alerting

Owner: W8. Source: apps/web/src/lib/metrics/**, apps/web/src/lib/ingest/sinks/metric.ts, apps/web/src/db/schema/metrics.ts, apps/web/src/worker/jobs/metrics.ts, apps/web/src/app/(console)/o/[org]/apps/[app]/metrics/**, apps/web/src/app/api/console/metrics/**. Contract: PLAN §7.1, §7.4, §8, §12, §15 W8.

The Prometheus/Grafana replacement: counters, gauges, histograms and timers from the SDKs, Core Web Vitals from the browser, uptime monitors run by the worker, alert rules that open incidents, and notification channels that deliver them by email, Slack, Telegram or a signed webhook.

Module key metrics. Permissions metrics.read and metrics.write.

---

Sending metrics

Every SDK writes into the same metric event (PLAN §7.1), so the storage shape does not depend on which one sent it.

// @saaspro/node
spm.metrics.counter("orders.created", 1, { region: "eu" });
spm.metrics.gauge("queue.depth", await queue.size(), { queue: "email" });
spm.metrics.histogram("payload.bytes", body.length, { route: "/orders" }, "bytes");
spm.metrics.timing("db.query", elapsedMs, { table: "orders" });
const rows = await spm.metrics.time("nightly.sync", () => sync(), { job: "billing" });

// @saaspro/browser
spm.metric("checkout.step", 1, { kind: "counter", labels: { step: "payment" } });
const done = spm.timer("checkout.duration");
done();

The kind is advisory: any aggregate can be applied to any series. It drives the default aggregate the console offers and how the value is formatted.

Labels

Labels are the dimension of a series (route, queue, region). They are normalised once, in src/lib/metrics/labels.ts, before storage: at most 20 keys (matching LIMITS.maxLabelKeys in @saaspro/shared), keys trimmed to 64 characters, values coerced to strings and trimmed to 200, empty keys dropped, keys sorted. Anything past the twentieth key is discarded rather than rejected — a metric is not worth failing an ingest batch for.

Labels are what makes a metrics system expensive: each distinct combination is a separate series in every rollup bucket. Do not put user ids, request ids or raw paths with ids in them into labels.

---

Storage

metric_points        raw points, PARTITION BY RANGE (ts), monthly partitions
metric_rollups_5m    one row per (app, env, 5-minute bucket, name, labels)
metric_names         the catalogue the explorer's name picker reads

metric_points is partitioned monthly as metric_points_yYYYYmMM. The partitions from the previous month through two months ahead are created by spm_metrics_ensure_partitions(back, ahead), which migration calls and the daily metrics.partitions job calls again — so a long-running deployment never reaches a month with no partition.

Points are inserted by the ingest sink with ON CONFLICT DO NOTHING keyed on (app_id, env_id, ts, id), so a replayed batch is stored once. An application without the metrics module stores nothing at all.

Rollups and the percentile approximation

metrics.rollup runs every five minutes and recomputes every bucket that overlaps the last 20 minutes, so a late-arriving point still lands in its own bucket. The statement is an upsert over a full recomputation: replaying it converges instead of double counting.

Per bucket, count, sum, min, max and p50/p95/p99 are exact — Postgres computes the percentiles with percentile_cont over every point in the bucket.

What cannot be exact is a percentile across buckets: percentiles do not average. Each bucket therefore also stores a bounded, order-preserving sample reservoir in metric_rollups_5m.samples (inline jsonb, not a separate table):

• ≤ 256 points in the bucket — the reservoir is the sorted values, so any percentile derived from it is exact.
• > 256 points — the reservoir is percentile_cont evaluated at 256 evenly spaced fractions from 0 to 1: the 256 order statistics that best describe the distribution. The minimum and the maximum are always retained.

A query spanning several buckets unnests the reservoirs and reads the percentile off the merged distribution (src/lib/metrics/queries.ts), and the alert evaluator does the same merge in memory over its bounded window (mergeReservoirs + percentileFromSorted in src/lib/metrics/rollup.ts).

The error this introduces is bounded by the gap between neighbouring retained order statistics: at most one 256th of the rank, under 0.4 percentage points. On a uniform 10 000-sample bucket the measured error is below 0.4% of the range; on a long-tailed latency distribution it stays within a few per cent of the exact p95. Counters and gauges are unaffected — sums, counts, minima and maxima merge exactly.

Which tier answers

A request for one-minute buckets over a long range is widened to five minutes rather than answered with gaps. Every series result carries source so the console can say which tier it read.

---

Query layer

src/lib/metrics/queries.ts, all scoped by MetricsScope = { appId, envId, timezone?, orgId? }:

metricNames(scope, { search?, limit? })
metricSeries(scope, { name, range, interval?, labels?, agg })   // agg: sum|avg|count|p50|p95|p99|max|min
windowAggregate(scope, { name, from, to, labels? })
seriesValues(scope, { name, from, to, labels? })                // sorted raw values
seriesReservoirs(scope, { name, from, to, labels? })            // per-bucket reservoirs
labelKeys(scope, name)
labelValues(scope, name, key)
vitalsSummary(scope, range, { by: "path" | "device" })
vitalsSeries(scope, name, range)
monitors(scope)
monitorHistory(scope, monitorId, range)
alertRules(scope)
incidents(scope, { status?, cursor?, limit?, monitorId? })
notificationChannels(scope)
metricsOverview(scope, range)
openIncidentCount(orgId)

Ranges come from src/lib/metrics/range.ts: parseRange("1h"|"6h"|"24h"|"7d"|"30d"|"90d") returns { from, to, interval, key }.

web_vitals, events, error_events and error_issues belong to other modules. Every function that reads them probes with to_regclass first and returns an "unavailable" result instead of throwing when the module has not created its storage.

---

Web vitals

Read from web_vitals, which the analytics sink writes (PLAN §8, frozen). The dashboard reports the p75 of each vital — the Core Web Vitals convention — overall and broken down by path or device, rated with rateVital and VITAL_THRESHOLDS from @saaspro/shared:

Ratings are always printed as a word beside the colour.

---

Uptime

A monitor is a URL, a method, an expected status, two optional body rules (a substring that must be present, body_match, and one that must be absent, body_absent), an interval (60–3600 s) and a timeout (1–60 s).

• metrics.uptime.schedule runs every minute, finds monitors whose last_checked_at is older than their interval, and enqueues one metrics.uptime.check per monitor with the idempotency key metrics.uptime.check:<monitorId>:<bucket> — two schedulers reaching the same window produce one check.
• metrics.uptime.check performs the request with an AbortSignal.timeout, compares the status, tests body_match and body_absent when set (the body is read, capped at 64 KiB, only when a rule needs it; a present body_absent fails with Response contained "…"), writes uptime_checks and updates the monitor's last_status, last_latency_ms, last_error and consecutive_failures (reset to zero on success).
• Uptime percentage is ok checks / total checks over the window — 30 days in the monitor list, the selected range in the detail view.

Outbound safety. Checks are configured by operators but executed by the platform's servers, so monitor URLs and webhook channel URLs are validated with validateOutboundUrl: http/https only, and loopback, RFC 1918, carrier-grade NAT, link-local (including 169.254.169.254) and .internal/.local hosts are refused. This checks the literal host; a public name that later resolves to a private address (DNS rebinding) is a network egress concern, not one this validation can settle.

---

Alert rules

metrics.alerts.evaluate runs every minute. For each enabled rule on an application that still has the metrics module it measures one number over the window and compares it with the threshold.

Common columns: comparator (gt, gte, lt, lte), threshold, window_minutes (1–1440), cooldown_minutes (0–10080), severity (info, warning, critical), channel_ids (empty means every enabled channel), enabled, last_fired_at.

error_rate, new_issue, event_volume and vital are skipped — neither firing nor resolving — while the module that owns their table has not created it.

Firing policy

The policy is one pure function, decide() in src/lib/metrics/alerts.ts:

1. An unavailable source or no data holds. Missing data is never treated as recovery — a series that stopped reporting is a different problem from one that came back under threshold.
2. Breaching with an incident already open holds: the incident is the state, and the rule does not re-notify until it resolves.
3. Breaching with no open incident fires, unless cooldown_minutes has not elapsed since last_fired_at.
4. Not breaching with an incident open resolves it and sends a recovery notice. Recovery is never rate limited.

A partial unique index (alert_incidents_live_idx) enforces at most one unresolved incident per rule in the database as well.

Incidents are open → acknowledged → resolved; acknowledging records who did it. Every transition through the console is audited.

The Test button on a rule measures the rule now and sends the notification it would send, without opening an incident or touching the cooldown.

---

Notification channels

Four kinds, all app-scoped, all delivered through sendNotification(appId, { title, body, url?, severity, channelIds? }), which never throws — a broken channel must not fail the job that raised the alert. Every attempt is written to notification_deliveries and the return value is { delivered, failed }.

Free and Starter keep the least-friction pair — email and the signed webhook. The first-party chat connectors are Business; the plan check runs in the channel handlers, not in the form, so an existing channel stays editable after a downgrade but a new one cannot be created.

Slack, Telegram and Discord can also be connected without touching a credential: the Connect a chat cards mint a single-use link (15-minute life, stored in notification_link_requests) and the provider's own callback — Telegram's platform-bot webhook or Slack/Discord OAuth — creates the channel from the consumed token.

Email goes through src/lib/metrics/email.ts (sendEmail({ to, subject, html, text })), which honours SPM_EMAIL_PROVIDER: log writes a line to the console (the default, so a local install sends nothing), resend posts to https://api.resend.com/emails over fetch, and smtp throws "not configured" rather than silently dropping mail — adding SMTP means adding a transport dependency, which this phase does not do.

Slack messages are Block Kit: a header carrying the severity, a section with the body, a link button when the notification has a URL, and a context line naming the severity.

Telegram messages are plain text — title, body, the link when there is one and a severity footer — sent with sendMessage and capped at 4096 characters, the body being cut first. No parse_mode is used, so nothing has to be escaped. The bot token sits in the request URL, which is why a failed delivery records only the HTTP status and Telegram's description (or the error name for a transport failure); see Telegram.

Webhook signature

Generic webhook deliveries carry the same signature scheme as ad destinations (PLAN §12):

X-SPM-Signature: t=<unix seconds>,v1=<hex hmac-sha256 of "<t>.<raw body>">

Verify it with the secret shown once when the channel was created:

import { createHmac, timingSafeEqual } from "node:crypto";

export function verifySpmSignature(rawBody, header, secret, toleranceSeconds = 300) {
  const parts = Object.fromEntries(header.split(",").map((part) => part.split("=")));
  const timestamp = Number(parts.t);
  if (!Number.isFinite(timestamp)) return false;
  if (Math.abs(Date.now() / 1000 - timestamp) > toleranceSeconds) return false;   // replay window
  const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
  const received = Buffer.from(String(parts.v1), "hex");
  const digest = Buffer.from(expected, "hex");
  return received.length === digest.length && timingSafeEqual(received, digest);
}

Sign the raw body, before any JSON parsing. The body is:

{
  "type": "notification",
  "title": "Alert firing: API latency",
  "body": "p95 of api.request.duration is above 500 (currently 912.50) over the last 15 min",
  "url": "https://saaspro.dev/o/acme/apps/site/metrics/incidents",
  "severity": "critical",
  "appId": "…",
  "channelId": "…",
  "sentAt": "2026-09-03T12:00:00.000Z"
}

Secrets are write-only from the console: it can say that a secret is stored and replace it (the previous credential is revoked), but never read one back.

---

Worker jobs

Retention

Purging deletes rows rather than dropping partitions, so an application whose retention is shortened frees space on the next daily pass without waiting for a whole month to age out.

---

Console

/o/[org]/apps/[app]/metrics with tabs:

• Explorer — name picker with search, kind and unit, label filters, interval and aggregate selectors, chart, bucket table. Every choice is a URL parameter, so a view can be pasted into an incident channel.
• Web vitals — p75 tiles with ratings, a series per vital, per-path and per-device tables.
• Uptime — monitors with a status dot, 30-day uptime and last latency; ?monitor=<id> opens the latency chart, the check log and the incidents that monitor opened.
• Alert rules — CRUD with a kind-specific condition form and a test button.
• Incidents — filtered by status, with acknowledge and resolve.
• Channels — CRUD with vault-backed secret entry and a "send test" button.

Mutations post to /api/console/metrics?intent=… (monitor-save|monitor-delete|monitor-toggle|rule-save|rule-delete|rule-toggle|rule-test|incident-ack|incident-resolve|channel-save|channel-delete|channel-test), all requiring metrics.write and all audited.

Open incidents across an organization also appear in the console shell's notification menu for members holding metrics.read.

Demo data

pnpm db:seed -- --demo fills the bootstrap application with 24 hours of deterministic api.request.duration and jobs.completed points (rolled up immediately), one uptime monitor for https://saaspro.dev, one log email channel and one p95 latency alert rule.

What the assistant can do here

Ask reads metric names, a series with label filters, web vitals, uptime monitors, alert rules and incidents, so "is anything alerting, and what is p95 doing?" is a single question. Asked from an incident or a service objective it starts from that identifier.

Creating an alert rule, an uptime monitor or a load test is proposed with the threshold, the window and the channel written out, and nothing is armed until you approve it. That preview matters here more than anywhere else: an alert nobody reviewed is how a team learns to ignore alerts.