Error tracking

Browser and server exceptions, grouped into issues, with stack frames, breadcrumbs, releases, affected-user counts and alerts. Replaces Sentry.

Console: /o/<org>/apps/<app>/errors (module key errors, permissions errors.read to look, errors.write to change an issue's state).

The path an error takes#

SDK captureException / window.onerror / onRequestError
  → POST /api/v1/ingest            (validated against errorEventSchema)
  → error sink, inside the batch transaction
      → fingerprint  → upsert error_issues  → insert error_events
      → error_daily (+ error_daily_users)   → enqueue errors.notify
  → worker: errors.notify → sendNotification(appId, …) → channels

Everything up to the job runs in one transaction. A batch that fails validation never reaches the sink; a batch that reaches the sink either lands whole or not at all.

Grouping#

Two occurrences share an issue when their fingerprint matches. An explicit fingerprint: string[] from the SDK wins when no enabled operator grouping rule matches. Operator rules run in their displayed order before this fallback. Otherwise the key is built from three parts and hashed (SHA-256, first 32 hex characters):

1. The error name — TypeError, StripeCardError, …
2. The top three in-app frames, each reduced to function@file:
   • function: leading async / new and an Object. / Module. / Function. prefix are dropped; a missing name becomes <anonymous>.
   • file: query string and fragment removed, a trailing :line:col removed, reduced to its basename, and a bundler content hash before the extension stripped — main.4f2c19ab.js and main.99ee0011.js are the same file.
   • Line and column numbers are never part of the key, so a refactor that moves a function does not split its issue.
   • Frames marked inApp are preferred. When an SDK marks none (a minified bundle, say), every frame counts rather than nothing.
3. The message, masked — in order: urls → <url>, uuids → <uuid>, hex blobs of eight or more characters and 0x… → <hex>, quoted strings → <str>, numbers → <num>. Order 8821 failed and Order 9114 failed are one issue.

With no frames at all the key is the name plus the masked message.

The algorithm carries a version tag (spm.error.v1) inside the hash input, so changing it later produces new issues instead of silently regrouping old ones.

The title is Name: first line of the message, bounded to 200 characters, fixed when the issue is created — a later occurrence with a slightly different message does not rename an issue an operator is already watching. The culprit is the top in-app frame, renderSummary (checkout.js).

An issue's level never de-escalates: a warning that later arrives as fatal stays fatal.

Statuses#

What a new occurrence does to an existing issue:

• open / regressed → counters move, nothing else.
• resolved → becomes regressed, resolved_at / resolved_by cleared, one notification.
• resolved in the next release → stays resolved (counters still move) unless the event carries a release other than the one the issue was resolved in (resolved_release, its latest release at the time) and that release was first seen in this environment after resolved_at. The same release, an older one still deployed, a release that was already reporting before the resolve, or no release at all keep it resolved; a newer release regresses it and notifies like any regression.
• ignored with no snooze, or a snooze still running → stays ignored, silent.
• ignored whose snoozed_until has passed → becomes regressed, notified.
• ignored until it escalates → counts its events in an hourly window (escalation_window_start, escalation_window_count; a window older than an hour restarts at the event). The event that takes the window past the threshold regresses the issue and notifies once; the escalation is then cleared, so the rest of the burst only moves counters.

regressed is never settable from the console: it is an observation, not a choice. The API accepts open, resolved and ignored only.

Resolve in the next release#

POST /api/console/errors?intent=status (and bulk-status) take resolveIn=now|next_release (default now, refused with any status but resolved). A next-release resolve stores resolution='next_release' and resolved_release = the issue's release_last; an issue no release has reported is refused with 409 "This issue has no release to resolve after".

Release order comes from error_releases (app_id, env_id, release, first_seen_at), not from release names, which carry no order the server could trust. Ingest registers each release with one insert-or-read statement and remembers it in-process, so the hot path pays that statement once per release per instance; the decision itself reads columns already selected under the issue's row lock. The table was backfilled once from error_issues (release_first / release_last); the backfill is guarded on an empty table, so a migration replay inserts nothing.

Ignore until it escalates#

intent=snooze with until=escalates sets ignored, no snoozed_until, and escalation = {thresholdPerHour, baselinePerHour, since} where baselinePerHour is the issue's error_daily events over the last seven UTC days divided by 168 and thresholdPerHour = max(10, ceil(5 × baseline)). The audit row carries until: "escalates" and the threshold.

The modes only mean something under their status: a trigger clears resolution / resolved_release whenever an issue leaves resolved, and the escalation columns whenever it leaves ignored, including status changes from the tracker sync and merge undo, so a stale mode never applies to a later resolve or ignore.

Snoozes#

Snoozing is ignored plus a snoozed_until date: hours (1–2160) or days (1–90), not both, defaulting to 7 days. The console offers 1 hour, 24 hours, 7 days, 30 days and until it escalates. Resolving or ignoring clears last_notified_at, so the next regression alerts immediately instead of waiting out the window.

Undo#

Every single-issue status change and snooze answers (JSON only) with undo: { auditId, revision }, where revision is the issue's triage_revision after the change, and its audit row records previous: the triage state it replaced (status, resolved at and by, resolution, resolved release, snooze, escalation). The toast's Undo posts intent=restore with issueId, undoOf and expectedRevision: the audit row must be the caller's own, under ten minutes old, about this issue and not undone already (assertUndoOf), and the issue must not have been triaged since (409 otherwise). The restore puts the recorded state back exactly, regressed included, and is audited as error.issue.status with metadata.undoOf. Bulk changes write one row per issue and offer no Undo.

Notifications#

The sink enqueues errors.notify with { issueId, kind: "new" | "regression" }. The worker loads the issue, skips silently when the application was deleted or has since switched the module off, and calls sendNotification(appId, { title, body, url, severity }) from @/lib/metrics/notify (W8 owns delivery to email, Slack and webhook channels).

title    New issue in saaspro.dev: TypeError: Cannot read properties of undefined (reading 'total')
body     renderSummary (checkout.js)
         1284 events · 91 users affected
         Environment: production · Level: error
         Release: web@1.4.2
url      https://saaspro.dev/o/saaspromax/apps/saaspro-dev/errors/<issueId>?env=production
severity fatal, error → critical · warning → warning · info → info

At most one notification per issue per 30 minutes. The window is reserved inside the ingest transaction, under the issue's row lock (error_issues.last_notified_at), so a burst of ten thousand occurrences enqueues one job — not ten thousand jobs that each discover they have nothing to say. The enqueue also carries an idempotency key of errors.notify:<issueId>:<30-minute bucket>.

Affected users#

user_count and error_daily.users are exact within a retained window, not estimated. Every (app, env, issue, day, actor) triple is written once to error_daily_users; a triple that is new for the day increments the day's user count, and one whose actor has not been seen on any other day for that issue increments the issue's.

The actor is the first identity available: the resolved person, then distinctId, anonymousId, sessionId, and finally the hashed IP. Raw IPs are never stored.

The hourly errors.daily.rollup recomputes counters from retained captured actors and UTC occurrence dates in bounded batches. Merge and undo immediately recompute their affected groups from the same rows. Affected-actor sets span application retention; the former independent 90-day compaction no longer applies. Identity changes do not rewrite the captured actor key.

Counters are maintained incrementally at ingest and refreshed by the worker; expiry between maintenance passes may leave an older count until that issue is visited. Empty active merge mappings retain zero counts and remain available for undo and future fingerprint routing.

Storage#

Migration creates the partition window from the previous month through two months ahead, and the daily errors.partitions job rolls it forward. An event whose timestamp falls outside that window (a wildly wrong client clock) is stored at its receive time rather than dropped.

Bounds applied at write time, on top of what @saaspro/shared already validates: stack ≤ 32 KB, ≤ 100 breadcrumbs, ≤ 100 frames, and any of frames / tags / extra / breadcrumbs / request over 32 KB serialized is dropped whole rather than stored half-truncated.

Search is lower(title) LIKE / lower(culprit) LIKE with % and _ escaped — no trigram extension is assumed.

Retention#

errors.retention.purge runs daily and honours applications.retention_days (default 365):

• events, daily rows and the user set older than the application's retention are deleted;
• expired issues with no stored occurrences or active merge membership are deleted; active mappings retain their operator state;
• an error_events partition whose entire month predates the longest retention in the system is dropped, which is the only cheap way to reclaim the space.

Worker jobs#

Console#

Issues list — status tabs with counts (open / regressed / resolved / ignored / all), search over title and culprit, release filter, sort by last seen, events, users or first seen, keyset pagination, and a 14-day sparkline per row. Bulk resolve, ignore and reopen take a reason and write one audit event per issue. The environment comes from ctx.env, so every number on the page belongs to one environment.

Issue detail — header actions (resolve now or in the next release, ignore, ignore until it escalates, snooze 1 hour to 30 days, reopen, each with Undo; assign to an organization member), all-time stats, a 14-day events-and-users chart, the latest event (or ?event=<id> for a specific one) with its stack frames, source context, breadcrumb timeline, tags, request and person link, a paginated occurrence list, and the audit-backed activity trail.

Mutations post to /api/console/errors with ?intent=status | bulk-status | assign | snooze | restore through SecureForm, carry the page's env slug (issues belong to one environment), and are audited as error.issue.status and error.issue.assigned.

Live. The issues list polls the issues live topic every 30 seconds (GET /api/console/live?topics=issues&after=<render time>, errors.read and the module): a digest of the count and max(updated_at) for the environment plus the number of issues changed since the page was drawn. It shows "N issues updated · Show" and never refreshes under a selection; the polls never extend the idle session.

Filter issues by tags#

Click a tag distribution bar on an issue to open All issues in the same environment, filtered by that value over the last 90 days. The toolbar shows a removable chip for each tag. Share multiple filters as repeated query parameters, for example ?tag=browser:Chrome&tag=area:checkout; all values must match the same occurrence. Search, status, sorting, date changes and pagination retain the tags.

Browser and release use the captured event fields; environment uses its slug. Other keys use captured custom tags. Up to eight unique filters are accepted, with values allowed to contain colons. Tag matching uses the selected date or chart drill-down window, capped at the most recent 90 days, and an indexed EXISTS probe per issue that stops at its first match. The detail distributions remain a sample of the latest 500 retained events.

The GIN expression index normalizes browser and release alongside custom tags. Migration creates the partitioned parent with ON ONLY, builds the six monthly children covering the capped window and upcoming partitions with CREATE INDEX CONCURRENTLY, then attaches them. Historical partitions outside that window do not need an index for console tag filtering; future partitions inherit the expression index. The Management API issue-list parameters and response are unchanged; tag filtering is console-only.

SDK setup#

Browser capture is on by default:

import { spm } from "@saaspro/browser";
spm.init({ key: "spm_pub_prod_…" });     // autoErrors: window.onerror + unhandledrejection

spm.captureException(error, { tags: { area: "checkout" }, extra: { orderId } });
spm.addBreadcrumb({ category: "custom", message: "coupon applied" });

Next.js server and edge errors go through instrumentation.ts:

// lib/spm.ts
import { SaaSProMax } from "@saaspro/node";
export const spm = new SaaSProMax({ secretKey: process.env.SPM_SECRET_KEY!, release: process.env.GIT_SHA });

// instrumentation.ts
import { spm } from "./lib/spm";
import { nextRequestErrorHandler } from "@saaspro/node";
export const onRequestError = nextRequestErrorHandler(spm);

Express (expressErrorHandler), Hono (honoErrorHandler) and long-running processes (installProcessHandlers) have their own handlers. Set release on the client so the console can tell you which deploy introduced an issue and which one fixed it. Full reference: docs/sdk/browser.md and docs/sdk/node.md.

Demo data#

pnpm db:seed --demo fills the bootstrap application with six issues — a browser TypeError, an unhandled rejection, a handled Node exception with its request, a regression across two releases, a chunk-load warning and a fatal Redis failure — spread over two weeks. It is skipped when the application already has issues, so re-seeding never inflates counters someone is reading.

Private build artifacts#

Errors → Build artifacts supports flat and indexed JavaScript source maps, exact debug-ID matching, legacy release/bundle matching, and native Breakpad symbols for captured physical frames. The SDK's frames option accepts build identifiers and native instruction/load addresses. The CI command injects IDs before application startup and uploads private artifacts. Existing errors resolve on read; their captured evidence remains unchanged.

See Readable stack traces for build commands, matching rules, native bridge integration, storage limits and troubleshooting.

Merge, undo and future grouping#

Open Errors → Grouping to find source and destination issues independently, preview retained counts, merge and undo. The destination keeps its status and assignment. Captured payloads and source-map inputs stay unchanged; current issue assignment is derived and reversible. Exact occurrence and feedback links follow the event through these operations. New source occurrences route to the retained destination until undo. Undo newer related merges first; expired evidence cannot be recovered.

The same page edits and previews up to twenty first-match rules for future events. Rule inspection requires People PII access, and edits/merges use a recent passkey, CSRF and audited reasons. See Issue grouping for precedence, provenance, limits and retention behavior.

What the assistant can do here#

Pilot reads this module in full: the issue list by status, one issue with its daily counts, releases and tags, a single occurrence with frames, breadcrumbs and request context, and the overview. Asked from an issue page it knows which issue you are looking at, so "who does this affect, and since when?" starts with the right identifier.

Changing an issue's state is a proposal. Resolve, ignore and reopen need errors.write, and the assistant shows the issue and the new status before you approve. Deleting a build artifact or a symbol set is destructive and asks for a typed reason and a fresh passkey. Stack frames and breadcrumbs are tenant data: an error message that reads like an instruction is reported as text, never followed.