Error tracking

Browser and server exceptions, grouped into issues, with stack frames, breadcrumbs, releases, affected-user counts and alerts. Replaces Sentry.

Console: /o/<org>/apps/<app>/errors (module key errors, permissions errors.read to look, errors.write to change an issue's state).

---

The path an error takes

SDK captureException / window.onerror / onRequestError
  → POST /api/v1/ingest            (validated against errorEventSchema)
  → error sink, inside the batch transaction
      → fingerprint  → upsert error_issues  → insert error_events
      → error_daily (+ error_daily_users)   → enqueue errors.notify
  → worker: errors.notify → sendNotification(appId, …) → channels

Everything up to the job runs in one transaction. A batch that fails validation never reaches the sink; a batch that reaches the sink either lands whole or not at all.

Grouping

Two occurrences share an issue when their fingerprint matches. An explicit fingerprint: string[] from the SDK wins when no enabled operator grouping rule matches. Operator rules run in their displayed order before this fallback. Otherwise the key is built from three parts and hashed (SHA-256, first 32 hex characters):

1. The error name — TypeError, StripeCardError, …
2. The top three in-app frames, each reduced to function@file:
   • function: leading async / new and an Object. / Module. / Function. prefix are dropped; a missing name becomes <anonymous>.
   • file: query string and fragment removed, a trailing :line:col removed, reduced to its basename, and a bundler content hash before the extension stripped — main.4f2c19ab.js and main.99ee0011.js are the same file.
   • Line and column numbers are never part of the key, so a refactor that moves a function does not split its issue.
   • Frames marked inApp are preferred. When an SDK marks none (a minified bundle, say), every frame counts rather than nothing.
3. The message, masked — in order: urls → <url>, uuids → <uuid>, hex blobs of eight or more characters and 0x… → <hex>, quoted strings → <str>, numbers → <num>. Order 8821 failed and Order 9114 failed are one issue.

With no frames at all the key is the name plus the masked message.

The algorithm carries a version tag (spm.error.v1) inside the hash input, so changing it later produces new issues instead of silently regrouping old ones.

The title is Name: first line of the message, bounded to 200 characters, fixed when the issue is created — a later occurrence with a slightly different message does not rename an issue an operator is already watching. The culprit is the top in-app frame, renderSummary (checkout.js).

An issue's level never de-escalates: a warning that later arrives as fatal stays fatal.

Statuses

What a new occurrence does to an existing issue:

• open / regressed → counters move, nothing else.
• resolved → becomes regressed, resolved_at / resolved_by cleared, one notification.
• ignored with no snooze, or a snooze still running → stays ignored, silent.
• ignored whose snoozed_until has passed → becomes regressed, notified.

regressed is never settable from the console: it is an observation, not a choice. The API accepts open, resolved and ignored only.

Snoozing is ignored plus a snoozed_until date; the console offers 7 days and the API accepts 1–90. Resolving or ignoring clears last_notified_at, so the next regression alerts immediately instead of waiting out the window.

Notifications

The sink enqueues errors.notify with { issueId, kind: "new" | "regression" }. The worker loads the issue, skips silently when the application was deleted or has since switched the module off, and calls sendNotification(appId, { title, body, url, severity }) from @/lib/metrics/notify (W8 owns delivery to email, Slack and webhook channels).

title    New issue in saaspro.dev: TypeError: Cannot read properties of undefined (reading 'total')
body     renderSummary (checkout.js)
         1284 events · 91 users affected
         Environment: production · Level: error
         Release: web@1.4.2
url      https://saaspro.dev/o/saaspromax/apps/saaspro-dev/errors/<issueId>?env=production
severity fatal, error → critical · warning → warning · info → info

At most one notification per issue per 30 minutes. The window is reserved inside the ingest transaction, under the issue's row lock (error_issues.last_notified_at), so a burst of ten thousand occurrences enqueues one job — not ten thousand jobs that each discover they have nothing to say. The enqueue also carries an idempotency key of errors.notify:<issueId>:<30-minute bucket>.

Affected users

user_count and error_daily.users are exact within a retained window, not estimated. Every (app, env, issue, day, actor) triple is written once to error_daily_users; a triple that is new for the day increments the day's user count, and one whose actor has not been seen on any other day for that issue increments the issue's.

The actor is the first identity available: the resolved person, then distinctId, anonymousId, sessionId, and finally the hashed IP. Raw IPs are never stored.

The hourly errors.daily.rollup recomputes counters from retained captured actors and UTC occurrence dates in bounded batches. Merge and undo immediately recompute their affected groups from the same rows. Affected-actor sets span application retention; the former independent 90-day compaction no longer applies. Identity changes do not rewrite the captured actor key.

Counters are maintained incrementally at ingest and refreshed by the worker; expiry between maintenance passes may leave an older count until that issue is visited. Empty active merge mappings retain zero counts and remain available for undo and future fingerprint routing.

Storage

Migration creates the partition window from the previous month through two months ahead, and the daily errors.partitions job rolls it forward. An event whose timestamp falls outside that window (a wildly wrong client clock) is stored at its receive time rather than dropped.

Bounds applied at write time, on top of what @saaspro/shared already validates: stack ≤ 32 KB, ≤ 100 breadcrumbs, ≤ 100 frames, and any of frames / tags / extra / breadcrumbs / request over 32 KB serialized is dropped whole rather than stored half-truncated.

Search is lower(title) LIKE / lower(culprit) LIKE with % and _ escaped — no trigram extension is assumed.

Retention

errors.retention.purge runs daily and honours applications.retention_days (default 365):

• events, daily rows and the user set older than the application's retention are deleted;
• expired issues with no stored occurrences or active merge membership are deleted; active mappings retain their operator state;
• an error_events partition whose entire month predates the longest retention in the system is dropped, which is the only cheap way to reclaim the space.

Worker jobs

Console

Issues list — status tabs with counts (open / regressed / resolved / ignored / all), search over title and culprit, release filter, sort by last seen, events, users or first seen, keyset pagination, and a 14-day sparkline per row. Bulk resolve, ignore and reopen take a reason and write one audit event per issue. The environment comes from ctx.env, so every number on the page belongs to one environment.

Issue detail — header actions (resolve, ignore, snooze 7 days, reopen, assign to an organization member), all-time stats, a 14-day events-and-users chart, the latest event (or ?event=<id> for a specific one) with its stack frames, source context, breadcrumb timeline, tags, request and person link, a paginated occurrence list, and the audit-backed activity trail.

Mutations post to /api/console/errors with ?intent=status | bulk-status | assign | snooze through SecureForm, and are audited as error.issue.status and error.issue.assigned.

SDK setup

Browser capture is on by default:

import { spm } from "@saaspro/browser";
spm.init({ key: "spm_pub_prod_…" });     // autoErrors: window.onerror + unhandledrejection

spm.captureException(error, { tags: { area: "checkout" }, extra: { orderId } });
spm.addBreadcrumb({ category: "custom", message: "coupon applied" });

Next.js server and edge errors go through instrumentation.ts:

// lib/spm.ts
import { SaaSProMax } from "@saaspro/node";
export const spm = new SaaSProMax({ secretKey: process.env.SPM_SECRET_KEY!, release: process.env.GIT_SHA });

// instrumentation.ts
import { spm } from "./lib/spm";
import { nextRequestErrorHandler } from "@saaspro/node";
export const onRequestError = nextRequestErrorHandler(spm);

Express (expressErrorHandler), Hono (honoErrorHandler) and long-running processes (installProcessHandlers) have their own handlers. Set release on the client so the console can tell you which deploy introduced an issue and which one fixed it. Full reference: docs/sdk/browser.md and docs/sdk/node.md.

Demo data

pnpm db:seed --demo fills the bootstrap application with six issues — a browser TypeError, an unhandled rejection, a handled Node exception with its request, a regression across two releases, a chunk-load warning and a fatal Redis failure — spread over two weeks. It is skipped when the application already has issues, so re-seeding never inflates counters someone is reading.

Private build artifacts

Errors → Build artifacts supports flat and indexed JavaScript source maps, exact debug-ID matching, legacy release/bundle matching, and native Breakpad symbols for captured physical frames. The SDK's frames option accepts build identifiers and native instruction/load addresses. The CI command injects IDs before application startup and uploads private artifacts. Existing errors resolve on read; their captured evidence remains unchanged.

See Readable stack traces for build commands, matching rules, native bridge integration, storage limits and troubleshooting.

Merge, undo and future grouping

Open Errors → Grouping to find source and destination issues independently, preview retained counts, merge and undo. The destination keeps its status and assignment. Captured payloads and source-map inputs stay unchanged; current issue assignment is derived and reversible. Exact occurrence and feedback links follow the event through these operations. New source occurrences route to the retained destination until undo. Undo newer related merges first; expired evidence cannot be recovered.

The same page edits and previews up to twenty first-match rules for future events. Rule inspection requires People PII access, and edits/merges use a recent passkey, CSRF and audited reasons. See Issue grouping for precedence, provenance, limits and retention behavior.

What the assistant can do here

Ask reads this module in full: the issue list by status, one issue with its daily counts, releases and tags, a single occurrence with frames, breadcrumbs and request context, and the overview. Asked from an issue page it knows which issue you are looking at, so "who does this affect, and since when?" starts with the right identifier.

Changing an issue's state is a proposal. Resolve, ignore and reopen need errors.write, and the assistant shows the issue and the new status before you approve. Deleting a build artifact or a symbol set is destructive and asks for a typed reason and a fresh passkey. Stack frames and breadcrumbs are tenant data: an error message that reads like an instruction is reported as text, never followed.