Evaluations lab

Owner: W11. Source: apps/web/src/lib/evals/**, apps/web/src/db/schema/evals.ts, apps/web/src/db/seeds/evals.ts, apps/web/src/worker/jobs/evals.ts, apps/web/src/app/(console)/o/[org]/apps/[app]/evals/**, apps/web/src/app/(console)/o/[org]/(org)/models/**, apps/web/src/app/api/console/{evals,credentials,models}/**. Contract: PLAN §10, §13, §15 W11.

An evaluation answers one question with evidence: is this prompt, on this model, at this reasoning level, better than what we run today? The same frozen set of cases is sent to every variant, each result keeps its trace, and the comparison is a matrix rather than a vibe.

Module key evals. Permissions evals.read, evals.write, evals.run, evals.review; the organization-level models and vault page uses credentials.read and credentials.write.

---

The model

Every page and route filters on org_id and app_id. A dataset, a run and a prompt belong to one application: two applications can hold the same prompt slug, and neither can read the other's cases.

---

Task types

A run's config.taskType decides how the runner executes a case.

A text case may also opt into live context: with reference_data.liveAgentContext = true the runner calls POST /agent-context first and merges the returned variables into the template values, so the prompt is graded against the context the application would really assemble.

---

Case contracts

A case is a JSON object. Imports accept a JSON array or JSONL (one object per line); both are deduplicated by checksum, so re-importing a file adds only what is new.

{
  "externalId": "faq-17",
  "input": { "question": "How do I cancel?" },
  "expected": { "text": "Open account settings" },
  "referenceData": { "relevantSourceIds": ["doc-1", "doc-2"] },
  "locale": "en",
  "tags": ["support", "billing"]
}

Per task type:

• text — input supplies the variables, expected.text the reference answer.
• tool — expected.tool names the tool that should be called; referenceData.toolSchema is the array of tool definitions passed to the provider.
• retrieval — input.siteId is required, input.question (or the rendered user template) is the query, and referenceData.relevantSourceIds lists the sources that should come back.
• voice_tts — the rendered user template is the speech text; input.voiceId (ElevenLabs) or input.voice (OpenAI) selects the voice.

No personal data. An import containing an email address, a phone number or a national identifier is refused before a single row is written. A case that somehow arrives with redaction_status = 'blocked' is excluded from execution.

---

Graders

src/lib/evals/graders.ts holds the deterministic ones; each returns a score in 0..1, a label, and a rationale that is stored with the result.

Automated scores are advisory. evals.review lets an operator attach a decision, a score, a confidence and notes to any single result; reviews are attributable and audited.

---

Running an evaluation

1. The console posts to /api/console/evals with evals.run. The route re-checks the dataset (same application, ready), every model (enabled in the platform catalog and not opted out by this organization), and the connector prompt link when one is named.
2. It writes the run, its variants, one eval_case_results row per (variant × case), and enqueues evals.run with idempotency_key = evals.run:<id>.
3. The worker claims each queued result with FOR UPDATE … SKIP LOCKED, so several workers can share one run.
4. Each result stores output, trace, token usage, latency and a list-price cost estimate derived from the catalog. A failing case is recorded with its error; the run keeps going.
5. When the queue is empty the run is summarized per variant (mean score, mean latency, cost, failures) with a plain-language conclusion.

Quality gates through the connector

A run can gate a release of a prompt the application actually serves.

1. Sync (POST /api/console/evals/prompts?intent=sync, evals.write) calls the application's connector GET /prompts. For each prompt the platform computes both checksums itself — the prompt checksum over system template + user template + runtime fingerprint, the benchmark checksum over the cases — and stores a prompt version plus a frozen dataset. Re-syncing an unchanged prompt writes nothing new; a changed benchmark snapshots a new dataset.
2. Benchmark opens the builder with the live prompt preloaded as the Candidate. The run carries productPromptSlug, the checksum it was opened with, and the gate policy from the link. ?audit=1 runs the same comparison without a gate.
3. Decide. After scoring, evaluationGateDecision compares the Candidate's mean score with minimumScore and its failure rate with maximumFailureRate. No score at all is review, never passed.
4. Report. The decision is written to prompt_quality_gates first, then sent to the application with POST /prompts/{slug}/gate ({ evalRunId, status, score, checksum, benchmarkChecksum }). A connector that refuses leaves the gate row with report_error set and the job retries. A passing gate promotes exactly the prompt version that was measured.

Because a gate names one checksum, the prompts page marks a prompt as drifted when its current checksum differs from the one the last gate measured — the application has changed the prompt since it was proven.

---

Credentials and the vault

One table, provider_credentials, holds every organization secret. The environment column says which module a row belongs to:

A label is unique per (org_id, provider, label, environment), so two organizations may both have a key labelled "Primary".

Reads go through src/lib/evals/credentials.ts:

credentialFor(orgId, provider, preferred?, environments = ["evaluation", "staging"]): Promise<string>
findCredential(orgId, provider, environments): Promise<CredentialMatch | null>
orgAiCredential(orgId, provider): Promise<CredentialMatch | null>   // "ai", then "evaluation"
listOrgCredentials(orgId): Promise<CredentialListRow[]>             // no ciphertext, last4 only

preferred is the id stored on a variant; it is honoured only when the row still belongs to that organization, so a stale or forged id cannot reach another tenant's secret. Plaintext is decrypted for exactly one provider call (withDecryptedCredential) and never returned to a page or an API response.

Storing a key (credentials.write, plus a passkey verified in the last ten minutes) queues credentials.validate when the provider is one the runner can call. The job stores the verdict on the row: active with validated_at, or invalid with the provider's message in validation_error. Revoking needs a reason and a fresh passkey, and is permanent.

The model catalog

src/lib/evals/model-catalog.ts is the reviewed catalog: official model ids, modalities, reasoning ladders and list prices, with CATALOG_VERIFIED_ON recording when the prices were last checked against the providers. The seed converges it into model_catalog (org_id NULL) on every run and disables — never deletes — a model that has left the catalog, so historical runs still resolve their rows.

Every entry also carries a pricingSource: the provider page its list price was read from. It is a required field, so a price cannot enter the catalog without somewhere to re-check it, and a review is a matter of opening each URL rather than guessing where a number came from. Prices were last re-fetched from those pages for all seven providers on 2026-09-03.

Two prices in the catalog are the full rate rather than the cheapest one a request might attract, and both say so in their note: DeepSeek publishes a peak and a half-price off-peak rate, and the catalog holds the peak one; OpenAI and xAI publish a second, higher rate for long prompts, and the catalog holds the standard one. estimateCostUsd models neither tier, which is why the run detail labels its number a list price and not a bill.

An organization narrows the catalog through model_catalog_overrides; enabling or disabling a model on /o/{org}/models writes one override row and never touches the shared catalog. A model is offered to a run only when the platform row is enabled and the organization has not opted out.

Judge configuration

The semantic grader uses a second model as a blinded judge.

• Provider and model come from SPM_EVAL_JUDGE_PROVIDER and SPM_EVAL_JUDGE_MODEL (default openai / gpt-5.6-luna).
• The key is the organization's own credential for that provider, looked up in ai then evaluation.
• With no such credential the platform key (SPM_AI_PROVIDER, SPM_AI_MODEL, SPM_AI_API_KEY) is used — and then the platform's provider and model are used with it, because a key only works for the provider that issued it.
• With neither, the grader fails loudly rather than silently skipping a signal.

---

Worker jobs

Neither is recurring: both are enqueued by a console mutation.

Pages

Voice artifacts are streamed by /api/console/evals/artifacts/{id}?org={slug}, which resolves the artifact through its run's organization and refuses anyone who is not a member with evals.read.

Verification

SPM_TEST_DATABASE_URL=… pnpm --filter @saaspro/web test -- tests/evals

The suite covers the catalog and its prices, dataset parsing and PII screening, the credential environment matrix, the runner's text, tool, retrieval, voice and gate paths against mocked providers and a mocked connector, the console mutation chain, cross-tenant refusals through the real routes, and the pages an operator reads.

Production quality review

Quality grades reviewed copies of observed production answers, without regenerating them. Sampling is paused by default and bounded by environment, retention and a daily case limit. See Production AI quality for the review queue, deterministic checks, separate optional judge approvals, version comparison and unknown cost coverage.

Publishing a reviewed corpus creates a frozen private dataset for the existing comparison lab. Its cases, results and any artifacts require people.pii.read and follow the original source's retention and deletion. These datasets cannot be appended to. The optional production judge budget does not govern new generation runs in the existing offline lab.

What the assistant can do here

Ask reads the production side of this module where the management API exposes it: AI traces with their costs, and sampled quality scores. That answers "what did conversations cost last week, and did quality move?" without leaving the page you are on.

Starting a run, editing a dataset, changing a grader or touching a model credential stays in the console. Those are authoring actions with a blast radius the assistant has no business guessing at, and the vault never hands a key to a tool in any case.