Sampled production AI quality

Open Evaluations → Quality in an application with Evaluations and Metrics enabled. This evaluates answers already delivered by the application. It does not regenerate production answers, block a bad answer, publish a connector release gate, or measure all application traffic.

Select, review, freeze

Enable production AI collection and send spans through the secret OTLP collector. Text evaluation requires the separate AI collection environment and exporter content opt-ins. No new SDK call or provider request is needed for selection.
On an observation, choose Select for quality review, or enable a sampling policy in Quality → Sampling. Sampling starts paused, applies to newly accepted observations only, and uses a stable hash with an environment salt. Service and response-model filters are exact matches. Sampling runs under the same trace admission lock; accepted exporter retries cannot add another case.
Reveal private evidence with a recent passkey, sanitize the retained input and observed output, supply an expected answer or tool, and accept the case. Missing content cannot be invented through metadata-only ingest. The reviewed copy may differ from the original because an operator deliberately edited it.
Create a quality run from up to 100 accepted cases. The corpus freezes exact case checksums and the evaluator configuration. Accepted cases are immutable; new grader choices produce a new run over the same case IDs and checksums.

The queue holds at most 2,000 retained cases per environment, including pending cleanup. Daily admission is 1–500 cases, shared by sampling and manual selection. The receipt shows the last recorded UTC day and admitted/budget-dropped counts. Sampling applies after OpenTelemetry and AI collection sampling, delivery and storage limits, so it is a sample of observed telemetry, not a traffic census or unbiased quality estimate. Changing the sampling policy does not relabel previous selections.

Private evidence and retention

Metadata reads use evals.read. Sampling, selection and deletion use evals.write. Accepting a case requires evals.review, evals.write and people.pii.read; starting a quality run requires evals.run and people.pii.read. Within Quality, private source text, reviewed copies, rubric, judge rationale and human notes are disclosed only through an audited reveal with people.pii.read. Mutations use the normal same-origin, CSRF, Google plus passkey session chain, a passkey verified within ten minutes and a reason.

Reviewers must remove personal data and secrets. Screening rejects some obvious identifiers; it cannot prove that all PII has been removed. Ingest technical IDs, model labels and release names are caller-supplied metadata, not anonymization guarantees or People authorization.

Private case retention is 1–7 days, capped by the application and original AI source retention. A reviewed text case additionally expires with its original preview. Reducing retention, disabling AI previews, explicit source deletion or source expiry hides affected evidence immediately. Cases and result copies cascade on physical source deletion. Unavailable cases are removed from score numerators while the run's original selected count remains visible. No-data is unscored, never healthy.

Maintenance visits 25 least-maintained environments every five minutes, removes at most 500 private cases and 100 runs per environment, and expires associated approvals. At most 200 runs and 20,000 result rows are retained per environment. Application/environment deletion cascades. The audit ledger contains identifiers, digests and counts; it does not contain private case text or judge payloads.

Deterministic and human grading

The existing evaluation exact-match and tool graders are reused. Exact-match normalizes case and whitespace; the tool check compares the expected name with the captured tool name. Missing expectations remain unscored. Optional nonempty and JSON checks assess the reviewed output's presence and format, not factual correctness or safety. Metadata-only tool cases cannot pass a text check.

Human accept/review/reject verdicts remain separate from automated scores. Comparison views show each grader's mean and scored/selected denominator. Only identical frozen corpora are offered for evaluator comparison; release/model cohorts are labeled observational because their inputs may differ. A model judge is advisory and can be biased or wrong. Review disagreements and calibration cases before using scores to make a release decision.

Optional model judge and budgets

Model grading is a separate, explicit request. Version 1 uses only the fixed OpenAI Responses HTTPS endpoint with an active organization evaluation credential from the vault, scoped to this application or the organization. There is no platform-key fallback. The exact model must remain enabled in the organization's catalog. An AI collection price profile must already cover that model, provider openai, tier default, text-token unit and the dispatch time.

The requested immutable digest includes corpus hashes, grader/rubric versions, rubric text, exact model, credential ID and revision, reviewed rate snapshot, input/output limits and the estimated dollar ceiling. The normal approval queue requires a different operator. Dispatch rechecks both actors' current permissions, active account/membership, module/environment, credential, approval expiry, case availability and the digest. Approval expires after at most one hour. Changing a bound input requires another run and approval.

Each selected case permits at most one network claim. Calls contain at most 16 KiB of input, request 256–2,048 output tokens, use store:false, the default service tier, structured score output, no tools, no redirects, a 15-second timeout and a 64 KiB response cap. The worker visits three least-attempted runs every thirty seconds and sends at most one judge call per visited run. A pending approval does not block the remaining runs.

Before approval, integer picodollar arithmetic reserves each call at reviewed rates using input UTF-8 bytes plus an 8,192-token allowance and the maximum output tokens. The sum must fit the chosen estimated-USD ceiling. This is a conservative token estimate, not a provider billing guarantee: provider fees, unsupported units, changed prices, tokenizer behavior or reported usage can differ. If a known charge exceeds its reservation, remaining calls stop. Provider responses with missing usage, a missing or different model/tier or incomplete rate coverage retain an unknown cost rather than zero. Cached input is part of total input; reasoning tokens are part of total output and are not charged twice.

A timeout, invalid response or abandoned claim is recorded as unknown. It is never automatically resent and its reservation is not refunded. Cancellation prevents new claims; a request already sent may complete and incur a charge. No provider traffic is necessary to verify the deterministic path; use local HTTP stubs for optional-provider development tests.

Reuse in the offline lab

Publish reviewed dataset copies only reviewed input and expected answer to a frozen dataset in the existing evaluation lab. All selected cases must still have retained reviewed text. This action makes no provider request. The corpus cannot be appended to through the ordinary import form. Its input template is {{question}}.

Published cases keep their source dependency. Source expiry is enforced on offline run admission, result reads and worker selection; physical case deletion removes its offline results too. These runs require people.pii.read in addition to the ordinary evaluation permissions. Text comparisons in the existing lab generate new candidate outputs using that lab's existing provider workflow; the production quality judge approval and budget apply to its judge runs only.

Private Management API

GET /api/v1/apps/{app}/evals/quality?env=production returns retained run metadata. Use id=<run-id> for per-grader evidence and missing-case counts, or group=cases&status=pending for the review queue. Queue pagination uses the returned opaque cursor. The token needs read scope and evals.read; both modules must be enabled. Ingest keys cannot read or administer this flow. Management responses never include private text, rubric, rationale or reviewer notes.

Primary references

The platform reuses its own evaluation graders. It does not require OpenAI's hosted Evals service or upload a dataset there.