Product experiments

---

title: Product experiments description: Run a randomized conversion experiment with a guardrail, exposure tracking and uncertainty.

Open Flags → Experiments. Enable Flags and Analytics for the application. Readers need flags.read and analytics.read; operators additionally need flags.write.

Design and launch

Create a draft with a unique SDK key, hypothesis, assignment identity, primary track event and guardrail track event. Higher primary conversion is better; lower guardrail occurrence is better. The guardrail margin is the allowed absolute increase in percentage points.

Choose an enrollment duration of 7–60 days, a conversion window of 1–168 hours and a minimum of 100–100,000 mature identities per variant. Determine the minimum from a power analysis using your baseline and smallest useful effect; the configured minimum alone does not guarantee adequate power. Audience inclusion is 1–100%; included identities split 50/50.

Validate both variants and event delivery in staging. Launch requires a reason and a passkey verified within ten minutes. The SDK key, metrics, audience, duration and allocation become immutable. A new hypothesis needs a new experiment. Early stop invalidates the result and cannot be reversed. Experiments have dedicated assignments; ordinary feature flag reads, overrides and targeting rules do not enroll anyone.

Browser

Use the current @saaspro/browser build with persistent analytics consent granted. Call exposure for both variants, at the point where the experience will be used. Do not call it only inside the treatment branch or when prefetching a screen.

const variant = await spm.experiments.expose("checkout-v2");
renderCheckout(variant === "treatment" ? "new" : "current");

// Send each event only when its outcome actually happens.
spm.track("checkout_completed");
spm.track("checkout_failed");

The returned value is control, treatment or null. Null means no enrollment because of consent, missing identity, audience sampling, lifecycle, limits or a request failure. Render your normal fallback. Never infer assignment from a failed request. Repeating exposure retains the first assignment and timestamp without increasing the sample count.

Browser experiments are disabled before consent and in ephemeral mode. Revocation/reset aborts in-flight requests and discards stale responses. An exposure already accepted by the server cannot be undone by aborting a later network response.

Node and mobile

// @saaspro/node: pass the same identifier to exposure and outcomes.
const variant = await server.experiments.expose("checkout-v2", {
  distinctId: user.id,
});
server.track({ event: "checkout_completed", distinctId: user.id });

// @saaspro/react-native: requires granted consent and persisted identity.
const mobileVariant = await mobile.exposeExperiment("checkout-v2");
await mobile.track("checkout_completed");

For a signed-in user experiment, identify first and use the same distinctId on every client and server outcome. For a browser/installation experiment, carry the same anonymousId to server outcomes if needed. Identity aliases and person merges do not change the assignment unit. Changing IDs creates a different unit; cross-device consistency requires a stable signed-in ID. Raw identifiers are not retained in experiment tables; per-experiment HMAC hashes are used.

Direct HTTP: POST /api/v1/experiments/expose accepts {experiment, anonymousId?, distinctId?, key?}. Use a public/secret ingest key in the body or Bearer header. Browser text/plain is supported with the application's origin allowlist. Responses are uncached. The endpoint accepts no client-supplied variant or exposure time.

Outcome contract

Only accepted ordinary track events with an exact configured name and the selected identifier count. Each metric is binary per exposed identity; repeats, duplicate deliveries and multiple purchases cannot inflate conversion. Events must occur after the first server-acknowledged exposure and before its conversion window ends. Outcomes arriving within 24 hours after that individual window closes are accepted; future-dated and pre-exposure events are excluded. Await exposure before sending outcomes, and keep SDK clocks accurate.

The existing ingest validation, deduplication, tracking plans, bot filtering, sampling and budgets still apply. Ensure both metric names are admitted without sampling or asymmetric filtering. Missing or selectively dropped outcomes can bias results even if allocation looks healthy. These metrics describe reported conversions; they are not verified revenue, amounts, averages or ratios. No automatic rollout decision changes a feature flag.

Read results

Only identities whose complete conversion window has elapsed enter the conversion-rate denominator. Allocation health uses all exposures. Enrollment stops at the scheduled end; existing identities retain their variant through the observation and arrival period. New identities receive null. After the full duration, conversion window and 24-hour arrival allowance, the report becomes final. The worker freezes aggregate results and removes expired identity evidence according to application retention. Aggregate final results remain until experiment/application deletion.

Each metric displays its observed rate, absolute difference, relative lift where the baseline is nonzero, and an approximate 97.5% Agresti–Caffo interval for treatment minus control. Two predeclared comparisons share a 5% family error budget via Bonferroni. These are fixed-horizon intervals; interim viewing is descriptive and does not justify stopping early. The method is an approximation, not a probability that a variant is best. See the NIST method.

A treatment improvement requires the primary interval wholly above zero and the guardrail upper bound within the allowed margin, after both variants reach the minimum sample. Otherwise the report identifies harm, insufficient data or an inconclusive result. It does not claim equivalence from a nonsignificant difference.

Allocation checks use a one-degree-of-freedom chi-square approximation for the configured 50/50 split, after at least 20 exposures. A p-value below 0.001 blocks a winner recommendation. Investigate eligibility, assignment and delivery; passing this check cannot prove that all instrumentation is unbiased. See Microsoft's SRM guidance.

Limits and management

Each environment permits 100 lifetime experiment keys and ten concurrently collecting experiments. Each experiment stores at most 200,000 identities; reaching the cap before enrollment ends invalidates the result. Exposure requests share a per-key, per-instance budget of 600 per ten seconds. Body size is limited to 2 KiB. Known bots are excluded for public keys, consistent with normal ingest.

Application retention must initially exceed the planned duration, conversion window and arrival allowance. Reducing retention below live evidence invalidates that experiment; expired evidence is excluded from reads immediately. Run the worker for final snapshots and physical cleanup. Stopping an experiment disables its exposure and outcome collection. Deletion requires an audited reason and a fresh console passkey; it removes design, measurements and snapshots while reserving the key against stale SDK calls.

The scoped Management API supports GET /apps/{app}/experiments, GET /experiments/{id}, and POST /experiments commands create, edit, start, stop, delete. Every command except create includes id and optimistic revision; launch, stop and deletion also require reason. Use ?env=production|staging|development. Returned reports contain aggregates, never identity hashes or assignment salts.