Quickstart

This walks you through the metric pipeline end to end: pick a metric, log the underlying events, create the metric definition, attach it to an experiment, and read the result. By the end you'll have a primary metric that drives a real ship/no-ship decision.

1. Pick what to measure

Before you create anything, answer one question: what number tells you the experiment worked?

For a checkout-flow rewrite: probably purchase_conversion — did exposed users buy? For a new paywall: probably subscription_conversion — did they sign up? For a homepage redesign: typically session_engagement — did they click past the fold?

Pick one. Two is fine if they're closely related. Five primary metrics means you haven't decided yet — go back and decide.

The metric needs to be:

• Computable from events you already log (or are willing to start logging).
• Specific to the change — not "DAU," which moves for a hundred reasons. "Did this exposed user convert after exposure."
• Reasonable to detect — a 1% conversion lift on 1,000 users a day will take a month. Check the power table before committing.

2. Define the metric

The simplest case — did event X happen for this user at least once?

shipeasy metrics create purchase_conversion \
  --event purchase \
  --query 'count_users(purchase)'

You now have a metric definition. It does nothing on its own; it tells the analysis pipeline how to aggregate the underlying events per user.

For a revenue metric (sum the revenueCents property across purchase events per user):

shipeasy metrics create revenue_per_user \
  --event purchase \
  --query 'sum(purchase, revenueCents)'

For more aggregation types, see Aggregations.

3. Log the underlying events

The metric is a rule for aggregating events. The events themselves come from your code:

import { flags } from "@shipeasy/sdk/server";

export default async function CheckoutSuccess({ order }: { order: Order }) {
  flags.track(order.userId, "purchase", {
    revenueCents: order.totalCents,
    currency: order.currency,
    channel: order.acquisitionChannel,
  });
  return <ThankYou />;
}

A few rules that matter:

• The first argument is the userId. Without it, the event can't be assigned to an experiment exposure — flags.track requires it as a positional arg.
• Properties become filterable. You can later add a metric like "organic-channel purchases" by filtering on channel inside the DSL selector (count_users(purchase{channel="organic"})).
• track() is fire-and-forget. It returns void; the event flushes asynchronously. Don't await it expecting a delivery guarantee — it's analytics, not transactional state.

Deploy this. Events start flowing. The metric definition will pick them up on the next analysis window (daily by default).

4. Attach the metric to an experiment

Attaching metrics (primary / guardrail / secondary) to an experiment is a dashboard operation today — open the experiment detail page, scroll to Metrics, click Attach on purchase_conversion, pick primary, save. The same can be done via the Admin API.

Create a guardrail alongside it — don't tank page load while you're at it:

shipeasy metrics create p95_page_load_ms \
  --event page_view \
  --query 'p95(page_view, loadTimeMs)'

Then attach it as guardrail on the same experiment detail page.

The metric you mark primary drives the ship/no-ship decision. A metric marked guardrail must not regress, even if the primary moves. See Guardrails.

5. Read the result

The dashboard's experiment page shows lift, confidence interval, and p-value per metric, refreshed once per analysis window:

purchase_conversion control v1 lift p 95% CI
4.8% 5.2% +8.3% 0.018 [1.2%, 15.6%]

p95_page_load_ms (guard) 860 ms 862 ms +0.2% 0.71 [-1.1%, 1.5%]

You read this as: paywall v1 lifted conversion by 8.3% (p=0.018, 95% CI excludes zero) and did not regress page-load time. Ship it.

What if the result is flat or noisy? Two checks:

1. Are events actually landing? Check the Events tab on the experiment — exposed users should have non-trivial event counts. If the count is zero, your track() call isn't running for the variant.
2. Is the experiment powered? The dashboard shows MDE alongside the lift. If MDE is ±10% and the real lift is +2%, you can't tell — run longer or accept the null.

See Power & sample size.

Where to next