Aggregation types

A metric is "events aggregated per user." The aggregation function turns each user's event stream into a single number that the analysis pipeline averages across the experiment arms. Pick the right one and your power calculations are honest; pick the wrong one and a single outlier can swing your result.

The five types

Conversion

The simplest and statistically the friendliest. Per user: did the event happen at least once during the analysis window?

shipeasy metrics create purchase_conversion \
  --event purchase \
  --query 'count_users(purchase)'

Each user contributes a 0 or 1. The metric mean is the fraction who converted. Variance is bounded by p(1-p), which caps at 0.25. Power calculations are cheap; the t-test behaves; you can't have an outlier.

Use this whenever the question is yes/no.

Count

Per user: how many of the event happened?

shipeasy metrics create sessions_per_user \
  --event session_start \
  --query 'count(session_start)'

Each user contributes an integer ≥ 0. Means and variances behave reasonably for low-volume events (0–5 per user) and get long-tailed for high-volume ones (a power user with 200 sessions skews the mean). For long-tailed counts, consider:

• Tightening the winsorize percentile (--winsorize 95 instead of the 99 default) to clamp the heavy tail.
• Switching to count_users(session_start) (did they have at least 1 session?) if the count distinction doesn't drive your decision.

Sum

Per user: sum a numeric property across all matching events.

shipeasy metrics create revenue_per_user \
  --event purchase \
  --query 'sum(purchase, revenueCents)'

The classic "did this experiment make us more money per user." Each user contributes their total revenue. Non-purchasers contribute 0.

Sums are heavy-tailed. One $50,000 enterprise purchase can swing the mean for thousands of users. The metric's --winsorize percentile clamps that — the CLI default is --winsorize 99 and is rarely wrong.

Mean

Per user: average a numeric property across their events.

shipeasy metrics create avg_order_value \
  --event purchase \
  --query 'avg(purchase, revenueCents)'

There's a trap with avg: should users with zero matching events count as 0, or be dropped from the denominator entirely? Today the DSL's avg averages across all exposed users — non-purchasers contribute 0 ("average revenue per exposed user"). To answer the "average across buyers only" question instead, define two metrics and divide via the ratio aggregation, or compute the per-buyer average offline.

Ratio

Per user: numerator-event count divided by denominator-event count. Express it with ratio(...) between two count arms — count or count_users are the only arm aggregations supported.

shipeasy metrics create click_through_rate \
  --event click \
  --query 'ratio(count(click), count(impression))'

Use ratios for inherently-ratio questions: clicks per impression, conversions per visit, errors per request. Don't compute the ratio yourself and store it as a mean — the math is different.

Why it matters: the naive ratio-of-means (mean of numerators divided by mean of denominators) under-states the variance. Shipeasy uses the delta method to compute the variance correctly — which means the p-values and confidence intervals you read on the dashboard are honest. If you had computed clicks/impressions yourself per user, then taken the mean, you'd be calculating the wrong thing.

The dashboard shows numerator and denominator means alongside the ratio so the math is auditable.

Outlier handling

Sums and means need outlier handling. Two options:

Winsorize at a percentile. Anything above is clamped to the value at that percentile.

shipeasy metrics create revenue_per_user \
  --event purchase \
  --query 'sum(purchase, revenueCents)' \
  --winsorize 99

99 (default), 95, 90. Lower percentiles clamp more aggressively. Trims the rarest 1% (or 5% / 10%) to the value of that percentile, keeps the body of the distribution intact.

For a hard ceiling (e.g. session length can't reasonably exceed 4 hours, revenue per user can't exceed your enterprise plan price.

The trade-off: clamping reduces variance (good — narrower CI, easier to detect lift) at the cost of slightly understating the true effect when the variant genuinely moves the tail (rare).

Filters

You can tighten what counts toward a metric with filters. Same shape as feature flag targeting rules, applied to the event's properties payload before aggregation:

shipeasy metrics create organic_purchase \
  --event purchase \
  --query 'count_users(purchase{channel="organic"})'

Now only purchase events with channel == "organic" count. Multiple predicates inside {} are ANDed (commas separate them). String values must be double-quoted in the DSL; =~ does regex match for numeric / pattern filters.

Common shapes:

# Web only (exclude mobile app purchases)
'count_users(purchase{platform="web"})'

# Geographic slice
'count_users(purchase{country=~"^(US|CA|GB)$"})'

# Multiple — ANDed inside the same {}
'count_users(purchase{platform="web", country="US"})'

Filters run cheaply during aggregation. You can have many metrics that share the same underlying event, each filtering differently — no need to log the same purchase twice with different names.

Direction

The dashboard infers each metric's "good" direction from its name + the metric type when you attach it to an experiment as primary or guardrail. Words like error, fail, latency, crash, churn, bounce are flagged as lower-is-better (see apps/ui/src/app/dashboard/[projectId]/metrics/metrics-list-view.tsx for the exact heuristic). Override on the experiment-metric attachment in the dashboard when the heuristic gets it wrong.

When you don't have the event yet

A metric definition can be created before any matching events exist. The pipeline picks them up once they start arriving. Use this to define metrics for an upcoming experiment ahead of time, then deploy the track() call as part of the same release.

What you can't do: create a metric, attach it to a running experiment, and have it back-fill against events from before the experiment started. Analysis runs forward from attachment time.

Inspecting a metric

# What does the metric definition look like?
shipeasy metrics get purchase_conversion

# What's the historical baseline distribution? (Powers the MDE calculator.)
shipeasy metrics baseline purchase_conversion --window 30d

# Dry-run aggregation against a recent sample
shipeasy metrics preview purchase_conversion --sample 1000

The baseline command is the one to run before launching an experiment — it tells you the metric's variance, which feeds the power calculation directly.