Metrics

A metric is a number computed per user from a set of events. Every experiment has at least one primary metric (the thing you're trying to move) and ideally a couple of guardrails (things you don't want to break). Get these two choices right and most of your experimentation problems go away on their own.

Aggregation types

Shipeasy ships four aggregation functions. Each one collapses a user's event stream into a single number per analysis window:

Conversion is the simplest and statistically the friendliest — the variance is bounded by p(1-p), so power calculations are cheap and the t-test behaves. Means and sums need more samples and benefit from outlier handling (see below).

When the metric is "no event"

For count_users (the binary "did it happen?" aggregation), a user with zero matching events contributes 0. For avg, the DSL averages across all exposed users — non-purchasers contribute 0 and pull the mean down. To answer "average among buyers only" instead, express it as a ratio (ratio(sum(purchase, revenueCents), count_users(purchase))) or compute the per-buyer cohort metric offline.

Creating a metric

Metrics are defined by a small query DSL — the same one the dashboard generates when you pick aggregation, source event, and filters in the "New metric" form. Run shipeasy metrics grammar for the full spec; the most common shapes:

# binary conversion on `purchase`
shipeasy metrics create purchase_conversion \
  --event purchase --query 'count_users(purchase)'

# revenue per user (includes non-buyers as $0)
shipeasy metrics create revenue_per_user \
  --event purchase --query 'sum(purchase, value)'

# average order value (averaged across buyers only)
shipeasy metrics create avg_order_value \
  --event purchase --query 'avg(purchase, value)'

# sessions per user
shipeasy metrics create sessions \
  --event session_start --query 'count(session_start)'

Or in the dashboard: Experiments → Metrics → New metric.

Filtering events into a metric

You can tighten what counts toward a metric with filters — written inline in the DSL selector as event{attr=value}. Filters run against the event's properties payload before the aggregation:

shipeasy metrics create organic_purchase \
  --event purchase \
  --query 'count_users(purchase{channel="organic"})'

Now organic_purchase only counts users with at least one purchase event whose channel property equals "organic". Compose multiple filters inside the same {} with commas — they're ANDed. Operators are = (equal), != (not equal), =~ (regex match), !~ (regex non-match). Common shapes:

# Web purchases only (exclude mobile app)
'count_users(purchase{platform="web"})'

# Orders priced above $10 (string-coerced, server compares numerically
# when the source event declares the label as numeric)
'sum(purchase{value=~"^[1-9][0-9]+$"}, value)'

# Multiple conditions — commas inside {} are ANDed
'count_users(purchase{platform="web", country="US"})'

Primary vs guardrail

When you attach metrics to an experiment, you mark each one as primary or guardrail.

• Primary: the metric you're trying to move. The experiment's "did it win?" decision is read off these. Pick one or two; if you have five primaries you have none.
• Guardrail: metrics that must not regress. A new checkout flow had better not tank page load time. Shipeasy flags any guardrail that moves significantly in the wrong direction, even if your primary won. By default, the dashboard treats a guardrail as failed when p < 0.05 and the lift is in the bad direction.

A common pattern for a checkout experiment:

The primary tells you "did the change work?", the guardrails tell you "did it work without breaking something else?". A win that ships is one where the primary moves and no guardrail regresses.

Outliers

For sum and mean, a single $50,000 enterprise purchase can swing the mean for thousands of users. Shipeasy supports two outlier handlers per metric:

• Winsorise at a configurable percentile (default p99). Anything above is clamped to the p99 value of the combined sample. Default for sum and mean.
• Cap at an absolute value. Use this when there's a domain-specific ceiling (e.g. a maximum plausible session length).

# Winsorise at p99 (this is the CLI default; --winsorize 99 is implicit)
shipeasy metrics create revenue_per_user \
  --event purchase --query 'sum(purchase, value)' \
  --winsorize 99

# Cap at p95 instead
shipeasy metrics create revenue_per_user \
  --event purchase --query 'sum(purchase, value)' \
  --winsorize 95

Winsorising is the default and is rarely wrong. The trade-off: clamping reduces variance (good — narrower CI, easier to detect lift) at the cost of slightly understating the true effect when the variant genuinely moves the tail (rare).

Ratio metrics

Some questions are inherently ratios — "clicks per impression", "conversion per visit". Express them with the over keyword between two selector arms in the DSL:

shipeasy metrics create click_through_rate \
  --event click \
  --query 'ratio(count(click), count(impression))'

The two ratio arms must each be count or count_users — the DSL doesn't allow sum/avg in ratio position.

Ratio metrics use the delta method to compute variance correctly (the naive ratio-of-means understates variance and inflates significance). The dashboard shows the per-user numerator and denominator alongside the ratio so the math is auditable.

Segmentation

Once results are in, slice them by user attribute:

purchase_conversion          control  v1     lift     p
  all                          4.8%   5.2%   +8.3%   0.018
  country = US                 5.1%   5.4%   +5.9%   0.041
  country = EU                 4.5%   4.9%   +8.9%   0.062
  plan = pro                   7.8%   9.4%   +20.5%  0.001
  plan = free                  3.9%   4.0%   +2.6%   0.401
  device = mobile              4.2%   4.9%  +16.7%   0.006
  device = desktop             5.6%   5.6%   +0.0%   0.974

Segmentation works on any attribute you register on the project. The platform stores attributes_at_exposure on the $exposure event, so segmentation is on the user's state at the moment they entered the experiment — a user who upgraded from free to pro mid-experiment stays in the free segment. This is the right behaviour: it's the only way segmentation is causal.

You don't need to ask for a segment up front — the platform recomputes them on demand from the existing exposure + event data, no re-bucketing required.

Sample size and power

You can't detect a 1% lift with 100 users. Shipeasy shows you a power estimate before you start the experiment, given your traffic and the metric's historical variance — so you know roughly how long the test will need to run.

Quick rules of thumb (80% power, α = 0.05, two-sided):

If your traffic is low, increase your effect-size requirement (don't bother shipping for 0.5% lift) or run the experiment longer. The platform will not stop you from running an underpowered test, but the dashboard will mark it insufficient_power and warn that the absence of a significant result doesn't mean the absence of an effect.

MDE (minimum detectable effect)

The flip side of "how many users do I need" is "given the users I have, what's the smallest lift I can reliably detect?" The dashboard surfaces an MDE alongside every running experiment, recomputed daily. If your MDE is +8% and the lift you're hoping to see is +2%, you're going to need to run the experiment for substantially longer — or accept that you won't be able to tell.

API · metrics.create

Where to next