Quickstart

Define the conversion metric

A metric wraps an event type and an aggregation function. For "did this user buy?" we use count_users over the purchase event (the binary did-it-happen-at-least-once aggregation).

shipeasy metrics create purchase_conversion \
  --event purchase \
  --query 'count_users(purchase)'

Or in the dashboard: Experiments → Metrics → New metric.

Tip: also add guardrails — secondary metrics that should not regress. For checkout, page load time and error rate are good guardrails.

Create the experiment

Two groups, equal weight, one parameter (label):

shipeasy experiments create checkout-cta \
  --allocation 100 \
  --groups '[
    {"name":"control","weight":50,"params":{"label":"Pay"}},
    {"name":"v1","weight":50,"params":{"label":"Buy now"}}
  ]' \
  --params '{"label":"string"}'

--allocation 100 means 100% of eligible users are in the experiment, split 50/50. Drop allocation to e.g. 20 if you want only a 20% sample of traffic to see any variant — the rest get neither, useful for a soft launch.

Attach purchase_conversion as the primary metric in the dashboard.

Wire the SDK

import { configure, Client } from "@shipeasy/sdk/server";

configure({
  apiKey: process.env.SHIPEASY_SERVER_KEY ?? "",
  attributes: (u) => ({ user_id: u.id, plan: u.plan, country: u.country }),
});

Then on the request path, bind a Client to the user:

const flags = new Client(currentUser);
const result = flags.getExperiment<{ label: string }>(
  "checkout-cta",
  { label: "Pay" }, // defaultParams
);

const label = result.params.label; // "Pay" or whatever the variant assigns

flags = shipeasy.Client(current_user)
result = flags.get_experiment(
    "checkout-cta",
    default_params={"label": "Pay"},
)
label = result.params["label"]

flags := shipeasy.NewClient(currentUser)
r := flags.GetExperiment("checkout-cta",
    map[string]any{"label": "Pay"})

label := r.Params["label"]

flags = Shipeasy::Client.new(current_user)
result = flags.get_experiment("checkout-cta", { label: "Pay" })
label  = result.params[:label]

Client flags = new Client(currentUser);
ExperimentResult r = flags.getExperiment("checkout-cta",
    Map.of("label", "Pay"));
Object label = r.params().get("label");

val flags = Client(currentUser)
val r = flags.getExperiment(
    "checkout-cta",
    mapOf("label" to "Pay"),
)
val label = r.params["label"]

$flags = new Shipeasy\Client($currentUser);
$r = $flags->getExperiment('checkout-cta', ['label' => 'Pay']);
$label = $r->params['label'];

let flags = try Client(currentUser)
let r = await flags.getExperiment(
    "checkout-cta",
    defaultParams: ["label": "Pay"],
)
let label = r.params["label"]

The first call to flags.getExperiment(...) for a given user logs an exposure event automatically. Subsequent calls in the same process don't re-log — exposures are deduplicated.

Track the conversion

Wherever the purchase succeeds:

flags.track(user_id, "purchase", { value: orderTotal });

The event lands in events store. The daily analysis job scans events and joins them against exposures by user_id.

Start the experiment

Until you start it, assign() returns inExperiment: false and the control params for everyone — safe.

shipeasy experiments start checkout-cta

Or click Start in the dashboard.

Read the results

Daily, the analysis cron computes per-metric, per-group results — lift, p-value, 95% CI — and writes them back to your project. Read them in the dashboard, or:

shipeasy experiments status checkout-cta

A typical result row:

purchase_conversion
  control  N=12,418  rate=4.8%
  v1       N=12,503  rate=5.2%   lift +8.3%   p=0.018   CI [+1.4%, +15.2%]

p < 0.05 and the CI excluding zero is your stop-condition. Promote v1 by adding a feature flag that mirrors the variant, then deleting the experiment.

Stop & clean up

shipeasy experiments stop checkout-cta

Stopping is safe at any time. Stopped experiments keep their config and results; they just stop assigning.