Running a low-traffic experiment

Most A/B advice assumes you have millions of users. You don't. You're testing a change on the settings page that maybe 400 accounts touch a week, or you're a B2B product bucketing by company where the unit count is small by construction. The experiment is real and worth running — it just needs different expectations than a high-traffic one.

The traps are: peeking at a tiny sample and calling a winner on noise, or running forever without ever crossing significance because the effect is smaller than your sample can resolve.

1. Size it honestly first

Before you start, run the math. The power calculator turns your baseline rate, the smallest effect worth shipping (MDE), and your weekly traffic into a runtime. With low traffic you'll usually find one of two things:

• The runtime is weeks, not days. Fine — plan for it, and don't peek early.
• The MDE you can detect is large. If you can only ever resolve a 10pp lift, accept that small wins are invisible to this test and only run it for changes you expect to be big.

2. Use sequential testing if you must watch

If you genuinely need to monitor the experiment as it runs (e.g. to bail on a regression), use sequential testing rather than repeatedly eyeballing a fixed-horizon p-value. Sequential tests are designed to be looked at continuously and keep the false-positive rate controlled across many peeks — they trade a little power for the right to stop early. Enable it on the experiment's stats settings.

import { configure, Client } from "@shipeasy/sdk/server";

configure({
  apiKey: process.env.SHIPEASY_SERVER_KEY!,
  attributes: (s) => ({
    user_id: s.userId,
    company_id: s.companyId, // bucket by account if units are accounts
  }),
});

const flags = new Client(session);
const { group } = flags.getExperiment("settings-redesign", { variant: "control" });

if (group === "treatment") renderRedesign();
else renderControl();

// fire the goal metric event on the action you care about
flags.track("settings_saved", { user_id: session.userId });

3. Give it runtime, then read it

Let it run for the full pre-computed horizon. When you read the result on the Analysis page:

• A non-significant result at low n is not "no effect" — it's "this test couldn't tell." Check the confidence interval: if it's wide and spans zero, you're underpowered, not flat.
• A significant result that survived the full run and a sequential boundary is trustworthy.

Rollout & measurement plan

• Maximize the units you have. Run a clean 50/50 split — uneven splits waste power you can't spare. Skip a holdout on this one; reserve scarce units for the comparison that matters.
• Pick a sensitive metric. A ratio or continuous metric (e.g. revenue per session) resolves faster at low n than a rare binary conversion. See Metrics.
• Pre-register the decision. Write down the MDE, the runtime, and "ship if X" before you start, so a noisy mid-run reading can't talk you into a bad call.
• Stack experiments instead of overlapping. With little traffic, run one test at a time per surface rather than diluting units across many.