Did the new button do that?
Teams ask it the week after every launch, and a before-and-after comparison cannot answer it. A recipe app moves its Save button into a bar that stays on screen while the reader scrolls, and saves go up on the week before. But it is early December, when saves always rise, and that week's newsletter featured twelve recipes. Everything that changed between the two weeks sits in the difference, the bar among them.
An A/B test, or controlled experiment, runs both versions at once and assigns each user at random: the control group sees the page as it was, the variant group sees the sticky bar. December and the newsletter reach both groups alike, so a difference in saves too large to be noise can be put down to the bar.
One person, one version
Random does not mean a coin flipped on every page view: a reader who sees the bar on Monday and the old page on Tuesday has had both versions, and their saves cannot be put down to either. The unit of assignment is the user, for the whole test; a logged-out reader on a phone and a laptop can still land in both groups, a leak that blurs the difference.
The usual assignment hashes the user id into one of 10,000 buckets, 0 to 4,999 for control and 5,000 to 9,999 for the variant, with a key unique to the experiment so the same users are not in the variant of every test; Feature flag services: targeting, rollouts & kill switches covers the hashing and the stable id behind it. The rule that belongs to experiments: once the test starts, nobody moves a bucket boundary or changes the key, because that sends users who saw one version into the other, and what they do afterwards reflects both versions, not the one they are counted in.
Counting the people who saw it
The hash can place every user of the app in a group, but the bar exists only on recipe pages, and 40% of the users who open the app during the test never reach one. Counted anyway, they spread the lift over people who could not have seen the bar, and any saves they make elsewhere in the app are noise the bar did not cause: the difference shrinks more than the noise does, and the test needs more users.
So each user counts from their first exposure, the moment a recipe page is actually drawn in their version, logged the same way for control users on the old page. If the variant changes who gets that far, the groups stop being comparable; the split check below can reveal that, but a balanced split does not prove it did not happen.
- The Recipe API hashes the user into a bucket in memory, with rules pushed from the experiment service, and serves that version.
- The app draws the page and sends an exposure event: user, experiment, variant, time.
- The Recipe API emits a save event for every save, as it would without any experiment, and both kinds of event flow through Kafka into the warehouse (Analytics data: OLTP vs OLAP & the warehouse).
- A results job joins each exposed user to their saves in the 7 days after first exposure, per group.
Assignment happens in the API's memory; exposures and saves arrive as events, and the results job joins them by user.
Pick the number before the test
The primary metric, chosen before the start, is the share of exposed users who save at least one recipe within 7 days. Chosen afterwards, it could be whichever number happened to move. The bar also covers a spot where an ad used to show, so guardrail metrics must not get worse beyond a margin set in advance: ad revenue per user, page load time, crash rate. One that breaks badly is a reason to switch the variant off, a rules change the flag service delivers within seconds.
Is 11% really more than 10%?
After two weeks of exposures and 7 more days, so the last users exposed have their full window, 1,500 of 15,000 exposed control users saved a recipe (10.0%), against 1,650 of 15,000 in the variant (11.0%). Random groups rarely come out exactly equal: is one point more than chance?
The p-value answers a narrow version of it: if the bar had no effect at all, how often would two random groups this size differ by at least this much, in either direction? Here about 0.5% of the time (p ≈ 0.005), under the 5% threshold fixed in advance, so the result is statistically significant. That is not a 99.5% chance that the variant is better: the p-value is computed assuming no effect, and says how surprising the data would be then. The 95% confidence interval for the difference, about 0.3 to 1.7 points, says more for a decision: the bar helps, by a little or by a lot.
The test's size was fixed first. The minimum detectable effect, the smallest lift the test is sized to detect, was one point, and the team wanted an 80% chance of detecting it if real (the test's power) at a 5% false-positive rate: the standard formula for two proportions gives about 14,750 exposed users per group. Halve the effect, 10% to 10.5%, and it takes about 57,800: about four times the users for half the effect, because noise shrinks only with the square root of the sample.
Green on day three
On day three the dashboard shows the variant ahead with p = 0.03. But the 5% false-positive rate holds for one reading at the planned sample. In a simulation of this test with no real effect, read daily for 14 days, about 22% of runs crossed the threshold on at least one day. Either wait for the planned sample, or use a method built for repeated looks, such as a sequential test, which adjusts the threshold so the false-positive rate stays where it was set.
Check the split first
Before reading the saves, compare the group sizes. A 50/50 split of 30,000 exposed users should come out close to 15,000 each; 15,600 against 14,400 is a gap chance produces at this size only a few times in a trillion. That is a sample ratio mismatch (SRM): something loses users from one group unevenly, perhaps a heavier variant page that some readers leave before the exposure event is sent. The missing users are not random, so the result cannot be trusted: fix the cause and run again.
Two tests on one recipe page
Another team wants to test a new ingredient list where the bar overlaps it, and a user in both variants would see a combination nobody designed. Tests that could interfere share a layer: the layer hands half its buckets to the bar test and half to the ingredient-list test, each splits its half into control and variant, and no user is in both. Each test now gets half the traffic and runs longer. Tests unlikely to interact, such as a ranking change on the search page, go in another layer, hashed with its own key, and overlap, since each of the bar's groups then gets about the same mix of search versions.
Broken, or better?
The build that carries the bar may first go out as a canary, a small slice watched for errors and rolled back when they rise (Progressive delivery: blue-green, canary & rollback): it asks whether the new version is broken, and a few minutes of errors can answer that. An experiment asks whether it is better, which needs a random split that holds still, a metric chosen in advance and a planned sample, because a one-point lift hides in noise where a crash does not.
Building the platform
A/B Experimentation Platform asks for this path in up to 16 nodes. Build it with bucketing inside the service, not a network call per page view, exposure sent from the screen that shows the version, a job that joins exposures to conversions and checks the split before significance and guardrails, and layers on the experiment configuration. Recommendation Engine asks for the same path with a ranking model as the change; the ranking itself is Recommendation systems: retrieval, ranking & cold start.
A clean two-week result still leaves two things unmeasured. A new bar draws taps partly because it is new, so a lift that fades from the first week to the second may not last. And the comparison assumes one user's version does not change another's outcome, which fails in a marketplace, where a variant that helps some buyers win auctions takes those wins from control buyers; such tests often assign whole cities or markets instead of users.