A sample ratio mismatch is a difference between the split a test was configured to produce and the split that actually arrived in the data.
An experiment configured to send half the traffic to each group will not produce two identical counts, because assignment is random and random counts wobble. The wobble has a known size, so the question of whether an observed split is ordinary has an exact answer. A split further from the target than chance allows means something in the assignment, the logging or the filtering went wrong.
The reason this matters more than any other check is what a mismatch implies about the users who went missing. They were not removed at random. A rule somewhere dropped a particular kind of person, such as everybody on a slow connection or everybody whose session ran long, and those people behave differently from the ones who remain. The comparison has stopped being between two randomly assigned groups, which was the only property that made it worth running.
The sections below define the check, give the observation from audience research that explains why teams resist it, work the arithmetic through on one test and then name the five stages of an experiment where a mismatch is created. The page closes on what a team does with a result that fails.
Twyman's law and why a surprising number invites a check
Tony Twyman worked in television and radio audience measurement in Britain, and the observation attributed to him is that any figure that looks interesting or different is usually wrong. Kohavi, Tang and Xu put it at the front of their 2020 book on controlled experiments, as the rule that governs how a surprising result gets treated.
The law is useful because it points at the exact moment a team is least inclined to investigate. A result showing a fourteen per cent lift on a two line copy change arrives as good news, and good news gets shared before it gets checked. A result showing a fourteen per cent collapse gets checked immediately, because somebody has to explain it.
That asymmetry is what a standing validity check corrects. The check runs on every experiment, in the same way, before anybody has seen the outcome, so the order of investigation no longer depends on whether the number was welcome.
The arithmetic behind the check
The test is a chi squared comparison of the counts observed against the counts configured. A trial signup experiment at a payroll company gives it a shape. The team configured an even split and ran for three weeks, and the analysis arrived with 412,340 users in control and 407,914 in treatment.
The numbers look unremarkable to a reader. The split is 50.27 against 49.73, and the total of 820,254 users means each group was expected to hold 410,127. Control is 2,213 users above that figure and treatment is 2,213 below it.
Squaring that gap, dividing by the expected count and adding the two sides gives a chi squared statistic of 23.9, which corresponds to a probability of about one in a million. A mismatch this size will not happen by chance in the working life of the product.
Two rules of thumb follow from the arithmetic. The first is that the threshold for declaring a mismatch is far stricter than the five per cent used elsewhere, because a check that runs on every experiment at five per cent would cry wolf once every twenty tests. Teams commonly use a probability below one in a thousand. The second is that a large sample makes the check more sensitive, so the biggest and most valuable experiments are the ones where a small percentage gap is most clearly a defect.
The five stages where a mismatch is created
Aleksander Fabijan and six colleagues published a taxonomy of causes in 2019, built from experiments at Microsoft, Booking.com, Outreach.io and Online Dialogue. They recorded twenty five distinct causes and grouped them by the stage of the experiment that produced each one. In the same study they found that roughly six per cent of experiments at Microsoft carried a mismatch.
- Assignment. The randomisation itself is uneven. One case in the paper involved a hashing scheme that divided users into a thousand buckets, and a bug that gave the control variant one bucket fewer than the treatment.
- Execution. One variant fails to start, redirects more slowly or crashes on a subset of devices, so the users affected never reach the point of being counted.
- Log processing. A pipeline stage drops events for one group more often than the other, through a bot filter, a deduplication rule or a join that silently discards rows.
- Analysis. The filter applied at analysis time removes users unevenly. A triggering condition that only one variant can satisfy is the common version.
- Interference. Something outside the experiment treats the two groups differently, including another experiment running on overlapping traffic.
Only the first of the five is what a product manager pictures when the phrase broken randomisation comes up. The other four sit in the pipeline and in the analysis code, which is where most real mismatches are found.
The published case that shows what a mismatch hides
The same authors describe an experiment on a news homepage in which a team increased the number of rotating cards on a carousel from twelve to sixteen. The result came back showing that users given more cards clicked less, which read as a clear argument against shipping.
The cause was in the log processing. The most engaged users in the treatment group interacted with the carousel so heavily that an automated bot detection rule classified them as machines and removed them from the analysis. Correcting for that removal reversed the result, and the team shipped the change.
That case is worth holding onto because it inverts the usual fear. A mismatch does not only manufacture false winners. Here it produced a false loser, and a team that had trusted the first reading would have thrown away a good change and recorded a wrong lesson about carousels.
Why a mismatched result is discarded
A team that finds a mismatch has two options and only one of them is honest. The tempting option is to explain the gap, decide it affects both groups equally and read the result anyway. The defensible option is to treat the result as void, fix the cause and run the test again.
The reason the first option fails is that the missing users are unknown by definition. If the team knew which people the filter had dropped and how they behave, it could correct for them, and the correction would be the fix rather than an argument for ignoring the problem. Every mismatch investigation that ends in a shrug is a team agreeing to read numbers produced by a process it has just proved it does not understand.
There is one piece of good news in the discipline. A mismatch is cheap to detect, costs nothing to check and catches a large family of faults that have no other symptom, so the check pays for itself many times over a year of experiments.
A test that passes the ratio check and holds up under a single examination has earned the reading its numbers claim. That reading still describes one window of time, and some effects are strongest in their first week and gone by their fourth.