When a winning test result should not be shipped at all

A test finishes, one version outperforms the other on the metric it was measuring, and the result gets shipped. That sequence is treated as the whole point of testing, and most of the time it is correct.

The cases where it is not correct share a pattern: the variant won on the metric it was pointed at while doing something the metric could not see.

Winning by moving the cost somewhere else

The most common version is a change that improves a near-term number by borrowing from a longer-term one.

More frequent offers raise offers-per-session revenue and reduce how often players open the store over the following weeks. Aggressive reminders raise return rate this week and raise uninstalls next month. A harder difficulty spike before a purchase point raises conversion at that point and increases drop-off just after it.

In every case the test measured the first effect and ran for too short a window to see the second. The result is real — the variant did win — and shipping it makes the game worse.

The defence is not more statistics. It is deciding before the test which secondary numbers would make you reject a winning result, and then actually looking at them. A test with one metric can only be wrong in one direction.

Winning on a population you did not intend to affect

The second pattern is an aggregate win built on a large effect in one group and a negative effect in another.

A change that substantially increases spending among a small number of high spenders and slightly reduces engagement across everyone else can win comfortably on total revenue. Whether that is a good trade depends on how much you value the engagement of the majority, and the aggregate number does not ask.

This is particularly worth checking when the winning margin is large. Big effects in aggregate tests frequently come from concentrated effects in small groups, and a change that works by pushing harder on a few players is a different proposition from one that works broadly — even when the revenue is identical.

Splitting the result by spending level and by tenure before shipping catches most of these, and it is a five-minute check.

Winning on something you would not defend

The third category is the one that does not show up in numbers at all.

Some changes win because they exploit something rather than improve something. An offer that is easy to buy by accident converts well. A confusing cancel flow reduces cancellations. A countdown that implies scarcity that does not exist raises urgency-driven purchases.

These produce genuine, reproducible wins on the target metric. They also produce refund requests, support load, negative reviews, and in several jurisdictions regulatory exposure — none of which the test measured, and some of which arrive months later attached to nothing in particular.

The check here is not statistical. It is whether you would be comfortable describing the change plainly to a player. A change that only works if the player misunderstands it is not a win; it is a liability that happened to score well.

See how we keep automated decisions reviewable →

The pre-registration habit

The practical discipline that prevents most of this costs almost nothing: before running the test, write down what you expect, what would make you ship it, and what would make you reject it despite a win.

That last item is the one that does the work. Naming the failure conditions in advance stops the post-hoc reasoning that turns every result into a justification. It is very easy, looking at a winning variant, to decide that the secondary metric that got worse does not really matter. It is much harder to decide that when you wrote down beforehand that it would be disqualifying.

The second habit is a mandatory pause between the result and the rollout. A day is usually enough. Results that are genuinely good survive a day; results that were exciting because they were surprising often look different after one.

The reason both of these matter more than they sound is that testing creates the appearance of rigour, and appearance of rigour is exactly what stops people asking obvious questions. A change that arrived with a p-value gets less scrutiny than one that arrived as a proposal, which is backwards — the tested change has been optimised against a metric, and optimising against a metric is precisely how you end up somewhere nobody intended.

The other thing worth building is a way to reverse a shipped result. A change that won its test and turns out badly over a longer horizon needs to be removable, and changes that are shipped by deleting the losing variant are not. Keeping the previous behaviour available for a period after rollout costs very little and preserves the option to go back when the longer-run data arrives — which is the only point at which some of these problems become visible at all.

Talk to us about evaluating changes properly →