Testing changes when your player base is still small

Advice about testing changes is mostly written for games with large populations. Split the traffic, wait for the result, ship the winner. That works when a week produces enough data to distinguish a real difference from noise.

Below a certain size it does not, and following the advice anyway produces confident conclusions from results that were mostly random.

What small numbers actually do to a result

The core problem is that with few players, the range of outcomes you would see from pure chance is wide.

If a hundred players see version A and a hundred see version B, and both are identical, the two groups will still produce different numbers. Sometimes noticeably different. A team looking at a five percentage point gap between two identical experiences will conclude one is better, because five points looks meaningful.

The gap needed to be confident shrinks as the population grows and stays large for a long time. With a few thousand players, only fairly large differences are distinguishable from chance — and large differences are rare, because most changes are incremental.

The practical consequence is uncomfortable: with a small base, most of your tests will be inconclusive, and the ones that look conclusive are disproportionately likely to be flukes. This is not a reason to stop testing. It is a reason to test different things.

Testing bigger things, less often

The adjustment that works is to test fewer, larger changes rather than many small ones.

A change that might move a metric by a small amount is undetectable at your size. Testing it consumes weeks and produces a shrug. Ten such tests produce ten shrugs and one apparent winner that is probably noise.

A change that restructures something — a different onboarding, a different economy shape, a genuinely different offer strategy — has a chance of producing a difference large enough to see. Those are the ones worth the measurement cost.

This also means accepting that many decisions will be made on judgement rather than data. That is not a failure of process; it is the correct response to not having enough information to do otherwise. The failure is running a test, getting a noisy result, and treating it as evidence — that is worse than judgement, because it carries false confidence.

Measurements that work better at small scale

Some questions are answerable with small numbers because they do not depend on detecting a small difference.

Whether something is broken is one. A funnel step where almost everyone drops out is visible with very few players, because the effect is enormous. Most genuinely important problems are large, and large effects are exactly what small samples can see.

Before-and-after on a big change is another, with care. If you change something substantially and the metric moves a lot, that is informative even without a control group — as long as you check that nothing else changed at the same time and that the population is comparable. This is weaker evidence than a proper split, and at small scale it is often the only evidence available.

Qualitative work is the third and the most underused. Watching ten players play, or reading what they write, produces information a small quantitative sample cannot. Ten sessions will tell you what is confusing, where people stall, and what they expected to happen. No amount of split testing at your size will tell you that.

See how we work with small-population games →

The trap of accumulating tests

The specific failure mode worth naming is running many small tests over time and treating the winners as a set of improvements.

If most of your tests are underpowered, the ones that appear to win are largely chance. Shipping all of them means shipping a collection of random changes with the appearance of a data-driven process. The game does not get better, and because each decision felt evidence-based, nobody revisits them.

A useful discipline is to write down, before running a test, how large a difference would need to be for you to believe it. If the answer is larger than the difference you expect the change to produce, do not run the test — either make a bigger change or make the decision on judgement and move on.

The second discipline is to re-check apparent wins. A result that was real will reproduce. A fluke will not. Re-running the one test that appeared to win costs the same as running another new one and is considerably more informative.

None of this argues against measuring. It argues for matching the method to the population. Small games have real advantages in understanding their players — you can talk to them, watch them, read everything they write — and those advantages are cheaper and more informative than a split test that cannot resolve the question being asked of it.

Talk to us about measurement at your scale →