How to run A/B tests on your mobile game offers

A/B testing is one of those practices that every studio says they do and almost none do correctly. The most common version looks like this: change the price on an offer, watch revenue for a week, decide based on whether the number went up. If it did, keep the new price. If it didn't, revert.

This is not an A/B test. It's before-and-after measurement, and it conflates the effect of your change with every other thing that changed during that week — seasonal player behavior, a competing game's launch, a platform feature that drove new installs, or simply random variance in a small sample.

A real A/B test shows version A to one group of players and version B to a different group simultaneously. The comparison is between groups, not between time periods. That distinction is what separates a finding you can act on from one that might just be noise.

What you can actually test

Offers have several variables worth testing, and not all of them are equally useful to start with.

Price is the most commonly tested variable and the one with the clearest business impact. A $0.99 offer versus a $1.99 offer for the same content tells you directly how price-elastic your players are at that moment. The result also informs everything downstream — bundle pricing, first-purchase targeting, re-engagement offers.

Content composition matters more than most studios realize. Two bundles at the same price with different item combinations often convert at very different rates. Players respond to specific items they want, not to abstract "value." Testing which items make an offer feel compelling versus which make it feel like filler is worth doing before you commit to a bundle design.

Timing — which in-game event triggers the offer — is the variable with the highest ceiling. Showing an offer after a level fail versus after a level complete versus after a player opens the in-game shop produces meaningfully different conversion rates. The same offer becomes more or less relevant depending on where the player is in their session.

Visual design and copy are worth testing only after you've settled on price and content. The gains from visual optimization are smaller and harder to isolate cleanly. Start with the variables that move the largest amount of revenue.

What makes a test valid

Three things a test needs to be worth acting on:

Random assignment. Players need to be randomly assigned to version A or version B, with the assignment persisting for the duration of the test. A player who sees version A on Monday should see version A on Friday. Splitting by day of the week, by install batch, or by device type introduces systematic biases that invalidate the comparison.

Sufficient sample size. The minimum sample size for a meaningful result depends on your current conversion rate and how large a difference you're trying to detect. As a rough guide: if your current offer converts at 3% and you want to detect a 1 percentage point improvement (a 33% relative increase), you need roughly 3,000 players per group who were shown the offer. Testing on 200 players total and claiming significance is how studios convince themselves of things that aren't true.

A defined endpoint. Decide before the test starts how long you'll run it and what metric you'll use to evaluate it. Stopping a test early because it's going the way you want — or extending it because it isn't — undermines the validity of the result. The test period should cover at least one full weekly cycle to account for day-of-week variation in player behavior.

The comparison group problem

The hardest part of offer testing is the comparison group. Unlike a website where 50% of visitors can see version A and 50% see version B with no interaction between them, mobile game players exist in a social context. If your game has any social features, referral mechanics, or chat — players talk. Version bleed is real.

More practically: the players who are in your game right now are not a random sample of all possible players. They're the ones who installed during a particular acquisition period, who play with a particular frequency, who are at a particular point in your content. Any test result is conditional on that player mix.

This is why the most reliable offer tests run for at least two weeks and why results from one game rarely transfer directly to another. The finding that "$0.99 converts better than $1.99 for our RPG players in month two" is a specific finding about a specific population. It's valid and useful — just not universally applicable.

What to do with the result

A test result tells you which version performed better with this player population over this time period. It doesn't tell you why.

If version B (higher price) converted at the same rate as version A (lower price), that's important — it means your players weren't price-sensitive at that point, and you left revenue on the table. But it doesn't mean you should raise all prices across the board. The same players at a different point in their lifecycle might respond very differently.

Use test results to build a model of your specific player base: which segments are price-sensitive, which respond to which content types, which triggers produce the highest intent. Over time, this model becomes more accurate and your offers become more targeted.

A single test is a data point. A testing practice is a competitive advantage.

See how we handle offer testing and measurement →

Running tests without a data team

The practical objection from most small studios is bandwidth. Running a valid A/B test requires random assignment infrastructure, sample size tracking, and result analysis. That's engineering work on top of everything else.

Modern monetization platforms handle this at the infrastructure level — random assignment, group persistence, result tracking, and significance calculation are built in. The studio decides what to test; the platform handles the mechanics of running it correctly.

This is the same shift that happened with analytics: studios used to build their own event pipelines, now they use platforms. Testing infrastructure is following the same path.

Book a demo to see how offer testing works →