Cohort tenure versus calendar time in live-ops reporting

Almost every dashboard is built on calendar time. This week against last week, this month against last. It is the natural way to look at a business and it is the wrong way to look at a live game.

The reason is that the players producing this month's numbers are not the same players who produced last month's, and the difference between the two groups is frequently larger than any change you made.

What calendar comparisons actually measure

A month-over-month comparison of any per-player metric is measuring three things mixed together: whatever changed in the game, whatever changed in your player mix, and whatever changed in the calendar.

The player mix moves constantly. A month with heavier acquisition has a younger population, and younger players convert differently, spend differently, and engage differently from established ones. Nothing about the game has to change for every per-player metric to move.

The calendar moves too. A month containing two events is not comparable to a month containing one, and a month with a holiday weekend is not comparable to one without. If your event schedule shifted at all, a month-over-month comparison is partly measuring the schedule.

The consequence is that calendar reporting produces movement constantly, and most of it is not about the game. Teams then explain the movement — usually by attributing it to whatever they shipped that month — and the explanation is unfalsifiable, because the comparison could not have isolated it.

What tenure comparisons measure instead

Comparing players at the same age removes the mix problem entirely.

If you look at every player's first week and compare the cohort that installed in June to the cohort that installed in July, both groups are being measured at the same point in their relationship with the game. Any difference between them is either a change in the game or a change in who you acquired — and you can usually separate those two with the acquisition data.

This is what makes tenure comparison the honest way to answer "is the game getting better". A change that improves the first week shows up as the newer cohort outperforming the older one at the same age. A calendar comparison would have buried that under population mix.

The same applies further out. Comparing month-three behaviour across cohorts tells you whether the mid-game is improving. Nothing in a calendar view can answer that, because any given month contains players at every tenure simultaneously.

The cost of switching, and the part that does not switch

Tenure-based reporting is more work than calendar reporting for a specific reason: results arrive late. To know how the June cohort performed at week eight, you have to wait until August. A change shipped in June cannot be evaluated on a tenure basis until enough time has passed for the affected cohort to reach the relevant age.

That lag is real and it is why calendar reporting persists. Businesses need a number this week.

The reasonable resolution is to use both for different questions rather than picking one. Calendar reporting answers operational questions — how much revenue arrived, is anything broken, did the event run. Those are legitimately about this week and the mix does not matter.

Tenure reporting answers product questions — is the early game better, is the mid-game retaining, did that change help. Those cannot be answered on a calendar basis at all, and answering them badly is worse than answering them slowly.

The failure is not using calendar reporting. It is using it for the second category of question, which is where most disputed conclusions come from.

See how we compare cohorts across a live game →

The early proxy that is usually good enough

Waiting eight weeks for every read is impractical, and the common workaround is to find an early signal that correlates with the later one.

For most games, behaviour in the first few days predicts a meaningful amount of what happens later. If a cohort's first-week behaviour is clearly better than the previous cohort's at the same point, that is usually directional even before the longer window closes.

The discipline is to treat that as provisional and go back and check. Early signals correlate with later outcomes until a change breaks the correlation — and a change that improves the first week while damaging the third is exactly the kind that early proxies miss, and exactly the kind that gets shipped confidently on the basis of one.

Recording the cohort's identity at install and keeping it attached to everything afterward is the whole technical requirement. It is cheap to add at the start and awkward to reconstruct later, which puts it in the same category as most economy telemetry: not hard, just impossible retroactively.

The short version is that calendar time tells you how the business is doing and tenure tells you how the game is doing, and confusing the two is why teams keep having the same argument about whether a change worked.

Talk to us about cohort-based measurement →