"Recommended for you" is easy to fake and hard to do well. A popular-now shelf is not personalisation; a nightly batch model is personalised but already stale by lunchtime; and a purely greedy recommender never discovers that a player would love a game it has simply never thought to show them. A better design runs two layers: an offline model that learns who likes what, and an online contextual bandit that decides what to actually surface right now and keeps learning from every interaction. This note is about how those layers fit together and why the second one matters most.
Layer one: learn the taste
The offline layer is a collaborative-filtering recommender trained on the customer's own catalogue and interaction history. It learns latent structure, which games co-occur in players' histories and which players resemble each other in taste, and produces, for any player, a ranked list of games they are likely to engage with. This is the workhorse. It covers the whole catalogue, it is retrained as new interactions accrue, and for a brand-new player or a freshly-added game it falls back to content features and popularity signals, so there is always something sensible to show rather than an empty shelf.
But an offline model trained on historical interactions has a blind spot baked into it: it can only learn from what players were actually shown, and players were shown whatever previous versions of the model recommended. That feedback loop quietly narrows the catalogue over time, as popular games get more exposure, get more interactions and look even more popular at the next retrain, while the model never tests the games it has decided not to show. Left alone, a recommender slowly convinces itself of its own priors.
Layer two: decide and explore
So on top of the offline model sits an online contextual bandit that makes the real-time call. For each placement, the bandit scores the candidate games using context, the features of the player and of each candidate, and selects with an exploration bonus in the spirit of upper-confidence-bound methods: a candidate the bandit is uncertain about gets a bump in its score, so the system will now and then surface something outside the greedy pick specifically in order to learn whether it was right. That is the mechanism that breaks the offline model's echo chamber. Exploitation serves the current best guess and captures value today; exploration deliberately spends a little of that value to buy information that improves every recommendation tomorrow. The upper-confidence-bound framing keeps the trade principled rather than a random shuffle, because uncertainty shrinks as evidence accrues, so a game gets explored until the bandit is confident about it, then quietly stops being force-shown.
The bandit does not run wild. It has a warmup, observing a minimum number of recommendation lists before it is permitted to reshape the live experience, so a cold bandit with no evidence yet does not degrade the feed while it has nothing to go on. And because it operates strictly on top of the collaborative-filtering candidates, its exploration is bounded to plausibly-good games the offline model already surfaced, never to random noise from across the catalogue. The player sees variety; they do not see nonsense.
The context in contextual bandit is worth being concrete about, because it is what separates this from a naive A/B test that shows everyone the same winner. The context is a feature vector describing this player at this moment, recent activity, taste affinities inferred by the offline model, session signals, joined with features of each candidate game. The bandit is therefore not learning which game is best on average; it is learning which game is best for a player who looks like this, and its uncertainty is conditional on that context. A newly-added game is uncertain for everyone at first, but the bandit can become confident it suits action-game players long before it has enough data to judge it for puzzle players, and it explores each population only as much as each still needs.
Closing the loop, safely
Every recommendation list the system serves is logged, the impressions and the clicks that follow, durably and, importantly, off the request path, so serving a recommendation never blocks on a write. That interaction log is the training data for the next iteration of the served ranker. Periodically a candidate ranker is retrained from the accumulated interactions and evaluated against the current live ranker on a held-out slice using a ranking metric that rewards putting the games a player actually engaged with near the top. The discipline that keeps the loop honest is this: the new ranker only replaces the live one if it actually wins on that held-out data. A retrain that fails to improve ranking quality is discarded and the incumbent stays. The loop is serve, log, retrain, validate, and promote only if better, never serve, log, retrain, deploy and hope.
There is an engineering footnote to that logging worth dwelling on, because it is the kind of thing that sinks real-time ML systems. An early version rewrote the entire impression log on every single recommendation to keep it internally consistent, an operation whose cost grew with the log and turned into the serving bottleneck under load, dragging recommendation latency from milliseconds into seconds as traffic climbed. The fix was to make the log append-only, an incremental insert instead of a full rewrite, with a single background writer draining a queue so concurrent requests never contend on the same file. Throughput jumped sharply and tail latency fell back to tens of milliseconds.
The lesson is one every real-time ML system relearns the hard way: the model is rarely your scaling problem; the write path wrapped around it almost always is. The result is a recommender that personalises from the customer's own data, keeps exploring so it never calcifies into its own history, learns from real behaviour continuously, and only ever ships an update that has measurably beaten what it already had.