After building out a full player-intelligence and recommendation platform, the thing I am most convinced of is counter-intuitive: the hard part of production machine learning is not the modelling. Gradient boosting, collaborative filtering and conformal prediction are well-trodden, with excellent tooling for all of them. The genuinely hard part is being honest about what the models do and do not know, consistently, in a hundred small places, so that a product team learns to trust the system and act on it rather than learning to route around it. This closing note gathers the honest machine learning patterns that run through everything else, because together they are not a nice-to-have; they are the actual product.
Dormant beats guessing, and assumptions wear a label
A model with no data to learn from should do nothing, loudly, rather than something, quietly. The uplift learner from the decisioning work stays inert until it has real treated-and-control outcomes, and falls back to a transparent, labelled baseline until then. The same rule governs any supervised head we would add later: it ships only when it beats the existing baseline on a held-out slice, and until then the baseline stands unchallenged. "We do not have the data for that yet" is a first-class answer the platform is architected to give, not an embarrassing edge case it tries to hide.
Wherever a number rests on an assumption, the population-level uplift, the action economics, the cost-per-contact in a what-if projection, the output says so. The decisioning layer's assumed-versus-learned source field is that pattern made concrete: never let a guess and a measurement look identical in the interface. A stakeholder who can see which inputs are assumed can reason correctly about how much to trust the output; one who cannot is being misled by omission, even if every individual number is technically defensible.
Uncertainty ships with the estimate, and signals must be real
Point estimates hide error; calibrated ranges expose it. Value predictions carry conformal confidence bands, and the console draws the band, not just the number, so a wide interval visibly says hint, not fact, without anyone having to read the docs. Exposing uncertainty is more work and looks less impressive in a screenshot than a single confident figure, and it is the difference between a system a team leans on correctly and one it eventually, rightly, ignores.
The at-risk and wellbeing scoring composes several behavioural signals, but only the ones that actually vary in the customer's current data. A signal with no variance across a population carries no information, so it is dropped from the composite rather than padded in to make the model look richer than it is. What the customer sees is which signals genuinely moved a given score, not a full slate of inputs half of which are inert noise dressed up as insight.
Honest denominators, and saying insufficient data
Cohort retention is a small, favourite example of how easy it is to mislead with arithmetic that is technically correct. Measure retention at thirty days naively across every cohort and your most recent cohorts always look terrible, not because those players churned but because most of them have not had thirty days yet, so they cannot possibly have been retained at that horizon. We measure each horizon only over the players who have had enough elapsed time to qualify, an eligible-only denominator, and return not-yet-due where nothing qualifies, rather than a misleadingly low number that a dashboard would happily render as a red arrow. The maths is trivial; choosing to do it is the point.
A small customer that genuinely cannot support a reliable model is told exactly that, in the console, rather than scored through a fabricated number that looks just like a real one. Cold-start honesty, the first note in this series, is the same principle applied at the scale of a whole customer: the platform would far rather admit a gap it can name than paper over it with a confident-looking guess.
Reason codes you can act on, and verifying like you mean it
Every prediction the platform makes carries its top drivers, the features that moved this particular score most, computed with attribution methods. But the framing matters as much as the numbers. A driver is reported as what moved the model, not the reason the player will churn. Those sound similar and are not: the first is an honest statement about the model's internals, the second is a causal claim the attribution does not support. Getting that wording right in the console is a small thing that prevents a large mistake, a product team confidently acting on a reason that was only ever a correlation the model happened to lean on.
None of the above survives contact with reality without a verification culture underneath it. Every capability is de-risked on real customer data and through a live, logged-in walkthrough before it ships; the codebase carries a large automated test suite that runs on every change; and the end-to-end system, from ingestion and scoring through the console, recommendations and decisioning, is exercised live, together, on a running deployment rather than only in isolated unit tests. The readiness discipline is not theatre: the pre-integration audit described in the privacy note caught a real ingestion defect that would have rejected a feed in production, before any customer could ever hit it.
Put together, these are not separate features. They are one stance, applied relentlessly: isolate hard, compute on the customer's own reality, price your uncertainty, label your assumptions, wait for real data before you claim to have learned anything, and admit plainly what you cannot yet know. A platform built that way is slower to look finished and far more likely to actually be trusted, which, for anything that makes decisions about real players and real money, is in the end the only thing that matters.