One model is easy to keep honest; two dozen, built as independent modules, is where entropy wins. Reproducible ML at this scale rests on three things: one registry that the code is forced to agree with, deterministic runs that make a changed number always a changed decision, and a continuous-integration aggregator that actually runs and re-checks every module in a single pass.
The consistency problem at scale
One model is easy to keep honest. More than twenty, built as independent modules so each can be owned and shipped on its own, is where entropy wins. One team renames a metric; another assumes a dependency that was never guaranteed; a third changes a model's risk tier to unblock a release and forgets to tell anyone. Individually harmless, collectively fatal: after a few months nothing agrees with anything, and no one can say with confidence what any given model does or how it is judged.
The fix is not more meetings. It is a single source of truth that the code is forced to agree with, and a build that refuses to go green when it does not.
The registry as single source of truth
One registry catalogues every model: its problem type, its governance tier, its primary metric, and its dependencies. Each module's manifest is wired to that registry, and a per-module validator fails the build on any drift, whether a metric that no longer matches, a tier that was changed locally, or a dependency that is not declared.
The registry is not documentation about the system; it is the contract the system is checked against. Independence of modules and consistency across them stop being in tension, because the shared contract is the one thing every module must honour.
Determinism on purpose
Reproducibility is a precondition for everything else. If a run does not reproduce, a metric change tells you nothing: it might be the code, it might be the dice. So determinism is enforced, with fixed seeds everywhere and no reliance on process-salted hashing, a subtle trap since a language's default hash of a string can vary between runs and silently make supposedly deterministic data non-deterministic.
A run reproduces bit for bit, which means a changed number is always a changed decision, and can be traced to one. That property is what makes every other check meaningful, because it rules out the dice as an explanation.
Beating a named baseline, and the aggregator
Every model is scored against an explicit baseline it must beat, on the standard metric for its problem type: a weighted absolute percentage error against a seasonal-naive baseline for forecasts, precision-recall area against prevalence for rare-event classifiers, mean absolute error against a segment median for regressions, and incremental uplift against the incumbent rule for optimisers. That the model trained without error is not success; that it beats the thing you would otherwise have used is.
All of it runs from one command. The CI aggregator executes every validator, every test suite, and the monitoring sweep, covering registry consistency, correctness, data-quality contracts, drift, and portfolio health, in a single pass. The result is one number: every check green and the portfolio healthy, or the build is red and something is wrong. Reproducibility, consistency, and governance are not aspirations pinned to a wiki; they are the exit code of a job that runs on every change.