A Practical Guide to Maintaining Machine Learning in Production

Eugene Yan's follow-up to his rundown of six hidden headaches in production ML skips theory and goes straight to fixes. Data drifts, models rot, codebases fragment, feedback loops warp reality, org charts slow everyone down, and customers keep filing tickets that eat engineering time. His answer isn't a new algorithm. It's discipline.

Most of the advice starts at the data layer, because that's where things quietly break first. Yan wants teams checking file schemas, encoding, row counts, and null rates before anything touches a model, and he points to Uber's use of multi-dimensional time series and PCA to catch quality problems at scale. He's especially insistent on training-serving skew, the gap between what a model learns on and what it actually sees in production. His fix: log the exact features used at serving time and train on those, not on a separately built historical dataset. It sounds small. It removes an entire class of mysterious performance drops.

Once data is clean, the model itself needs watching. Yan likes permutation importance for pruning dead features, validation holdouts before every retrain, and shadow releases that run a new model on live traffic without exposing users to it. He cites Etsy's finding that AUC tracked click-through rate well, and a KDD paper showing the opposite in other settings, as a reminder that offline metrics only mean something if you've verified they correlate with real outcomes. Tools like SHAP and LIME get a mention too, mainly for catching bias or data leaks by watching which features suddenly matter more than they should.

The engineering section is where Yan gets almost stubborn about simplicity. Containerize everything, log every hyperparameter and commit hash through something like MLflow, and always keep old model images around so a bad deploy can be rolled back in minutes rather than debugged in a panic. He holds up Instagram's 13-person engineering team, still running Postgres and Redis at the time of its 2012 acquisition while competitors chased trendy NoSQL stacks, as proof that boring technology scales fine when the team stays lean and skips unnecessary complexity.

The last stretch covers people and bias. Positional features can absorb ranking bias so a recommendation model doesn't just reward whatever was already ranked first. Thresholds should be learned per retrain instead of hardcoded, since fixed cutoffs drift with the data. And Yan pushes back on hyper-specialized ML teams, arguing that data scientists who own a problem end-to-end iterate faster than teams split into narrow silos waiting on each other.