How to Write Design Docs for Machine Learning Systems
Eugene Yan
Eugene Yan lays out a practical framework for writing design docs specifically for ML projects. It matters because most ML failures trace back to skipped planning, not bad models.
Based on reporting by Eugene Yan — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Eugene Yan's latest piece isn't really about templates. It's about forcing yourself to think before you build, and using a document as the mechanism to do it. His argument, borrowed loosely from Jeff Bezos's famous stance on narrative memos over slide decks, is that full sentences with real verbs expose fuzzy thinking in a way bullet points never do. If you can't write clearly about why you're solving a problem, you probably don't understand it yet.
Yan structures his approach around a Why-What-How framework. The Why and What sections force you to justify the project's existence: why this problem, why now, what does success look like, and critically, what's explicitly out of scope. He's blunt about tech debt too — it's fine to take shortcuts under time pressure, as long as you write down that you're doing it and have a plan to pay it back.
The How section is where things get ML-specific, and it's split into methodology and implementation. Methodology covers problem framing (is your recommender item-to-item or user-to-item, is it candidate generation or ranking), the data you're using, and how you'll validate things offline before ever touching an A/B test. Yan flags a real trap here: surrogate problems. Netflix's famous prize assumed that predicting star ratings would translate into better movie picks, but the actual metric that matters — watch time, engagement — might diverge from whatever proxy you're optimizing in training. He also insists on separating success metrics from guardrail metrics, since a model that boosts clicks while quietly tanking customer trust is not a win.
Implementation is the unglamorous half: infra choices, latency budgets, security, GDPR compliance, monitoring, and — the part executives actually read — cost, including engineer-hours, not just cloud bills. Yan's line about operating a system without monitoring being like driving at night with no headlights is a fair description of how a lot of ML systems actually get shipped.
He closes with a section most people skip: alternatives considered and rejected. Writing down why you didn't pick the other approach isn't busywork — it's insurance. When the environment changes six months later, that paragraph tells you whether your old assumptions still hold or whether it's time to revisit the whole design.
My take — AI-written commentary, not fact-checked reporting
I've watched more ML projects die from a vague problem statement than from a bad model, so Yan's framework reads less like advice and more like a post-mortem checklist nobody wanted to write in advance. The alternatives-considered section is the one everyone skips and the one that saves you six months later — treat it as mandatory, not optional homework.
Read more about this at: Eugene Yan