Mechanisms for Effective Machine Learning Projects

Eugene Yan has spent years running machine learning projects and watching other people's projects, and he's landed on a handful of mechanisms that keep showing up as useful no matter the team. The core problem he's describing is a familiar one: too many problems, too few people, and nobody checking anyone else's work until a standup meeting surfaces a disaster that's been compounding for weeks.

His fix for that is a pilot-and-copilot structure, borrowed loosely from aviation. One person owns the project and does the bulk of the design and coding. A second person, usually more experienced, checks in periodically, reviews drafts and prototypes, and acts as a mandatory reviewer on the actual code. Yan pegs the overhead at roughly 10 percent of the pilot's time — for every 10 hours the owner spends, the copilot spends one. That sounds like a tax, but he argues it's cheap insurance against a bad train-validation split or a mismatched design going unnoticed until the project has already burned a month.

Before any of that, though, he pushes for a literature review at the very start, not as an afterthought. Early in his career he'd dive straight into the data and start training models, only to have a senior colleague hand him a paper a week or two later that would have saved him the detour. Now he reads with a specific lens: how the problem was framed, how the input data was cleaned and labeled, and how the model was evaluated offline versus how that evaluation correlated with real-world performance. Architecture details, oddly, matter less to him than the boring plumbing around the model.

Once there are actual results, Yan runs what he calls a methodology review — essentially a code review for experiments. The questions are pointed: is the model accidentally using information it wouldn't have at inference time, is a forecasting model split by time instead of randomly, and what's the theoretical ceiling if the model were allowed to overfit. He notes tools like DagsHub let teams do this asynchronously, the same way they'd review pull requests.

The piece ties together with timeboxing, which he treats as separate from estimation. Timeboxes are stretch goals; estimates, which he derives by multiplying timeboxes by 1.5 to 3x, are what actually go into planning. He lays out three flavors: halve the time spent on a similar past project for startup-style urgency, trim 10 to 20 percent for seasoned teams wanting a gentle push, or fall back on rough defaults — two weeks for a literature review, a couple months for a prototype — when the project is genuinely new territory.