Bootstrapping Labels via ___ Supervision & Human-In-The-Loop

Every ML tutorial you've ever read assumes labeled data just exists. Someone hands you SuperGLUE or a pile of click logs and you get to work. But real problems rarely start that way. Try labeling COVID misinformation, or tagging book reviews by whether they're about pricing versus writing style, and you quickly realize the labels don't exist yet. Somebody has to invent them.

Eugene Yan's writeup lays out three ways companies actually solve this, and the DoorDash example is the most concrete. They needed to tag millions of menu items with attributes like regional cuisine or flavor, with no labeled data to start. So they trained a rough, high-precision classifier just to pick good candidates for annotation, then layered on data augmentation — swapping sentence order, stripping out menu categories — at a ratio of 100 synthetic samples for every real labeled one. Single-layer LSTMs with FastText embeddings did the heavy lifting. When they needed more labels, they didn't grab samples randomly; they specifically hunted for cases where the model disagreed with an annotator, or where confidence was low, then routed the trickiest ones through Mechanical Turk first and professional annotators second.

Facebook's contribution to this puzzle is a technique called SEALS, which adds nearest-neighbor search as a filter before active learning even starts. The insight is blunt: in most datasets, the thing you care about is maybe 1 in 1,000 examples, so why waste annotation budget scanning the other 999. On ImageNet, this got them performance matching full supervision using just 2,000 labels per concept — roughly 5% of the candidate pool. On OpenImages they matched standard active learning while touching as little as 2% of the data.

Then there's weak supervision, which Google operationalized through something called Snorkel DryBell. Instead of labeling examples by hand, engineers write labeling functions — heuristics based on URLs, keyword matches, internal topic models, knowledge graphs — that each vote yes, no, or abstain. A generative model then estimates how reliable each function is based on where they agree and disagree, and blends them into probabilistic labels. Google says this beat models trained on hand-labeled data alone, and matching its accuracy the old-fashioned way would have required 80,000 hand-labeled examples for topic classification and 12,000 for product classification. Apple runs something similar internally called Overton, notably for privacy-sensitive products where traditional supervised data isn't really an option at all.

What gets less attention, and probably deserves more, is the unglamorous work of writing labeling guidelines. Andrej Karpathy tweeted that after four years running Tesla's Autopilot labeling pipeline, he still hasn't fully solved workflow, QA, or escalation processes. DoorDash's advice here is refreshingly practical: keep tags mutually exclusive, always include an 'Other' bucket you can refine later, and avoid subjective tags like 'popular' that drift in meaning over time. None of this is glamorous machine learning work, but it's the difference between a labeling pipeline that scales and one that quietly rots.