Active Learning with AutoNLP and Prodigy

There's a version of this story that's just a product demo, but the more interesting part is the graph buried near the end showing accuracy climbing as the labeled dataset grew from 20 samples to 250. That curve is the whole pitch for active learning, and Hugging Face uses AutoNLP and Explosion's Prodigy to walk through it step by step instead of just asserting it works.

They started with the BBC News Classification dataset from Kaggle, five categories (business, entertainment, politics, sport, tech), a pile of raw article text. Classification itself was almost boring by comparison: upload the data, let AutoNLP auto-split and train 15 candidate models, wait about 15 minutes, and the best one hit 98.67% accuracy. At AutoNLP pricing that can run as low as $10 per model, that's a startlingly cheap way to get a production-ready classifier without touching a training loop.

The real experiment was named entity recognition, since the BBC dataset has no entity labels at all. So they fired up Prodigy's ner.manual mode, tagging PERSON, ORG, PRODUCT and LOCATION by hand in a browser interface, then wrote a short script to convert Prodigy's span annotations into IOB-tagged JSONL that AutoNLP could actually ingest. At 20 labeled examples, the resulting model scored 86% accuracy but with 0 precision and recall — a model that had memorized nothing useful, just as you'd expect. At 70 samples, accuracy hit 92%, precision 0.52, recall 0.42; the model could finally pick out names like Bruce and David from unseen text, though clumsily. By roughly 150 samples, accuracy reached 95.7% with precision at 0.64 and recall at 0.76, and the qualitative jump was obvious enough that the authors reacted with genuine surprise in their own writeup.

They stopped at around 250 labeled examples, where precision and recall peaked at 0.73 and 0.79 with accuracy near 95.9%. That's not a huge dataset by deep learning standards, and that's rather the point: the bottleneck in this whole pipeline was never compute or model architecture, it was the tedious human work of drawing entity spans in Prodigy. AutoNLP handled model selection, hyperparameter tuning, and evaluation automatically, leaving humans to do the one thing they're actually needed for.

Hugging Face open-sourced both the final model and the labeled dataset, which matters more than it might seem. A tool stack that turns 250 manually labeled news articles into a working NER model, with zero lines of training code, lowers the bar for building specialized datasets that otherwise wouldn't exist publicly at all.