Product Classification API Part 1: Data Acquisition

Eugene Yan set out to build something practical: an API that takes a product title and spits back the three most likely categories it belongs to. No manual browsing through thousands of category options. Just a title in, predictions out. He later shut the API down to cut cloud costs, but the write-up on how he built it is the interesting part, especially the unglamorous first step everyone skips past in their head: getting the data into shape.

He almost built a scraper for Amazon and Alibaba, then talked himself out of it. Scraping and wrangling scraped data, he figured, would eat up something like 30% of the whole project's effort, and it's not a skill he'd lean on much at his day job anyway, where data usually already lives in a database. So he went hunting for open product data instead, nearly gave up, and eventually found Julian McAuley's UCSD dataset: 9.4 million Amazon products, 3.1 gigabytes zipped, complete with titles, prices, images, sales rank, and nested category lists.

That 9.4 million number is deceptive, though. Once you strip out products missing a title or category, you lose over a million right away, down to 7.98 million. Then entire categories get tossed because titles alone don't reveal what they are — books, movies, CDs — since those get classified by things like ISBNs or ratings, not product names. That cut takes it to 5.59 million. Then there's the problem of category depth: some products are tagged with a shallow path like