Scraping 107M rows of data to build this
Ben's Bites
A scraper dug through 107 million rows of UK council spending data and turned it into a map. It shows what councils paid, and how messy public data becomes when it’s spread everywhere.
Based on reporting by Ben's Bites — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Ben’s Bites spent this build on a very specific kind of problem: public data that exists, but is awkward to find, awkward to clean, and harder still to make sense of. The result is an “Apple Maps” style view of UK council spending, built from 107 million rows of data scraped from council sites and pulled into one place.
The starting point was England’s council payment data, where councils have to publish spending over £500. Codex helped search for sources across scattered websites, pulled together a catalogue of 31 official sources, and then fetched real data from five councils while logging each download with a checksum fingerprint. After that came the bigger grind: flattening everything into one format and pulling in more councils, including Bedfordshire and its separate pieces, Bedford, Central Bedfordshire and Luton.
By the time that first pass was done, the project had 8 councils, 1.8 million rows and £9 billion of spend in a clean CSV on a laptop. The setup then moved to a Mac Mini, using Parquet and DuckDB to handle larger tables in a single-file database that could answer questions without reading more data than needed. The plan widened to 339 councils in England and Wales, though Wales later got dropped because its reporting rules are different.
Then the automation really started humming. A single /goal ran for nearly two days, with Codex repeatedly finding spending pages, downloading data, flattening and formatting it, checking it through an auditor subagent, and loading it into DuckDB. After 552 threads, 319 of the 339 councils were collected or parked because there was no data, or no way to get it.
The visual side went through its own pile of experiments: receipts, bank statements, county grids, a mini-site, comparison cards, charts and maps. The direction that stuck was the Money Map, first as a Google Maps clone, then as an Apple Maps-style version with clickable council outlines, category filters, a ranked list, and search for councils and suppliers. Along the way, the data got cleaned further, including 2,000 unknown vendors, duplicate suppliers like Tesco appearing under six names, and a strange £5m threshold filter that had to be tracked down during testing.
My take — AI-written commentary, not fact-checked reporting
This is the kind of public-data project that makes a better argument for AI than a thousand chatbot demos. The real win isn’t the map; it’s the ugly middle where agents do the tedious scraping, cleaning and checking humans usually hate. If open systems are going to matter, this is the lane: boring, messy, useful.
Read more about this at: Ben's Bites
