Scraping 107M rows of data to build this
Ben's Bites 1 month ago 10
A developer scraped 107 million rows of UK council spending data and built a map-style website showing how each council spent money and on whom. The project processed 8.2 billion tokens and ultimately collected spending data from 319 of 339 England councils after some councils blocked automated scraping. The developer consolidated and cleaned the dataset, created visual prototypes including map versions, and is preparing to open-source the site and data after final tweaks and testing.