If You Think You Can Do Real-World Text-to-SQL
cacm.acm.org
A new benchmark called Beaver tests LLMs on real enterprise data warehouses, not clean textbook databases. Turns out the models everyone's hyping for text-to-SQL fall apart once schema rot and messy legacy data enter the picture.
Based on reporting by cacm.acm.org — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Most text-to-SQL benchmarks live in a fantasy world. Clean schemas, sensible column names, tidy foreign keys — the kind of database you'd build from scratch in a tutorial, not the one your company has actually been running since 2011. Beaver, a newly released benchmark, throws that fantasy out and tests large language models against genuine enterprise data warehouses instead.
The difference in results is stark. Models that post impressive numbers on Spider or BIRD, the go-to academic benchmarks, stumble badly when asked to write SQL against real schemas riddled with what researchers call schema rot: tables renamed years ago but never cleaned up, columns that mean something different depending on which team touched them last, and business logic baked into cryptic abbreviations nobody documented. Add in idiosyncratic data quirks specific to a single company's history, and the gap between benchmark performance and production readiness widens fast.
This matters because text-to-SQL is one of the more concrete, sellable promises of the current LLM wave. Give the model a plain-English question, get back a working query against your warehouse — no analyst required. It's a pitch that shows up in demos from nearly every major AI vendor right now. Beaver's results suggest those demos are quietly cherry-picking friendly conditions, and that the leap from a polished sandbox to an actual Snowflake or Redshift instance with fifteen years of accumulated technical debt is much larger than vendors let on.
The researchers behind Beaver aren't arguing LLMs are useless for this task. They're arguing the field has been grading itself on an easy test. Real data warehouses are messy because organizations are messy, and any benchmark that ignores that is measuring something other than the skill companies actually need.
My take — AI-written commentary, not fact-checked reporting
I've said for a while that text-to-SQL demos are the AI industry's favorite party trick precisely because they're staged on toy schemas — Beaver just put numbers behind what any data engineer already suspected. If a vendor's pitch deck doesn't mention schema rot, assume their benchmark didn't either, and go test it on your own ugly warehouse before you believe a word of it.
Read more about this at: cacm.acm.org
