Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks
MarkTechPost Michal Sutter
Supabase open-sourced a benchmark that tests coding agents on real database and auth tasks, not toy problems. It exposes which AI agents actually read docs versus guess, and where they still hand-roll risky auth code.
Based on reporting by MarkTechPost, Michal Sutter — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Supabase just released Evals, an open source benchmark built to answer a question every backend team eventually asks: can the AI agent I'm trusting with my database actually do the job? The framework runs agents like Claude Code, Codex, and OpenCode against tasks pulled straight from real support tickets and GitHub issues — building a schema, debugging a broken Edge Function, patching a leaky RLS policy. It's Apache-2.0, runs locally through pnpm, and feeds both a public leaderboard at supabase.com/evals and an internal regression suite that refreshes daily.
What makes this more than a marketing leaderboard is how literal the testing is. Every scenario spins up a real environment in Docker — a hosted-like Supabase stack plus a local CLI project — so agents are hitting the actual MCP server and CLI, not some sandboxed approximation. Supabase mapped scenarios across three axes (products like auth and storage, topics like migrations and security, and stages from build to resolve) and picked the minimal set that touches each once. Scoring blends deterministic checks with an LLM acting as judge, and agents get exactly one retry before the grade locks in.
The results say less about raw model intelligence and more about habits. In the Build stage, Opus 5 and Kimi K3 nailed 100% with zero extra guidance. But smaller or mid-tier models leaned hard on
My take — AI-written commentary, not fact-checked reporting
the packaged skill files Supabase ships: Sonnet 5 jumped from 78% to 100% once skills were loaded, GPT-5.6 Sol went from 89% to 100%, and GPT-5.4 mini climbed from 78% to 89%. In other words, skills matter most exactly where you'd expect — for the models that aren't already frontier-tier. Three recurring failure patterns emerged, and they're the kind of thing that should worry anyone shipping agent-written backend code to production. Agents kept hand-writing migrations instead of using Supabase's declarative schema tooling, which pushed Supabase to update its skill guidance. They also verified auth manually rather than reaching for the @supabase/server package, prompting a new package-selection guide. And documentation habits varied wildly between tools: Codex running on GPT-5.6 pulled roughly eight docs pages per scenario, while Claude Code averaged about two and consulted docs in under 40% of scenarios even with skills switched on. None of this is abstract for regulated industries. A wrong RLS policy isn't a bug report, it's a data exposure incident, and that's exactly the kind of failure this benchmark is designed to catch before it reaches a fintech or healthcare backend.
Read more about this at: MarkTechPost
Related stories
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
Amazon Science · 3 weeks ago ·
32
Introducing the LiveCodeBench Leaderboard - Holistic and Contamination-Free Evaluation of Code LLMs
Hugging Face · 2 years ago ·
24