Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks
MarkTechPost Michal Sutter
Supabase open sourced Evals, a benchmark framework that tests AI coding agents like Claude Code and GPT models on real Supabase development tasks such as schema building and policy debugging. The benchmark uses three dimensions (products, topics, stages) with separate benchmark and regression test suites, combining deterministic checks with LLM-based scoring. The results show top models like Opus 5 and Kimi K3 achieve 100% on build tasks unaided, while skills improve smaller models' performance and reveal agents under-utilize declarative schemas and documentation.
Why it matters
Supabase has open sourced supabase/evals, an Apache-2.0 benchmark and framework that runs coding agents including Claude Code, Codex and OpenCode against real Supabase tasks — building schemas, debugging Edge Functions, fixing RLS policies — inside containerized stacks, then scores them with deterministic checks and LLM-as-a-judge. The post Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks appeared first on MarkTechPost.
Related stories
AstaBench update: New results, plus adoption from industry
Allen Institute (AI2) · 3 months ago ·
27
Introducing the LiveCodeBench Leaderboard - Holistic and Contamination-Free Evaluation of Code LLMs
Hugging Face Blog · 2 years ago ·
22
smevals - a small eval suite for evaluating models, prompts, and harnesses
Simon Willison · 2 days ago ·
44