TLDRocket
Sign in

AI Benchmarking

290 summarised stories about AI Benchmarking, each linking back to the original source. Browse all topics →

+ Follow this topic

Monday, 14 September 2026

AI’s best coding agent fails 60% of the time — and the data backs it up

The New Stack 1 month ago 27 ● 2 sources

Claude Fable 5.1 topped the Real-SWE coding benchmark while failing on most attempts in private, company-code-style tests. The agent scored 38.8%, meaning it missed more than 60% of tasks (6 out of 10 tries). Real-SWE’s shift to private code and full tool setups makes coding-agent results drop quickly in unfamiliar production-like codebases and shows no model was consistently reliable.

Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks

MarkTechPost 1 month ago 26

Sakana AI researchers introduced Augmented Lagrangian Predictive Coding (PC-ALM), a layer-local learning method meant to replace backpropagation’s global credit assignment with predictive-coding updates. They report training 1000-layer residual MLPs on MNIST for 5 epochs while staying within about 2 percentage points of backprop accuracy. The method adds per-layer Lagrange multipliers and improves accuracy over standard predictive coding, with released MIT-licensed JAX reference code reproducing results on CPU.

If coding is solved, what now?: Measuring the sloppiness of code

Earendil 1 month ago 15

The article argues that LLM code can pass hidden tests yet still be sloppy, and it proposes concrete ways to measure that sloppiness beyond correctness. It reports SlopCodeBench comparisons where agent-generated code has average verbosity 0.33 ± 0.10 versus 0.15 ± 0.06 in established repos, and erosion 0.68 ± 0.20 versus 0.31 ± 0.17. The evaluation method uses multiple instruction/test rounds with context erased, so bad decisions accumulate and strict solve rates drop to a 0% pass rate at checkpoints, making “agents will fix it” less reliable.

A Severe Misalignment of AI in Mathematics

What's new 1 month ago 30 ● 21 sources

A group of 25 Fields Medallists signed and publicized a declaration warning that rapid AI results in mathematics reflect severe misalignment between AI-company goals and the mathematics community’s goals. The declaration was released after the authors said discussions in the preceding week led to the statement’s urgency, prompting them to post it sooner than with the more consultative Leiden process. The mathematical community is urged to address attribution, writeup quality, and human transmission so that AI accelerates genuine study rather than overwhelming conceptual understanding and long-term integration of ideas.

How GPT-6 Became No. 1: Artificial Analysis CEO Explains The Much-Debated Index Updates

Trending Topics 1 month ago 12 ● 2 sources

Artificial Analysis updated its Intelligence Index twice within days after GPT-6 Astra launched, moving GPT-6 Astra to a shared first place with Anthropic’s Claude Fable 5.1. The displayed score for both models was 53 after Index v4.3, down from GPT-6 Astra’s 61 under Index v4.1. The changes reflect revised test methodology—adding new task benchmarks, removing one that had saturated, and increasing private-evaluation weighting from 40% to 45%—so the leaderboard can shift quickly and comparisons must use the same index version.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.