291 summarised stories about AI Benchmarking, each linking back to the original source. Browse all topics →
Simon Willison’s Weblog 1 month ago 7 ● 6 sources
Anthropic released Claude Fable 5.1 and the article tests it by generating an animated SVG of a pelican riding a bicycle. Terminal-Bench-Science 0.1 is reported at a 52.6% score for Fable 5.1. Higher reasoning-effort settings produce much longer outputs and a better-quality pelican SVG, enabling a separate run that animates the benchmark “pelican” without paying the full cost again.
Hugging Face 1 month ago 35 ● 2 sources
BenchMIRT was introduced as a method to audit LLM benchmarks at the level of individual prompts, estimating which underlying capabilities each question measures. BenchMIRT was trained on benchmarking results from 100 LLMs across more than 34K questions. It recovered two stable capability dimensions—safety and general reasoning—showing that averaging benchmark items can hide mixed signals and that using only 10% to 50% of questions often preserves the same capability picture while predicting held-out answers 79% of the time.
MarkTechPost 1 month ago 9 ● 2 sources
Anthropic released Claude Fable 5.1 and Claude Mythos 5.1 as two safeguard-layer variants of the same underlying model. Fable 5.1 scores 52.6% on Terminal-Bench-Science 0.1. Cache reads were cut 75% to $0.25 per million tokens, and accounts created on or after 2026-08-31 face breaking API changes for agents that edit conversation history.
TechCrunch 1 month ago 42 ● 5 sources
Anthropic released Fable and Mythos 5.1, updated versions of its most advanced AI model, with Fable intended to be less restrictive and both versions improving performance.
The New Stack 1 month ago 15 ● 6 sources
Z.AI released GLM-5.3-Flash on August 26 alongside earlier GLM-5.3, and one tester compared both models on three real-style tasks to check whether the cheaper model sacrifices accuracy. Flash was priced at $0.075 per million input tokens and used 38,677 output tokens on a hardest coding spec, taking 455.8 seconds versus GLM-5.3’s 174.7 seconds at 14,801 tokens. Accuracy stayed equal at 27/27 for both, but the outcome for users shifts by workload: Flash is cheaper and faster on easier extraction while the flagship is faster for difficult coding and may change cost-effectiveness if pricing changes.
Allen Institute (AI2) 1 month ago 31 ● 2 sources
BenchMIRT audits large language model benchmark scores at the level of individual prompts by using multidimensional item-response theory to estimate the capabilities each task tests. It was trained on results from 100 LLMs covering 16 benchmarks and more than 34,000 questions. Benchmark scoring becomes more interpretable by separating signals like safety and general reasoning, revealing cases where “safety” benchmarks are driven largely by reasoning or other mixed effects, and enabling smaller question sets while largely preserving the underlying capability picture.
The daily briefing
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.
TLDRocket
19 October 2026
Nvidia’s latest open-model push set the tone for AI’s next phase: less “big chat,” more dependable agent infrastructure. Alongside its enterprise pitch, the company released Nemotron 3.5 Super VL, a 120-billion-parameter multimodal model aimed at always-on corporate agents, and followed with Nemotron 3.5 Bolt, a locally runnable, laptop-focused variant. Artificial Analysis reports Super VL lands 26 points on its 10-benchmark Intelligence Index while using about 60% less text than the prior Nemotron 3 Super—an efficiency bump that matters when tokens are your operating cost, not your poetry. Nvidia also shifted availability toward faster, leaner deployments, including new quantized variants and updated API token pricing designed for long-running workflows.
Read the full briefing →