359 summarised stories about AI Evaluation, each linking back to the original source. Browse all topics →
Simon Willison’s Weblog 2 weeks ago 42 ● 2 sources
Anthropic Frontier Red Team evaluated several models on 100 internal binary exploitation tasks. GLM-5.3 achieved full control flow hijacks in 4% of trials, while Claude Mythos Preview did so in 6%. Earlier models such as Claude Opus 4.6 and GLM-5.2 showed 0 successes, indicating capabilities have increased.
MarkTechPost 2 weeks ago 5
Liquid AI released d1, a decision model that takes context plus typed questions and returns calibrated probabilities for a fixed set of options without generating text. It reports usage.output_tokens as 0 in every response. The model is deployable today via a hosted API (model name d1:free) rather than self-hosted weights, shifting common classification and routing use cases away from general text-generating LLMs.
Google Research 2 weeks ago 31
Diffusion Controller reframes diffusion-based text-to-image generation as a continuous control problem and adds a lightweight “steering damper” network to steer outputs toward user targets while protecting base-model image quality. It reports a 90% win rate over the baseline in its fully unlocked (“white-box”) version. This unifies previously separate inference guidance and fine-tuning methods under one control framework and enables runtime adjustment of control strength without degrading visuals.
Product Hunt 2 weeks ago 22
Pheebs, an open-source AI telemetry tool by Eversynced, was launched as an AI coding-agent hook that captures lightweight session-shape signals without storing contents. It launched on a page showing it had 8 followers. As a result, teams can measure how they work with AI coding agents using telemetry from tools like Claude Code, Cursor, and Codex.
The New Stack 2 weeks ago 33 ● 13 sources
Anthropic released Claude Opus 5.5 and the author compared it against Claude Fable 5.1 with identical coding benchmarks, focusing on accuracy, cost, and runtime. Opus 5.5 costs $4 per million input tokens and $20 per million output tokens, while the author’s tests found overall cost of $22.07 versus $28.14 for Fable 5.1. The results lead the author to conclude neither model should be billed as a premium choice: Opus 5.5 better handles agentic tasks that can self-check, while Fable 5.1 is faster and more reliable on concurrency-heavy one-shot problems.
Hugging Face 2 weeks ago 2
ProvenanceGuard was proposed as a post-generation verification layer for MCP-based LLM agents to prevent cross-source conflation, where evidence supports a claim but the answer attributes it to the wrong MCP tool output. In a medical-agent evaluation with 281 real traces and a test set where 361 claims were reviewed by experts, experts said 139 claims should not pass and ProvenanceGuard caught 138. It keeps tool-source identity through claim checking, emits per-claim source verdicts plus an allow/block decision, and runs a RARR-style repair loop when answers are blocked, instead of relying on source-blind faithfulness scoring.
Alex Ewerlu00F6f Notes 2 weeks ago 49 ● 5 sources
The author argues that coding is not solved by LLMs and that teams should still prioritize reading, testing, and accountability for production software. The post notes that on Hacker News it reached spot 2 with 473 points and 476 comments as of 2026-09-29. As a result, the author calls for narrower, well-scoped AI use cases (like translation and summarization) rather than pushing AI into every engineering surface where non-functional requirements dominate costs.
claude.dev Blog 2 weeks ago 46
Claude Code’s claude-api skill added guided workflows for building evals and improving apps via /claude-api build-eval and /claude-api hillclimb. The hillclimb workflow warns when a baseline score is around 95% or higher. It now produces structured eval cases, grader checks, and an iterated patch process that splits data into train and test to reduce overfitting before keeping changes.
MarkTechPost 2 weeks ago 54
Google Research and university partners released RRSI (Regularized Recursive Self-Improvement), a framework that lets an LLM agent rewrite its own harness while keeping model weights frozen. On Terminal-Bench 2.1 (evolve split), performance rose from 74.2% to 80.2%. Because RRSI regularizes the self-improvement loop with leakage checks, noise-adjusted acceptance, and a token cost rule, improvements transfer to benchmarks the agent never optimized against and the harness evolution uses fewer tokens per trial.
Apple Machine Learning Research 2 weeks ago 2
The study measures how well language models preserve tree-structured expressions when they are serialized into word problems and then extracted back. It evaluates all pairwise combinations of sixteen models. The result is an empirical test of the “communication bottleneck,” showing how much of that structured content survives the round trip.
The daily briefing
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.
TLDRocket
18 October 2026
The day’s biggest AI thread wasn’t model quality—it was accountability, in public. Anthropic announced it will commit 5% of its shares to a new Anthropic Public Benefit Fund, with the option to add 2% for every 50% increase in company value above a baseline, effectively steering more of its long-term upside toward mission-driven giving. It’s pairing that financial structure with operational one: METR, an outside AI watchdog, now gets “employee-like” access, including an embedded team at Anthropic’s office with access to hardware and internal systems for risk evaluation. It’s a rare instance where governance isn’t just a policy PDF; it looks like an audit workflow with teeth.
Read the full briefing →