366 summarised stories about AI Evaluation, each linking back to the original source. Browse all topics →
The Register 1 month ago 49
Piotr Migdał tested leading AI models for identifying mushrooms and found they give unreliable results for safe versus deadly species. The best model, Gemini-3.8-flash, was correct on its first guess only 65% of the time. As a result, the dataset and findings push people to not eat mushrooms based on AI outputs and instead rely on manual expertise or safer species until identification accuracy is proven.
Simon Willison’s Weblog 1 month ago 30
llm released version 0.34, adding response-duration logging to Markdown output and the llm logs --short output. The update reports duration in milliseconds via a new duration_ms field. This changes the logs by including both a human-readable duration and duration_ms, alongside bug fixes and a performance improvement to llm logs.
The daily briefing
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.
TLDRocket
19 October 2026
The day’s biggest thread was the messy middle between “agents that act” and the systems that prove they won’t act badly. Hugging Face policy chief Irene Solaiman warned that embedded AI evaluators and today’s benchmarking are still too immature to trust without better reproducibility and clearer comparisons, especially as evaluation is shifting from model-only tests to full-system setups that include the harness and agent—raising cost and forcing standardization. That concern landed alongside a Princeton-led paper on emerging agent-evaluation trends: ExploitBench runs can burn tens of thousands of credits, and the authors argue the field needs more independent, multi-layer evaluation, including real-world and human-impact studies, plus a better evaluation ecosystem.
Read the full briefing →