TLDRocket
Sign in

AI Evaluation

358 summarised stories about AI Evaluation, each linking back to the original source. Browse all topics →

+ Follow this topic

Friday, 2 October 2026

Circuit Breaker Labs hopes to make AI safer for your kids (and you)

TechCrunch 2 weeks ago 33

Circuit Breaker Labs is building AI-safety testing agents to find harmful, psychologically risky interactions in AI chat and mental-health style apps. The company runs tens of thousands to hundreds of thousands of simulated conversations per day. It plans to expand its “red-team” style platform beyond early mental health use cases by scoring model responses for explainable, auditable safety weaknesses across ages, languages, and slang.

Fine-tune a search agent with multi-turn RL on Amazon SageMaker AI

Amazon Web Services 2 weeks ago 29

Amazon SageMaker AI fine-tuned a Qwen3.6-27B search agent using multi-turn reinforcement learning to improve how it searches across multiple interaction rounds. The failure rate on BrowseComp-Plus fell from 22.89% to 0.68%. Multi-turn RL raised retrieval quality on three of four held-out benchmarks and made the agent substantially more reliable by reducing task-completion errors within the turn and token limits.

Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks

MarkTechPost 2 weeks ago 6

Datalab released OmniExtractBench, an open benchmark that scores structured extraction from PDFs by comparing predicted JSON fields to gold using an auditable deterministic scorer. The benchmark pools 620 documents from 4 existing extraction suites and packages a PyPI scorer (omni-extract-bench v0.1.7) under Apache 2.0. Vendors’ extraction leaderboards become more comparable and easier to audit because scoring is transparent, content-based for table row alignment, and rules drop null/empty-field padding that can otherwise inflate results.

Deep Learning Weekly: Issue 475

Deep Learning Weekly 2 weeks ago 26 ● 20 sources

Deep Learning Weekly Issue 475 rounds up multiple deep-learning updates including OpenAI’s GPT-6.1 Sol, Anthropic’s Claude Sonnet 5.5, agent tools, MLOps/agent evaluation articles, and two new research papers on LLM agent memory and deep search agents. OpenAI’s GPT-6.1 Sol is positioned at 1/5th the Astra-like price, alongside a new Ultrafast tier at 300 tokens per second. The result is a consolidated guide to what to read next, with specific performance/cost comparisons and research methods (read-time memory curation and role-decoupled iterative synthesis) to apply or evaluate.

Don’t be fooled—LLMs don’t reason

MIT Technology Review 2 weeks ago 21

Thore Graepel argues that AlphaGo’s famous move 37 reflected true reasoning from an explicit search over future possibilities, not “intuition” alone, and contrasts this with how today’s large language models generate answers token by token. He says LLM reasoning attempts via techniques like “chain of thought” still rely on the same next-token prediction process and often lack an explicit epistemic state, citing move 37 as a case where the policy ranked it as about a 1 in 10,000 chance. He concludes that trustworthy AI systems need a new machine-reasoning approach modeled on AlphaGo’s architecture, including an auditable state of beliefs that gets updated with evidence rather than post-hoc rationalizations.

AutoSynthData: Generating Training Data for Enterprise Agents

Hugging Face 2 weeks ago 25

ServiceNow CoreAI built AutoSynthData to turn enterprise agent capability gaps—identified from a target model’s environment failures—into executable training tasks generated and validated using a stronger teacher’s successes. AutoSynthData’s EnterpriseOps Gym pipeline uses a task-selection rule that favors target-model solves on no more than 1 of 3 trials while requiring the stronger solver to succeed on at least 2 of 3 trials. As the model improves, AutoSynthData shifts the curriculum toward tasks the model still struggles with, using iterative evaluation and feedback loops to expand and rebalance the training dataset.

Language Discrimination Improves Linguistic Learning in Multilingual Speech Models

Apple Machine Learning Research 2 weeks ago 18

Multilingual HuBERT-style self-supervised speech models improved linguistic learning when pretraining was made to better discriminate between languages. Phone-ABX error dropped from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%) while lexical (sWUGGY) and prosodic performance also increased. The gains were largest when language discrimination was introduced in the first training iteration, whereas later or repeated use caused more language-wise segregation.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.