TLDRocket
Sign in

AI Evaluation

362 summarised stories about AI Evaluation, each linking back to the original source. Browse all topics →

+ Follow this topic

Tuesday, 8 September 2026

Pathway’s brain-inspired architecture development on Amazon SageMaker HyperPod

Amazon Web Services 1 month ago 46 ● 2 sources

Pathway developed its brain-inspired BDH model architecture and trained its BDH-CQ reasoning system on Amazon SageMaker HyperPod. BDH-CQ reached 29.5% pass@2 on ARC-AGI-1 at US$0.0007 per task. The work shifts reasoning from token-by-token text traces to iterative latent computation with fewer active neurons, aiming to improve cost efficiency and scalability in production training and inference.

Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions

Amazon Web Services 1 month ago 51

Amazon Bedrock AgentCore is used with a GitHub Actions pipeline to deploy an AI agent plus an OAuth-protected MCP server, evaluate agent sessions with the AgentCore Evaluate API, and fail PRs when scores regress. The quality gate is set to require at least 0.8 out of 1.0 for overall evaluation before merges proceed. CI can authenticate headlessly by using an M2M token flow that bypasses role checks in the MCP layer while preserving role enforcement for interactive user tokens, so PRs are blocked based on the current code changes.

How DiDi built intelligent contact center QA with Amazon Bedrock

Amazon Web Services 1 month ago 49

DiDi partnered with AWS to replace an opaque third-party contact center QA tool with a self-owned system built on Amazon Bedrock for its International Business Group CX team. In production validation, intent verification accuracy rose from 38% to 86% while compliance scoring accuracy exceeded 90%. The pipelines now generate auditable, structured results for intent checks, compliance evaluation, and Voice of Customer trend analysis, and manual summarization was compressed from hours to minutes.

Safety for Whom? Refusing the Right Subset of a Topic, Not the Whole Topic

Hugging Face 1 month ago 9

A research paper argues that LLM safety work should refuse only the harmful subset of a topic, not the whole topic, because topic-level guards can mis-handle deployment-specific boundaries. It reports that single-shot self-generation for harmful political prompts drops 19.88% of prompts (8,009) but an escalating retry strategy reduces residual failures to 0.20% (79). It introduces boundary-aware self-distillation and corresponding measurements so training data composition can be tuned to reduce over-refusals on benign prompts near the boundary while keeping harmful refusals high.

The Year Finding and Exploiting Bugs Became Cheap, and What to Do About It

ZK/SEC Quarterly 1 month ago 25

Security researchers increasingly use LLM-based tooling to find bugs and build exploits, and the article says this has made exploit activity grow because both discovery and exploitation have become much cheaper for attackers and defenders. In 2026, two exploits against live ZK circuits are cited, including one that rescued about $1.5 million and another drained 5 ETH. The change pushes security toward continuous, layered defenses—updating tests, AI harnesses, audits, and formal verification—rather than relying on audits done once in the past.

How well do agents use test/verification techniques?

danluu.com 1 month ago 5

The author evaluated agent-generated Rust implementations by running 26 different prompt conditions that asked agents to use specific test/verification techniques or libraries when implementing Zstd. The study tested 26 prompt conditions across different techniques and skills. Overall, no approach wildly outperformed the baseline, and agents often applied named techniques superficially or wrote poor tests rather than using verification effectively.

Insilico’s AI-designed drug made patients look biologically younger on six aging clocks

Nature Publishing Group US 1 month ago 8 ● 3 sources

Insilico’s AI-designed TNIK inhibitor rentosertib (INS018_055) was evaluated in a phase 2a IPF trial using serum proteomics analyzed by six proteomic aging clocks. At 12 weeks, 21 of 54 treatment-vs-placebo clock comparisons were statistically significant, with most occurring at week 4. All six clocks showed reduced predicted biological age in treated groups versus minimal change in placebo, shifting trial readouts toward measurable aging-rate deceleration alongside disease effects.

GPT-6 Astra Now on Par With Claude Fable 5.1 in Updated Artificial Analysis Index

Trending Topics 1 month ago 31 ● 19 sources

Artificial Analysis updated its Intelligence Index to version 4.3, tying OpenAI’s GPT-6 Astra (max) and Anthropic’s Claude Fable 5.1 (max with fallback) at the top with 53 points. Terminal-Bench was upgraded from version 2.1 to 4.0 using 66 tasks, where GPT-6 Astra scores 59.1% and Claude Fable 5.1 scores 52.0%. The rankings change within a few days because two test evaluations were replaced/upgraded and the index was recalculated after each revision, so Astra moved from fifth to first while some competitors dropped.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.