Anthropic granted six METR researchers employee-level access to independently assess its frontier AI models, safety practices, and risk management. The access includes Anthropic-issued company laptops and internal tools roughly on par with what its own risk assessment teams use. METR will publish its own evaluations and Anthropic will not be able to veto publication, only require redactions for categories like customer data and attorney-client privileged information.
Anthropic began giving employee-like access to a METR nonprofit team to evaluate its models and safety practices under third-party oversight. The update says six METR employees were onboarded with company hardware and internal tool access. As a result, METR will publish risk assessment findings, while Anthropic can require redactions for narrow categories of sensitive information and expands other evaluator partnerships in tiers.
Agentic software tooling increasingly produces code changes faster than teams can review them line by line, shifting the bottleneck from generating code to ensuring confidence in what ships. The article highlights an explicit intent×guardrails×verification×evidence formula as the concrete framework for reducing the code confidence gap. Organizations should redesign delivery systems to require clear standards, independent checks, and auditable records rather than relying on PR completion or the same probabilistic model that generated the change.
BrowseComp-Plus evaluations showed that agentic systems using BM25 to retrieve evidence fell sharply in accuracy compared with a setting where GPT-4.1 was given the supporting documents directly. The score dropped from 93.49% to 14.58% when GPT-4.1 switched from oracle evidence to BM25 search on the benchmark. The article concludes that retrieval quality, query interface, and retrieval-layer capacity/latency become the key design focus, because better models cannot compensate for missing or stale evidence.
LangSmith’s team explains how to run agent evals to check agent behavior in development and on real user traffic, since agent outputs can vary and multiple paths can be correct. They recommend starting with 10 to 20 dataset examples drawn from traces and using a baseline experiment to compare future agent versions. The guidance shifts teams from exact-output unit tests to an offline/online eval loop that adds production failures back into the dataset and uses different evaluator types to score runs.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.
