BenchMIRT was introduced as a method to audit LLM benchmarks at the level of individual prompts, estimating which underlying capabilities each question measures. BenchMIRT was trained on benchmarking results from 100 LLMs across more than 34K questions. It recovered two stable capability dimensions—safety and general reasoning—showing that averaging benchmark items can hide mixed signals and that using only 10% to 50% of questions often preserves the same capability picture while predicting held-out answers 79% of the time.
Researchers from Princeton University, Ant Group, and Stanford University introduced AQuA, a two-part framework where language-model agents iteratively improve quantitative research while the evaluation setup stays fixed to prevent evidence leakage. On a crypto five-minute universe across 20 research epochs, AQuA’s combined validation Spearman IC rose to about 0.190 (vs 0.171 for AlphaMemo and lower for several baselines). The approach changes research by making leakage-inducing actions unavailable and by having agents emit only constrained factor expressions or configuration diffs, yielding a positive equity long/short book with about +2.50 Sharpe after a volatility-targeting overlay.
AI companies in the West paused training or published safety plans while trust-and-safety frameworks still fail users in many countries due to safety decisions being defined and evaluated with a narrow set of assumptions. More than two-thirds of chatbots do not adequately account for dialects or recognize urgency cues, which leads to errors that can affect health and other decisions. The result is a widening “AI divide” where English-speaking users are safer than users of low-resource languages, pushing governments and UN-linked efforts to invest in local safety infrastructure and oversight.
BenchMIRT audits large language model benchmark scores at the level of individual prompts by using multidimensional item-response theory to estimate the capabilities each task tests. It was trained on results from 100 LLMs covering 16 benchmarks and more than 34,000 questions. Benchmark scoring becomes more interpretable by separating signals like safety and general reasoning, revealing cases where “safety” benchmarks are driven largely by reasoning or other mixed effects, and enabling smaller question sets while largely preserving the underlying capability picture.
Ai2 and the Paul G. Allen Research Center held an August 27 event to review what modern AI still cannot do for steering live scientific research as evidence and hypotheses change. The discussion centered on five recurring challenges, including steerable agents and delegation choices, rather than improvements to day-to-day analysis. The takeaway was that future scientific AI work should focus on adaptability, human steering and verification, and tighter AI-to-lab feedback loops instead of only generating hypotheses or accelerating computation.
Gradium AI released a new default text-to-speech model across its API and Studio. It reported an 81.0% human-rated pass rate on a 500-sentence hard-case set and a 216 ms P50 time-to-first-audio. The default model switch went live on August 31, 2026 with no migration and existing (including custom) voices continuing to work unchanged.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.
