TLDRocket
Sign in

AI Benchmarking

294 summarised stories about AI Benchmarking, each linking back to the original source. Browse all topics →

+ Follow this topic

Wednesday, 26 August 2026

GlucoFM: Foundation model for continuous glucose monitoring

Google Research 1 month ago 37 ● 2 sources

GlucoFM was developed as a self-supervised foundation model for continuous glucose monitoring that separates slow glycemic trends from short-term deviations while accounting for time of day and missing data. It was pre-trained on 109,066 hours of unlabeled CGM data and, across 14 cohort–task evaluations, achieved an average PR-AUC of 58.8% versus 54.7% for the best GluFormer baseline variant, and 21.88 mg/dL MAE for two-hour post-meal glucose-change forecasting. The model improved diabetes-risk, beta-cell-dysfunction, and most insulin-resistance predictions, transferred well across cohorts, and adapted with very limited labeled data.

When LLM judges agree, should we believe them?

Amazon Science 1 month ago 28 ● 2 sources

Researchers propose a dependence-aware way to aggregate outputs from multiple LLM judges so that agreement counts don’t overstate independent evidence when judges share biases. The method’s strongest results using 10 judge models at temperature 0 reach 0.912 accuracy for relevance, up from 0.820 with weighted majority vote and 0.804 with uniform majority vote. As a result, evaluation scores adjust for learned pairwise correlation between judges rather than assuming independent errors, improving accuracy across three binary tasks.

Preparing data for supervised fine-tuning Part 2: Advanced data strategies

Amazon Web Services 1 month ago 31 ● 2 sources

The article lays out advanced strategies to improve supervised fine-tuning datasets after formatting and quality checks. It recommends planning for roughly 2,000 high-quality training samples as a starting point, with learning-curve analysis using checkpoints saved every 10–20 percent of training to find when gains saturate. As a result, teams can decide when to stop, curate a smaller higher-value subset, use augmentation (including synthetic reasoning traces) to expand coverage, and apply controlled data mixing to reduce catastrophic forgetting.

What Would Have to Be True for Agentic Coding to Replace Junior Engineers

MarkTechPost 1 month ago 31

Agentic coding would replace junior engineers only if four conditions hold, but the article argues three key conditions fail. The most specific cited benchmark issue is that OpenAI audited 27.6% of SWE-bench Verified and found flawed tests rejecting correct solutions in 59.4% of the audited problems. The result, it claims, is that agentic coding is more likely to replace junior-level tasks and increase reliance on senior verification and judgment rather than eliminating junior hiring.

The Sequence Learning Loop - Issue #921: Learn About DeepSeek New Model, the Env Harness Paper and the Amazing Etched

TheSequence 1 month ago 16

DeepSeek added vision to its fast V4 model, while Google Cloud AI Research introduced EnvHarness and Etched shipped its first inference rack to Jane Street. EnvHarness was described as a framework for adapting training environments using an agent’s weaknesses, rather than a fixed setup. These updates tighten the loop across model inputs, training environments, and serving hardware to make perception, learning, and inference costs more tightly coupled.

AI models flub these intelligence tests. Can you fare any better?

MIT Technology Review 1 month ago 14

AI models have performed poorly on multiple puzzle-based intelligence tests, especially those involving spatial/visual reasoning and small changes that break memorized patterns. In late 2024, top models solved only 18% of the New York Times Connections puzzles. The article changes by reframing puzzles as a way to pinpoint specific model weaknesses and invites readers to try the same challenges that tripped up AI.

PROOF-Gen: From Optimized Data to Better Distillation

Apple Machine Learning Research 1 month ago 30

Tool-calling distillation pipelines that repeatedly re-run supervised fine-tuning on teacher-generated trajectories keep paying the frontier-teacher cost while generating-and-filtering only passing cases. On τ 2-bench, 57% of teacher trials fail, with two-thirds of those failures being near-misses, so the current process leaves the hardest scenarios without useful training signal. PROOF-Gen changes the post-training approach by turning those optimized data signals (including near-misses) into better distillation for deployable models.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.