llm-meta-ai 0.2
Simon Willison’s Weblog 6 days ago 46 ● 27 sources
Meta AI released llm-meta-ai 0.2, adding support for its Muse Wonder and Muse Wonder Full models.
Model Evaluation
203 summarised stories about Model Evaluation, each linking back to the original source. Browse all topics →
Key players
Monday, 12 October 2026
Five Months Treating Bugs Like Patients and Coding Agents Like a Medical Team
Cockroach Labs 6 days ago 36
Cockroach Labs’ MOLT Sinai “coding hospital” pipeline added Db2 support to its MOLT migration tooling without human-written code during execution. The Db2 migration support work finished in less than two days and the Claude token bill was $4,172 (164x faster and 38x cheaper than the prior Oracle effort). After this pilot, the team enabled human approval mode for merges and used the same system to build the Migration Assistant, which is in preview for Postgres migrations and largely authored by the pipeline.
Build Your Own Decision Model
Another Dev's Two Cents 6 days ago 33
A developer showed how to build decision-model style multiple-choice inference with an LLM by constraining outputs to fixed option tokens and using their softmax scores as pseudo confidence. They tested Qwen/Qwen3-1.7B on a CommonsenseQA sample of 1221 items and reported 725/1221 accuracy (0.5938). The approach changed from overconfident, uncalibrated scores to better-calibrated probabilities after applying temperature scaling with a fitted temperature of 3.797280788421631, and they released scripts to reproduce evaluation, finetuning, and calibration.
Self-Compacting Language Model Agents
Apple Machine Learning Research 1 week ago 13
Self-compacting language model agents were proposed to prevent long agent traces from accumulating stale context that pushes earlier content out of the context window. The method, SELFCOMPACT, reports up to 18.1 point gains on math and uses 30–70% fewer tokens per question. It changes compaction from fixed token-interval summarization to a model-invoked summarization tool plus a lightweight rubric that decides when to summarize or suppress during trajectories.