OpenAI Blog
·
2 weeks ago
● 2 sources
OpenAI's analysis identified flaws in SWE-Bench Pro, a widely-used benchmark for evaluating coding AI systems, questioning whether its results accurately reflect model performance. The benchmark contains ambiguous test cases and inconsistent evaluation criteria that affect reliability of the results. This finding prompts developers to scrutinize existing coding benchmarks more carefully and potentially revise evaluation methodologies before trusting benchmark rankings.
Mistral AI
·
2 weeks ago
Robostral Navigate is an 8-billion parameter model that enables robots to autonomously navigate complex environments using only a single RGB camera and natural language instructions. The model achieved 76.6% success on unseen R2R-CE benchmarks, outperforming multi-sensor systems by 4.5 percentage points while using neither depth sensors nor LiDAR. The approach generalizes across different robot types and enables autonomous navigation in real-world settings like offices and commercial buildings without encountering obstacles during training.
TLDR Dev
·
2 weeks ago
● 2 sources
Anthropic's Claude Sonnet 5 has lower per-token pricing but consumes 10x to 12x more tokens on common agent tasks, making it 3.7x more expensive on code upgrades despite the 33% per-token discount. On architecture tasks, Sonnet 5 produced lower quality output (78% vs 90% on idiomatic evaluation) while consuming vastly more tokens, though it excelled at precise instruction-following on code upgrade tasks. Neither model overcomes undocumented information gaps, so improving agent grounding content delivers more value than upgrading models.
TheSequence
·
2 weeks ago
Google Research released TabFM, a foundation model for tabular data that performs classification and regression tasks without training, tuning, or feature engineering by treating the entire problem as a single prompt. TabFM builds on the approach of TimesFM, Google's time-series foundation model, and uses in-context learning to make predictions on unseen tabular data in a single forward pass. This approach disrupts the traditional enterprise ML workflow of manual feature engineering and hyperparameter tuning that has dominated for over a decade.
Allen Institute (AI2)
·
2 weeks ago
MolmoAct 2, an open-source AI model for robotics, enabled developers at a hackathon to build voice-controlled robots that performed tasks without specific training in just ten days. The model includes complete artifacts such as weights, training code, and fine-tuning scripts on public platforms like Hugging Face and GitHub. The availability of this general-purpose model represents a shift in robotics by providing the community with accessible tools for building adaptable robot systems.
Latent Space
·
2 weeks ago
Lilian Weng published a research summary covering 35 papers on harness engineering for recursive self-improvement, framing the field's shift toward optimizing prompts and task specifications rather than direct model weight modification. The post synthesizes design trends from papers including the well-known ACE work through recent approaches like Meta-Harnesses, demonstrating that goal and context specification will remain necessary even as harness improvements get absorbed into core models. This signals a consolidation around harness-based agent design as the primary optimization frontier, influencing product development at companies including Anthropic, Google, and LangChain.
