Revalvo
Product Hunt 1 month ago 23
Revalvo launched a local-first workbench for prompt engineering and LLM evaluation that runs prompts across models and scores responses with built-in evaluators.
AI Evaluation
368 summarised stories about AI Evaluation, each linking back to the original source. Browse all topics →
Key players
Thursday, 27 August 2026
Google found a way to test Gemini without seeing the questions
The New Stack 1 month ago 17 ● 2 sources
Google DeepMind demonstrated a double-blind evaluation for Gemini where evaluators cannot see Gemini’s weights and Gemini cannot see the test questions. The pilot evaluated Gemini 2.5 Flash Lite against private MLCommons and Singapore AI Safety Institute benchmarks using an NVIDIA H100 Confidential GPU with Intel TDX and Confidential Space, with evaluation overhead reported as under 5%. This shifts focus from publishing a benchmark score to showing how to run tests with reduced benchmark leakage risk by keeping both sides’ data protected inside an encrypted enclave.
Aider, Claude Code, and OpenClaw ran an identical model. Token use varied 70-fold.
The New Stack 1 month ago 8 ● 2 sources
A set of AI coding-agent benchmarking efforts found that using different harnesses with the same underlying model can cause very large differences in token usage and cost. One result showed token use per solved task ranging from about 3,500 for Aider (architect mode) to 292,000 for OpenClaw. The findings shift optimization toward harness prompt/context overhead and cache-hit behavior—so enterprises should measure cost per verified outcome rather than raw token counts.
ThunderPhone
Product Hunt 1 month ago 45
ThunderPhone launched a self-serve platform for building AI phone agents. Pricing starts at 2¢/min and its Storm tier with extra-intelligence option scored 99.4% on Big Bench Audio. Users can build agents, test them via AI-caller simulations, monitor live calls, and get auto-detected issues with proposed fixes.
Deep Learning Weekly: Issue 470
Deep Learning Weekly 1 month ago 45 ● 2 sources
Z.ai open-sourced GLM-5.3-Flash, a stealth “Ox Alpha” mixture-of-experts model with hybrid sparse-plus-linear attention and a 1M-token multimodal context, and multiple other deep-learning updates were shared alongside it. The issue’s FreeToken paper reports edge-native MoE serving that supports running a 753B GLM-5.2 on a single workstation GPU. Benchmarks and tool writeups in the issue shift focus toward more practical local/edge deployment and evaluating real-world LLM/agent behavior rather than only final answers.
Piloting the world's first double-blind AI evaluations
Google 1 month ago 17 ● 2 sources
Google introduced the world’s first double-blind evaluation of a proprietary frontier AI model, using a cryptographic “box” so external evaluators’ benchmarks can’t be used for later optimization. On August 27, 2026, Google said it would test a Gemini Flash Lite model against confidential benchmarks with partners including the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. This adds cryptographic safeguards to reduce benchmark contamination and improve trust in the measured model results.
From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers
Apple Machine Learning Research 1 month ago 46
The authors propose a rubric-based reward framework that generates query-specific, evidence-grounded rubrics and uses them for post-training to supervise open-domain question answering. Averaged across three evaluation axes (composition, grounding, and instruction-following), it improves 6.5% over an instruction-tuned baseline. The approach changes training by replacing a single scalar reward with multi-dimensional rubrics conditioned on retrieved evidence, yielding more factual support and better coherence and instruction adherence.