AI Uncertainty & Reliability

2 summarised stories about AI Uncertainty & Reliability, each linking back to the original source. Browse all topics →

Thursday, 16 July 2026

Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

arXiv cs.CL 18 hours ago

Researchers introduced Code-MUE, a black-box framework that measures uncertainty in code-generating language models by analyzing runtime behavior through Semantic Interaction Graphs rather than text similarity. The method achieved Spearman's correlation of up to -0.98 with functional correctness across eight state-of-the-art code LLMs, substantially outperforming text-based and embedding-based alternatives. This enables better risk detection for deploying code models in production, where distinguishing confident predictions from stochastic guessing is critical for safety and security.

AI Code Generation AI & Cybersecurity AI Evaluation AI Uncertainty & Reliability

When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents

arXiv cs.AI 18 hours ago

Researchers found that running the same large language model agent multiple times on identical inputs produces 2.3-4.2 different action sequences per 10 runs, and this behavioral variance can serve as an uncertainty signal for detecting when agents will fail. On HotpotQA tasks, consistent ones achieving at most 2 unique paths reached 82-87% accuracy while inconsistent ones with 4+ paths reached only 41-65%, with divergence clustering at step 2 for about 50% of tasks. By using selective prediction to answer only when 3 runs agree, accuracy improved to 87-88% at 54-62% coverage, a gain of 6-14 percentage points over single-run baselines, while cross-validation on SWE-bench showed single evaluations misrank models 29.3% of the time.

AI Agents AI Behavior & Psychology AI Evaluation AI Uncertainty & Reliability