Information-Theoretic Limits of Reliability and Scaling in Language Models

arXiv cs.CL · 6 hours ago

Researchers established information-theoretic limits showing that large language models cannot achieve perfect reliability on any task, with maximum performance determined by how much output uncertainty is resolvable from context. The framework decomposes reliability into a resolvable component addressable through additional context and a subjective component tied to inherent task ambiguity, with autoregressive generation further reducing performance ceilings at rates governed by inter-token correlations. This theoretical framework recovers existing scaling laws as special cases and explains when additional model capacity or training data improves performance, unifying phenomena like retrieval-augmentation benefits and catastrophic forgetting.