AdapTive-LeArning Speculator System (ATLAS): A New Paradigm in LLM Inference via Runtime-Learning Accelerators

Together AI · 1 year ago · 42

Together AI introduced ATLAS, an adaptive-learning speculator system that accelerates LLM inference by dynamically improving at runtime without manual tuning. ATLAS achieves up to 500 tokens per second on DeepSeek-V3.1 and 460 TPS on Kimi-K2, representing a 4x speedup over baseline decoding and outperforming static speculators. The system adapts continuously to changing workloads and input distributions, enabling sustained performance improvements as usage patterns evolve.