Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

Apple Machine Learning Research · 2 months ago · 48

Researchers from UC Berkeley and affiliated labs developed ARBITRAGE, a technique that accelerates large language model reasoning by using a lightweight router to dynamically choose between a fast draft model and a slower but more capable target model based on their relative performance. The method achieves up to 2× speedup in inference latency on mathematical reasoning benchmarks compared to prior step-level speculative decoding approaches. This allows LLMs to generate longer reasoning chains more efficiently without sacrificing accuracy.