Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation
Together AI
AutoJudge accelerates large language model inference by automatically identifying which mismatched tokens between draft and target models don't affect task correctness, eliminating the need for manual annotation. The method achieves 1.5–2x speedups over standard speculative decoding while accepting up to 40 draft tokens per verification cycle with minimal accuracy loss, and integrates into existing frameworks like vLLM and TensorRT-LLM. Inference speed improves across benchmarks with only 2–4% accuracy drops: on GSM8K, the Llama-3.1-70B/8B pair reaches 107.4 tokens/s (1.49x faster), and on code tasks, acceptance rates increase 2.3–3.5x.
Why it matters
AutoJudge accelerates LLM inference by identifying which token mismatches actually matter. Using self-supervised learning to train a lightweight classifier, it accepts up to 40 draft tokens per cycle—delivering 1.5–2× speedups over standard speculative decoding with minimal accur
Related stories
Liquid AI Releases LFM2.5-DSpark Draft Models That Deliver Up to 3.18x Faster Decoding Without Changing Model Outputs
MarkTechPost · 3 weeks ago ·
37
Speculative cascades — A hybrid approach for smarter, faster LLM inference
Google Research · 1 year ago ·
47