Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation
Together AI
AutoJudge accelerates large language model inference by automatically identifying which mismatched tokens between draft and target models don't affect task correctness, eliminating the need for manual annotation. The method achieves 1.5–2x speedups over standard speculative decoding while accepting up to 40 draft tokens per verification cycle with minimal accuracy loss, and integrates into existing frameworks like vLLM and TensorRT-LLM. Inference speed improves across benchmarks with only 2–4% accuracy drops: on GSM8K, the Llama-3.1-70B/8B pair reaches 107.4 tokens/s (1.49x faster), and on code tasks, acceptance rates increase 2.3–3.5x.
Why it matters
AutoJudge accelerates LLM inference by identifying which token mismatches actually matter. Using self-supervised learning to train a lightweight classifier, it accepts up to 40 draft tokens per cycle—delivering 1.5–2× speedups over standard speculative decoding with minimal accur
