Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

Together AI · 7 months ago

AutoJudge accelerates large language model inference by automatically identifying which mismatched tokens between draft and target models don't affect task correctness, eliminating the need for manual annotation. The method achieves 1.5–2x speedups over standard speculative decoding while accepting up to 40 draft tokens per verification cycle with minimal accuracy loss, and integrates into existing frameworks like vLLM and TensorRT-LLM. Inference speed improves across benchmarks with only 2–4% accuracy drops: on GSM8K, the Llama-3.1-70B/8B pair reaches 107.4 tokens/s (1.49x faster), and on code tasks, acceptance rates increase 2.3–3.5x.