Capturing token IDs during agentic interactions for better reinforcement learning

Amazon Science · 3 months ago · 37

Anthropic released Turnstile, a proxy tool written in Rust that captures exact token-level data during reinforcement learning training of language models on multi-step tasks. Turnstile records token IDs, log probabilities, and loss masks at the moment of generation without modifying existing agent harnesses, solving the problem that transcript-based data loses critical information needed for effective RL training. The system enables RL training runs with existing agent harnesses as black boxes while handling complexities like mixture-of-experts routing and multimodal inputs from vision-language models.