ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
Together AI ● Covered by 3 sources
Together AI built a scheduler called ThunderAgent that stops AI agents from thrashing each other's memory while running in parallel. Result: over 2x faster inference and near-linear scaling across GPU clusters, just by adding one extra field to your requests.
Anyone who's tried to run hundreds of AI agents at once has hit the same wall: the more you throw at the cluster, the worse things get. Together AI's new paper, accepted as an ICML 2026 Spotlight, names the culprit precisely. It's called KV cache thrashing, and it happens because today's inference engines like vLLM and SGLang treat every LLM call as its own isolated request, with zero memory that it's actually part of a long-running agent workflow.
Here's the mechanics of the problem. An agent alternates between bursts of GPU-heavy reasoning and idle stretches where it's waiting on a tool, say a compiler or a search API. During that idle wait, a request-level scheduler sees free memory and evicts the agent's cached context to make room for someone else. When the tool finally responds, the engine has to recompute that entire conversation from scratch, which promptly evicts a different agent's cache. Multiply that by hundreds of concurrent agents and you get a cascading loop that tanks throughput. Together AI says this is exactly what bit them while generating agentic training data for their CoderForge dataset, where coding agents run dozens of turns against sandboxed environments.
Their fix, ThunderAgent, is a scheduling layer that sits between the client and the inference backend and treats each multi-turn agent as a single trackable
My take
The clever part here isn't the speedup numbers, it's the realization that the wrong abstraction was baked into every popular inference engine from the start. Request-level scheduling made sense for chatbots; it was never going to survive contact with agents that pause for tools mid-thought. Open-sourcing this and getting it into SkyRL and NVIDIA Dynamo so fast is the right move, and it's a good sign for anyone worried that all the interesting infra work would stay locked inside closed labs.
Read more about this at: Together AI
