NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework

Agentic reinforcement learning research is basically an endless cycle of tweaking algorithms — new estimators, new pipeline stages, new ways of handling rollouts. The problem is that in most existing frameworks, every one of those tweaks has to thread through layers of trainer code, distributed backend logic, and rollout plumbing. Researchers pay that tax on every single iteration. NVIDIA's NeMo team built Molt specifically to cut that tax down.

The pitch is simplicity as a feature. Molt's RL code comes in at about 8,600 lines, measured by tracing the import graph from the framework's entry point. Compare that to verl at roughly 62,000 lines, slime at 25,000, and OpenRLHF at 7,200. NVIDIA's goal was a codebase small enough for a human researcher to hold in their head — and small enough for an AI coding assistant to read start to finish and actually reason about, rather than just autocomplete around the edges.

Under the hood, Molt stitches together three existing tools rather than reinventing them: Ray handles placement and async queues, vLLM runs the rollouts, and NVIDIA AutoModel with FSDP2 handles training. Crucially, none of these get forked, so when upstream ships an improvement, Molt just pins a new container version instead of undergoing a painful rebase. The runtime itself is a pool of vLLM engines sitting behind a request router, feeding a single trainable policy actor, with a streaming pool that keeps prompts flowing so the engines never sit idle while training happens.

Writing an agent for Molt is refreshingly boring, in a good way. You just point the framework at a Python module exporting an AgentRunner, and everything else — including the reward function — is ordinary code. You can let Molt own the LLM loop through a Gymnasium-style step() call, or keep control yourself using a standard OpenAI or Anthropic SDK, which Molt intercepts through a loopback server that decodes every request into an exact token accumulation. That detail matters a lot for correctness: three invariants govern the whole design, ensuring the framework trains only on tokens the model actually generated, preserves the right log-probabilities for async rollouts, and keeps rollout and training in agreement on model behavior — a particularly thorny issue for mixture-of-experts models, where Molt replays vLLM's expert-routing choices during training to avoid mismatches.

None of this is free, though. The shipped recipes assume two nodes of eight H100 GPUs apiece, split evenly between training and rollout. That puts Molt squarely in the hands of frontier labs, well-capitalized startups doing post-training, enterprise research teams in finance or healthcare or robotics, and academic groups lucky enough to have multi-node H100 or H200 clusters. It's released under Apache 2.0 with Slurm scripts and a prebuilt container, and NVIDIA is careful to frame it as research infrastructure rather than a production training service — a distinction that matters once you look at the hardware bill.