How to run TorchForge reinforcement learning pipelines in the Together AI Native Cloud

Reinforcement learning has always been the messier cousin of standard model training. Instead of one clean loop, you're juggling policy servers, reference models, replay buffers, reward functions, and environments that need to run code or call tools mid-training. Together AI is betting that a lot of teams are stuck wrestling with that plumbing instead of doing actual RL research, and its answer is to make the infrastructure disappear.

The company has gotten the full PyTorch RL stack — TorchForge and Monarch included — running natively on its Instant Clusters, with distributed training support out of the box. That means InfiniBand and NVLink setups tuned for the RDMA-heavy actor messaging these frameworks lean on, clusters that boot pre-loaded with NCCL, CUDA, and GPU operators so nobody's hand-configuring drivers at 2am, and scheduling that mixes GPU nodes for policy and training work with CPU nodes for environments and tool execution. That last part matters more than it sounds: RL pipelines are inherently heterogeneous, and treating everything as a GPU job wastes money and time.

The tool-use piece is arguably the more interesting bet. Together is pairing this with CodeSandbox, which spins up full microVMs for anything too complex or risky to run in a container, and Code Interpreter, a fast sandboxed Python runner meant for reward functions built on unit tests or code evaluation. Both plug into OpenEnv and TorchForge's environment services, so a rollout worker can hand off a coding task or a tool call mid-episode without leaving the pipeline.

To prove it's not just architecture diagrams, Together shipped a working demo adapted from a Meta reference implementation: a 1.5B-parameter Qwen model learning to play BlackJack via GRPO, with a vLLM policy server, a reference model, an off-policy replay buffer, and a TorchTitan trainer all wired together through Monarch's actor mesh, with TorchStore handling weight syncing. Spinning it up is three kubectl commands, and swapping in a different model or rewriting the reward function is a config-file edit. Together also released a standalone wrapper that treats Code Interpreter as an OpenEnv environment on its own, so the same BlackJack pipeline could just as easily be pointed at math problems or coding benchmarks — anything scoreable through executed code.

None of this is the RL service itself yet — Together is explicit that this is groundwork for something bigger being built jointly with Meta. But getting a multi-component RL job to deploy as easily as a single-GPU training script is a real milestone, not a marketing line, and it signals where the next layer of AI infrastructure competition is heading: not just who can serve inference cheapest, but who can make the messy business of training an agent to actually act in the world feel routine.