Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

Generalist agents have been a nice idea on paper since DeepMind's Gato showed up in 2022, but nobody outside DeepMind could actually poke at the thing. Hugging Face just changed that with JAT, short for Jack of All Trades, an open reproduction that trains a single Transformer to handle Atari games, robotic manipulation in Meta-World, locomotion in MuJoCo, navigation instructions in BabyAI, and even basic vision-and-language tasks like image captioning.

The approach starts unglamorously: train a separate expert policy for each of 157 tasks until it hits state-of-the-art performance, then harvest hundreds of thousands of trajectories from those experts. That collection became the JAT dataset, which the team calls the first dataset built specifically for generalist agent training. It's not just robot logs either — Wikipedia, OSCAR, OK-VQA, and Conceptual Captions are stuffed in there too, so the same model that dodges asteroids can also answer a question about a picture.

Architecturally, JAT leans on EleutherAI's GPT-Neo, but the interesting part is how it stitches observations, actions, and rewards together as interleaved embeddings, decoding each with whatever encoder fits the data type: CNNs for images, linear layers for continuous vectors, projection layers for discrete values. A causal mask forces the model to predict the next action from everything it's seen so far, which is the same trick that makes GPT models write coherent paragraphs, just repurposed for deciding whether to jump over a Goomba or twist a robot wrist.

The numbers are a mixed bag, which is honestly refreghing given how much AI reporting rounds up. JAT hits 99% of expert performance on BabyAI, 84.8% on MuJoCo, 65.5% on Meta-World, but only 14.1% on Atari 57 — though that still translates to beating human players on 21 of the 57 games. Average that across domains and you get 65.8%, a number the researchers are upfront about rather than dressing up. On the language and vision side, they're blunt too: capabilities are rudimentary, full stop.

The more quietly fascinating result came from a side experiment on whether forcing the model to predict future observations, not just actions, helps or hurts. Crank that auxiliary objective too high and it drags performance down. Set it near zero and it does nothing. But tune the weighting parameter to around 0.005 and the agent actually learns faster, a small tuning knob with an outsized effect. The team is now asking the community to pile on with better data, offline RL instead of plain behavioral cloning, and smarter task sampling instead of uniform sampling across all 157 environments.