The Data Pyramid in Robotics

LLMs got lucky. Humanity had spent decades typing text onto the web, and that pile became free pre-training fuel for GPT and everything after it. Robotics has no such gift. Open X-Embodiment, the biggest open robot dataset around, is a patchwork of 60 datasets from dozens of labs, and it still only adds up to about a million trajectories across 22 robot types. Sergey Levine of Physical Intelligence put the gap in blunt terms on the Dwarkesh podcast: robot data trails multimodal training data by one to two orders of magnitude, and that's before you account for how sparse robot footage actually is. Thirty frames a second of a gripper inching toward a mug carries a lot less signal than thirty tokens of text.

So the field has stopped waiting for an internet-scale miracle and started building a pyramid instead. Nvidia's GR00T team frames it as web data at the bottom, synthetic data in the middle, real robot data at the top — volume shrinking as fidelity climbs. Jaipuria breaks that into seven concrete buckets: raw YouTube footage, egocentric video from head cameras, richer human demos with motion capture and tactile sensors, UMI-style capture where humans wear robot end-effectors, simulation and emerging world models like Nvidia's Cosmos or Google's Genie 3, expensive teleoperation, and finally real deployment data from robots doing paid work in the field.

Each layer solves a different piece of the puzzle and comes with its own catch. YouTube gives models a sense of what objects and actions look like, but it's unlabeled and mostly enters training indirectly through vision-language backbones, though startups like Rhoda AI and projects like DreamZero are trying to squeeze direct policy signal out of it. Teleoperation gives you gold-standard, embodiment-specific data, but it costs over $100 an hour and needs skilled operators, which caps how fast anyone can scale it. Simulation is theoretically unlimited since you're only bound by compute, but the sim-to-real gap is real, and physics engines still struggle with things like deformable objects.

Sunday Robotics, founded by the researchers behind the UMI method, is betting on a middle path: they shipped skill-capture gloves into people's homes and say their ACT-1 model trained on roughly 10 million household episodes without touching a single robot. Toyota Research Institute mixes UMI capture with teleop and simulation for its Large Behavior Models. Meanwhile Physical Intelligence's π0 reportedly trained on about 10,000 hours of robot data, and closing Levine's orders-of-magnitude gap would mean something closer to a million hours — a number nobody has actually assembled yet.

The honest answer to how much data robots eventually need is that nobody has it. Levine said as much when asked whether robots need 100x or 1,000x more data than today: "we don't know that." What's changing is that progress is arriving on every layer at once — better ways to mine human video, richer tactile and motion-capture rigs, world models chipping away at sim-to-real, and the first real deployment flywheels, where robots doing actual paid work with a human backstop start generating the exact data that improves them. That last piece, deployment-driven data with human corrections when the robot fails, borrows straight from the old DAgger idea in imitation learning, and it may end up mattering more than any single dataset on this pyramid.