NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

NVIDIA’s latest Cosmos3-H-Dreams model takes a striking step from tabletop robotics into clinical surgery. Instead of learning only from simple lab setups, it is trained on video and robot kinematics recorded during cholecystectomy on the CMR Surgical Versius platform, and the company says it can now generate interactive surgical scenes in real time.

The pitch is simple enough, even if the machinery behind it is not. Given an initial frame and a stream of tool actions, the model has to keep anatomy, instruments, reflections, smoke, and tissue deformation coherent as it rolls forward. A plausible-looking clip is not enough; the scene has to respond to translation, rotation, and gripper commands without drifting off into nonsense. That’s the hard part, and it’s why NVIDIA frames visual realism as only the starting point.

Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. It uses curated cholecystectomy data, with sample selection focused on two visible instruments: a left Cadiere grasper and a right monopolar hook. The action space is broad, with 28 dimensions covering pose and gripper signals for both arms, plus energy controls and thumbstick inputs. Training uses 832 x 480 video aligned with actions at 10 Hz.

The system also leans on a teacher-to-student setup meant to make long rollouts workable. NVIDIA first adapts a bidirectional teacher on 24 future frames, then extends that to 72 frames, or 7.2 seconds at 10 Hz. From there, distillation turns it into a causal student that can keep generating as new actions arrive. The first released student uses four denoising steps per block, and a second checkpoint is trained with a two-step schedule for faster inference at lower visual quality.

FlashDreams provides the streaming inference layer, while an adapted FadeMem module helps manage the growing key/value cache during long generations. NVIDIA says the whole thing runs interactively on a single NVIDIA RTX PRO 6000 GPU. It also says plainly that this is research software, not a validated system for patient care. That matters because the visuals are impressive, but the company is still selling a simulator, not a surgeon.