Dreaming in Voxels: How AI is Generating Playable Minecraft Worlds

Sakana AI has a habit of poking at generative AI from odd angles, and its newest project, built with New York University, points the technique at something nobody quite expects: Minecraft. The idea is called Dream-Cubed, and the pitch is straightforward once you hear it. Language models got good because words and tokens are natural, discrete units you can predict one after another. Minecraft, it turns out, offers the same kind of structure, just made of cubes instead of words. So the team trained large transformers directly on billions of blocks pulled from the game, treating each cube the way a language model treats a token.

That framing matters because generative AI has mostly stayed in flat territory. Text, images, video, all of it renders eventually as a 2D output or a linear sequence. Fully structured, playable 3D worlds have been the harder problem, partly because there was no obvious unit to tokenize. Cubes solve that awkwardness for voxel-based games specifically, giving researchers something as clean as a word or a pixel to build a training objective around.

The results, according to Sakana, go beyond just spitting out static block layouts. The models were trained with both continuous and discrete diffusion objectives, which opened up a handful of practical tricks: filling in missing chunks of a world with targeted inpainting, extending a map outward through large-scale outpainting, and generating new terrain that responds to user input at the level of individual blocks. Put together, that's a system that can build worlds of effectively unlimited size while still letting a person steer what gets placed and where.

What's notable here isn't the graphics or the gameplay, since Minecraft's blocky aesthetic was never going to wow anyone visually. It's the demonstration that the token-based recipe behind large language models generalizes to a domain most people wouldn't think of as text-like at all. If cubes can act as tokens for a transformer, the same logic presumably extends to any environment built from discrete, game-engine-style primitives, not just the one Mojang shipped in 2011.