Learning to play Minecraft with Video PreTraining
OpenAI Blog
Researchers trained a neural network to play Minecraft by learning from unlabeled videos of human gameplay combined with a small labeled dataset, then fine-tuned it for specific tasks. The model learned to craft diamond tools—a goal requiring over 20 minutes of play and 24,000 actions from skilled humans—using only keyboard and mouse inputs. This approach demonstrates a method for teaching agents to perform complex computer tasks by leveraging large amounts of unlabeled video data rather than detailed task instructions.
Why it matters
We trained a neural network to play Minecraft by Video PreTraining (VPT) on a massive unlabeled video dataset of human Minecraft play, while using only a small amount of labeled contractor data. With fine-tuning, our model can learn to craft diamond tools, a task that usually takes proficient humans over 20 minutes (24,000 actions). Our model uses the native human interface of keypresses and mouse movements, making it quite general, and represents a step towards general computer-using agents.
