Learning to play Minecraft with Video PreTraining

OpenAI · 4 years ago · 59

Researchers trained a neural network to play Minecraft by learning from unlabeled videos of human gameplay combined with a small labeled dataset, then fine-tuned it for specific tasks. The model learned to craft diamond tools—a goal requiring over 20 minutes of play and 24,000 actions from skilled humans—using only keyboard and mouse inputs. This approach demonstrates a method for teaching agents to perform complex computer tasks by leveraging large amounts of unlabeled video data rather than detailed task instructions.