Image GPT

OpenAI's newest experiment takes the same transformer architecture that powers GPT-2 and points it at pixels instead of words. Feed it a sequence of pixel values the way you'd feed GPT-2 a sequence of tokens, and something strange happens: the model starts finishing images the way GPT-2 finishes sentences.

Show it half a photo and it will paint in the rest, guessing at shapes, colors, and textures based on nothing but statistical patterns learned from pixel sequences. No labels, no supervision, no hand-built rules about what an eye or a horizon looks like. Just a model trained to predict the next pixel, over and over, until it has absorbed something like visual intuition.

The more interesting claim buried in the release is about what this teaches the model, not just what it can draw. OpenAI found that the better a version of the model got at generating convincing image completions, the better its internal features became at a separate job: classifying images without ever being trained explicitly to do so. That's the generative-quality-tracks-classification-accuracy link the researchers highlight, and it's the kind of correlation that turns a fun demo into a research result.

What makes that notable is the comparison point. The features this pixel-predicting transformer learns on its own turn out to be competitive with those from leading convolutional networks, the architecture that has dominated computer vision for the better part of a decade, and it gets there in an unsupervised setting where nobody is telling the model what it's looking at.

It's a small, almost quiet paper about a big idea: the recipe that made large language models work, predict the next token and let scale do the rest, doesn't seem to care whether the tokens are words or pixels.