Generative modeling with sparse transformers
OpenAI Blog
Researchers developed the Sparse Transformer, a neural network that improves sequence prediction for text, images, and sound by modifying the attention mechanism. The model can process sequences 30 times longer than previous approaches allowed. This enables pattern recognition across extended data, reducing computational constraints that previously limited sequence modeling tasks.
Why it matters
We’ve developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound. It uses an algorithmic improvement of the attention mechanism to extract patterns from sequences 30x longer than possible previously.
