Generative modeling with sparse transformers

OpenAI Blog · 7 years ago

Researchers developed the Sparse Transformer, a neural network that improves sequence prediction for text, images, and sound by modifying the attention mechanism. The model can process sequences 30 times longer than previous approaches allowed. This enables pattern recognition across extended data, reducing computational constraints that previously limited sequence modeling tasks.