Block-sparse GPU kernels
OpenAI Blog
Optimized GPU kernels for block-sparse neural networks have been released, enabling faster computation than standard libraries like cuBLAS and cuSPARSE. The kernels achieved state-of-the-art performance on text sentiment analysis and generative image and text modeling tasks. Networks using these kernels can execute orders of magnitude faster depending on their sparsity configuration.
Why it matters
We’re releasing highly-optimized GPU kernels for an underexplored class of neural network architectures: networks with block-sparse weights. Depending on the chosen sparsity, these kernels can run orders of magnitude faster than cuBLAS or cuSPARSE. We’ve used them to attain state-of-the-art results in text sentiment analysis and generative modeling of text and images.
