Boosting Wav2Vec2 with n-grams in 🤗 Transformers

Hugging Face Blog · 4 years ago

The Hugging Face Transformers library now supports integrating n-gram language models with Wav2Vec2 speech recognition through the pyctcdecode library. The integration allows users to combine a fine-tuned Wav2Vec2 checkpoint with a language model using the Wav2Vec2ProcessorWithLM class, which applies beam search across a matrix of character probabilities weighted by language model predictions. This approach significantly reduces spelling errors in transcriptions: a test case corrected "christmaus" to "christmas" and "simalyis" to "similes," though ambiguities between valid words like "rose" versus "roast" persist based on acoustic probability weighting.