Powerful ASR + diarization + speculative decoding with Hugging Face Inference Endpoints

Whisper gets deployed constantly, but the moment you want more than plain transcription, things get annoying fast. Say you need to know who said what, or you want faster inference without burning more GPU. Suddenly you're stitching together multiple models behind one API, and that stitching is exactly what Hugging Face's new custom inference handler solves.

The setup splits cleanly into three files: handler.py for initialization and inference, diarization_utils.py for pre- and post-processing around speaker identification, and config.py for settings. The diarization piece borrows heavily from the Insanely Fast Whisper project and leans on a Pyannote model, currently the strongest open source option for figuring out who's talking. None of this is mandatory, though — ASR is the only required component. Diarization and an assistant model for speculative decoding are optional add-ons you toggle through environment variables.

Speculative decoding is the more interesting trick here. A smaller, faster model proposes transcriptions and the larger model checks them, which speeds things up, but only under specific conditions. The assistant model's decoder needs to match the main model's architecture, and batch size has to stay at 1. Hugging Face's own benchmarks on an A10 GPU make the tradeoff obvious: an 8-second clip transcribed with distil-whisper/distil-large-v3 assisting whisper-large-v3 dropped from roughly 784 milliseconds to about 327 milliseconds. But feed it a 60-second clip, which gets auto-chunked into batches, and assisted generation actually slows things down slightly, averaging 4.15 seconds versus 3.48.

Deployment still runs through Inference Endpoints, though getting the diarization token in safely requires calling the API directly instead of clicking through the web interface, since hardcoding auth tokens is off the table for obvious reasons. Everything else, including which models to load, gets configured through Pydantic settings and passed as environment variables at container build time. Once running, you send base64-encoded audio and a parameters dictionary, and the endpoint hands back transcription, speaker labels, or both, depending on what you configured.

Hugging Face is upfront that this is glue work built on existing open models rather than something novel from scratch. Whisper does the heavy lifting, Pyannote handles diarization, and the whole architecture takes cues from Insanely Fast Whisper. What's new is the packaging: a modular, single-endpoint pipeline you can adjust without rebuilding from zero.