Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models
Apple Machine Learning Research
A lightweight method detects code-switching directly from speech by using language-selective hidden-state dimensions in pretrained speech encoders. Whisper-small reached a mean F1 of 0.798 across 14 language pairs on CS-FLEURS after tuning a decision rule with a small CS set. As a result, pretrained speech encoders can be used to label CS utterances without multilingual transcription, helping build more representative multilingual training corpora.
Why it matters
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly from speech. It uses frame-level language information in pretrained speech…