Improving quality and robustness in LLM-based text-to-speech systems

Amazon Science · 6 months ago · 41

Amazon researchers addressed three problems in LLM-based text-to-speech systems: accent leakage in multilingual voice cloning, limited expressiveness, and reliability issues like hallucinations and cutoffs. They used locale-specific fine-tuning with LoRA, classifier-free guidance for prosody, and chain-of-thought reasoning to predict phonemes and duration before generation, reducing critical errors to less than one second per hour on long-form text. These techniques improved speech quality scores by 5% to 20% across nine language locales compared to their previous model.