LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement

arXiv cs.AI · 18 hours ago

Researchers developed an audio-visual speech enhancement system that uses reinforcement learning guided by an LLM-based reward model, where an audio LLM generates natural language descriptions of enhanced speech that are converted to quality ratings for training. The method was evaluated on the AVSEC-4 dataset and outperformed supervised baselines and DNSMOS-based RL baselines across PESQ, STOI, and subjective listening test metrics. This approach replaces traditional scalar loss functions with semantically interpretable LLM-generated feedback to better optimize for perceived speech quality.