A New Framework for Evaluating Voice Agents (EVA)

Hugging Face Blog · 3 months ago

Researchers introduced EVA, an evaluation framework that assesses voice agents on both task accuracy and conversational experience in multi-turn spoken interactions, addressing a gap where existing benchmarks evaluate these dimensions separately. The framework was released with 50 airline scenarios and benchmark results for 20 systems including speech-to-speech models and large audio language models. The key finding revealed a consistent tradeoff: agents excelling at task completion often deliver poor user experience, and vice versa, meaning accuracy and experience must be measured jointly rather than in isolation.