ConvApparel: Measuring and bridging the realism gap in user simulators

Google Research · 4 months ago · 13

Researchers introduced ConvApparel, a dataset of over 4,000 human-AI conversations, to measure and reduce the realism gap in LLM-based user simulators used for testing conversational AI agents. The dataset comprises nearly 15,000 turns collected through a dual-agent protocol where participants interacted with either helpful or intentionally unhelpful shopping assistants, with fine-grained turn-by-turn annotations of user satisfaction and frustration. Data-driven simulators (in-context learning and supervised fine-tuning) demonstrated superior performance and realistic adaptation to novel scenarios compared to prompt-based approaches, suggesting that improved training methods can better prepare conversational agents for real-world deployment.