Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Apple Machine Learning Research · 2 days ago · 21

The paper evaluated large language models as alternatives to human judgment for machine-translation quality annotation by comparing agreement with human annotators. It used a long-context test set covering 70 language pairs and also the WMT23 and WMT25 datasets. Results varied by evaluation scheme, language pair, and domain, so LLMs were not reliably consistent for most settings and the authors call for improvements, possibly via human–LLM collaborative annotation pipelines.