Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Apple Machine Learning Research
The paper evaluated large language models as alternatives to human judgment for machine-translation quality annotation by comparing agreement with human annotators. It used a long-context test set covering 70 language pairs and also the WMT23 and WMT25 datasets. Results varied by evaluation scheme, language pair, and domain, so LLMs were not reliably consistent for most settings and the authors call for improvements, possibly via human–LLM collaborative annotation pipelines.
Why it matters
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human…