In the previous article, we saw how automatic metrics and LLM-as-a-judge approaches allow machine translation to be evaluated quickly and at scale. But when we need to know whether a system truly meets our needs—particularly before making important decisions, such as launching a product or switching translation provider—human evaluation remains the benchmark for assessing machine translation quality, just as it does in other natural language processing (NLP) tasks.
In this article we go a step further and focus on the main human evaluation methods: what each one measures, what advantages and limitations it presents, and what best practices should be followed to obtain reliable results.
1. Types of human evaluation
1.1. Ranking
Ranking methods do not measure the absolute quality of a translation, but rather its relative quality compared with others. The simplest form is pairwise evaluation: a source sentence is presented alongside two translations—one from a reference system and one from a new system—and the evaluator decides which they consider better, or whether both are of equivalent quality (Mukherjee, 2025).
Relative Ranking (RR) extends this idea: evaluators see the source text, a reference translation and five translations generated by different systems, and rank them from best to worst, allowing for ties. To simplify the task, identical translations produced by different systems are grouped into a single option.
There are also finer-grained variants. In constituent ranking, only a single, automatically selected syntactic constituent is ranked, rather than the whole sentence. In constituent judgement, by contrast, the evaluator decides with a yes or no whether that constituent is acceptable. The final score corresponds to the percentage of constituents deemed acceptable (Mukherjee, 2025).
Finally, the DARR (Direct Assessment Relative Ranking) method derives rankings from direct assessment scores. Obtaining reliable segment-level DA scores requires each translation to be evaluated by several people, which is not always feasible. DARR addresses this limitation by generating all possible pairs of translations of the same sentence and considering one better than the other only when their scores differ by at least 25 points. Pairs with a smaller difference are treated as ties and discarded from the ranking (Mukherjee, 2025).
The main advantage of ranking is that it is simple and inexpensive: comparing translations is usually easier than assigning an absolute score. Its main limitation is that it only allows preferences to be established. Knowing that one translation is better than another does not tell us whether either reaches a sufficient level of quality. It is therefore particularly well suited to comparing systems, but less so to measuring translation quality in absolute terms.
1.2. Error categorisation
Error categorisation methods seek not only to measure the quality of a translation, but also to explain where and why it fails. The reference framework is MQM (Multidimensional Quality Metrics), used in WMT since 2020 (Lommel et al., 2013; Lommel et al., 2024).
In MQM, expert annotators identify errors at segment level, classify them according to a taxonomy—which may include categories such as accuracy, fluency, style, terminology or locale conventions—and assign them a severity level: critical, major, minor or neutral. Each severity level carries an associated weight, and the weighted sum of errors yields a score for the segment or for the system. Categories can be added, removed or adapted to the needs of each task, and the method can be applied to both human and machine translation.
MQM offers a high level of detail and explainability, but also entails a higher evaluation cost. It requires specifically trained annotators, and classifying each error within a detailed taxonomy can be slow and cognitively demanding.
In response to this limitation, Error Span Annotation (ESA) emerged, adopted as the main human evaluation method in WMT in 2024 (Kocmi et al., 2024). ESA combines error identification with scoring in two steps.
First, the evaluator marks the spans of the translation that contain errors and indicates their severity—neutral, minor or major—without classifying them by type. They may also flag omitted content. Then, with the identified errors in view, they assign an overall score from 0 to 100 to the segment.
By requiring specific errors to be identified first, the final score is evidence-based and can be more consistent than a direct assessment. At the same time, by dispensing with the error taxonomy, the process is faster than MQM and reduces the training requirements for evaluators. ESA thus seeks a balance between the explainability of MQM and the efficiency of direct assessment.
1.3. Scoring
Scoring methods assign each translation a numerical value representing its absolute quality. One of the most classic approaches is the use of Likert scales (Joshi et al., 2015), in which evaluators score aspects such as fluency—linguistic correctness in the target language—and adequacy—faithfulness to the meaning of the source—typically on a scale of 1 to 5 or 1 to 7.
Direct Assessment (DA) estimates the absolute quality of a translation on a scale of 0 to 100, taking adequacy and fluency into account. Raw scores are subsequently converted into z-scores, normalised according to each evaluator's mean and standard deviation. This is intended to compensate for the fact that some people may systematically score more generously or more strictly than others.
There are three main variants. Ref-DA, monolingual, compares the translation with a reference; src-DA, bilingual, compares it directly with the source text, which reduces the bias that a particular reference might introduce; and contrastive DA, in which translations from several systems are presented simultaneously for comparative evaluation (Stanchev et al., 2020).
SQM (Scalar Quality Metric) uses a seven-point scale and evaluates each segment within the context of the document to which it belongs. Scores may come from non-expert crowdworkers (cSQM) or from professional translators (pSQM).
The DA+SQM variant combines both approaches: it retains the 0–100 scale of direct assessment, but incorporates seven labelled reference points that serve as guidance for the evaluation. This helps stabilise scores across evaluators compared with plain DA (Mukherjee et al., 2025).
Scoring is more costly and demanding than ranking, but offers a fundamental advantage: it allows quality to be quantified. Moreover, a ranking can be derived from a score, as DARR does, whereas deriving an absolute score from a ranking is far more difficult.
For this reason, Mukherjee et al. (2025) propose that, whenever resources allow, scoring should be preferred over ranking. The authors also present a decision tree that classifies methods according to their explainability, cognitive load and cost, helping to select the most suitable one based on the granularity of the evaluation, the type of assessment required, the availability of references and the profile of the evaluators.
2. Best practices in human evaluation
A poorly designed human evaluation can lead to misleading conclusions. Regardless of the method chosen, there are a number of factors that determine the reliability of the results and that must be taken into account throughout the process.
2.1. Inter-annotator agreement
Evaluations should be carried out with at least two evaluators. In general, involving more people produces more robust results. Inter-annotator agreement measures the extent to which evaluators agree when assessing the same translations. Low agreement may indicate that the task is ambiguous, that the guidelines are not sufficiently clear, or that the evaluators are not adequately trained—all of which call the reliability of the results into question.
This is measured using coefficients that account for agreement expected by chance, such as Cohen's kappa, for two people (Cohen, 1960); Fleiss' kappa, for more than two (Fleiss, 1971); or Krippendorff's alpha, which accommodates different data types, more than two annotators, and incomplete annotations (Krippendorff, 2011). For continuous scores, such as those from DA, correlation measures between evaluators are also used, such as Pearson's (Pearson, 1895) or Spearman's (Spearman, 1961) coefficients.
In error annotation tasks such as MQM and ESA, the situation is more complex, and there is no standardised method for measuring agreement. Each annotator may mark a different number of errors, the flagged spans may only partially overlap, and, in the case of MQM, the same problem may be classified under different but equally defensible categories. Early studies on MQM-based error annotation already found low agreement in both the detection and classification of specific errors (Lommel et al., 2014), and later work showed that many discrepancies are not necessarily due to annotator error, but to different, equally valid interpretations of the same phenomenon (Popović, 2021).
For this reason, the most advisable approach is to measure agreement at several levels, from strictest to most lenient. First, agreement on error detection can be measured—that is, whether evaluators agree on which segments contain errors (Castilho, 2021). Second, in the case of MQM, agreement on the categories assigned to the spans marked by each evaluator can be measured (Popović and Belz, 2022). Third, agreement on the segment-level score can be measured—that is, the score derived from errors in MQM or the 0–100 score in ESA—using correlations or Krippendorff's alpha for interval data (Freitag et al., 2021). Finally, agreement on segment-level ranking can be measured: given two translations of the same sentence, whether evaluators agree on which is better, which is worse, or whether it is a tie—something that can be quantified using Krippendorff's alpha (Song et al., 2025).
These levels provide complementary information. It is common for two evaluators to disagree on the exact spans they flag or the categories they assign, yet agree on the overall assessment or on the ranking of the translations—which is, ultimately, what matters for comparing systems. This was shown, for example, by Freitag et al. (2021) in their large-scale analysis of MQM annotation, and by Kocmi et al. (2024) in their comparison of ESA with MQM. Reporting agreement at multiple levels therefore allows for a better interpretation of the robustness of the results and helps identify where in the task discrepancies are concentrated.
2.2. Intra-annotator consistency
Intra-annotator consistency measures how consistent a single person is with themselves: if they evaluate the same translation twice, they should provide a similar assessment. To check this, repeated segments are usually included in the evaluation set without informing the evaluator in advance.
In DA, quality control is additionally based on trap items: deliberately degraded translations that should receive a clearly lower score than the original. Evaluators who fail these checks are discarded. Low intra-annotator consistency may indicate fatigue, lack of attention or unstable criteria, and is particularly relevant in evaluations carried out with non-expert crowdworkers.
2.3. Type of evaluator
The profile of the evaluators has a direct influence on the results. A first distinction is between monolingual evaluators, who know only the target language and compare the translation with a reference, and bilingual evaluators, who can compare it directly with the source text. Monolingual evaluation is more economical, but depends on the quality of the reference and may inherit its biases.
Another important distinction is between non-expert crowdworkers and professional translators or linguists. The former allow large volumes to be evaluated at low cost, but require a larger number of people per segment—in the case of DA, at least fifteen to ensure reproducibility—and strict quality controls. Professional evaluators are more costly, but can detect subtle errors that go unnoticed by non-expert evaluators, something increasingly relevant as translation systems improve and their errors become less obvious.
Methods such as MQM necessarily require expert annotators, whereas ESA was designed, among other things, to reduce this requirement.
2.4. Developing the guidelines
Annotation guidelines are essential to ensure that all evaluators interpret the task in the same way. Good guidelines should clearly define each criterion—for example, what is meant by adequacy, fluency or a major error—explain the meaning of each point on the scale, and include concrete examples, particularly for borderline cases.
It is advisable to run a pilot phase with a small dataset, analyse disagreements and revise the guidelines before beginning the full evaluation. It is also worth providing a training session and, where possible, a space to resolve queries during the process. Guidelines should be published alongside the results to facilitate reproducibility of the evaluation (Hovy and Lavid, 2010).
2.5. Amount of text to evaluate
The amount of text evaluated largely determines the statistical reliability of the results (Lommel et al., 2024). If too few segments are evaluated, observed differences between systems may be due to chance, so a sufficient volume is needed to apply significance tests. At the same time, too great a volume per person can cause fatigue and reduce annotation quality. It is therefore advisable to divide the work into sufficiently short blocks.
Another relevant aspect, as we saw in the first article in this series on evaluation datasets, is context. Evaluating isolated sentences is faster, but prevents the detection of errors that only become visible at document level (Castilho, 2021), such as terminological inconsistencies, poorly resolved pronouns or register shifts. For this reason, recent methods such as SQM, DA+SQM and MQM in WMT evaluate segments within the context of the document.
The right balance will depend on the resources available, the number of systems being compared, and the level of granularity sought in the evaluation.
3. Conclusions
As with the two previous articles in this series—and particularly the first, on datasets—reliable human evaluation requires careful design and well-grounded methodological decisions.
Firstly, it is necessary to determine what we want to measure, since the most suitable method will depend on this. If the aim is to compare systems with one another, a ranking method may suffice; if we need to quantify quality, we can turn to scoring methods such as DA or SQM; and if we want to understand where and why translations fail, error categorisation using MQM or ESA offers more detailed information.
Secondly, the available resources must be taken into account, particularly the profile of the evaluators. Working with professional translators is not the same as working with non-expert crowdworkers, nor is evaluating with a reference the same as evaluating without one. Each of these decisions affects both the cost and the type of errors that can be detected, as well as the reliability of the results.
Thirdly, the evaluation must be carefully prepared: clear guidelines, with examples and tested in a pilot phase; a sufficient volume of text for differences to be statistically significant, without causing fatigue; and, wherever possible, evaluation of segments within their context.
Finally, results are only reliable if their consistency can be demonstrated. This requires having several evaluators, measuring their agreement—ideally at different levels—and having sufficient data to draw meaningful conclusions.
Only when all these aspects are addressed can human evaluation fulfil its role as the gold standard and provide a solid basis for making decisions about machine translation systems.
BIBLIOGRAPHY
Lommel, Arle, et al. "The Multi-Range Theory of Translation Quality Measurement: MQM Scoring Models and Statistical Quality Control." Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), edited by Marianna Martindale et al., Association for Machine Translation in the Americas, 2024, pp. 75-94. ACL Anthology, aclanthology.org/2024.amta-presentations.6/.
Lommel, Arle Richard, et al. "Multidimensional Quality Metrics: A Flexible System for Assessing Translation Quality." Proceedings of Translating and the Computer 35, Aslib, 28-29 Nov. 2013. ACL Anthology.
Lommel, Arle, et al. "Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics." Revista tradumàtica: traducció i tecnologies de la informació i la comunicació, no. 12, 2014.
Kocmi, Tom, et al. "Error span annotation: A balanced approach for human evaluation of machine translation." Proceedings of the Ninth Conference on Machine Translation. 2024.
Mukherjee, Ananya, and Manish Shrivastava. "Lost in translation? Found in evaluation: A comprehensive survey on sentence-level translation evaluation." ACM Computing Surveys 58.1 (2025): 1-47.
Joshi, Ankur, et al. "Likert scale: Explored and explained." British journal of applied science & technology 7.4 (2015): 396.
Stanchev, Peter, Weiyue Wang, and Hermann Ney. "Towards a better evaluation of metrics for machine translation." Proceedings of the Fifth Conference on Machine Translation. 2020.
Cohen, Jacob. "A coefficient of agreement for nominal scales." Educational and psychological measurement 20.1 (1960): 37-46.
Fleiss, Joseph L. "Measuring nominal scale agreement among many raters." Psychological bulletin 76.5 (1971): 378.
Krippendorff, Klaus. "Computing Krippendorff's alpha-reliability." (2011).
Pearson, Karl. "Note on regression and inheritance in the case of two parents." Proceedings of the royal society of London 58 (1895): 240-242.
Spearman, Charles. "The proof and measurement of association between two things." (1961).
Popović, M. (2021). Agree to disagree: Analysis of inter-annotator disagreements in human evaluation of machine translation output. In Proceedings of CoNLL 2021.
Song, Yixiao, et al. "Enhancing human evaluation in machine translation with comparative judgement." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Popović, Maja, and Anja Belz. "On reporting scores and agreement for error annotation tasks." Proceedings of the Second Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). 2022.
Castilho, Sheila. "Towards document-level human MT evaluation: On the issues of annotator agreement, effort and misevaluation." Proceedings of the workshop on human evaluation of NLP systems (HumEval). 2021.
Freitag, Markus, et al. "Experts, errors, and context: A large-scale study of human evaluation for machine translation." Transactions of the Association for Computational Linguistics 9 (2021): 1460-1474.
Hovy, Eduard, and Julia Lavid. "Towards a ‘science’ of corpus annotation: a new methodological challenge for corpus linguistics." International journal of translation 22.1 (2010): 13-36.
