In the previous article in this series, we saw that reliable evaluation of machine translation begins with building a good dataset: you need to define what you want to measure, choose the appropriate level of analysis, and make sure the data truly represents the context in which the system will be used.
Once we have this data, the next question is: how do we measure the quality of translations?
Today, human evaluation remains the benchmark for determining the quality of a translation, whether machine-produced or human-produced. However, it is a costly process, both in time and resources, and it is not always possible to have enough professionals, time, or technical means to evaluate large volumes of data. For this reason, in practice, automatic metrics play a fundamental role, especially when systems need to be evaluated at scale.
In this article, we review the main types of automatic metrics, their advantages and limitations, some best practices for avoiding mistaken conclusions, and the increasingly relevant role of LLMs as judges.
1. Types of metrics
Automatic metrics can be classified according to two main criteria: whether or not they require a human reference, and what type of information they use to compare translations, ranging from surface-level matching between texts to their semantic representation.
1.1. Reference-dependent and reference-independent metrics
Reference-based metrics: they compare the system's output with one or more human reference translations. This is the most common approach and includes metrics such as BLEU (Papineni et al., 2002), chrF (Popović, 2015), or COMET (Rei et al., 2020).
Their main weakness is obvious: they need a high-quality reference translation. As we saw in the previous article, building these references is a fundamental aspect of evaluation, and it is not always possible to have them available.
Reference-free metrics (also known as quality estimation): they estimate the quality of the translation by directly comparing the source text with the produced translation, without needing a human reference. These metrics are trained on data derived from human evaluations. Some examples are COMETKiwi (Rei et al., 2022) and MetricX-QE (Juraska et al., 2025).
This type of metric is especially useful when:
- no human references exist, for example, in highly specialised domains;
- evaluation is needed in production or in real time, with no references available;
- the goal is to detect low-quality translations before they go through human review—known as applied quality estimation—for example, in post-editing workflows.
Their main risk is that, since they lack a human reference as a point of comparison, they can be more sensitive to the model's own systematic errors, to biases present in the data they were trained on, or to a lack of resources for certain languages, especially low-resource ones.
1.2. Lexical metrics: from tokens and characters to edit distance
Lexical metrics compare the system's output and the reference at the level of textual surface form, without directly representing meaning. Within this group we find different approaches:
- Based on token (word) n-grams: these compare sequences of words between the translation and the reference. The best-known example is BLEU, which measures n-gram overlap. There are also variants such as METEOR (Banerjee et al., 2005), which adds matching via synonyms and word stems.
- Based on characters: metrics such as chrF work with character n-grams instead of words. This can make them more robust for morphologically rich languages or those with high inflectional variation, where small changes in a word's form—for example, in gender, number, or verb tense—could be unfairly penalised by a token-based metric.
- Based on distance measures: metrics such as TER (Translation Edit Rate) (Snover et al., 2006) calculate the minimum number of edits—insertions, deletions, substitutions, or reorderings—needed to convert the system's output into the reference. The fewer edits required, the better the score.
The common problem with these metrics is that they mainly measure surface-level matching: a correct translation phrased with different words than the reference may receive a low score, while a translation that shares the same terms may score well even if it has syntactic problems.
In addition, it has traditionally been difficult to compare BLEU results obtained across different studies, since small differences in implementation—especially in tokenisation—could cause significant variations in the final score. For this reason, it is recommended to use SacreBLEU (Post, 2018) to calculate BLEU, chrF, and TER, as it standardises calculation conditions and facilitates comparison between results.
1.3. Embedding-based metrics
To overcome some of the limitations of lexical metrics, metrics have emerged that aim to compare meaning rather than surface form, using vector representations (embeddings) generated by language models.
Well-known examples are BERTScore (Zhang et al., 2020), COMET, and BLEURT (Sellam et al., 2020). These metrics represent the translation and the reference—and, in some cases, also the source text—as vectors and calculate their similarity in that space.
Among other things, this allows:
- detecting correct paraphrases even when they don't share exactly the same words as the reference;
- being more sensitive to certain meaning-related errors that a purely lexical comparison might overlook;
- achieving, in certain contexts, a higher correlation with human judgements.
2. Advantages and limitations of automatic metrics
Automatic metrics have a clear and useful role in evaluation, but they do not on their own provide a complete picture of quality.
Advantages
- Speed: they allow thousands or even millions of segments to be evaluated in very little time, which would not be feasible through human evaluation.
- Reproducibility: given the same system and the same dataset, the result is consistent, which makes it easier to compare systems, versions, or configurations over time.
- Cost: they don't require mobilising human evaluators for each run, making it possible to evaluate large volumes of data with few resources.
Limitations
- Opacity: a score doesn't always reveal why a system obtained that result. This makes it harder to diagnose specific problems.
- Difficulty identifying specific errors: a metric may provide an overall score without indicating whether the problems relate to gender agreement, terminology, omissions, or other types of error. Thus, a good score can mask issues relevant to deployment. To partially address this limitation, metrics such as xCOMET (Guerreiro et al., 2024) have emerged in recent years, which, in addition to providing a score, can detect errors and classify them by severity.
- Sensitivity to reference quality: in reference-based metrics, a poor reference can directly distort the result.
- Imperfect correlation with human judgements: even the most advanced metrics do not fully replace human evaluation, especially when factors such as stylistic nuance, cultural appropriateness, or long-range discourse coherence come into play.
In short, automatic metrics are especially useful for tracking progress, comparing systems, and detecting potential problems at scale, but they should not be seen as a definitive substitute for human judgement when making high-impact decisions.
3. Best practices in automatic evaluation
It is not enough to choose a metric and calculate a score. How the evaluation is designed and interpreted also determines the validity of the results.
In 2021, Marie et al. analysed 769 scientific papers on machine translation evaluation published between 2010 and 2020. The study identified several recurring shortcomings in the methodologies used and the conclusions drawn, including exclusive reliance on BLEU, the absence of statistical significance testing, and comparisons with results from earlier work obtained under different conditions.
Based on these issues, several recommendations stand out:
- Don't rely exclusively on BLEU or a single metric. It is advisable to combine different metrics or supplement them with human evaluation, especially when solid conclusions about a system's quality are needed.
- Run statistical significance tests. When comparing two systems, a difference in score does not necessarily mean that one is significantly better. Statistical tests make it possible to determine whether the observed difference could be due to chance.
- Don't directly compare scores taken from other studies. Results obtained in earlier work should only be compared when the calculation conditions are equivalent and the information needed to reproduce them is available.
- Compare systems under equal conditions. When attributing a difference in results to the systems or methods themselves, it is necessary to control for factors such as training, validation, and test data, and the preprocessing used.
These practices—variety of metrics, statistical rigour, and comparison under equivalent conditions—are essential for correctly interpreting the results of an automatic evaluation.
4. LLMs as judges
One of the most recent trends in evaluation is the use of large language models (LLMs) to evaluate translation quality, alongside traditional metrics. This approach, commonly known as LLM-as-a-judge, is gaining traction across different natural language processing tasks.
What options are there?
- Generic LLMs used directly as judges, through prompts instructing them to evaluate specific aspects of the translation, such as adequacy, fluency, or error severity, with no additional training required (Kocmi and Federmann, 2023a; Kocmi and Federmann, 2023b; Fu et al., 2024).
- Models fine-tuned specifically for evaluation, trained on human judgements to predict scores close to those of expert evaluators (Treviso et al., 2024).
- Agentic approaches, which combine the outputs of several LLMs, either because each one evaluates a specific aspect of the translation—such as adequacy, fluency, style, or terminology—(Feng et al., 2025), or because the different outputs are used contrastively to identify those that provide the most information or best capture certain nuances (Wang et al., 2025).
Advantages
- They can generate explanations of errors, not just a score, which can make them useful for diagnosis.
- They can help identify certain critical errors that traditional metrics don't easily detect.
Limitations
- Cost and latency: they are slower and more expensive to run than many traditional metrics, especially at scale.
- Reproducibility: results can vary depending on the model, the prompt, or even the order in which the translations are presented (Ye et al., 2025; Lin et al., 2025).
- Model biases: an LLM may show systematic preferences that don't necessarily correspond to higher translation quality.
- Correlation with human judgements: although they may show good correlation at the system level, behaviour can differ at the segment level, meaning aggregated results may obscure relevant differences (Gain et al., 2026).
LLMs as judges thus represent a promising tool for expanding and speeding up evaluation. But, like traditional automatic metrics, they do not eliminate the need for human evaluation in contexts where decisions have a high impact.
5. Conclusion
Automatic evaluation is essential when machine translation needs to be evaluated at scale, but obtaining a score is not enough.
A solid evaluation requires combining metrics of different kinds, using representative datasets, applying appropriate statistical methods, and understanding the limitations of each tool. New LLM-based approaches expand the possibilities for analysis, but they also introduce new challenges that must be taken into account.
Ultimately, evaluating is not simply a matter of obtaining a score, but of understanding what that score means and what information is needed to make a decision. This perspective will be especially important when comparing different machine translation systems and models and deciding which one best fits each use case.
BIBLIOGRAPHY
Papineni, Kishore, et al. "Bleu: a method for automatic evaluation of machine translation." Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002.
Popović, Maja. "chrF: character n-gram F-score for automatic MT evaluation." Proceedings of the tenth workshop on statistical machine translation. 2015.
Rei, Ricardo, et al. "COMET: A neural framework for MT evaluation." Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp). 2020.
Rei, Ricardo, et al. "COMET-22: Unbabel-IST 2022 submission for the metrics shared task." Proceedings of the Seventh Conference on Machine Translation (WMT). 2022.
Juraska, Juraj, et al. "Metricx-25 and gemspaneval: Google translate submissions to the wmt25 evaluation shared task." Proceedings of the Tenth Conference on Machine Translation. 2025.
Banerjee, Satanjeev, and Alon Lavie. "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments." Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 2005.
Snover, Matthew, et al. "A study of translation edit rate with targeted human annotation." Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers. 2006.
Post, Matt. "A call for clarity in reporting BLEU scores." Proceedings of the third conference on machine translation: Research papers. 2018.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert.
Sellam, Thibault, Dipanjan Das, and Ankur Parikh. "BLEURT: Learning robust metrics for text generation." Proceedings of the 58th annual meeting of the association for computational linguistics. 2020.
Marie, Benjamin, Atsushi Fujita, and Raphael Rubino. "Scientific credibility of machine translation research: A meta-evaluation of 769 papers." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Guerreiro, Nuno M., Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. "xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection." Transactions of the Association for Computational Linguistics 12. 2024. 979–995.
Kocmi, Tom, and Christian Federmann. "Large language models are state-of-the-art evaluators of translation quality." Proceedings of the 24th Annual Conference of the European Association for Machine Translation. 2023a.
Kocmi, Tom, and Christian Federmann. "GEMBA-MQM: Detecting translation quality error spans with GPT-4." Proceedings of the Eighth Conference on Machine Translation. 2023b.
Treviso, Marcos V., et al. "xtower: A multilingual llm for explaining and correcting translation errors." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
Fu, Jinlan, et al. "Gptscore: Evaluate as you desire." Proceedings of the 2024 conference of the north american chapter of the association for computational linguistics: Human language technologies (volume 1: Long papers). 2024.
Feng, Zhaopeng, et al. "M-MAD: Multidimensional multi-agent debate for advanced machine translation evaluation." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Wang, Xiao, et al. "ContrastScore: Towards higher quality, less biased, more efficient evaluation metrics with contrastive evaluation." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 2025.
Shi, Lin, et al. "Judging the judges: A systematic study of position bias in llm-as-a-judge." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 2025.
Gain, Baban, et al. "Bridging the linguistic divide: a survey on leveraging large language models for machine translation." Language Resources and Evaluation 60.2 (2026): 40.
