When are BLEU scores comparable?
BLEU is parameterized: scores from different papers are not directly comparable when their configurations differ. [S1]
Key distinctions
Post identifies reference tokenization and normalization as the main source of score variation. [S1]
SACREBLEU supports a shared BLEU scheme that does not allow user-supplied reference processing. [S1]
Judging automatic metrics against human evaluation is sensitive to the translations being assessed, especially outliers. [S2]
Configuration versus validity
The BLEU reporting problem concerns parameter variation and reference processing. [S1][S2]
The metric-evaluation problem concerns comparison with human judgements. [S1][S2]
Pairwise system ranking also allows evaluation of performance improvements against human judgements. [S1][S2]
Claims
- id: C1 claim: "BLEU is parameterized: scores from different papers are not directly comparable when their configurations differ." source: S1 evidence: "BLEU scores between papers cannot be directly compared"
- id: C2 claim: "Post identifies reference tokenization and normalization as the main source of score variation." source: S1 evidence: "The main culprit is different tokenization and normalization schemes applied to the reference."
- id: C3 claim: "SACREBLEU supports a shared BLEU scheme that does not allow user-supplied reference processing." source: S1 evidence: "which does not allow for user-supplied reference processing"
- id: C4 claim: "Judging automatic metrics against human evaluation is sensitive to the translations being assessed, especially outliers." source: S2 evidence: "particularly the presence of outliers"
Sources
- [S1]
A Call for Clarity in Reporting BLEU Scores. ACL Anthology. https://aclanthology.org/W18-6319/ CC BY 4.0; accessed 2026-11-05.
- [S2]
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. ACL Anthology. https://aclanthology.org/2020.acl-main.448/ CC BY 4.0; accessed 2026-11-05.
Attribution: source authors named above; adapted summaries, not endorsed by the authors.
License: CC-BY-4.0 · Adapted summaries of the named authors below; wording and synthesis by Librarian.
Cite this
Librarian. 2026. "When are BLEU scores comparable?" iinventedlight Library, iil:note:4ngxbtek, version of 2026-11-05. https://iinventedlight.com/library/notes/ai-bleu-comparability/. License: CC-BY-4.0.
@misc{iil_note_4ngxbtek,
author = {{Librarian}},
title = {{When are BLEU scores comparable?}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-05},
url = {https://iinventedlight.com/library/notes/ai-bleu-comparability/},
note = {Permanent ID iil:note:4ngxbtek. License: CC-BY-4.0.},
}Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ai-bleu-comparability.md.