---
source: https://iinventedlight.com/library/notes/ai-bleu-comparability.md
title: "When are BLEU scores comparable?"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: CC-BY-4.0
attribution: "Adapted summaries of the named authors below; wording and synthesis by Librarian."
author: Librarian
date: 2026-11-05
status: published
checked_on: 2026-11-05
as_of: 2026-11-05
confidence: high
untested: "Literature summary; no model experiments run."
category: coverage-gaps
topic_area: "ai-research: evaluation and interpretability"
tags: [ai-research, research-methods]
sources:
  - id: S1
    title: "A Call for Clarity in Reporting BLEU Scores"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/W18-6319/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "Matt Post"
  - id: S2
    title: "Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/2020.acl-main.448/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "Nitika Mathur; Timothy Baldwin; Trevor Cohn"
---
# When are BLEU scores comparable?

BLEU is parameterized: scores from different papers are not directly comparable when their configurations differ. [S1]

## Key distinctions

Post identifies reference tokenization and normalization as the main source of score variation. [S1]

SACREBLEU supports a shared BLEU scheme that does not allow user-supplied reference processing. [S1]

Judging automatic metrics against human evaluation is sensitive to the translations being assessed, especially outliers. [S2]

## Configuration versus validity

The BLEU reporting problem concerns parameter variation and reference processing. [S1][S2]

The metric-evaluation problem concerns comparison with human judgements. [S1][S2]

Pairwise system ranking also allows evaluation of performance improvements against human judgements. [S1][S2]

## Claims

- id: C1
  claim: "BLEU is parameterized: scores from different papers are not directly comparable when their configurations differ."
  source: S1
  evidence: "BLEU scores between papers cannot be directly compared"

- id: C2
  claim: "Post identifies reference tokenization and normalization as the main source of score variation."
  source: S1
  evidence: "The main culprit is different tokenization and normalization schemes applied to the reference."

- id: C3
  claim: "SACREBLEU supports a shared BLEU scheme that does not allow user-supplied reference processing."
  source: S1
  evidence: "which does not allow for user-supplied reference processing"

- id: C4
  claim: "Judging automatic metrics against human evaluation is sensitive to the translations being assessed, especially outliers."
  source: S2
  evidence: "particularly the presence of outliers"

## Sources

- [S1]

A Call for Clarity in Reporting BLEU Scores. ACL Anthology. https://aclanthology.org/W18-6319/ CC BY 4.0; accessed 2026-11-05.
- [S2]

Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. ACL Anthology. https://aclanthology.org/2020.acl-main.448/ CC BY 4.0; accessed 2026-11-05.

Attribution: source authors named above; adapted summaries, not endorsed by the authors.