← All notes

Attention weights alone are not a settled explanation: the relationship between weights and model outputs needs testing. [S1]

Key distinctions

Jain and Wallace found very different attention distributions that produced equivalent predictions. [S1]

The alternative account makes explanation depend on its definition and on testing all elements of the model. [S2]

Their alternative tests include uniform weights, multiple random seeds, frozen pretrained weights, and end-to-end adversarial attention training. [S2]

Scope

The alternative tests concern attention in RNN models. [S2]

Its diagnostic framework tests whether attention supports meaningful interpretation in RNN models. [S2]

Claims

• id: C1 claim: "Attention weights alone are not a settled explanation: the relationship between weights and model outputs needs testing." source: S1 evidence: "it is unclear what relationship exists between attention weights and model outputs"

• id: C2 claim: "Jain and Wallace found very different attention distributions that produced equivalent predictions." source: S1 evidence: "very different attention distributions that nonetheless yield equivalent predictions"

• id: C3 claim: "The alternative account makes explanation depend on its definition and on testing all elements of the model." source: S2 evidence: "such a claim depends on one’s definition of explanation"

• id: C4 claim: "Their alternative tests include uniform weights, multiple random seeds, frozen pretrained weights, and end-to-end adversarial attention training." source: S2 evidence: "a simple uniform-weights baseline"

Sources

• [S1]

Attention is not Explanation. ACL Anthology. https://aclanthology.org/N19-1357/ CC BY 4.0; accessed 2026-11-05.

• [S2]

Attention is not not Explanation. ACL Anthology. https://aclanthology.org/D19-1002/ CC BY 4.0; accessed 2026-11-05.

Attribution: source authors named above; adapted summaries, not endorsed by the authors.

@misc{iil_note_z2vawvry,
  author = {{Librarian}},
  title = {{Does attention explain a prediction?}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-05},
  url = {https://iinventedlight.com/library/notes/ai-attention-explanation-tests/},
  note = {Permanent ID iil:note:z2vawvry. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ai-attention-explanation-tests.md.