---
source: https://iinventedlight.com/library/notes/ai-attention-explanation-tests.md
title: "Does attention explain a prediction?"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: CC-BY-4.0
attribution: "Adapted summaries of the named authors below; wording and synthesis by Librarian."
author: Librarian
date: 2026-11-05
status: published
checked_on: 2026-11-05
as_of: 2026-11-05
confidence: high
untested: "Literature summary; no model experiments run."
category: coverage-gaps
topic_area: "ai-research: evaluation and interpretability"
tags: [ai-research, research-methods]
sources:
  - id: S1
    title: "Attention is not Explanation"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/N19-1357/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "Sarthak Jain; Byron C. Wallace"
  - id: S2
    title: "Attention is not not Explanation"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/D19-1002/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "Sarah Wiegreffe; Yuval Pinter"
---
# Does attention explain a prediction?

Attention weights alone are not a settled explanation: the relationship between weights and model outputs needs testing. [S1]

## Key distinctions

Jain and Wallace found very different attention distributions that produced equivalent predictions. [S1]

The alternative account makes explanation depend on its definition and on testing all elements of the model. [S2]

Their alternative tests include uniform weights, multiple random seeds, frozen pretrained weights, and end-to-end adversarial attention training. [S2]

## Scope

The alternative tests concern attention in RNN models. [S2]

Its diagnostic framework tests whether attention supports meaningful interpretation in RNN models. [S2]

## Claims

- id: C1
  claim: "Attention weights alone are not a settled explanation: the relationship between weights and model outputs needs testing."
  source: S1
  evidence: "it is unclear what relationship exists between attention weights and model outputs"

- id: C2
  claim: "Jain and Wallace found very different attention distributions that produced equivalent predictions."
  source: S1
  evidence: "very different attention distributions that nonetheless yield equivalent predictions"

- id: C3
  claim: "The alternative account makes explanation depend on its definition and on testing all elements of the model."
  source: S2
  evidence: "such a claim depends on one’s definition of explanation"

- id: C4
  claim: "Their alternative tests include uniform weights, multiple random seeds, frozen pretrained weights, and end-to-end adversarial attention training."
  source: S2
  evidence: "a simple uniform-weights baseline"

## Sources

- [S1]

Attention is not Explanation. ACL Anthology. https://aclanthology.org/N19-1357/ CC BY 4.0; accessed 2026-11-05.
- [S2]

Attention is not not Explanation. ACL Anthology. https://aclanthology.org/D19-1002/ CC BY 4.0; accessed 2026-11-05.

Attribution: source authors named above; adapted summaries, not endorsed by the authors.