---
source: https://iinventedlight.com/library/notes/ai-behavioral-testing-hans.md
title: "Why test NLP behavior beyond held-out accuracy?"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: CC-BY-4.0
attribution: "Adapted summaries of the named authors below; wording and synthesis by Librarian."
author: Librarian
date: 2026-11-05
status: published
checked_on: 2026-11-05
as_of: 2026-11-05
confidence: high
untested: "Literature summary; no model experiments run."
category: coverage-gaps
topic_area: "ai-research: evaluation and interpretability"
tags: [ai-research, research-methods]
sources:
  - id: S1
    title: "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/2020.acl-main.442/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "Marco Tulio Ribeiro; Tongshuang Wu; Carlos Guestrin; Sameer Singh"
  - id: S2
    title: "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference"
    publisher: "ACL Anthology"
    url: https://aclanthology.org/P19-1334/
    accessed: 2026-11-05
    license: CC-BY-4.0
    attribution: "R. Thomas McCoy; Ellie Pavlick; Tal Linzen"
---
# Why test NLP behavior beyond held-out accuracy?

Held-out accuracy can overestimate NLP generalization; CheckList offers task-agnostic behavioral testing. [S1]

## Key distinctions

CheckList organizes test ideation around linguistic capabilities and test types, rather than only a task score. [S1]

HANS tests natural language inference with examples where syntactic heuristics fail. [S2]

In the HANS study, models trained on MNLI, including BERT, performed very poorly on HANS. [S2]

## What each contributes

CheckList is a testing methodology across tasks. [S1][S2]

HANS is a controlled evaluation set for natural language inference, targeting lexical overlap, subsequence, and constituent heuristics. [S1][S2]

## Claims

- id: C1
  claim: "Held-out accuracy can overestimate NLP generalization; CheckList offers task-agnostic behavioral testing."
  source: S1
  evidence: "a task-agnostic methodology for testing NLP models"

- id: C2
  claim: "CheckList organizes test ideation around linguistic capabilities and test types, rather than only a task score."
  source: S1
  evidence: "a matrix of general linguistic capabilities and test types"

- id: C3
  claim: "HANS tests natural language inference with examples where syntactic heuristics fail."
  source: S2
  evidence: "a controlled evaluation set called HANS"

- id: C4
  claim: "In the HANS study, models trained on MNLI, including BERT, performed very poorly on HANS."
  source: S2
  evidence: "perform very poorly on HANS"

## Sources

- [S1]

Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL Anthology. https://aclanthology.org/2020.acl-main.442/ CC BY 4.0; accessed 2026-11-05.
- [S2]

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. ACL Anthology. https://aclanthology.org/P19-1334/ CC BY 4.0; accessed 2026-11-05.

Attribution: source authors named above; adapted summaries, not endorsed by the authors.