← All notes

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Summary

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022) isolated the hardest tasks in BIG-bench and asked whether chain-of-thought prompting helps on them. The resulting slice, usually called BIG-Bench Hard or BBH, became the most used subset of the original suite, because overall BIG-bench averages had already started to look too easy.

What it introduced: a focused hard-task subset, plus an analysis of where step-by-step prompting still fails and where it helps, instead of treating the whole BIG-bench average as the headline number.

What it led to: BBH as a default challenging reasoning eval, often reported alongside MMLU or MATH. Live notes: 2206-04615-big-bench and 2201-11903-chain-of-thought-prompting. This week's drafts also include 2103-03874-math-dataset.

Use it when you want the hard-slice paper rather than the full BIG-bench suite. Where to read it: https://arxiv.org/abs/2210.09261 (v1 (17 Oct 2022)).

The paper is posted under CC BY-NC-SA 4.0. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: CC BY-NC-SA 4.0; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Mirac Suzgun, Nathan Scales, Nathanael Schärli, and 8 coauthors (listed on the arXiv abs page), Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (CC BY-NC-SA 4.0). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:f8e813xq

'iinventedlight (AI-assisted)'. 2026. "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them." iinventedlight Library, iil:note:f8e813xq, version of 2026-11-12. https://iinventedlight.com/library/notes/2210-09261-big-bench-hard/. License: CC-BY-4.0.

@misc{iil_note_f8e813xq,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2210-09261-big-bench-hard/},
  note = {Permanent ID iil:note:f8e813xq. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2210-09261-big-bench-hard.md.