---
title: 'Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY-NC-SA 4.0 (verified on arXiv abs page 2026-11-12); this pointer note is our own words and is CC BY 4.0; we link to the paper rather than reproduce it'
source: https://iinventedlight.com/library/notes/2210-09261-big-bench-hard.md
source_license: 'CC BY-NC-SA 4.0'
attribution: 'Pointer by iinventedlight (AI-assisted). Source: Mirac Suzgun, Nathan Scales, Nathanael Schärli, and 8 coauthors (listed on the arXiv abs page), Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (CC BY-NC-SA 4.0). This note links to the paper rather than reproducing it.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: 'Pointer note only. Paper not reproduced. Abs page and license checked 2026-11-12.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, evals, benchmarks, chain-of-thought
arxiv_id: 2210.09261
alias: 2210.09261-big-bench-hard
type: pointer
format: pointer
sources:
  - id: S1
    title: 'Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them'
    publisher: arXiv
    authors: 'Mirac Suzgun, Nathan Scales, Nathanael Schärli, and 8 coauthors (listed on the arXiv abs page)'
    url: https://arxiv.org/abs/2210.09261
    version_read: 'v1 (17 Oct 2022)'
    accessed: 2026-11-12
    license: 'CC BY-NC-SA 4.0'
---
# Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

## Summary
[Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://arxiv.org/abs/2210.09261) (2022) isolated the hardest tasks in BIG-bench and asked whether chain-of-thought prompting helps on them. The resulting slice, usually called BIG-Bench Hard or BBH, became the most used subset of the original suite, because overall BIG-bench averages had already started to look too easy.

What it introduced: a focused hard-task subset, plus an analysis of where step-by-step prompting still fails and where it helps, instead of treating the whole BIG-bench average as the headline number.

What it led to: BBH as a default challenging reasoning eval, often reported alongside MMLU or MATH. Live notes: [2206-04615-big-bench](https://iinventedlight.com/library/notes/2206-04615-big-bench/) and [2201-11903-chain-of-thought-prompting](https://iinventedlight.com/library/notes/2201-11903-chain-of-thought-prompting/). This week's drafts also include [2103-03874-math-dataset](https://iinventedlight.com/library/notes/2103-03874-math-dataset/).

Use it when you want the hard-slice paper rather than the full BIG-bench suite. Where to read it: https://arxiv.org/abs/2210.09261 (v1 (17 Oct 2022)).

The paper is posted under CC BY-NC-SA 4.0. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: CC BY-NC-SA 4.0; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
