← All notes

Big Bird: Transformers for Longer Sequences

Summary

Big Bird: Transformers for Longer Sequences (2020) introduces a sparse attention pattern that reduces the usual quadratic memory cost of Transformers to linear, so much longer sequences can fit on similar hardware.

What it introduced: a combination of local, global, and random attention, including a small number of global tokens (such as CLS) that attend over the whole sequence. The authors argue that this pattern remains a universal approximator of sequence functions and is Turing complete, so it keeps important properties of full attention. They report that the same hardware can handle sequences about eight times longer, with gains on question answering and summarization, and they also propose applications to genomics sequences.

What it led to: a standard sparse-attention design, usually cited together with Longformer, and a precursor to later long-context architectures. Live notes: 1706-03762-attention-is-all-you-need and 1810-04805-bert. Longformer is a pointer note in this batch.

Where to read it: https://arxiv.org/abs/2007.14062 (v2, last revised 8 Jan 2021).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Manzil Zaheer, Guru Guruganesh, Avinava Dubey, and 8 coauthors, Big Bird: Transformers for Longer Sequences (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:kfxx7dd8

'iinventedlight (AI-assisted)'. 2026. "Big Bird: Transformers for Longer Sequences." iinventedlight Library, iil:note:kfxx7dd8, version of 2026-11-12. https://iinventedlight.com/library/notes/2007-14062-big-bird/. License: CC-BY-4.0.

@misc{iil_note_kfxx7dd8,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Big Bird: Transformers for Longer Sequences}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2007-14062-big-bird/},
  note = {Permanent ID iil:note:kfxx7dd8. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2007-14062-big-bird.md.