---
title: "Big Bird: Transformers for Longer Sequences"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is arXiv non-exclusive distribution license (verified on arXiv abs page 2026-11-12); this pointer note is our own words and is CC BY 4.0; we link to the paper rather than reproduce it'
source: https://iinventedlight.com/library/notes/2007-14062-big-bird.md
source_license: "arXiv non-exclusive distribution license"
attribution: 'Pointer by iinventedlight (AI-assisted). Source: Manzil Zaheer, Guru Guruganesh, Avinava Dubey, and 8 coauthors, Big Bird: Transformers for Longer Sequences (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: 'Pointer note only. Paper not reproduced. Abs page and license checked 2026-11-12.'
category: arxiv
topic_area: 'ai-research: memory-and-context'
tags: arxiv, long-context, sparse-attention, big-bird, transformers
arxiv_id: 2007.14062
alias: 2007-14062-big-bird
type: pointer
format: pointer
sources:
  - id: S1
    title: "Big Bird: Transformers for Longer Sequences"
    publisher: arXiv
    authors: "Manzil Zaheer and 10 coauthors (listed on the arXiv abs page)"
    url: https://arxiv.org/abs/2007.14062
    version_read: "v2 (8 Jan 2021)"
    accessed: 2026-11-12
    license: "arXiv non-exclusive distribution license"
---
# Big Bird: Transformers for Longer Sequences

## Summary
[Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062) (2020) introduces a sparse attention pattern that reduces the usual quadratic memory cost of Transformers to linear, so much longer sequences can fit on similar hardware.

What it introduced: a combination of local, global, and random attention, including a small number of global tokens (such as CLS) that attend over the whole sequence. The authors argue that this pattern remains a universal approximator of sequence functions and is Turing complete, so it keeps important properties of full attention. They report that the same hardware can handle sequences about eight times longer, with gains on question answering and summarization, and they also propose applications to genomics sequences.

What it led to: a standard sparse-attention design, usually cited together with Longformer, and a precursor to later long-context architectures. Live notes: [1706-03762-attention-is-all-you-need](https://iinventedlight.com/library/notes/1706-03762-attention-is-all-you-need/) and [1810-04805-bert](https://iinventedlight.com/library/notes/1810-04805-bert/). Longformer is a pointer note in this batch.

Where to read it: https://arxiv.org/abs/2007.14062 (v2, last revised 8 Jan 2021).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
