← All notes

Summary

Grouped-query attention (GQA) is a generalization of multi-query attention (MQA). MQA uses a single key-value head and can speed up decoder inference, but it can reduce quality. GQA uses an intermediate number of key-value heads: more than one, but fewer than the number of query heads. The authors also describe a recipe for uptraining existing multi-head checkpoints into MQA or GQA models using 5% of original pre-training compute. They report that uptrained GQA achieves quality close to multi-head attention with speed comparable to MQA. [S1]

Claims

• id: C1 claim: 'GQA is a generalization of multi-query attention that uses an intermediate number of key-value heads.' source: S1 evidence: "a generalization of multi-query attention"
• id: C2 claim: 'They propose uptraining existing multi-head checkpoints using 5% of original pre-training compute.' source: S1 evidence: "using 5% of original pre-training compute"
• id: C3 claim: 'They report that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.' source: S1 evidence: "uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA"

Verification

• Code available: The authors say they implemented experiments with JAX, Flax, and Flaxformer (https://github.com/google/flaxformer, HTTP 200 on 2026-11-11). No separate GQA repository was named as this paper's release. Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes 2309-06180-pagedattention-vllm and 2401-04088-mixtral-of-experts. This is the paper-verification note for GQA itself.
• Limits: Quality and speed claims are from uptrained T5-scale experiments in the paper, not a guarantee for every later decoder.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 22 May 2023 (v1); last revised 23 Dec 2023 (v3). Sources: [S1] https://arxiv.org/abs/2305.13245, v3 (23 Dec 2023). checked_on: 2026-11-11.

When to apply

Use when reducing key-value cache cost at inference without fully collapsing to a single MQA head, or when converting an existing multi-head checkpoint.

Facts

• id: F1 claim: 'GQA uses an intermediate number of key-value heads, between one (MQA) and the full number of query heads.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors describe uptraining existing multi-head checkpoints with 5% of original pre-training compute.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They report that uptrained GQA is close in quality to multi-head attention and comparable in speed to MQA.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2305.13245, https://arxiv.org/abs/2305.13245. License: CC BY 4.0. v3 (23 Dec 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_wkhr5r39,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2305-13245-gqa/},
  note = {Permanent ID iil:note:wkhr5r39. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2305-13245-gqa.md.