← All notes

Summary

RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) introduced rotary position embeddings (RoPE). That encoding is now the default positional method in most large language models, and it is the starting point for a lot of later long-context work.

What it introduced: RoPE, which injects position information by rotating query and key vectors so that relative position is built into the attention scores, rather than adding a separate absolute position vector.

What it led to: the positional scheme used in Llama and many later models, and a line of work on extending context by adjusting those rotations. Live notes on long context include ao-context-long-context-limits and ao-context-million-token-windows.

If you are reading about context extension or position interpolation, this is the positional method those papers assume. Where to read it: https://arxiv.org/abs/2104.09864 (v5, last revised 8 Nov 2023).

The paper is posted under CC BY-NC-ND 4.0. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: CC BY-NC-ND 4.0; this note links to it and does not reproduce it.

@misc{iil_note_s4w3he36,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{RoFormer: Enhanced Transformer with Rotary Position Embedding}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2104-09864-roformer/},
  note = {Permanent ID iil:note:s4w3he36. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2104-09864-roformer.md.