← All notes

Answer: Transformers supports three subword tokenization algorithms (BPE, Unigram, WordPiece); subwords keep the vocabulary compact while rare or unknown words decompose into known pieces.

BPE

Byte pair encoding is the most popular algorithm, used by Llama, Gemma, and Qwen2. BPE starts with individual characters and iteratively merges the most frequent adjacent pair. It learns merge rules until it reaches the target vocabulary size.

Byte-level BPE

Byte-level BPE uses 256 byte values as the base vocabulary, so every word can be tokenized without an unknown token. GPT-2 uses byte-level BPE with a vocabulary size of 50,257.

Unigram

Unigram (T5, BigBird, Pegasus) starts with a large set of candidate subwords. It removes the tokens with the lowest loss increase, usually the bottom 10-20%.

WordPiece

WordPiece (BERT-family models) merges pairs that maximize the likelihood of the training data instead of simply merging the pair that appears the most.

Where it runs

Most tokenizers resolve to TokenizersBackend, a fast Rust-based tokenizer from the Tokenizers library.

Artifacts

• Topic packet: huggingface

See also

• hf-tokenizers-padding-truncation

Claims

• id: C1 claim: Transformers supports three subword algorithms. source: S1 evidence: "Transformers support three subword tokenization algorithms: Byte pair encoding (BPE), Unigram, and WordPiece."

• id: C2 claim: BPE merges the most frequent adjacent pair. source: S1 evidence: "iteratively merges the most frequent adjacent pair."

• id: C3 claim: Byte-level BPE uses 256 byte values as base. source: S1 evidence: "Byte-level BPE uses 256 byte values as the base vocabulary instead"

• id: C4 claim: WordPiece maximizes training-data likelihood. source: S1 evidence: "WordPiece merges pairs that maximize the likelihood of the training data."

• id: C5 claim: Most tokenizers use the Rust TokenizersBackend. source: S2 evidence: "a fast Rust-based tokenizer from the Tokenizers library."

Sources

• [S1] Tokenization algorithms. Hugging Face docs. https://huggingface.co/docs/transformers/tokenizer_summary . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] Tokenizers. Hugging Face docs. https://huggingface.co/docs/transformers/fast_tokenizers . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_yek5qktw,
  author = {{iinventedlight}},
  title = {{Subword tokenization: BPE, byte-level BPE, Unigram, WordPiece}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-tokenization-subword-algorithms/},
  note = {Permanent ID iil:note:yek5qktw. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-tokenization-subword-algorithms.md.