---
source: https://iinventedlight.com/library/notes/hf-tokenization-subword-algorithms.md
title: "Subword tokenization: BPE, byte-level BPE, Unigram, WordPiece"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/transformers, huggingface_hub, diffusers docs; LICENSE verified in each repo); prose restates cited source lines in own words"
license_family: hf-apache
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Tokenization algorithms, Apache-2.0 (huggingface docs repos); Hugging Face docs, Tokenizers, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: [hugging-face, transformers, tokenization, bpe]
client: Librarian
sources:
  - id: S1
    title: "Tokenization algorithms"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/tokenizer_summary
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Tokenizers"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/fast_tokenizers
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
---
# Subword tokenization: BPE, byte-level BPE, Unigram, WordPiece

**Answer:** Transformers supports three subword tokenization algorithms (BPE, Unigram, WordPiece); subwords keep the vocabulary compact while rare or unknown words decompose into known pieces.

## BPE

Byte pair encoding is the most popular algorithm, used by Llama, Gemma, and Qwen2. BPE starts with individual characters and iteratively merges the most frequent adjacent pair. It learns merge rules until it reaches the target vocabulary size.

## Byte-level BPE

Byte-level BPE uses 256 byte values as the base vocabulary, so every word can be tokenized without an unknown token. GPT-2 uses byte-level BPE with a vocabulary size of 50,257.

## Unigram

Unigram (T5, BigBird, Pegasus) starts with a large set of candidate subwords. It removes the tokens with the lowest loss increase, usually the bottom 10-20%.

## WordPiece

WordPiece (BERT-family models) merges pairs that maximize the likelihood of the training data instead of simply merging the pair that appears the most.

## Where it runs

Most tokenizers resolve to `TokenizersBackend`, a fast Rust-based tokenizer from the Tokenizers library.

## Artifacts

- [Topic packet: huggingface](https://iinventedlight.com/library/packets/huggingface.json)

## See also

- [hf-tokenizers-padding-truncation](https://iinventedlight.com/library/notes/hf-tokenizers-padding-truncation/)

## Claims

- id: C1
  claim: Transformers supports three subword algorithms.
  source: S1
  evidence: "Transformers support three subword tokenization algorithms: Byte pair encoding (BPE), Unigram, and WordPiece."

- id: C2
  claim: BPE merges the most frequent adjacent pair.
  source: S1
  evidence: "iteratively merges the most frequent adjacent pair."

- id: C3
  claim: Byte-level BPE uses 256 byte values as base.
  source: S1
  evidence: "Byte-level BPE uses 256 byte values as the base vocabulary instead"

- id: C4
  claim: WordPiece maximizes training-data likelihood.
  source: S1
  evidence: "WordPiece merges pairs that maximize the likelihood of the training data."

- id: C5
  claim: Most tokenizers use the Rust TokenizersBackend.
  source: S2
  evidence: "a fast Rust-based tokenizer from the Tokenizers library."

## Sources

- [S1] Tokenization algorithms. Hugging Face docs. https://huggingface.co/docs/transformers/tokenizer_summary . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

- [S2] Tokenizers. Hugging Face docs. https://huggingface.co/docs/transformers/fast_tokenizers . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).