---
source: https://iinventedlight.com/library/notes/hf-tokenizers-padding-truncation.md
title: "Tokenizer padding, truncation, and special tokens"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/transformers, huggingface_hub, diffusers docs; LICENSE verified in each repo); prose restates cited source lines in own words"
license_family: hf-apache
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Tokenizers, Apache-2.0 (huggingface docs repos); Hugging Face docs, Text generation, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: [hugging-face, transformers, tokenizer, padding]
client: Librarian
sources:
  - id: S1
    title: "Tokenizers"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/fast_tokenizers
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Text generation"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/llm_tutorial
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
---
# Tokenizer padding, truncation, and special tokens

**Answer:** A tokenizer converts text into tensors; to batch varying-length sequences, use padding (shorter sequences gain pad tokens) and truncation (long sequences lose trailing tokens).

## Call, encode, decode

- Calling the tokenizer encodes text into `input_ids`, `attention_mask`, and other model inputs, and controls padding, truncation, and special tokens.
- `encode` only returns the `input_ids`.
- `decode` converts tokenized `input_ids` back to text; `skip_special_tokens` strips special tokens from the output.

## Padding

In the attention mask, padding positions are `0`, so the model ignores them. `padding=True` pads to the longest sequence; `max_length` pads to a fixed size.

## Truncation

Set `truncation=True` and specify `max_length` to clip tokens so a sequence fits within a maximum length.

## Left padding for generation

Large language models pad on the left side because generation predicts the next token from the right side, and LLMs aren't trained to continue generation from padding tokens.

## Artifacts

- [Topic packet: huggingface](https://iinventedlight.com/library/packets/huggingface.json)

## See also

- [hf-transformers-auto-classes](https://iinventedlight.com/library/notes/hf-transformers-auto-classes/)
- [hf-tokenization-subword-algorithms](https://iinventedlight.com/library/notes/hf-tokenization-subword-algorithms/)
- [hf-chat-templates](https://iinventedlight.com/library/notes/hf-chat-templates/)

## Claims

- id: C1
  claim: Padding and truncation make a rectangular batch.
  source: S1
  evidence: "Together, they produce a packed rectangular tensor."

- id: C2
  claim: Padding and truncation handle varying-length sequences.
  source: S1
  evidence: "Padding and truncation are strategies to handle varying-length sequences."

- id: C3
  claim: LLMs pad on the left for generation.
  source: S1
  evidence: "Large language models pad on the *left* side to avoid disrupting generation"

- id: C4
  claim: Set padding_side to left for LLM batches.
  source: S2
  evidence: "LLMs aren't trained to continue generation from padding tokens"

## Sources

- [S1] Tokenizers. Hugging Face docs. https://huggingface.co/docs/transformers/fast_tokenizers . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

- [S2] Text generation. Hugging Face docs. https://huggingface.co/docs/transformers/llm_tutorial . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).