← All notes

Answer: A tokenizer converts text into tensors; to batch varying-length sequences, use padding (shorter sequences gain pad tokens) and truncation (long sequences lose trailing tokens).

Call, encode, decode

• Calling the tokenizer encodes text into input_ids, attention_mask, and other model inputs, and controls padding, truncation, and special tokens.
• encode only returns the input_ids.
• decode converts tokenized input_ids back to text; skip_special_tokens strips special tokens from the output.

Padding

In the attention mask, padding positions are 0, so the model ignores them. padding=True pads to the longest sequence; max_length pads to a fixed size.

Truncation

Set truncation=True and specify max_length to clip tokens so a sequence fits within a maximum length.

Left padding for generation

Large language models pad on the left side because generation predicts the next token from the right side, and LLMs aren't trained to continue generation from padding tokens.

Artifacts

• Topic packet: huggingface

See also

• hf-transformers-auto-classes
• hf-tokenization-subword-algorithms
• hf-chat-templates

Claims

• id: C1 claim: Padding and truncation make a rectangular batch. source: S1 evidence: "Together, they produce a packed rectangular tensor."

• id: C2 claim: Padding and truncation handle varying-length sequences. source: S1 evidence: "Padding and truncation are strategies to handle varying-length sequences."

• id: C3 claim: LLMs pad on the left for generation. source: S1 evidence: "Large language models pad on the left side to avoid disrupting generation"

• id: C4 claim: Set padding_side to left for LLM batches. source: S2 evidence: "LLMs aren't trained to continue generation from padding tokens"

Sources

• [S1] Tokenizers. Hugging Face docs. https://huggingface.co/docs/transformers/fast_tokenizers . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] Text generation. Hugging Face docs. https://huggingface.co/docs/transformers/llm_tutorial . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_v9w0ww7y,
  author = {{iinventedlight}},
  title = {{Tokenizer padding, truncation, and special tokens}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-tokenizers-padding-truncation/},
  note = {Permanent ID iil:note:v9w0ww7y. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-tokenizers-padding-truncation.md.