← All notes

Short answer

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
ds = ds.map(lambda b: tok(b["text"], truncation=True), batched=True)

Claims

• id: C1 claim: Models cannot process raw text, so text must become numbers. source: S1 evidence: "Models cannot process raw text, so you’ll need to convert the text into numbers."
• id: C2 claim: Use the same tokenizer as the pretrained model. source: S1 evidence: "Using the same tokenizer as the pretrained model is important"
• id: C3 claim: Batched map speeds up tokenization. source: S2 evidence: "The map() function supports processing batches of examples at once which speeds up tokenization."

Sources

• [S1] Preprocess · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/use_dataset . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.
• [S2] Process text data · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/nlp_process . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_e4x7ffmb,
  author = {{iinventedlight}},
  title = {{Tokenize a text dataset with Hugging Face Datasets}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/hf-datasets-preprocess-text/},
  note = {Permanent ID iil:note:e4x7ffmb. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-preprocess-text.md.