---
source: https://iinventedlight.com/library/notes/hf-datasets-preprocess-text.md
title: "Tokenize a text dataset with Hugging Face Datasets"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/datasets docs); prose restates cited source lines"
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face (datasets docs), Preprocess · Hugging Face, Apache-2.0 (huggingface/datasets docs); Hugging Face (datasets docs), Process text data · Hugging Face, Apache-2.0 (huggingface/datasets docs)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No code was run for this note; only the fetched Hugging Face datasets docs pages were read."
category: hf-datasets
topic_area: "code-ai-eng: Hugging Face datasets library"
tags: huggingface, datasets, tokenization, nlp
sources:
  - id: S1
    title: "Preprocess · Hugging Face"
    publisher: Hugging Face (datasets docs)
    url: https://huggingface.co/docs/datasets/use_dataset
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface/datasets docs)"
  - id: S2
    title: "Process text data · Hugging Face"
    publisher: Hugging Face (datasets docs)
    url: https://huggingface.co/docs/datasets/nlp_process
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface/datasets docs)"
---
# Tokenize a text dataset with Hugging Face Datasets

## Short answer

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
ds = ds.map(lambda b: tok(b["text"], truncation=True), batched=True)
```

## Claims
- id: C1
  claim: Models cannot process raw text, so text must become numbers.
  source: S1
  evidence: "Models cannot process raw text, so you’ll need to convert the text into numbers."
- id: C2
  claim: Use the same tokenizer as the pretrained model.
  source: S1
  evidence: "Using the same tokenizer as the pretrained model is important"
- id: C3
  claim: Batched map speeds up tokenization.
  source: S2
  evidence: "The map() function supports processing batches of examples at once which speeds up tokenization."

## Sources
- [S1] Preprocess · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/use_dataset . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.
- [S2] Process text data · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/nlp_process . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).