← All notes

Summary

Datasets supports image transforms both through map and through access-time transforms. The distinction is when the work runs and whether the transformed result is cached. [S1, S2]

Details

The image guide uses map for a resize performed once for training. It uses set_transform for augmentation applied as examples are read. Large image batches can consume substantial memory, so the guide identifies batch_size and writer_batch_size as controls. The general processing guide also distinguishes a returned formatted dataset from an in-place formatting change. [S1, S2]

Claims

• id: C1 claim: "map suits image work performed once rather than each epoch." source: S1 evidence: "operations you only run once per training"
• id: C2 claim: "set_transform applies augmentation as batches are accessed." source: S1 evidence: "Transforms can be applied on-the-fly on batches of data"
• id: C3 claim: "set_format changes formatting in place." source: S2 evidence: "except it runs in-place"

Sources

• [S1] Process image data. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.
• [S2] Process. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.

Attribution and notices

This note is CC BY 4.0. Source material has been condensed and restated; short evidence excerpts retain their source license.

Hugging Face Datasets documentation is Apache-2.0. This note restates the documentation and includes only short evidence excerpts. See the accompanying source license.

@misc{iil_note_d6mn00pd,
  author = {{Librarian}},
  title = {{Image preprocessing separates cached work from access-time transforms}},
  howpublished = {iinventedlight Library},
  url = {https://iinventedlight.com/library/notes/hf-datasets-image-preprocess/},
  note = {Permanent ID iil:note:d6mn00pd. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-image-preprocess.md.