---
title: Image preprocessing separates cached work from access-time transforms
slug: hf-datasets-image-preprocess
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://iinventedlight.com/library/notes/hf-datasets-image-preprocess.md
source_license: Apache-2.0
attribution: Original synthesis by iinventedlight, drafted by Librarian with AI assistance; based on the attributed sources
  below.
author: Librarian
client: Librarian
date: '2026-11-05'
checked_on: '2026-11-05'
as_of: '2026-11-05'
status: published
confidence: medium
category: coding-references
tags:
- hf
- datasets
- image
untested: Reference note. No code or commands were run to validate behavior.
sources:
- id: S1
  title: Process image data
  publisher: Hugging Face contributors
  url: https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_process.mdx
  accessed: '2026-11-05'
  license: Apache-2.0
  license_url: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE
  license_verified: '2026-11-05'
  green: true
  attribution: Hugging Face contributors; Process image data; Apache-2.0
  snapshot: _fetched/hf-image.txt
- id: S2
  title: Process
  publisher: Hugging Face contributors
  url: https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx
  accessed: '2026-11-05'
  license: Apache-2.0
  license_url: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE
  license_verified: '2026-11-05'
  green: true
  attribution: Hugging Face contributors; Process; Apache-2.0
  snapshot: _fetched/hf-process.txt
---
# Image preprocessing separates cached work from access-time transforms

## Summary
Datasets supports image transforms both through `map` and through access-time transforms. The distinction is when the work runs and whether the transformed result is cached. [S1, S2]

## Details
The image guide uses `map` for a resize performed once for training. It uses `set_transform` for augmentation applied as examples are read. Large image batches can consume substantial memory, so the guide identifies `batch_size` and `writer_batch_size` as controls. The general processing guide also distinguishes a returned formatted dataset from an in-place formatting change. [S1, S2]

## Claims
- id: C1
  claim: "map suits image work performed once rather than each epoch."
  source: S1
  evidence: "operations you only run once per training"
- id: C2
  claim: "set_transform applies augmentation as batches are accessed."
  source: S1
  evidence: "Transforms can be applied on-the-fly on batches of data"
- id: C3
  claim: "set_format changes formatting in place."
  source: S2
  evidence: "except it runs in-place"

## Sources
- [S1] Process image data. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.
- [S2] Process. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.

## Attribution and notices
This note is CC BY 4.0. Source material has been condensed and restated; short evidence excerpts retain their source license.

Hugging Face Datasets documentation is Apache-2.0. This note restates the documentation and includes only short evidence excerpts. See the accompanying source license.