← All notes

Short answer

Downloads go to the Hub cache (~/.cache/huggingface/hub); Arrow versions and processed results go to ~/.cache/huggingface/datasets. Move both by setting HF_HOME. Re-running the same map reuses cached results because each transform has a fingerprint. [S1][S2]

Gotchas

• A function that cannot be pickled gets a random fingerprint, so it is recomputed every time. [S1]
• With caching off, results go to a temp folder; call save_to_disk() to keep them. [S1]

Claims

• id: C1 claim: The cache reloads previously processed datasets. source: S1 evidence: "It stores previously downloaded and processed datasets"
• id: C2 claim: Non-hashable transforms get a random fingerprint. source: S1 evidence: "uses a random fingerprint instead and raises a warning"
• id: C3 claim: Use save_to_disk when caching is disabled. source: S1 evidence: "When caching is disabled, use Dataset.save_to_disk() to save your transformed dataset"
• id: C4 claim: Hub files are cached under ~/.cache/huggingface/hub. source: S2 evidence: "which is at ~/.cache/huggingface/hub by default"
• id: C5 claim: HF_HOME changes the cache location. source: S2 evidence: "Change the cache location by setting the shell environment variable, HF_HOME"

Sources

• [S1] The cache · Hugging Face. Hugging Face (Datasets docs). https://huggingface.co/docs/datasets/about_cache . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.
• [S2] Cache management · Hugging Face. Hugging Face (Datasets docs). https://huggingface.co/docs/datasets/cache . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_acpcgkdw,
  author = {{iinventedlight}},
  title = {{How the Hugging Face Datasets cache works and where it lives}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/hf-datasets-cache/},
  note = {Permanent ID iil:note:acpcgkdw. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-cache.md.