Move dataset files from cloud storage with fsspec

iinventedlight (synthesis of cited sources); text drafted with AI assistance

Hugging Face Datasets: cloud storage and streaming

Short answer

Load from cloud filesystems through fsspec-compatible stores, or stream with IterableDataset when you do not want a full local Arrow conversion. [S1][S2]

Points

Most cloud providers have an fsspec FileSystem implementation. [S1]
Non-versioned buckets lack the git-style history Hub repos provide. [S1]
Streaming avoids waiting for huge local conversions and saves disk. [S2]

Claims

id: C1 claim: Most cloud providers have an fsspec FileSystem. source: S1 evidence: "Most cloud storage providers have a fsspec FileSystem implementation"
id: C2 claim: Hub repos provide data versioning via git. source: S1 evidence: "provides data versioning thanks to git"
id: C3 claim: Streaming works without downloading the whole dataset. source: S2 evidence: "work with a dataset without downloading it"
id: C4 claim: Streaming helps when converted size would exceed disk space. source: S2 evidence: "exceed the amount of available disk space on your computer"

Sources

[S1] Cloud storage · Hugging Face. Hugging Face (Datasets docs). https://huggingface.co/docs/datasets/filesystems . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-05.
[S2] Stream. Hugging Face. https://huggingface.co/docs/datasets/stream . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

Source notice: Hugging Face Datasets documentation, Apache License 2.0 (https://github.com/huggingface/datasets/blob/main/LICENSE).

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Sources: Hugging Face (Datasets docs), Cloud storage · Hugging Face, Apache-2.0 (huggingface/datasets docs); Hugging Face, Stream, Apache-2.0 (huggingface/datasets docs).

Cite this

Permanent ID: iil:note:rrkmwe7w

iinventedlight. 2026. "Move dataset files from cloud storage with fsspec." iinventedlight Library, iil:note:rrkmwe7w, version of 2026-11-04. https://iinventedlight.com/library/notes/hf-datasets-cloud-storage/. License: CC-BY-4.0.

@misc{iil_note_rrkmwe7w,
  author = {{iinventedlight}},
  title = {{Move dataset files from cloud storage with fsspec}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/hf-datasets-cloud-storage/},
  note = {Permanent ID iil:note:rrkmwe7w. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-cloud-storage.md.