← All notes

Summary

load_dataset reads a Hub repository, local or remote files, or tables already in memory. Stream when the data is too large to download or to convert to Arrow.

Loading

Pass the namespace and dataset name. revision selects a Git tag, branch, or commit. Rows go into the train split unless file names mention train, test, or validation. data_files and data_dir limit which files are read. If you name no files, every file comes back, which can take a long time on a very large set.

Formats include csv, json, txt, and parquet. One JSON object per line is the efficient layout; pass field when the rows are nested. https URLs load public files. hf:// URLs load a file in a dataset repository or a storage bucket. Dataset.from_file memory-maps a local Arrow file and skips cache preparation. Only the Arrow streaming format is supported, not Arrow IPC (Feather V2). Dataset.from_sql reads a table or a query. from_dict, from_list, from_pandas, and from_generator build a dataset from memory, and a generator can exceed RAM.

A Hub dataset already downloaded stays in the cache. Set HF_HUB_OFFLINE to 1 so the library reads that cache instead of waiting for a download to time out. num_proc prepares many shards in parallel. A split string can join splits, take a row range, or take a percentage. Percent bounds round to the nearest integer, so slices of an uneven set can differ in length. pct1_dropremainder equalizes them and may drop leftover rows.

Streaming

streaming=True yields rows without a full download, and it can stream local files without an Arrow conversion. Use it when you will not wait, the data will not fit on disk, or you only want a few samples. Parquet streams can also keep selected columns or apply filters.

The result is an IterableDataset, meant for a loop such as training, not for random access. to_iterable_dataset on a dataset already on disk is faster than streaming=True. shuffle draws from a buffer of 1,000 examples by default and also reorders shards. set_epoch makes the seed the original seed plus the epoch number. Shuffle before take or skip: those two lock shard order. concatenate_datasets needs matching column types. interleave_datasets alternates inputs, and probabilities set the mix. first_exhausted stops when any input runs out; all_exhausted waits until every row has been used at least once.

map runs as rows arrive (default batch size 1,000). Resume with state_dict and load_state_dict: they record the shard and the position in it. A shuffle buffer is not restored. push_to_hub uploads the stream, and export helpers write CSV, JSON, Parquet, or SQL.

Claims

• id: C1 claim: load_dataset loads a Hub dataset from its namespace and name. source: S1 evidence: "use the load_dataset() function to load the dataset"
• id: C2 claim: revision selects a Git tag, branch, or commit. source: S1 evidence: "Use the revision parameter to specify the dataset version you want to load"
• id: C3 claim: HF_HUB_OFFLINE set to 1 reads the cache instead of waiting out a download. source: S1 evidence: "Set the environment variable HF_HUB_OFFLINE to 1 to enable full offline mode."
• id: C4 claim: Streaming iterates a dataset without downloading it. source: S2 evidence: "Dataset streaming lets you work with a dataset without downloading it."
• id: C5 claim: Streaming returns an IterableDataset, not a random-access Dataset. source: S2 evidence: "known as an IterableDataset"
• id: C6 claim: take and skip lock shard order, so shuffle before splitting the stream. source: S2 evidence: "take and skip prevent future calls to shuffle"

@misc{iil_note_cwnvfg3b,
  author = {{iinventedlight}},
  title = {{Loading and streaming Hugging Face datasets}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-03},
  url = {https://iinventedlight.com/library/notes/hf-datasets-load-and-stream/},
  note = {Permanent ID iil:note:cwnvfg3b. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-load-and-stream.md.