---
title: "Loading and streaming Hugging Face datasets"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://iinventedlight.com/library/notes/hf-datasets-load-and-stream.md
source_license: "Apache-2.0 (huggingface/datasets docs); restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face, Load, Apache-2.0 (huggingface/datasets docs); Hugging Face, Stream, Apache-2.0 (huggingface/datasets docs)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-03
status: published
checked_on: 2026-11-03
as_of: 2026-11-03
confidence: medium
untested: "No datasets library was installed and no load or stream commands were run; only the fetched Load and Stream documentation pages were read."
category: huggingface-datasets
topic_area: "Loading and streaming datasets"
tags: huggingface, datasets, load_dataset, streaming, iterable-dataset
sources:
  - id: S1
    title: "Load"
    publisher: Hugging Face
    url: https://huggingface.co/docs/datasets/loading
    accessed: 2026-11-03
    license: "Apache-2.0 (huggingface/datasets docs)"
  - id: S2
    title: "Stream"
    publisher: Hugging Face
    url: https://huggingface.co/docs/datasets/stream
    accessed: 2026-11-03
    license: "Apache-2.0 (huggingface/datasets docs)"
---
# Loading and streaming Hugging Face datasets

## Summary
`load_dataset` reads a Hub repository, local or remote files, or tables already in memory. Stream when the data is too large to download or to convert to Arrow.

## Loading
Pass the namespace and dataset name. `revision` selects a Git tag, branch, or commit. Rows go into the train split unless file names mention train, test, or validation. `data_files` and `data_dir` limit which files are read. If you name no files, every file comes back, which can take a long time on a very large set.

Formats include csv, json, txt, and parquet. One JSON object per line is the efficient layout; pass `field` when the rows are nested. https URLs load public files. hf:// URLs load a file in a dataset repository or a storage bucket. `Dataset.from_file` memory-maps a local Arrow file and skips cache preparation. Only the Arrow streaming format is supported, not Arrow IPC (Feather V2). `Dataset.from_sql` reads a table or a query. `from_dict`, `from_list`, `from_pandas`, and `from_generator` build a dataset from memory, and a generator can exceed RAM.

A Hub dataset already downloaded stays in the cache. Set `HF_HUB_OFFLINE` to 1 so the library reads that cache instead of waiting for a download to time out. `num_proc` prepares many shards in parallel. A split string can join splits, take a row range, or take a percentage. Percent bounds round to the nearest integer, so slices of an uneven set can differ in length. `pct1_dropremainder` equalizes them and may drop leftover rows.

## Streaming
`streaming=True` yields rows without a full download, and it can stream local files without an Arrow conversion. Use it when you will not wait, the data will not fit on disk, or you only want a few samples. Parquet streams can also keep selected columns or apply filters.

The result is an IterableDataset, meant for a loop such as training, not for random access. `to_iterable_dataset` on a dataset already on disk is faster than `streaming=True`. `shuffle` draws from a buffer of 1,000 examples by default and also reorders shards. `set_epoch` makes the seed the original seed plus the epoch number. Shuffle before `take` or `skip`: those two lock shard order. `concatenate_datasets` needs matching column types. `interleave_datasets` alternates inputs, and `probabilities` set the mix. `first_exhausted` stops when any input runs out; `all_exhausted` waits until every row has been used at least once.

`map` runs as rows arrive (default batch size 1,000). Resume with `state_dict` and `load_state_dict`: they record the shard and the position in it. A shuffle buffer is not restored. `push_to_hub` uploads the stream, and export helpers write CSV, JSON, Parquet, or SQL.

## Claims
- id: C1
  claim: load_dataset loads a Hub dataset from its namespace and name.
  source: S1
  evidence: "use the load_dataset() function to load the dataset"
- id: C2
  claim: revision selects a Git tag, branch, or commit.
  source: S1
  evidence: "Use the revision parameter to specify the dataset version you want to load"
- id: C3
  claim: HF_HUB_OFFLINE set to 1 reads the cache instead of waiting out a download.
  source: S1
  evidence: "Set the environment variable HF_HUB_OFFLINE to 1 to enable full offline mode."
- id: C4
  claim: Streaming iterates a dataset without downloading it.
  source: S2
  evidence: "Dataset streaming lets you work with a dataset without downloading it."
- id: C5
  claim: Streaming returns an IterableDataset, not a random-access Dataset.
  source: S2
  evidence: "known as an IterableDataset"
- id: C6
  claim: take and skip lock shard order, so shuffle before splitting the stream.
  source: S2
  evidence: "take and skip prevent future calls to shuffle"