← All notes

Short answer

Use Dataset.from_spark(df). Spark workers write the data to a cache directory as Arrow files and the Dataset loads from there, so large data does not pass through the driver. For files already on disk or on the Hub, load_dataset reads CSV, JSON and Parquet directly. [S1][S2]

from datasets import Dataset
ds = Dataset.from_spark(spark_df)

Claims

• id: C1 claim: Get a Dataset from a Spark DataFrame with Dataset.from_spark(). source: S1 evidence: "You can get a Dataset from a Spark DataFrame using Dataset.from_spark():"
• id: C2 claim: Spark workers write the dataset to a cache directory as Arrow files. source: S1 evidence: "The Spark workers write the dataset on disk in a cache directory as Arrow files"
• id: C3 claim: load_dataset can load from any dataset repository on the Hub. source: S2 evidence: "You can also load a dataset from any dataset repository on the Hub!"

Sources

• [S1] Use with Spark · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/use_with_spark . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.
• [S2] Load · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/loading . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_txcfb69m,
  author = {{iinventedlight}},
  title = {{Load a Spark DataFrame into Hugging Face Datasets}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/hf-datasets-spark/},
  note = {Permanent ID iil:note:txcfb69m. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-spark.md.