---
source: https://iinventedlight.com/library/notes/hf-datasets-spark.md
title: "Load a Spark DataFrame into Hugging Face Datasets"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/datasets docs); prose restates cited source lines"
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face (datasets docs), Use with Spark · Hugging Face, Apache-2.0 (huggingface/datasets docs); Hugging Face (datasets docs), Load · Hugging Face, Apache-2.0 (huggingface/datasets docs)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No code was run for this note; only the fetched Hugging Face datasets docs pages were read."
category: hf-datasets
topic_area: "code-ai-eng: Hugging Face datasets library"
tags: huggingface, datasets, spark
sources:
  - id: S1
    title: "Use with Spark · Hugging Face"
    publisher: Hugging Face (datasets docs)
    url: https://huggingface.co/docs/datasets/use_with_spark
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface/datasets docs)"
  - id: S2
    title: "Load · Hugging Face"
    publisher: Hugging Face (datasets docs)
    url: https://huggingface.co/docs/datasets/loading
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface/datasets docs)"
---
# Load a Spark DataFrame into Hugging Face Datasets

## Short answer
Use `Dataset.from_spark(df)`. Spark workers write the data to a cache directory as Arrow files and the Dataset loads from there, so large data does not pass through the driver. For files already on disk or on the Hub, `load_dataset` reads CSV, JSON and Parquet directly. [S1][S2]

```python
from datasets import Dataset
ds = Dataset.from_spark(spark_df)
```

## Claims
- id: C1
  claim: Get a Dataset from a Spark DataFrame with Dataset.from_spark().
  source: S1
  evidence: "You can get a Dataset from a Spark DataFrame using Dataset.from_spark():"
- id: C2
  claim: Spark workers write the dataset to a cache directory as Arrow files.
  source: S1
  evidence: "The Spark workers write the dataset on disk in a cache directory as Arrow files"
- id: C3
  claim: load_dataset can load from any dataset repository on the Hub.
  source: S2
  evidence: "You can also load a dataset from any dataset repository on the Hub!"

## Sources
- [S1] Use with Spark · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/use_with_spark . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.
- [S2] Load · Hugging Face. Hugging Face (datasets docs). https://huggingface.co/docs/datasets/loading . License: Apache-2.0 (huggingface/datasets docs). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).