---
source: https://iinventedlight.com/library/notes/hf-datasets-pyarrow.md
title: "Hugging Face Datasets and Apache Arrow"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface docs repos); prose restates cited source lines"
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Use with PyArrow · Hugging Face, Apache-2.0 (huggingface docs repos); Hugging Face docs, Datasets 🤝 Arrow · Hugging Face, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: ['hugging-face', 'datasets', 'arrow', 'pyarrow']
sources:
  - id: S1
    title: "Use with PyArrow · Hugging Face"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/datasets/use_with_pyarrow
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Datasets 🤝 Arrow · Hugging Face"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/datasets/about_arrow
    accessed: 2026-11-04
    license: "Apache-2.0 (huggingface docs repos)"
---
# Hugging Face Datasets and Apache Arrow

**Answer:** 

```python
from datasets import load_dataset
ds = load_dataset("imdb", split="train")

ds = ds.with_format("arrow")
for table in ds.iter(batch_size=1000):   # pyarrow.Table batches
    print(table.num_rows)

import pyarrow.compute as pc
ds = ds.map(lambda t: t.append_column("len", pc.utf8_length(t["text"])), batched=True)
```

## Why Arrow

- **Zero-copy reads:** almost no serialization overhead.
- **Language-agnostic:** the same data works from many languages.
- This is possible because the Arrow data is actually memory-mapped from disk, and not loaded in memory. (S2)

## Claims
- id: C1
  claim: You can set the dataset format to pyarrow with with_format.
  source: S1
  evidence: "you can set the format of the dataset to pyarrow using Dataset.with_format():"
- id: C2
  claim: Datasets uses Arrow for its local caching system.
  source: S2
  evidence: "Datasets uses Arrow for its local caching system."
- id: C3
  claim: Arrow stores data in a columnar memory layout.
  source: S2
  evidence: "It is a specific data format that stores data in a columnar memory layout."
- id: C4
  claim: Arrow's format allows zero-copy reads.
  source: S2
  evidence: "Arrow’s standard format allows zero-copy reads which removes virtually all serialization overhead."
- id: C5
  claim: Arrow data is memory-mapped from disk, not loaded in memory.
  source: S2
  evidence: "the Arrow data is actually memory-mapped from disk, and not loaded in memory."

## Sources
- [S1] Use with PyArrow · Hugging Face. Hugging Face docs. https://huggingface.co/docs/datasets/use_with_pyarrow . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-04.
- [S2] Datasets 🤝 Arrow · Hugging Face. Hugging Face docs. https://huggingface.co/docs/datasets/about_arrow . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).