← All notes

Summary

Datasets can expose reads as Pandas objects or convert a dataset to a DataFrame. with_format("pandas") configures reads, while to_pandas performs an export. [S1]

Details

The Pandas guide also provides from_pandas for importing a DataFrame. In formatted mode, map and filter can operate on Pandas batches. Passing batched=True lets a function process multiple rows together, and batch_size controls the batch size. The general guide notes that formatting happens as data is read and can be limited to selected columns. [S1, S2]

Claims

• id: C1 claim: "Dataset.from_pandas imports a Pandas DataFrame." source: S1 evidence: "To import data from Pandas, you can use [Dataset.from_pandas]"
• id: C2 claim: "Dataset.to_pandas exports a Dataset as a DataFrame." source: S1 evidence: "export a Dataset to a Pandas DataFrame"
• id: C3 claim: "Formatting is applied at read time." source: S2 evidence: "Formatting is applied on-the-fly"

Sources

• [S1] Use with Pandas. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/use_with_pandas.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.
• [S2] Process. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.

Attribution and notices

This note is CC BY 4.0. Source material has been condensed and restated; short evidence excerpts retain their source license.

Hugging Face Datasets documentation is Apache-2.0. This note restates the documentation and includes only short evidence excerpts. See the accompanying source license.

@misc{iil_note_8ssy8rb0,
  author = {{Librarian}},
  title = {{Pandas formatting and conversion serve different uses}},
  howpublished = {iinventedlight Library},
  url = {https://iinventedlight.com/library/notes/hf-datasets-pandas/},
  note = {Permanent ID iil:note:8ssy8rb0. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-pandas.md.