← All notes

Summary

with_format("polars") makes dataset reads return Polars DataFrames or Series. Datasets and Polars both use Arrow, which enables zero-copy operations described in the guide. [S1]

Details

A formatted dataset can run Polars expressions inside batched map or filter. from_polars and to_polars handle import and export instead of changing the read format. The processing guide lists Polars as a tabular format and explains that columns= selects the fields to format. No fixed speedup follows from choosing Polars; performance depends on the operation and data. [S1, S2]

Claims

• id: C1 claim: "Both libraries use Arrow internally." source: S1 evidence: "both datasets and Polars use Arrow under the hood"
• id: C2 claim: "Dataset.from_polars imports a Polars DataFrame." source: S1 evidence: "To import data from Polars, you can use [Dataset.from_polars]"
• id: C3 claim: "columns= selects which columns to format." source: S2 evidence: "choose which the columns you want to format using columns="

Sources

• [S1] Use with Polars. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/use_with_polars.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.
• [S2] Process. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.

Attribution and notices

This note is CC BY 4.0. Source material has been condensed and restated; short evidence excerpts retain their source license.

Hugging Face Datasets documentation is Apache-2.0. This note restates the documentation and includes only short evidence excerpts. See the accompanying source license.

@misc{iil_note_n1kcjqpj,
  author = {{Librarian}},
  title = {{Polars batches can process Datasets without row-by-row Python}},
  howpublished = {iinventedlight Library},
  url = {https://iinventedlight.com/library/notes/hf-datasets-polars/},
  note = {Permanent ID iil:note:n1kcjqpj. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/hf-datasets-polars.md.