---
title: Polars batches can process Datasets without row-by-row Python
slug: hf-datasets-polars
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://iinventedlight.com/library/notes/hf-datasets-polars.md
source_license: Apache-2.0
attribution: Original synthesis by iinventedlight, drafted by Librarian with AI assistance; based on the attributed sources
  below.
author: Librarian
client: Librarian
date: '2026-11-05'
checked_on: '2026-11-05'
as_of: '2026-11-05'
status: published
confidence: medium
category: coding-references
tags:
- hf
- datasets
- polars
untested: Reference note. No code or commands were run to validate behavior.
sources:
- id: S1
  title: Use with Polars
  publisher: Hugging Face contributors
  url: https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/use_with_polars.mdx
  accessed: '2026-11-05'
  license: Apache-2.0
  license_url: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE
  license_verified: '2026-11-05'
  green: true
  attribution: Hugging Face contributors; Use with Polars; Apache-2.0
  snapshot: _fetched/hf-polars.txt
- id: S2
  title: Process
  publisher: Hugging Face contributors
  url: https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx
  accessed: '2026-11-05'
  license: Apache-2.0
  license_url: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE
  license_verified: '2026-11-05'
  green: true
  attribution: Hugging Face contributors; Process; Apache-2.0
  snapshot: _fetched/hf-process.txt
---
# Polars batches can process Datasets without row-by-row Python

## Summary
`with_format("polars")` makes dataset reads return Polars DataFrames or Series. Datasets and Polars both use Arrow, which enables zero-copy operations described in the guide. [S1]

## Details
A formatted dataset can run Polars expressions inside batched `map` or `filter`. `from_polars` and `to_polars` handle import and export instead of changing the read format. The processing guide lists Polars as a tabular format and explains that `columns=` selects the fields to format. No fixed speedup follows from choosing Polars; performance depends on the operation and data. [S1, S2]

## Claims
- id: C1
  claim: "Both libraries use Arrow internally."
  source: S1
  evidence: "both `datasets` and Polars use Arrow under the hood"
- id: C2
  claim: "Dataset.from_polars imports a Polars DataFrame."
  source: S1
  evidence: "To import data from Polars, you can use [`Dataset.from_polars`]"
- id: C3
  claim: "columns= selects which columns to format."
  source: S2
  evidence: "choose which the columns you want to format using `columns=`"

## Sources
- [S1] Use with Polars. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/use_with_polars.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.
- [S2] Process. Hugging Face contributors. https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/process.mdx. Apache-2.0; license: https://raw.githubusercontent.com/huggingface/datasets/main/LICENSE. Accessed 2026-11-05.

## Attribution and notices
This note is CC BY 4.0. Source material has been condensed and restated; short evidence excerpts retain their source license.

Hugging Face Datasets documentation is Apache-2.0. This note restates the documentation and includes only short evidence excerpts. See the accompanying source license.