---
source: https://iinventedlight.com/library/notes/hf-transformers-big-model-loading.md
title: "Loading big models: device_map, meta device, shards"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/transformers, huggingface_hub, diffusers docs; LICENSE verified in each repo); prose restates cited source lines in own words"
license_family: hf-apache
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Loading models, Apache-2.0 (huggingface docs repos); Hugging Face docs, Text generation, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: [hugging-face, transformers, device-map, memory]
client: Librarian
sources:
  - id: S1
    title: "Loading models"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/models
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Text generation"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/llm_tutorial
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
---
# Loading big models: device_map, meta device, shards

**Answer:** Set `device_map="auto"` in `from_pretrained` to enable Accelerate's Big Model Inference, which avoids holding two copies of the weights and dispatches them across devices.

## Why big models run out of memory

Loading creates a model with random weights, loads the pretrained weights, then places them on the model. You need enough memory to hold two copies of the model weights.

## What Big Model Inference does

- It creates a model skeleton on the PyTorch meta device, which stores only metadata, not real data.
- Maximum memory usage is only the size of the model.
- Weights go to the fastest device first (usually the GPU); remaining weights are offloaded to slower devices (CPU and hard drive).
- When the model is too large for GPUs and CPU RAM combined, set `offload_folder` for disk offloading.
- Access `hf_device_map` to see how a model is distributed across devices.

## Sharded checkpoints

`save_pretrained` automatically shards checkpoints larger than 50GB; `max_shard_size` sets the maximum size before sharding.

## Artifacts

- [Hugging Face model load choices (table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.json)
- [Hugging Face model load choices (markdown table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.md)
- [Topic packet: huggingface](https://iinventedlight.com/library/packets/huggingface.json)

## See also

- [hf-transformers-auto-classes](https://iinventedlight.com/library/notes/hf-transformers-auto-classes/)
- [hf-transformers-quantization-overview](https://iinventedlight.com/library/notes/hf-transformers-quantization-overview/)

## Claims

- id: C1
  claim: Naive loading needs two copies of the weights.
  source: S1
  evidence: "You need enough memory to hold two copies of the model weights"

- id: C2
  claim: The skeleton lives on the PyTorch meta device.
  source: S1
  evidence: "Big Model Inference creates a *model skeleton* on the PyTorch meta device."

- id: C3
  claim: device_map="auto" turns on Big Model Inference.
  source: S2
  evidence: "enables Accelerates' Big Model Inference feature for automatically initiating the model skeleton"

- id: C4
  claim: Checkpoints over 50GB are sharded automatically.
  source: S1
  evidence: "automatically shards checkpoints larger than 50GB."

## Sources

- [S1] Loading models. Hugging Face docs. https://huggingface.co/docs/transformers/models . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

- [S2] Text generation. Hugging Face docs. https://huggingface.co/docs/transformers/llm_tutorial . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).