← All notes

Answer: Set device_map="auto" in from_pretrained to enable Accelerate's Big Model Inference, which avoids holding two copies of the weights and dispatches them across devices.

Why big models run out of memory

Loading creates a model with random weights, loads the pretrained weights, then places them on the model. You need enough memory to hold two copies of the model weights.

What Big Model Inference does

• It creates a model skeleton on the PyTorch meta device, which stores only metadata, not real data.
• Maximum memory usage is only the size of the model.
• Weights go to the fastest device first (usually the GPU); remaining weights are offloaded to slower devices (CPU and hard drive).
• When the model is too large for GPUs and CPU RAM combined, set offload_folder for disk offloading.
• Access hf_device_map to see how a model is distributed across devices.

Sharded checkpoints

save_pretrained automatically shards checkpoints larger than 50GB; max_shard_size sets the maximum size before sharding.

Artifacts

• Hugging Face model load choices (table)
• Hugging Face model load choices (markdown table)
• Topic packet: huggingface

See also

• hf-transformers-auto-classes
• hf-transformers-quantization-overview

Claims

• id: C1 claim: Naive loading needs two copies of the weights. source: S1 evidence: "You need enough memory to hold two copies of the model weights"

• id: C2 claim: The skeleton lives on the PyTorch meta device. source: S1 evidence: "Big Model Inference creates a model skeleton on the PyTorch meta device."

• id: C3 claim: device_map="auto" turns on Big Model Inference. source: S2 evidence: "enables Accelerates' Big Model Inference feature for automatically initiating the model skeleton"

• id: C4 claim: Checkpoints over 50GB are sharded automatically. source: S1 evidence: "automatically shards checkpoints larger than 50GB."

Sources

• [S1] Loading models. Hugging Face docs. https://huggingface.co/docs/transformers/models . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] Text generation. Hugging Face docs. https://huggingface.co/docs/transformers/llm_tutorial . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_bj5q4j84,
  author = {{iinventedlight}},
  title = {{Loading big models: device\_map, meta device, shards}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-transformers-big-model-loading/},
  note = {Permanent ID iil:note:bj5q4j84. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-transformers-big-model-loading.md.