← All notes

Answer: Quantization stores weights (and sometimes activations) in fewer bits so the model needs less memory; in Transformers you either pass a quantization config to from_pretrained or load a checkpoint that is already quantized.

Memory rule of thumb

Models usually load in bf16 at 2 bytes per parameter. An 8B model needs about 16GB in bf16. At 8 bits it needs about 8GB; at 4 bits, about 4–5GB. Each drop in bits costs some accuracy.

Two paths

• Full-precision checkpoint: pass a config such as BitsAndBytesConfig to quantization_config; weights are quantized as they load.
• Pre-quantized checkpoint: call from_pretrained without a quantization config; the method is read from the checkpoint's config.json.
• AWQ, AutoRound, AQLM, compressed-tensors, and GGUF can only be loaded from a checkpoint.
• Most GGUF models are dequantized to full precision as they load.

bitsandbytes

bitsandbytes offers LLM.int8() for 8-bit inference and QLoRA, a 4-bit technique that stays trainable through small LoRA weights. Quantizing in 8-bit halves the memory usage.

Artifacts

• Hugging Face model load choices (table)
• Hugging Face model load choices (markdown table)
• Topic packet: huggingface

See also

• hf-transformers-big-model-loading
• hf-safetensors

Claims

• id: C1 claim: Quantization stores weights in fewer bits. source: S1 evidence: "Quantization stores a model's weights, and sometimes its activations, in fewer bits"

• id: C2 claim: An 8B model needs about 16GB in bf16. source: S1 evidence: "An 8B model needs about 16GB for its weights in bf16"

• id: C3 claim: Pre-quantized checkpoints pick their quantizer automatically. source: S1 evidence: "the matching quantizer is selected automatically."

• id: C4 claim: 8-bit quantization halves memory. source: S2 evidence: "Quantizing a model in 8-bit halves the memory-usage"

Sources

• [S1] Quantization overview. Hugging Face docs. https://huggingface.co/docs/transformers/quantization/overview . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] Bitsandbytes. Hugging Face docs. https://huggingface.co/docs/transformers/quantization/bitsandbytes . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_8szmwr6d,
  author = {{iinventedlight}},
  title = {{Quantization in Transformers: load-time vs pre-quantized}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-transformers-quantization-overview/},
  note = {Permanent ID iil:note:8szmwr6d. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-transformers-quantization-overview.md.