---
source: https://iinventedlight.com/library/notes/hf-transformers-quantization-overview.md
title: "Quantization in Transformers: load-time vs pre-quantized"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/transformers, huggingface_hub, diffusers docs; LICENSE verified in each repo); prose restates cited source lines in own words"
license_family: hf-apache
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, Quantization overview, Apache-2.0 (huggingface docs repos); Hugging Face docs, Bitsandbytes, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: [hugging-face, transformers, quantization, bitsandbytes]
client: Librarian
sources:
  - id: S1
    title: "Quantization overview"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/quantization/overview
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Bitsandbytes"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/quantization/bitsandbytes
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
---
# Quantization in Transformers: load-time vs pre-quantized

**Answer:** Quantization stores weights (and sometimes activations) in fewer bits so the model needs less memory; in Transformers you either pass a quantization config to `from_pretrained` or load a checkpoint that is already quantized.

## Memory rule of thumb

Models usually load in bf16 at 2 bytes per parameter. An 8B model needs about 16GB in bf16. At 8 bits it needs about 8GB; at 4 bits, about 4–5GB. Each drop in bits costs some accuracy.

## Two paths

- Full-precision checkpoint: pass a config such as `BitsAndBytesConfig` to `quantization_config`; weights are quantized as they load.
- Pre-quantized checkpoint: call `from_pretrained` without a quantization config; the method is read from the checkpoint's `config.json`.
- AWQ, AutoRound, AQLM, compressed-tensors, and GGUF can only be loaded from a checkpoint.
- Most GGUF models are dequantized to full precision as they load.

## bitsandbytes

bitsandbytes offers LLM.int8() for 8-bit inference and QLoRA, a 4-bit technique that stays trainable through small LoRA weights. Quantizing in 8-bit halves the memory usage.

## Artifacts

- [Hugging Face model load choices (table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.json)
- [Hugging Face model load choices (markdown table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.md)
- [Topic packet: huggingface](https://iinventedlight.com/library/packets/huggingface.json)

## See also

- [hf-transformers-big-model-loading](https://iinventedlight.com/library/notes/hf-transformers-big-model-loading/)
- [hf-safetensors](https://iinventedlight.com/library/notes/hf-safetensors/)

## Claims

- id: C1
  claim: Quantization stores weights in fewer bits.
  source: S1
  evidence: "Quantization stores a model's weights, and sometimes its activations, in fewer bits"

- id: C2
  claim: An 8B model needs about 16GB in bf16.
  source: S1
  evidence: "An 8B model needs about 16GB for its weights in bf16"

- id: C3
  claim: Pre-quantized checkpoints pick their quantizer automatically.
  source: S1
  evidence: "the matching quantizer is selected automatically."

- id: C4
  claim: 8-bit quantization halves memory.
  source: S2
  evidence: "Quantizing a model in 8-bit halves the memory-usage"

## Sources

- [S1] Quantization overview. Hugging Face docs. https://huggingface.co/docs/transformers/quantization/overview . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

- [S2] Bitsandbytes. Hugging Face docs. https://huggingface.co/docs/transformers/quantization/bitsandbytes . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).