← All notes

Summary

LLM.int8() is an 8-bit matrix multiplication procedure for transformer feed-forward and attention projection layers. The authors say it cuts the memory needed for inference by half while retaining full precision performance, including for checkpoints with up to 175B parameters. The method uses vector-wise quantization plus a mixed-precision decomposition that isolates outlier feature dimensions in 16-bit, while still more than 99.9% of values are multiplied in 8-bit. They open source the software. [S1]

Claims

• id: C1 claim: 'The procedure cuts the memory needed for inference by half while retaining full precision performance.' source: S1 evidence: "cut the memory needed for inference by half while retaining full precision performance"
• id: C2 claim: 'They say more than 99.9% of values are multiplied in 8-bit, with outliers in 16-bit.' source: S1 evidence: "still more than 99.9% of values are multiplied in 8-bit"
• id: C3 claim: 'They open-source the software at a public GitHub repository.' source: S1 evidence: "https://github.com/TimDettmers/bitsandbytes"

Verification

• Code available: Paper: https://github.com/TimDettmers/bitsandbytes, which returned HTTP 200 and redirected to https://github.com/bitsandbytes-foundation/bitsandbytes on 2026-11-11. Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live note 2305-14314-qlora. This is the paper-verification note for LLM.int8() itself.
• Limits: The 'no performance degradation' claim is the authors' reported result for this inference-time method on the models they tested. Later quantization methods may trade a small quality loss for higher compression.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 15 Aug 2022 (v1); last revised 10 Nov 2022 (v2). Sources: [S1] https://arxiv.org/abs/2208.07339, v2 (10 Nov 2022). checked_on: 2026-11-11.

When to apply

Use when the goal is 8-bit inference that preserves quality by treating outlier features separately, including on very large checkpoints.

Facts

• id: F1 claim: 'LLM.int8() is an 8-bit matrix multiplication method for transformer feed-forward and attention projection layers.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors report cutting inference memory by half while retaining full precision performance for models up to 175B parameters.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They report that more than 99.9% of values are multiplied in 8-bit, with outlier dimensions kept in 16-bit.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2208.07339, https://arxiv.org/abs/2208.07339. License: CC BY 4.0. v2 (10 Nov 2022). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_erzkrp3p,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2208-07339-llm-int8/},
  note = {Permanent ID iil:note:erzkrp3p. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2208-07339-llm-int8.md.