HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Summary
HarmBench is a standardized evaluation framework for automated red teaming and for testing whether models refuse disallowed requests. The authors identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. They compare 18 red-teaming methods and 33 target models and defenses, and they also introduce an adversarial training method aimed at broader robustness. The authors open source HarmBench at https://github.com/centerforaisafety/HarmBench. HarmBench is a standardized evaluation framework for automated red teaming. [S1]
Claims
- id: C1 claim: "HarmBench is a standardized framework for evaluating automated red teaming and robust refusal." source: S1 evidence: "a standardized evaluation framework for automated red teaming"
- id: C2 claim: "The paper compares 18 red-teaming methods and 33 target models and defenses." source: S1 evidence: "comparison of 18 red teaming methods and 33 target LLMs and defenses"
- id: C3 claim: "The authors open-source HarmBench on GitHub." source: S1 evidence: "https://github.com/centerforaisafety/HarmBench"
Verification
- Code available: Paper: https://github.com/centerforaisafety/HarmBench and https://www.harmbench.org, both HTTP 200 on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Related notes include 2307-02483-jailbroken and 2212-08073-constitutional-ai. Constitutional Classifiers is in this batch.
- Limits: The 18-by-33 comparison is as of v2 (27 Feb 2024). New attacks and defenses may need new evaluations. The note does not reproduce attack procedures.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 6 Feb 2024 (v1); last revised v2 (27 Feb 2024). Sources: [S1] https://arxiv.org/abs/2402.04249, v2 (27 Feb 2024). checked_on: 2026-11-12.
When to apply
Use when asking what HarmBench measures, or how automated red-teaming methods have been compared under one protocol.
Facts
- id: F1 claim: "HarmBench provides a shared evaluation framework for automated red teaming and refusal." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F2 claim: "The authors compare 18 red-teaming methods and 33 target models and defenses." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F3 claim: "Code is at https://github.com/centerforaisafety/HarmBench; the project site is https://www.harmbench.org." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
Sources
- [S1] arXiv 2402.04249, https://arxiv.org/abs/2402.04249. License: CC BY 4.0. v2 (27 Feb 2024). Accessed 2026-11-12.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.
License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Mantas Mazeika, Long Phan, Xuwang Yin, and 9 coauthors, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (CC BY 4.0).
Cite this
'iinventedlight (AI-assisted)'. 2026. "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal." iinventedlight Library, iil:note:xtcdmy5q, version of 2026-11-12. https://iinventedlight.com/library/notes/2402-04249-harmbench/. License: CC-BY-4.0.
@misc{iil_note_xtcdmy5q,
author = {{'iinventedlight (AI-assisted)'}},
title = {{HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2402-04249-harmbench/},
note = {Permanent ID iil:note:xtcdmy5q. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2402-04249-harmbench.md.