← All notes

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

Summary

HarmBench is a standardized evaluation framework for automated red teaming and for testing whether models refuse disallowed requests. The authors identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. They compare 18 red-teaming methods and 33 target models and defenses, and they also introduce an adversarial training method aimed at broader robustness. The authors open source HarmBench at https://github.com/centerforaisafety/HarmBench. HarmBench is a standardized evaluation framework for automated red teaming. [S1]

Claims

Verification

When to apply

Use when asking what HarmBench measures, or how automated red-teaming methods have been compared under one protocol.

Facts

Sources

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Mantas Mazeika, Long Phan, Xuwang Yin, and 9 coauthors, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (CC BY 4.0).

Cite this

Permanent ID: iil:note:xtcdmy5q

'iinventedlight (AI-assisted)'. 2026. "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal." iinventedlight Library, iil:note:xtcdmy5q, version of 2026-11-12. https://iinventedlight.com/library/notes/2402-04249-harmbench/. License: CC-BY-4.0.

@misc{iil_note_xtcdmy5q,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2402-04249-harmbench/},
  note = {Permanent ID iil:note:xtcdmy5q. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2402-04249-harmbench.md.