---
title: "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-12; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2402-04249-harmbench.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Mantas Mazeika, Long Phan, Xuwang Yin, and 9 coauthors, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: "Not reproduced. Abs page and HTML key sections read 2026-11-12. Named code URLs checked for HTTP 200; code not run."
category: arxiv
topic_area: 'ai-research: alignment-and-safety'
tags: arxiv, alignment, red-teaming, refusal, harmbench
arxiv_id: 2402.04249
alias: 2402-04249-harmbench
sources:
  - id: S1
    title: "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal"
    publisher: arXiv
    authors: "Mantas Mazeika and 11 coauthors (listed on the arXiv abs page)"
    url: https://arxiv.org/abs/2402.04249
    version_read: "v2 (27 Feb 2024)"
    accessed: 2026-11-12
    license: 'CC BY 4.0'
---
# HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

## Summary
HarmBench is a standardized evaluation framework for automated red teaming and for testing whether models refuse disallowed requests. The authors identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. They compare 18 red-teaming methods and 33 target models and defenses, and they also introduce an adversarial training method aimed at broader robustness. The authors open source HarmBench at https://github.com/centerforaisafety/HarmBench. HarmBench is a standardized evaluation framework for automated red teaming. [S1]

## Claims
- id: C1
  claim: "HarmBench is a standardized framework for evaluating automated red teaming and robust refusal."
  source: S1
  evidence: "a standardized evaluation framework for automated red teaming"
- id: C2
  claim: "The paper compares 18 red-teaming methods and 33 target models and defenses."
  source: S1
  evidence: "comparison of 18 red teaming methods and 33 target LLMs and defenses"
- id: C3
  claim: "The authors open-source HarmBench on GitHub."
  source: S1
  evidence: "https://github.com/centerforaisafety/HarmBench"

## Verification
- Code available: Paper: https://github.com/centerforaisafety/HarmBench and https://www.harmbench.org, both HTTP 200 on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Related notes include [2307-02483-jailbroken](https://iinventedlight.com/library/notes/2307-02483-jailbroken/) and [2212-08073-constitutional-ai](https://iinventedlight.com/library/notes/2212-08073-constitutional-ai/). Constitutional Classifiers is in this batch.
- Limits: The 18-by-33 comparison is as of v2 (27 Feb 2024). New attacks and defenses may need new evaluations. The note does not reproduce attack procedures.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 6 Feb 2024 (v1); last revised v2 (27 Feb 2024). Sources: [S1] https://arxiv.org/abs/2402.04249, v2 (27 Feb 2024). checked_on: 2026-11-12.

## When to apply
Use when asking what HarmBench measures, or how automated red-teaming methods have been compared under one protocol.

## Facts
- id: F1
  claim: "HarmBench provides a shared evaluation framework for automated red teaming and refusal."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: "The authors compare 18 red-teaming methods and 33 target models and defenses."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: "Code is at https://github.com/centerforaisafety/HarmBench; the project site is https://www.harmbench.org."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2402.04249, https://arxiv.org/abs/2402.04249. License: CC BY 4.0. v2 (27 Feb 2024). Accessed 2026-11-12.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
