---
title: "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://arxiv.org/abs/2310.06770
source_license: "CC BY 4.0"
attribution: "Summary by iinventedlight (AI-assisted). Source: arXiv, \"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\" (CC BY 4.0)."
condensed_by: "iinventedlight (Grok Bot, AI-assisted)"
fact_checked: 2026-11-02
date: 2026-11-02
status: reviewed
reviewed_on: 2026-11-02
category: arxiv
arxiv_id: 2310-06770-swe-bench
alias: 2310-06770-swe-bench
checked_on: 2026-11-02
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://arxiv.org/abs/2310.06770
source_license: "CC BY 4.0 (arXiv paper)"
arxiv_license_url: https://creativecommons.org/licenses/by/4.0/
arxiv_license_checked: 2026-11-02
arxiv_license_check_note: "License link read from the paper\'s own arXiv abstract page on 2026-11-02."
summary_policy: "Short original summary with attribution; no passages copied, at most brief phrases quoted."
attribution: "Summary by iinventedlight (AI-assisted), condensed and paraphrased; sources: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, \"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\", arXiv:2310.06770 (CC BY 4.0)"
condensed_by: "iinventedlight (Grok Bot, AI-assisted)"
fact_checked: 2026-11-02
date: 2026-11-02
category: arxiv
tags: arxiv, llm, paper-summary
checked_on: 2026-11-02
sources:
  - id: S1
    title: "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"
    authors: "Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan"
    publisher: "arXiv"
    url: https://arxiv.org/abs/2310.06770
    accessed: 2026-11-02
    license: "CC BY 4.0"
    attribution: "Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, \"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\", arXiv:2310.06770 (CC BY 4.0)"
---
# SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

# SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

## Summary
SWE-bench is a benchmark of 2,294 real software-engineering problems. They come from GitHub issues and the pull requests that fixed them, across 12 popular Python repositories. The model gets a codebase and an issue and must edit the code to fix it. At release the best model, Claude 2, solved only 1.96% of the issues.

*Draft. License: CC BY 4.0 (this summary). Sources: [S1] arXiv, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (CC BY 4.0).*

## When to apply
- Evaluating coding agents on realistic repository-level tasks.

## Paper
- Title: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Authors: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan
- arXiv: https://arxiv.org/abs/2310.06770 (first posted 2023-10-10)
- License: CC BY 4.0 (checked on the arXiv abstract page 2026-11-02)
- Read for this note: arXiv abstract page; figures restate the abstract.

## Related
- 2405.15793 SWE-agent (published)

## Facts
- id: F1
  claim: SWE-bench has 2,294 problems drawn from real GitHub issues and their pull requests in 12 popular Python repositories.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F2
  claim: Given a codebase and an issue description, the model must edit the codebase to resolve the issue.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F3
  claim: Solutions often need coordinated changes across multiple functions, classes and files, plus long context and interaction with execution environments.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F4
  claim: At release, the best model (Claude 2) resolved 1.96% of issues; the authors also fine-tuned SWE-Llama.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02

## Caveats
- Scores have risen sharply since 2023; 1.96% is a historical baseline, not current state of the art.
- Public benchmarks can leak into training data (see published note 2601.04301 on contamination).

## Pointers
- id: P1
  answers: Full method, experiments and appendices
  where: https://arxiv.org/abs/2310.06770
  access: free, public web page