SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Summary by iinventedlight (AI-assisted), condensed and paraphrased; sources: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770 (CC BY 4.0)

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Summary

SWE-bench is a benchmark of 2,294 real software-engineering problems. They come from GitHub issues and the pull requests that fixed them, across 12 popular Python repositories. The model gets a codebase and an issue and must edit the code to fix it. At release the best model, Claude 2, solved only 1.96% of the issues.

Draft. License: CC BY 4.0 (this summary). Sources: [S1] arXiv, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (CC BY 4.0).

When to apply

Evaluating coding agents on realistic repository-level tasks.

Paper

Title: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Authors: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan
arXiv: https://arxiv.org/abs/2310.06770 (first posted 2023-10-10)
License: CC BY 4.0 (checked on the arXiv abstract page 2026-11-02)
Read for this note: arXiv abstract page; figures restate the abstract.

Facts

id: F1 claim: SWE-bench has 2,294 problems drawn from real GitHub issues and their pull requests in 12 popular Python repositories. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
id: F2 claim: Given a codebase and an issue description, the model must edit the codebase to resolve the issue. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
id: F3 claim: Solutions often need coordinated changes across multiple functions, classes and files, plus long context and interaction with execution environments. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
id: F4 claim: At release, the best model (Claude 2) resolved 1.96% of issues; the authors also fine-tuned SWE-Llama. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02

Caveats

Scores have risen sharply since 2023; 1.96% is a historical baseline, not current state of the art.
Public benchmarks can leak into training data (see published note 2601.04301 on contamination).

Pointers

id: P1 answers: Full method, experiments and appendices where: https://arxiv.org/abs/2310.06770 access: free, public web page

Source: https://arxiv.org/abs/2310.06770 · License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted), condensed and paraphrased; sources: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, \"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\", arXiv:2310.06770 (CC BY 4.0)

Cite this

Permanent ID: iil:note:c9ed0g8c

iinventedlight Library. 2026. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" iinventedlight Library, iil:note:c9ed0g8c, version of 2026-11-02. https://iinventedlight.com/library/reference/arxiv/2310-06770-swe-bench/. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2310.06770.

@misc{iil_note_c9ed0g8c,
  author = {{iinventedlight Library}},
  title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-02},
  url = {https://iinventedlight.com/library/reference/arxiv/2310-06770-swe-bench/},
  note = {Permanent ID iil:note:c9ed0g8c. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2310.06770.},
}

For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2310-06770-swe-bench.md.