← All notes

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Summary

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023) trains a process reward model that scores each step of a math solution, using automatically built step-level data instead of human step labels.

What it introduced: two uses for that step scorer. One is verification: rerank several model samples. The other is reinforcement learning: step-by-step PPO against the scorer. The authors report gains on GSM8K and MATH for open models such as Mistral-7B, with further gains when the same scorer is also used to rerank.

What it led to: later automated process-supervision work, including OmegaPRM in this week's batch F, and process rewards in math-specialized models. Live notes: 2110-14168-gsm8k, 2402-03300-deepseekmath, and 1707-06347-ppo.

Where to read it: https://arxiv.org/abs/2312.08935 (v3, last revised 19 Feb 2024).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Peiyi Wang, Lei Li, Zhihong Shao, and 6 coauthors, Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:0mg6xcb6

'iinventedlight (AI-assisted)'. 2026. "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations." iinventedlight Library, iil:note:0mg6xcb6, version of 2026-11-12. https://iinventedlight.com/library/notes/2312-08935-math-shepherd/. License: CC-BY-4.0.

@misc{iil_note_0mg6xcb6,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2312-08935-math-shepherd/},
  note = {Permanent ID iil:note:0mg6xcb6. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2312-08935-math-shepherd.md.