← All notes

Summary

Program-Aided Language models (PAL) use a language model to read a natural-language problem and generate a program, then offload the solution step to a runtime such as a Python interpreter. The authors evaluate this approach on 13 mathematical, symbolic, and algorithmic reasoning tasks. They report that PAL using Codex reached state-of-the-art few-shot accuracy on GSM8K, surpassing PaLM-540B with chain-of-thought by an absolute 15% top-1. Code and data are publicly available. [S1]

Claims

• id: C1 claim: 'PAL generates programs as intermediate reasoning steps and offloads solving to an interpreter.' source: S1 evidence: "offloads the solution step to a runtime such as a Python interpreter"
• id: C2 claim: 'They evaluate across 13 mathematical, symbolic, and algorithmic reasoning tasks.' source: S1 evidence: "across 13 mathematical, symbolic, and algorithmic reasoning tasks"
• id: C3 claim: 'They report PAL with Codex surpasses PaLM-540B chain-of-thought by an absolute 15% top-1 on GSM8K.' source: S1 evidence: "surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1"

Verification

• Code available: Comments and paper: http://reasonwithpal.com/ (HTTPS https://reasonwithpal.com/ returned HTTP 200 on 2026-11-11). Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes 2201-11903-chain-of-thought-prompting and 2402-01030-codeact. This is the paper-verification note for PAL itself.
• Limits: The 15% GSM8K gap is a reported few-shot result against PaLM-540B with chain-of-thought, using Codex, and may not transfer to later models. Interpreter offloading helps arithmetic; it does not by itself fix a wrong program.
• License: CC0 1.0 on the abs page (2026-11-11; icon zero-1.0.png; http://creativecommons.org/publicdomain/zero/1.0/). Submitted 18 Nov 2022 (v1); last revised 27 Jan 2023 (v2). Sources: [S1] https://arxiv.org/abs/2211.10435, v2 (27 Jan 2023). checked_on: 2026-11-11.

When to apply

Use when a quantitative problem is better solved by generating code than by doing the arithmetic in free text.

Facts

• id: F1 claim: 'PAL uses a language model to generate a program and a Python interpreter to compute the answer.' confidence: medium date: 2026-11-11 sources: [S1] license: CC0 1.0
• id: F2 claim: 'The authors evaluate on 13 mathematical, symbolic, and algorithmic reasoning tasks.' confidence: medium date: 2026-11-11 sources: [S1] license: CC0 1.0
• id: F3 claim: 'They report that PAL using Codex surpasses PaLM-540B with chain-of-thought by an absolute 15% top-1 on GSM8K.' confidence: medium date: 2026-11-11 sources: [S1] license: CC0 1.0

Sources

• [S1] arXiv 2211.10435, https://arxiv.org/abs/2211.10435. License: CC0 1.0. v2 (27 Jan 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC0 1.0.

@misc{iil_note_nhjcdbws,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{PAL: Program-aided Language Models}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2211-10435-pal/},
  note = {Permanent ID iil:note:nhjcdbws. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2211-10435-pal.md.