← All notes

Summary

This paper presents MultiMedQA, a combined question-answering benchmark, and evaluates PaLM, a 540-billion parameter language model, plus its instruction-tuned variant Flan-PaLM. Using a combination of prompting strategies, Flan-PaLM achieves 67.6% accuracy on MedQA, surpassing prior state-of-the-art by over 17%, and state-of-the-art accuracy on every MultiMedQA multiple-choice dataset. Human evaluation reveals key gaps in Flan-PaLM responses. They then introduce instruction prompt tuning, a parameter-efficient way to align a model to a new domain using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. [S1]

Claims

• id: C1 claim: 'Flan-PaLM reaches 67.6% accuracy on MedQA (USMLE-style questions).' source: S1 evidence: "67.6% accuracy on MedQA (US Medical License Exam questions)"
• id: C2 claim: 'That MedQA result surpasses the prior reported state of the art by over 17%.' source: S1 evidence: "surpassing prior state-of-the-art by over 17%"
• id: C3 claim: 'Med-PaLM, after instruction prompt tuning, still remains inferior to clinicians.' source: S1 evidence: "The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians"

Verification

• Code available: The paper does not point to a public training-code repository. Not run.
• Reproduced: not reproduced (2026-11-11). No public code URL was found on the abs page or in the HTML sections read.
• Related work: No live library note is a paper-verification page for this work.
• Limits: The authors say the combined benchmark is not exhaustive, and they discuss limits of the evaluation setup and of current model behavior, including that Med-PaLM still trails clinicians.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 26 Dec 2022 (v1). Sources: [S1] https://arxiv.org/abs/2212.13138, v1 (26 Dec 2022). checked_on: 2026-11-11.

When to apply

Use when asking how a language model was scored on MultiMedQA and how instruction prompt tuning produced Med-PaLM. Not guidance for care.

Facts

• id: F1 claim: 'Flan-PaLM reaches 67.6% accuracy on MedQA (USMLE).' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors report that this MedQA score surpasses the prior state of the art by over 17%.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'After instruction prompt tuning, Med-PaLM still remains inferior to clinicians.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2212.13138, https://arxiv.org/abs/2212.13138. License: CC BY 4.0. v1 (26 Dec 2022). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_g1wxpd58,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Large Language Models Encode Clinical Knowledge}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2212-13138-llms-encode-clinical-knowledge/},
  note = {Permanent ID iil:note:g1wxpd58. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2212-13138-llms-encode-clinical-knowledge.md.