← All notes

Answer: Inspect is an open-source framework for LLM evaluations from the UK AI Security Institute. It builds evaluations from datasets, agents, tools, and scorers, and can run external agents like Codex CLI and Gemini CLI.

Building blocks

• Composable datasets, agents, tools, and scorers make evaluations easy to write and reuse.
• Over 200 pre-built evaluations run on any model.
• A sandboxing system runs untrusted model code in Docker, Kubernetes, and other systems.
• Inspect View is a web-based tool for monitoring and visualizing evaluations.

Scorers

Scorers evaluate whether solvers found the right output for the dataset's target. Common forms:

1. Extract a specific answer from the completion using heuristics.
2. Apply a text similarity algorithm against the target.
3. Use another model to assess whether the completion satisfies a description of the ideal answer.
4. Apply another rubric, such as whether a file format is valid.

Scorers define metrics that aggregate scores, for example accuracy() or mean().

See also

• ao-evals-agent-benchmarks-and-judges
• ao-evals-llm-judges

Claims

• id: C1 claim: Inspect is a framework for frontier AI evaluations. source: S1 evidence: "Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute"

• id: C2 claim: Inspect can run external agents like Codex CLI. source: S1 evidence: "the ability to run arbitrary external agents like Claude Code, Codex CLI, and Gemini CLI."

• id: C3 claim: Scorers check output against the target. source: S2 evidence: "Scorers evaluate whether solvers were successful in finding the right"

• id: C4 claim: Scorers can use another model as judge. source: S2 evidence: "Using another model to assess whether the model's completion satisfies a description"

Sources

• [S1] Inspect. UK AI Security Institute, Inspect docs (GitHub). https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd . License: MIT (UKGovernmentBEIS/inspect_ai LICENSE). Accessed 2026-11-09.

• [S2] Scorers. UK AI Security Institute, Inspect docs (GitHub). https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/scorers.qmd . License: MIT (UKGovernmentBEIS/inspect_ai LICENSE). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: GitHub repository docs under Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0) or MIT License (https://opensource.org/licenses/MIT), as listed per source; copyright notices remain with the original authors. Text was condensed and changed from the originals.

@misc{iil_note_qw5rpj8q,
  author = {{iinventedlight}},
  title = {{Checking agent work with Inspect: tasks, agents, scorers}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/ao-evals-inspect-scorers/},
  note = {Permanent ID iil:note:qw5rpj8q. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/ao-evals-inspect-scorers.md.