---
source: https://iinventedlight.com/library/notes/ao-evals-inspect-scorers.md
title: "Checking agent work with Inspect: tasks, agents, scorers"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 or MIT (repo docs; LICENSE file verified in each GitHub repo); prose restates cited source lines in own words"
license_family: agentops-green
attribution: "Summary by iinventedlight (AI-assisted). Sources: UK AI Security Institute, Inspect docs (GitHub), Inspect, MIT (UKGovernmentBEIS/inspect_ai LICENSE); UK AI Security Institute, Inspect docs (GitHub), Scorers, MIT (UKGovernmentBEIS/inspect_ai LICENSE)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No tool or code was run for this note; only the fetched repository documentation was read."
category: agent-operations
topic_area: "agent operations: evals and verification"
tags: [evals, inspect, scorers, llm-as-judge]
client: Librarian
sources:
  - id: S1
    title: "Inspect"
    publisher: UK AI Security Institute, Inspect docs (GitHub)
    url: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd
    accessed: 2026-11-09
    license: "MIT (UKGovernmentBEIS/inspect_ai LICENSE)"
  - id: S2
    title: "Scorers"
    publisher: UK AI Security Institute, Inspect docs (GitHub)
    url: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/scorers.qmd
    accessed: 2026-11-09
    license: "MIT (UKGovernmentBEIS/inspect_ai LICENSE)"
---
# Checking agent work with Inspect: tasks, agents, scorers

**Answer:** Inspect is an open-source framework for LLM evaluations from the UK AI Security Institute. It builds evaluations from datasets, agents, tools, and scorers, and can run external agents like Codex CLI and Gemini CLI.

## Building blocks

- Composable datasets, agents, tools, and scorers make evaluations easy to write and reuse.
- Over 200 pre-built evaluations run on any model.
- A sandboxing system runs untrusted model code in Docker, Kubernetes, and other systems.
- Inspect View is a web-based tool for monitoring and visualizing evaluations.

## Scorers

Scorers evaluate whether solvers found the right output for the dataset's target. Common forms:
1. Extract a specific answer from the completion using heuristics.
2. Apply a text similarity algorithm against the target.
3. Use another model to assess whether the completion satisfies a description of the ideal answer.
4. Apply another rubric, such as whether a file format is valid.

Scorers define metrics that aggregate scores, for example `accuracy()` or `mean()`.

## See also

- [ao-evals-agent-benchmarks-and-judges](https://iinventedlight.com/library/notes/ao-evals-agent-benchmarks-and-judges/)
- [ao-evals-llm-judges](https://iinventedlight.com/library/notes/ao-evals-llm-judges/)

## Claims

- id: C1
  claim: Inspect is a framework for frontier AI evaluations.
  source: S1
  evidence: "Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute"

- id: C2
  claim: Inspect can run external agents like Codex CLI.
  source: S1
  evidence: "the ability to run arbitrary external agents like Claude Code, Codex CLI, and Gemini CLI."

- id: C3
  claim: Scorers check output against the target.
  source: S2
  evidence: "Scorers evaluate whether solvers were successful in finding the right"

- id: C4
  claim: Scorers can use another model as judge.
  source: S2
  evidence: "Using another model to assess whether the model's completion satisfies a description"

## Sources

- [S1] Inspect. UK AI Security Institute, Inspect docs (GitHub). https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd . License: MIT (UKGovernmentBEIS/inspect_ai LICENSE). Accessed 2026-11-09.

- [S2] Scorers. UK AI Security Institute, Inspect docs (GitHub). https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/scorers.qmd . License: MIT (UKGovernmentBEIS/inspect_ai LICENSE). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: GitHub repository docs under Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0) or MIT License (https://opensource.org/licenses/MIT), as listed per source; copyright notices remain with the original authors. Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).