← All notes

Working with AI agents and, in future, superintelligence (SI)

Note, 2026-10-14. Status: reviewed. Tags: ai, agents, research. Authorship: synthesis by iinventedlight from the cited sources; text drafted with AI assistance (Grok Bot), reviewed by Cole Withington on 2026-10-14.

Summary

Work with AI agents goes best when you delegate tasks whose results can be checked, build verification before you scale up, give the agent the right context, and protect your own understanding when you need to learn the material [F1–F12]. "SI" in this note means superintelligence. No SI exists today, so the SI section covers research on supervising AI on tasks people can't easily judge, and is partly speculative [F13–F15].

Sections

Delegation

• Delegate work that can be checked automatically. Keep humans in the loop where it can't [F1].
• Humans keep the spec, the invariants and the taste. Write those down explicitly before an agent builds [F2].
• Raise autonomy step by step: start with small, reviewable changes and move the "autonomy slider" up over time [F3].

Verification

• How much an agent can usefully produce depends on how much you trust it. Build verification that the agent can run itself before running many agents in parallel [F4, F5].
• Verification checks correctness, not quality. Hard guardrails (architecture, lint/CI) are stronger than soft ones (rules, style guides) [F6].
• Test agent skills with small evals, using a judge from a different model to reduce bias [F7].
• Put budgets and permission checks on expensive agent actions [F8].

Context

• Context means what the agent knows about the user, the task, its tools and its memory. Treat it as something to design: collect, organize, compress and share it on purpose [F9].
• For very large inputs, letting a model work through the data with code and sub-calls beat compaction and retrieval baselines on long-context benchmarks [F10].
• Agents copy patterns already in a codebase, so the codebase acts as their memory [F11].

Skill formation

• In one randomized study, people learning a new library with a chat assistant scored lower on a later quiz than a control group, and were not significantly faster [F12].
• Interaction style mattered. Generating code and then asking for explanations went with the highest scores; full delegation went with the lowest [F12].
• Practical rule (our recommendation): first decide whether the goal is learning or getting the job done, and choose delegation or explanation to match.

Toward superintelligence (partly speculative)

• Research framing: supervising AI smarter than us is the same problem as supervising AI on hard tasks without ground truth [F13].
• Open problem: weak supervisors only partly bring out a strong model's abilities on reward-modeling tasks [F14].
• Speculation: the habits above (checkable delegation, independent verification, keeping human understanding) will still matter with much stronger systems. Nothing in these sources demonstrates this.

Facts

• id: F1 claim: Karpathy argues that ordinary programs automate what you can specify, while LLMs automate what you can verify, because labs train them with RL on checkable rewards. confidence: low date: 2026-04-29 sources: [S4] url: https://www.youtube.com/watch?v=96jN2OCOfLs checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F2 claim: Karpathy says humans still own taste, judgment, the spec and invariants. His example is an agent that linked accounts by email address instead of a persistent user ID. confidence: low date: 2026-04-29 sources: [S4] url: https://www.youtube.com/watch?v=96jN2OCOfLs checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F3 claim: Karpathy recommends partial-autonomy tools with an adjustable autonomy level and small, incremental AI changes that humans can verify quickly, rather than huge diffs. confidence: low date: 2025-06-19 sources: [S5] url: https://www.youtube.com/watch?v=LCEmiRjPEtQ checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F4 claim: Lauren Tan argues that agent output is limited by how much you trust the agents, not by how many you run, and that running many before you trust them gives sloppy PRs and regressions. confidence: low date: 2026-09-22 sources: [S6] url: https://www.youtube.com/watch?v=NjoZoUm85x0 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F5 claim: Tan recommends building verification locally, where you can watch it, before scaling to many cloud agents. confidence: low date: 2026-09-26 sources: [S7] url: https://www.youtube.com/watch?v=7jeWwLvx0F0 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F6 claim: Tan says verification proves correctness but not quality. She ranks enforcement from strongest to weakest as architecture, then lint/CI, then rules and skills, then human style guides. confidence: low date: 2026-09-22 sources: [S6, S7] url: https://www.youtube.com/watch?v=NjoZoUm85x0 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement) note: Two talks by the same speaker; not independent.
• id: F7 claim: Tan maintains agent skills with evals (a rubric plus blind runs) and uses a judge from a different model to reduce bias. confidence: low date: 2026-09-26 sources: [S7] url: https://www.youtube.com/watch?v=7jeWwLvx0F0 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F8 claim: RSIGym wraps expensive agent actions as metered services, checking permissions and the remaining budget on every call. confidence: medium date: 2026-10-07 sources: [S9] url: https://arxiv.org/abs/2610.10310 checked_on: 2026-10-14 license: CC BY 4.0
• id: F9 claim: Context Engineering 2.0 is a position paper with no experiments. It frames context as information about relevant entities (user, tools, environment, memory) and discusses how to collect, manage and use it. confidence: medium date: 2025-10-30 sources: [S3] url: https://arxiv.org/abs/2510.26493 checked_on: 2026-10-14 license: CC BY 4.0
• id: F10 claim: Recursive Language Models put a long input in a code environment and let the model inspect it and call sub-models on pieces. The authors report handling inputs over 10 million tokens and beating compaction and retrieval baselines on four benchmarks. confidence: medium date: 2025-12-31 sources: [S2] url: https://arxiv.org/abs/2512.24601 checked_on: 2026-10-14 license: CC BY 4.0 note: The eval sets are small. Mean cost exceeds the median because of outlier runs, and the method only consistently helps beyond about 16K tokens.
• id: F11 claim: Tan observes that agents copy existing codebase patterns, so a single workaround can spread quickly. confidence: low date: 2026-09-22 sources: [S6] url: https://www.youtube.com/watch?v=NjoZoUm85x0 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F12 claim: In a randomized study (n=52) of developers learning an unfamiliar Python library, the group with a chat AI scored about 17% lower on a later no-AI quiz and was not significantly faster. Generating code and then asking for explanations went with the highest scores; full delegation went with the lowest. confidence: low date: 2026-01-28 sources: [S1] url: https://arxiv.org/abs/2601.20245 checked_on: 2026-10-14 license: CC BY 4.0 (our restatement; source is CC BY-NC-ND, not quoted) note: A single one-hour task with a chat interface (no agentic tools tested). The interaction-pattern clusters are small and were defined after the fact. Rated low only because the license is restricted; the study itself is a randomized controlled experiment (arXiv preprint).
• id: F13 claim: Jan Leike treats supervising smarter-than-human AI as the same problem as supervising AI on hard tasks that have no ground-truth labels. confidence: low date: 2024-09-10 sources: [S8] url: https://www.youtube.com/watch?v=PuESFNSh_Qo checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F14 claim: Leike reports that weak-to-strong supervision generalized somewhat on NLP tasks but was nearly flat on reward-modeling tasks. He reads this as some evidence that plain human-feedback training may not scale to much stronger models (an argument by analogy). confidence: low date: 2024-09-10 sources: [S8] url: https://www.youtube.com/watch?v=PuESFNSh_Qo checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)
• id: F15 claim: Leike reports that training a model to make its solutions checkable helped time-limited humans judge them more accurately, at a "legibility tax" in performance. confidence: low date: 2024-09-10 sources: [S8] url: https://www.youtube.com/watch?v=PuESFNSh_Qo checked_on: 2026-10-14 license: CC BY 4.0 (our restatement)

Pointers

• answers: Interaction patterns and per-question-type results in the skill-formation RCT where: https://arxiv.org/abs/2601.20245 (§4–5); data https://github.com/safety-research/how-ai-impacts-skill-formation access: open (NC-ND; read, do not copy)
• answers: Reference implementation of Recursive Language Models where: https://github.com/alexzhang13/rlm access: open

Disagreements

• Speed: Tan and Karpathy describe large productivity gains for experts [F4]. The skill-formation RCT found no significant speedup for learners [F12]. The settings differ: experts using agents versus novices using chat.

Markdown

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/working-with-superintelligence-and-ai-agents.md.