---
title: "Working with AI agents and, in future, superintelligence (SI)"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://iinventedlight.com/library/notes/working-with-superintelligence-and-ai-agents.md
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-10-14
status: reviewed
reviewed_on: 2026-10-14
checked_on: 2026-10-14
sources:
  - id: S1
    title: "How AI Impacts Skill Formation (arXiv 2601.20245, v2)"
    publisher: arXiv (Shen & Tamkin)
    url: https://arxiv.org/abs/2601.20245
    accessed: 2026-10-14
    license: "restricted: CC BY-NC-ND 4.0, restated, not copied"
  - id: S2
    title: "Recursive Language Models (arXiv 2512.24601, v3)"
    publisher: arXiv (Zhang, Kraska, Khattab)
    url: https://arxiv.org/abs/2512.24601
    accessed: 2026-10-14
    license: CC BY 4.0
  - id: S3
    title: "Context Engineering 2.0 (arXiv 2510.26493)"
    publisher: arXiv (Hua et al.)
    url: https://arxiv.org/abs/2510.26493
    accessed: 2026-10-14
    license: CC BY 4.0
  - id: S4
    title: "Andrej Karpathy: From Vibe Coding to Agentic Engineering (Sequoia, 2026)"
    publisher: Sequoia Capital (YouTube)
    url: https://www.youtube.com/watch?v=96jN2OCOfLs
    accessed: 2026-10-14
    license: "restricted: restated, not copied"
  - id: S5
    title: "Andrej Karpathy: Software Is Changing (Again) (YC AI Startup School, 2025)"
    publisher: Y Combinator (YouTube)
    url: https://www.youtube.com/watch?v=LCEmiRjPEtQ
    accessed: 2026-10-14
    license: "restricted: restated, not copied"
  - id: S6
    title: "How I Shipped 2000 PRs Last Month: Trusting AI Agents (Lauren Tan, 2026)"
    publisher: YouTube
    url: https://www.youtube.com/watch?v=NjoZoUm85x0
    accessed: 2026-10-14
    license: "restricted: restated, not copied"
  - id: S7
    title: "How a SpaceX AI Engineer Builds a Team of AI Agents (Lauren Tan, Perfology, 2026)"
    publisher: Perfology (YouTube)
    url: https://www.youtube.com/watch?v=7jeWwLvx0F0
    accessed: 2026-10-14
    license: "restricted: restated, not copied"
  - id: S8
    title: "Supervising AI on hard tasks (Jan Leike, FAR.AI, 2024)"
    publisher: FAR.AI (YouTube)
    url: https://www.youtube.com/watch?v=PuESFNSh_Qo
    accessed: 2026-10-14
    license: "restricted: restated, not copied"
  - id: S9
    title: "RSIGym (arXiv 2610.10310, v1)"
    publisher: arXiv (Meng et al.)
    url: https://arxiv.org/abs/2610.10310
    accessed: 2026-10-14
    license: CC BY 4.0
---
# Working with AI agents and, in future, superintelligence (SI)

*Note, 2026-10-14. Status: reviewed. Tags: ai, agents, research.*
*Authorship: synthesis by iinventedlight from the cited sources; text drafted with AI assistance (Grok Bot), reviewed by Cole Withington on 2026-10-14.*

## Summary
Work with AI agents goes best when you delegate tasks whose results can be checked, build verification before you scale up, give the agent the right context, and protect your own understanding when you need to learn the material [F1–F12]. "SI" in this note means superintelligence. No SI exists today, so the SI section covers research on supervising AI on tasks people can't easily judge, and is partly speculative [F13–F15].

## Sections
### Delegation
- Delegate work that can be checked automatically. Keep humans in the loop where it can't [F1].
- Humans keep the spec, the invariants and the taste. Write those down explicitly before an agent builds [F2].
- Raise autonomy step by step: start with small, reviewable changes and move the "autonomy slider" up over time [F3].

### Verification
- How much an agent can usefully produce depends on how much you trust it. Build verification that the agent can run itself before running many agents in parallel [F4, F5].
- Verification checks correctness, not quality. Hard guardrails (architecture, lint/CI) are stronger than soft ones (rules, style guides) [F6].
- Test agent skills with small evals, using a judge from a different model to reduce bias [F7].
- Put budgets and permission checks on expensive agent actions [F8].

### Context
- Context means what the agent knows about the user, the task, its tools and its memory. Treat it as something to design: collect, organize, compress and share it on purpose [F9].
- For very large inputs, letting a model work through the data with code and sub-calls beat compaction and retrieval baselines on long-context benchmarks [F10].
- Agents copy patterns already in a codebase, so the codebase acts as their memory [F11].

### Skill formation
- In one randomized study, people learning a new library with a chat assistant scored lower on a later quiz than a control group, and were not significantly faster [F12].
- Interaction style mattered. Generating code and then asking for explanations went with the highest scores; full delegation went with the lowest [F12].
- Practical rule (our recommendation): first decide whether the goal is learning or getting the job done, and choose delegation or explanation to match.

### Toward superintelligence (partly speculative)
- **Research framing:** supervising AI smarter than us is the same problem as supervising AI on hard tasks without ground truth [F13].
- **Open problem:** weak supervisors only partly bring out a strong model's abilities on reward-modeling tasks [F14].
- **Speculation:** the habits above (checkable delegation, independent verification, keeping human understanding) will still matter with much stronger systems. Nothing in these sources demonstrates this.

## Facts
- id: F1
  claim: Karpathy argues that ordinary programs automate what you can specify, while LLMs automate what you can verify, because labs train them with RL on checkable rewards.
  confidence: low
  date: 2026-04-29
  sources: [S4]
  url: https://www.youtube.com/watch?v=96jN2OCOfLs
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F2
  claim: Karpathy says humans still own taste, judgment, the spec and invariants. His example is an agent that linked accounts by email address instead of a persistent user ID.
  confidence: low
  date: 2026-04-29
  sources: [S4]
  url: https://www.youtube.com/watch?v=96jN2OCOfLs
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F3
  claim: Karpathy recommends partial-autonomy tools with an adjustable autonomy level and small, incremental AI changes that humans can verify quickly, rather than huge diffs.
  confidence: low
  date: 2025-06-19
  sources: [S5]
  url: https://www.youtube.com/watch?v=LCEmiRjPEtQ
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F4
  claim: Lauren Tan argues that agent output is limited by how much you trust the agents, not by how many you run, and that running many before you trust them gives sloppy PRs and regressions.
  confidence: low
  date: 2026-09-22
  sources: [S6]
  url: https://www.youtube.com/watch?v=NjoZoUm85x0
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F5
  claim: Tan recommends building verification locally, where you can watch it, before scaling to many cloud agents.
  confidence: low
  date: 2026-09-26
  sources: [S7]
  url: https://www.youtube.com/watch?v=7jeWwLvx0F0
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F6
  claim: Tan says verification proves correctness but not quality. She ranks enforcement from strongest to weakest as architecture, then lint/CI, then rules and skills, then human style guides.
  confidence: low
  date: 2026-09-22
  sources: [S6, S7]
  url: https://www.youtube.com/watch?v=NjoZoUm85x0
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
  note: Two talks by the same speaker; not independent.
- id: F7
  claim: Tan maintains agent skills with evals (a rubric plus blind runs) and uses a judge from a different model to reduce bias.
  confidence: low
  date: 2026-09-26
  sources: [S7]
  url: https://www.youtube.com/watch?v=7jeWwLvx0F0
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F8
  claim: RSIGym wraps expensive agent actions as metered services, checking permissions and the remaining budget on every call.
  confidence: medium
  date: 2026-10-07
  sources: [S9]
  url: https://arxiv.org/abs/2610.10310
  checked_on: 2026-10-14
  license: CC BY 4.0
- id: F9
  claim: Context Engineering 2.0 is a position paper with no experiments. It frames context as information about relevant entities (user, tools, environment, memory) and discusses how to collect, manage and use it.
  confidence: medium
  date: 2025-10-30
  sources: [S3]
  url: https://arxiv.org/abs/2510.26493
  checked_on: 2026-10-14
  license: CC BY 4.0
- id: F10
  claim: Recursive Language Models put a long input in a code environment and let the model inspect it and call sub-models on pieces. The authors report handling inputs over 10 million tokens and beating compaction and retrieval baselines on four benchmarks.
  confidence: medium
  date: 2025-12-31
  sources: [S2]
  url: https://arxiv.org/abs/2512.24601
  checked_on: 2026-10-14
  license: CC BY 4.0
  note: The eval sets are small. Mean cost exceeds the median because of outlier runs, and the method only consistently helps beyond about 16K tokens.
- id: F11
  claim: Tan observes that agents copy existing codebase patterns, so a single workaround can spread quickly.
  confidence: low
  date: 2026-09-22
  sources: [S6]
  url: https://www.youtube.com/watch?v=NjoZoUm85x0
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F12
  claim: In a randomized study (n=52) of developers learning an unfamiliar Python library, the group with a chat AI scored about 17% lower on a later no-AI quiz and was not significantly faster. Generating code and then asking for explanations went with the highest scores; full delegation went with the lowest.
  confidence: low
  date: 2026-01-28
  sources: [S1]
  url: https://arxiv.org/abs/2601.20245
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement; source is CC BY-NC-ND, not quoted)
  note: A single one-hour task with a chat interface (no agentic tools tested). The interaction-pattern clusters are small and were defined after the fact. Rated low only because the license is restricted; the study itself is a randomized controlled experiment (arXiv preprint).
- id: F13
  claim: Jan Leike treats supervising smarter-than-human AI as the same problem as supervising AI on hard tasks that have no ground-truth labels.
  confidence: low
  date: 2024-09-10
  sources: [S8]
  url: https://www.youtube.com/watch?v=PuESFNSh_Qo
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F14
  claim: Leike reports that weak-to-strong supervision generalized somewhat on NLP tasks but was nearly flat on reward-modeling tasks. He reads this as some evidence that plain human-feedback training may not scale to much stronger models (an argument by analogy).
  confidence: low
  date: 2024-09-10
  sources: [S8]
  url: https://www.youtube.com/watch?v=PuESFNSh_Qo
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)
- id: F15
  claim: Leike reports that training a model to make its solutions checkable helped time-limited humans judge them more accurately, at a "legibility tax" in performance.
  confidence: low
  date: 2024-09-10
  sources: [S8]
  url: https://www.youtube.com/watch?v=PuESFNSh_Qo
  checked_on: 2026-10-14
  license: CC BY 4.0 (our restatement)

## Pointers
- answers: Interaction patterns and per-question-type results in the skill-formation RCT
  where: https://arxiv.org/abs/2601.20245 (§4–5); data https://github.com/safety-research/how-ai-impacts-skill-formation
  access: open (NC-ND; read, do not copy)
- answers: Reference implementation of Recursive Language Models
  where: https://github.com/alexzhang13/rlm
  access: open

## Disagreements
- Speed: Tan and Karpathy describe large productivity gains for experts [F4]. The skill-formation RCT found no significant speedup for learners [F12]. The settings differ: experts using agents versus novices using chat.