← Reference notes

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Summary

Research ideation as a standalone task with three RL-trained roles (gap finder, cross-domain innovator, report writer) over a 2.77M-idea store. Reports +14% over the best open baseline, mostly novelty; RL adds little, proposals were never executed, and judging is subjective.

When to apply

• An agent is asked to propose research directions / solutions for a stated problem + challenge, before any experiments.
• Especially when novelty matters and you can retrieve from other fields' literature.

Procedure

1. Write the problem setting (input/output) and the challenge current solutions fail on.
2. Gap finder: retrieve close prior work, list concrete limitations.
3. Innovator: query a cross-domain idea store for problems with an analogous challenge; extract the mechanism and why it worked there.
4. Report writer: develop into a proposal (problem formalisation, central novelty, distinction from prior mechanisms, testable decision rules), try to falsify novelty against same-problem prior art.
5. Evaluate with an LLM-judge rubric (novelty vs all / pre-cutoff literature, feasibility, soundness, impact) and, ideally, human review.

Costs and limits

• Main judge is Qwen3.6-27B — same family as the main backbone; authors report negligible self-preference, but proposal scoring is subjective (r = 0.56 between experts) (§5.4).
• Measures proposal quality only; no proposals were executed, so real-world feasibility is untested (paper notes novelty may trade off with feasibility, §2).
• RL training adds little over the untrained pipeline.

Evidence (detail)

• Qwen3.6-27B backbone: +14.0% overall vs strongest open autoresearch baseline, mainly from novelty; beats Claude Code SDK (Claude-4.8-Opus) by 5.9% and Codex SDK (GPT-5.4) by 5.1% on the same metrics (Abstract; §5.1, Table 1, held-out 277-paper test split).
• Base system novelty gains up to +23.1% on Qwen3.6-27B; quality gains modest (+1.8% to +2.5%) (§5.1).
• RL training: role rewards 56.6→66.7% (gap finder), 58.8→69.5% (innovator), 78.6→89.1% (writer); Trained vs Base only +1.9% novelty, +0.4% quality (§5.1).
• Cross-domain retrieval improves all novelty measures (e.g. All +5.3%, Cutoff +3.7%; in-domain novelty +30.7% in ablation) (§5.3, Fig. 3).
• Human blind pairwise eval on 50 papers, 2 annotators: Trained significantly beats every baseline except DeepScientist; vs Base not significant (§5.2, Fig. 2b).
• Judge validity: inter-annotator r = 0.56; top LLM judges vs human average r = 0.51–0.59 (§5.4, Fig. 4).

Facts

• id: F1 claim: Three roles: gap finder (limits of prior work), innovator (borrows mechanisms via cross-domain retrieval), report writer (full proposal). confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 Abstract] license: CC BY 4.0
• id: F2 claim: Backed by the Svalbard Idea Vault of 2.77M decomposed ideas, evaluated with a temporal cutoff against 15K later papers. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 Abstract] license: CC BY 4.0
• id: F3 claim: With a Qwen3.6-27B backbone, +14.0% overall vs the strongest open baseline, mainly from novelty; +5.9% vs Claude Code SDK and +5.1% vs Codex SDK. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 Abstract; §5.1, Table 1] license: CC BY 4.0
• id: F4 claim: RL training adds only +1.9% novelty and +0.4% quality over the untrained pipeline. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 §5.1] license: CC BY 4.0
• id: F5 claim: Cross-domain retrieval improves all novelty measures in ablation. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 §5.3, Fig. 3] license: CC BY 4.0
• id: F6 claim: Human blind eval (50 papers, 2 annotators): trained system beats all baselines except DeepScientist; not significant vs base. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 §5.2, Fig. 2b] license: CC BY 4.0
• id: F7 claim: Inter-annotator agreement r = 0.56; top LLM judges vs human average r = 0.51–0.59. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 §5.4, Fig. 4] license: CC BY 4.0
• id: F8 claim: Proposals were not executed, so real-world feasibility is untested. confidence: medium # single source: author-reported results date: 2026-10-14 sources: [S1 §2; limits] license: CC BY 4.0

Related

• 2610.10310 RSIGym; 2606.01444 self-revising discovery systems; The AI Scientist (Lu et al., 2024); DeepScientist.

Review notes

• Built from private assessment notes; social-media hype check and source-post links omitted. All content is paraphrase; no verbatim quotes.
• Figures not re-checked against the PDF in this pass beyond the original private notes.

For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2610-04074-ideascientist-grounded-ideation.md.