WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Summary
WebShop is a simulated shopping website for grounded language agents, with 1.18 million real-world products and 12,087 crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate webpages and issue actions to find, customize, and purchase an item. The authors collected over 1,600 human demonstrations and trained agents with reinforcement learning, imitation learning, and pretrained image and language models. Their best model reached a 29% task success rate, above a 9.6% rule-based heuristic but well below 59% human expert performance. Agents trained in the simulator also showed non-trivial transfer when tried on amazon.com and ebay.com. Code, data, and demos are pointed to a project site. [S1]
Claims
- id: C1 claim: "WebShop is a simulated store with 1.18 million products and 12,087 crowd-sourced instructions." source: S1 evidence: "1.18$ million real-world products and $12,087$ crowd-sourced text instructions"
- id: C2 claim: "The best model reached 29% task success, versus 9.6% for heuristics and 59% for human experts." source: S1 evidence: "Our best model achieves a task success rate of $29\%$"
- id: C3 claim: "Code, data, and demos are pointed to a public project page." source: S1 evidence: "https://webshop-pnlp.github.io"
Verification
- Code available: Paper comments: https://webshop-pnlp.github.io and https://github.com/princeton-nlp/WebShop, both HTTP 200 on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Later web-agent evals include 2307-13854-webarena. One of the authors later introduced 2210-03629-react-reasoning-and-acting. This week's pointer notes cover Mind2Web, VisualWebArena, and WebVoyager.
- Limits: The 29% figure is for the authors' best agent in the simulator, not on the live web.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 4 Jul 2022 (v1); last revised v4 (8 Feb 2023). Sources: [S1] https://arxiv.org/abs/2207.01206, v4 (8 Feb 2023). checked_on: 2026-11-12.
When to apply
Use when asking for an early scalable shopping environment for language agents, or how later web-agent benchmarks relate to WebShop.
Facts
- id: F1 claim: "WebShop provides 1.18 million products and 12,087 crowd-sourced instructions in a simulated store." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F2 claim: "The authors report 29% success for their best model, 9.6% for a rule-based heuristic, and 59% for human experts." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F3 claim: "The project page is https://webshop-pnlp.github.io, and the authors also released GitHub code." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
Sources
- [S1] arXiv 2207.01206, https://arxiv.org/abs/2207.01206. License: CC BY 4.0. v4 (8 Feb 2023). Accessed 2026-11-12.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.
License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan, WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (CC BY 4.0).
Cite this
'iinventedlight (AI-assisted)'. 2026. "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents." iinventedlight Library, iil:note:6phy913f, version of 2026-11-12. https://iinventedlight.com/library/notes/2207-01206-webshop/. License: CC-BY-4.0.
@misc{iil_note_6phy913f,
author = {{'iinventedlight (AI-assisted)'}},
title = {{WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2207-01206-webshop/},
note = {Permanent ID iil:note:6phy913f. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2207-01206-webshop.md.