---
title: "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-12; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2207-01206-webshop.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan, WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: "Not reproduced. Abs page and HTML key sections read 2026-11-12. Named code URLs checked for HTTP 200; code not run."
category: arxiv
topic_area: 'ai-research: agents'
tags: arxiv, agents, web-agents, grounded-language, webshop
arxiv_id: 2207.01206
alias: 2207-01206-webshop
sources:
  - id: S1
    title: "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents"
    publisher: arXiv
    authors: "Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan"
    url: https://arxiv.org/abs/2207.01206
    version_read: "v4 (8 Feb 2023)"
    accessed: 2026-11-12
    license: 'CC BY 4.0'
---
# WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

## Summary
WebShop is a simulated shopping website for grounded language agents, with 1.18 million real-world products and 12,087 crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate webpages and issue actions to find, customize, and purchase an item. The authors collected over 1,600 human demonstrations and trained agents with reinforcement learning, imitation learning, and pretrained image and language models. Their best model reached a 29% task success rate, above a 9.6% rule-based heuristic but well below 59% human expert performance. Agents trained in the simulator also showed non-trivial transfer when tried on amazon.com and ebay.com. Code, data, and demos are pointed to a project site. [S1]

## Claims
- id: C1
  claim: "WebShop is a simulated store with 1.18 million products and 12,087 crowd-sourced instructions."
  source: S1
  evidence: "1.18$ million real-world products and $12,087$ crowd-sourced text instructions"
- id: C2
  claim: "The best model reached 29% task success, versus 9.6% for heuristics and 59% for human experts."
  source: S1
  evidence: "Our best model achieves a task success rate of $29\%$"
- id: C3
  claim: "Code, data, and demos are pointed to a public project page."
  source: S1
  evidence: "https://webshop-pnlp.github.io"

## Verification
- Code available: Paper comments: https://webshop-pnlp.github.io and https://github.com/princeton-nlp/WebShop, both HTTP 200 on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Later web-agent evals include [2307-13854-webarena](https://iinventedlight.com/library/notes/2307-13854-webarena/). One of the authors later introduced [2210-03629-react-reasoning-and-acting](https://iinventedlight.com/library/notes/2210-03629-react-reasoning-and-acting/). This week's pointer notes cover Mind2Web, VisualWebArena, and WebVoyager.
- Limits: The 29% figure is for the authors' best agent in the simulator, not on the live web.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 4 Jul 2022 (v1); last revised v4 (8 Feb 2023). Sources: [S1] https://arxiv.org/abs/2207.01206, v4 (8 Feb 2023). checked_on: 2026-11-12.

## When to apply
Use when asking for an early scalable shopping environment for language agents, or how later web-agent benchmarks relate to WebShop.

## Facts
- id: F1
  claim: "WebShop provides 1.18 million products and 12,087 crowd-sourced instructions in a simulated store."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: "The authors report 29% success for their best model, 9.6% for a rule-based heuristic, and 59% for human experts."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: "The project page is https://webshop-pnlp.github.io, and the authors also released GitHub code."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2207.01206, https://arxiv.org/abs/2207.01206. License: CC BY 4.0. v4 (8 Feb 2023). Accessed 2026-11-12.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
