# Sayash Kapoor and Arvind Narayanan argue for open-world evaluations [Canonical reading](https://escapemeridian.com/readings/sayash-kapoor-and-arvind-narayanan-argue-for-open-world-15ccc4b2) Published April 16, 2026, Sayash Kapoor and Arvind Narayanan argue that open-world evaluations can complement benchmarks by testing AI agents on long, messy tasks in real-world settings. Benchmarks can overstate capabilities when systems are optimized for narrowly specified tests, and understate them when incidental obstacles such as CAPTCHA block a task. The authors say open-world evaluations can expose capabilities and failure modes that benchmarks miss, but they are less reproducible, harder to compare, and often need human intervention and expert review. The authors introduce CRUX, a project to conduct these evaluations regularly. In its first experiment, an AI agent developed and submitted an iOS app; the app was published after the agent made errors, including fabricating a phone number, and needed human intervention. The authors call this an early warning about possible app store spam, not evidence that agents can already complete the process without help. They say the experiment cost about $1,000, most of it monitoring the app’s review status. Their conclusion is that evaluators should specify what they measure, document human help, analyze and release logs, conduct dry runs, and measure cost. How often will the agent succeed, and how much human help will it need, across repeated app submissions? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Arvind Narayanan and Sayash Kapoor · Article Original source date: Apr 16, 2026 (2026-04-16T17:47:29.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-07T11:18:23.305Z. First observed: 2026-10-24T01:04:15.084Z. Retrieved: 2026-10-24T01:05:41.050Z. - [Open-world evaluations for measuring frontier AI capabilities](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Limits Text as retrieved on October 24, 2026; originally published April 16, 2026. The app experiment was a single evaluation; the authors describe the result as an early indication and warning, not a measure of performance across repeated submissions. The agent needed human intervention and made errors, including fabricating a fictional phone number; the authors say they could not test how it would perform in communications with Apple reviewers. ## Open question How often will the agent succeed, and how much human help will it need, across repeated app submissions? ## Source-stated Kapoor and Narayanan argue that benchmarks can misrepresent AI capabilities in both directions, and that open-world evaluations offer a complementary way to assess them. > But benchmarks can both overestimate and underestimate progress. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors say open-world evaluations can offer early warnings about emerging capabilities, while acknowledging that these evaluations lack reproducibility and standardization. > Despite these limitations, we think such evaluations are important for collecting evidence about AI capabilities. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated In CRUX’s first experiment, an AI agent developed and submitted an iOS app, which was published after errors and human intervention. > The agent was successful after making two errors, one of which required manual intervention - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors warn that app store operators may face a surge in agent-submitted spam apps. > App store operators should prepare for and police spam submissions, as they might soon see thousands of applications submitted autonomously using agents. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors recommend documenting human intervention and analyzing agent logs as part of open-world evaluations. > To increase the usefulness of open-world evaluations, evaluators should specify what and how much human intervention is allowed, release logs collected while the agent was solving the task, and analyze logs to report what an agent did in the course of solving the task. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source and contributor context Arvind Narayanan and Sayash Kapoor · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Editorial provenance Version 1. Publication ID: ed571f16-0941-6787-2fae-2103a212be4c. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T11:13:43.449Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 518852dfaf4e699c63731811548b38935ae5a785e0454c7ec67ca8bd6d9c54d9. Source text hash: f2b5573fecb025ed7f7f4a9d62a613eae227a835139de3acaaec7d98fd10180e. ## Revision and source-check history ### Version 1 — 2026-11-07T11:18:23.305Z Published April 16, 2026, Sayash Kapoor and Arvind Narayanan argue that open-world evaluations can complement benchmarks by testing AI agents on long, messy tasks in real-world settings. Benchmarks can overstate capabilities when systems are optimized for narrowly specified tests, and understate them when incidental obstacles such as CAPTCHA block a task. The authors say open-world evaluations can expose capabilities and failure modes that benchmarks miss, but they are less reproducible, harder to compare, and often need human intervention and expert review. The authors introduce CRUX, a project to conduct these evaluations regularly. In its first experiment, an AI agent developed and submitted an iOS app; the app was published after the agent made errors, including fabricating a phone number, and needed human intervention. The authors call this an early warning about possible app store spam, not evidence that agents can already complete the process without help. They say the experiment cost about $1,000, most of it monitoring the app’s review status. Their conclusion is that evaluators should specify what they measure, document human help, analyze and release logs, conduct dry runs, and measure cost. How often will the agent succeed, and how much human help will it need, across repeated app submissions? ## Original source Arvind Narayanan and Sayash Kapoor · Article Original source date: Apr 16, 2026 (2026-04-16T17:47:29.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-07T11:18:23.305Z. First observed: 2026-10-24T01:04:15.084Z. Retrieved: 2026-10-24T01:05:41.050Z. - [Open-world evaluations for measuring frontier AI capabilities](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Limits Text as retrieved on October 24, 2026; originally published April 16, 2026. The app experiment was a single evaluation; the authors describe the result as an early indication and warning, not a measure of performance across repeated submissions. The agent needed human intervention and made errors, including fabricating a fictional phone number; the authors say they could not test how it would perform in communications with Apple reviewers. ## Open question How often will the agent succeed, and how much human help will it need, across repeated app submissions? ## Source-stated Kapoor and Narayanan argue that benchmarks can misrepresent AI capabilities in both directions, and that open-world evaluations offer a complementary way to assess them. > But benchmarks can both overestimate and underestimate progress. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors say open-world evaluations can offer early warnings about emerging capabilities, while acknowledging that these evaluations lack reproducibility and standardization. > Despite these limitations, we think such evaluations are important for collecting evidence about AI capabilities. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated In CRUX’s first experiment, an AI agent developed and submitted an iOS app, which was published after errors and human intervention. > The agent was successful after making two errors, one of which required manual intervention - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors warn that app store operators may face a surge in agent-submitted spam apps. > App store operators should prepare for and police spam submissions, as they might soon see thousands of applications submitted autonomously using agents. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source-stated The authors recommend documenting human intervention and analyzing agent logs as part of open-world evaluations. > To increase the usefulness of open-world evaluations, evaluators should specify what and how much human intervention is allowed, release logs collected while the agent was solving the task, and analyze logs to report what an agent did in the course of solving the task. - [Supporting source](https://www.normaltech.ai/p/open-world-evaluations-for-measuring) ## Source and contributor context Arvind Narayanan and Sayash Kapoor · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Editorial provenance Version 1. Publication ID: ed571f16-0941-6787-2fae-2103a212be4c. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T11:13:43.449Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 518852dfaf4e699c63731811548b38935ae5a785e0454c7ec67ca8bd6d9c54d9. Source text hash: f2b5573fecb025ed7f7f4a9d62a613eae227a835139de3acaaec7d98fd10180e. ### passed — 2026-11-07T11:13:43.449Z — anthropic/claude-opus-5.5 Publication: ed571f16-0941-6787-2fae-2103a212be4c attribution — pass: The post's byline names Sayash Kapoor and Arvind Narayanan, so attributing the argument to them is supported. The piece is one collaborative paper, not a listing. The paper has 17 authors, which the reading does not mention, but this does not misattribute the argument. > Sayash Kapoor and Arvind Narayanan Apr 16, 2026 summary — pass: The reading accurately conveys the thesis that benchmarks can over- and underestimate capabilities. It also covers the stated limits of open-world evaluations, the CRUX iOS app experiment and its errors, the cost, the spam warning and the best practices. Open-world evaluation is explained in plain language, and benchmark and agent are defined. The authors' own framing that the agent was extremely close to full automation is softened but not contradicted. The question follows from the single-run limitation. > In short, the agent couldn’t completely automate the task, but it was extremely close to being able to do so. date — pass: The reading gives April 16, 2026, which matches the dateline and the originalPublished instant (UTC). The retrieval date is disclosed separately. There is no hindsight framing, and the spam surge is reported as the authors' forecast. The kind is publisher-page, so no archive disclosure is needed. modifiedAt equals the publication time. > Apr 16, 2026 support — pass: Each claim is grounded in source text: the two-directional benchmark critique, early-warning value with reproducibility limits, two errors including a fabricated phone number, and one manual intervention. The source also supports the roughly $1,000 cost driven by monitoring, the spam warning framed as a possibility, the recommended practices, and the untested communication with reviewers. No unsupported benefits or costs are implied. > The agent was successful after making two errors, one of which required manual intervention media — pass: No media is selected and no verified candidates were supplied, so there is no media identity or relevance issue. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/15ccc4b244b41ba90ee00d82f3aed28593cfe34ba8aaa4f6716977932187a4cd/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/sayash-kapoor-and-arvind-narayanan-argue-for-open-world-15ccc4b2.json)