Sayash Kapoor and Arvind Narayanan argue for open-world evaluations

Sayash Kapoor and Arvind Narayanan argue for open-world evaluations

Published April 16, 2026, Sayash Kapoor and Arvind Narayanan argue that open-world evaluations can complement benchmarks by testing AI agents on long, messy tasks in real-world settings. Benchmarks can overstate capabilities when systems are optimized for narrowly specified tests, and understate them when incidental obstacles such as CAPTCHA block a task. The authors say open-world evaluations can expose capabilities and failure modes that benchmarks miss, but they are less reproducible, harder to compare, and often need human intervention and expert review. The authors introduce CRUX, a project to conduct these evaluations regularly. In its first experiment, an AI agent developed and submitted an iOS app; the app was published after the agent made errors, including fabricating a phone number, and needed human intervention. The authors call this an early warning about possible app store spam, not evidence that agents can already complete the process without help. They say the experiment cost about $1,000, most of it monitoring the app’s review status. Their conclusion is that evaluators should specify what they measure, document human help, analyze and release logs, conduct dry runs, and measure cost. How often will the agent succeed, and how much human help will it need, across repeated app submissions?

Publication: ed571f16-0941-6787-2fae-2103a212be4c

attribution — pass: The post's byline names Sayash Kapoor and Arvind Narayanan, so attributing the argument to them is supported. The piece is one collaborative paper, not a listing. The paper has 17 authors, which the reading does not mention, but this does not misattribute the argument.

Sayash Kapoor and Arvind Narayanan Apr 16, 2026

summary — pass: The reading accurately conveys the thesis that benchmarks can over- and underestimate capabilities. It also covers the stated limits of open-world evaluations, the CRUX iOS app experiment and its errors, the cost, the spam warning and the best practices. Open-world evaluation is explained in plain language, and benchmark and agent are defined. The authors' own framing that the agent was extremely close to full automation is softened but not contradicted. The question follows from the single-run limitation.

In short, the agent couldn’t completely automate the task, but it was extremely close to being able to do so.

date — pass: The reading gives April 16, 2026, which matches the dateline and the originalPublished instant (UTC). The retrieval date is disclosed separately. There is no hindsight framing, and the spam surge is reported as the authors' forecast. The kind is publisher-page, so no archive disclosure is needed. modifiedAt equals the publication time.

Apr 16, 2026

support — pass: Each claim is grounded in source text: the two-directional benchmark critique, early-warning value with reproducibility limits, two errors including a fabricated phone number, and one manual intervention. The source also supports the roughly $1,000 cost driven by monitoring, the spam warning framed as a possibility, the recommended practices, and the untested communication with reviewers. No unsupported benefits or costs are implied.

The agent was successful after making two errors, one of which required manual intervention

media — pass: No media is selected and no verified candidates were supplied, so there is no media identity or relevance issue.