arXiv paper introduces ScamWorld to test web agents against online fraud

arXiv paper introduces ScamWorld to test web agents against online fraud

Authors Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang and Xingjun Ma introduce ScamWorld, an interactive environment for testing how web agents handle online fraud. They argue that this is an underexplored problem: agents encounter fraudulent stores, fake investment platforms and advance-fee schemes, while scam websites are often short-lived and difficult to evaluate systematically. ScamWorld reconstructs reported scam websites in different scenarios, with documented fraud mechanisms made observable before an agent’s final action. The authors say this enables controlled, repeatable evaluation. In tests of agents powered by six frontier LLMs, baseline deception rates ranged from 11% to 35% across models. Agents proceeded more readily with payments than with requests for credentials or wallet information. The authors also report that some agents acknowledged suspicious conditions but proceeded anyway, sometimes interpreting a user’s instructions as permission to accept additional terms imposed by a website. The environment is presented as a reusable testbed for evaluating and stress-testing web agents against online fraud. Escape Meridian’s interpretation: the abstract reports these results but does not identify the six models, so the rates cannot be assigned to particular models from this evidence.

Publication: 15690c40-3e8d-43b8-8945-316c9af7c0c8

attribution — pass: The draft correctly names the six listed authors and attributes the environment, findings and the 'underexplored' framing to them. Results are hedged as reported by the authors, and the closing point is labeled as Escape Meridian interpretation.

Authors:Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma

summary — uncertain: The summary is faithful, but its central term is never explained. It does not say what a web agent is or why fraud matters for one. The abstract supplies this: agents browse, shop and manage accounts for users, so being deceived could mean paying or handing over credentials on a user's behalf. Without it, a non-specialist misses the stake, and 'baseline deception rate' has no context. Smallest repair: in the first sentence, add the source's description that web agents browse, shop and manage accounts on behalf of users. It could also note that the reconstructions span multiple languages. Add 'web agent' and 'frontier LLM' to the term library.

Web agents increasingly browse, shop, and manage accounts on behalf of their users.

date — pass: originalPublished 2026-11-03 matches the v1 submission date. The v2 revision on 4 Nov and the 5 Nov feed entry are later, so they correctly do not set the date.

[Submitted on 3 Nov 2026 (v1), last revised 4 Nov 2026 (this version, v2)]

support — pass: Each claim matches the abstract's wording, including the 11%–35% range, the payment-versus-credentials contrast and agents proceeding despite suspicion. The interpretation that model-specific rates are unavailable is correctly labeled and supported. The question follows from that gap without leading. The draft makes no unsupported claims about benefits or real-world reliability.

some agents acknowledge suspicious conditions yet proceed, at times treating the user's instructions as authorisation to accept additional terms imposed by the website.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.