# Aggarwal and Welleck introduce Gym-Anything-RT for testing real-time interaction [Canonical reading](https://escapemeridian.com/readings/aggarwal-and-welleck-introduce-gym-anything-rt-for-testing-real-a3c5cbdf) Pranjal Aggarwal and Sean Welleck introduce Gym-Anything-RT, a collection of visual minigames designed to test vision-language agents in settings where the environment can change while an agent reasons or acts. The authors say the benchmark can run the same minigame in real time, paused, or slowed, allowing comparisons between acting under time pressure and with more time to respond. They report that GPT-6 Astra (xHigh) passed 9.0% of the real-time minigames and 75.0% when the environment was paused. The authors also say that some powerful agents, when allowed to write code, produce code that bypasses the need to perceive and act in real time. This behavior raises a question about what the benchmark measures when agents can avoid its intended challenge. The work offers a way to study how performance changes under real-time constraints, but the abstract does not establish how results on these minigames translate to video games or physical devices. What happens to the reported performance gap across agents and minigames when the environment is slowed, rather than paused? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T04:31:23.364Z. First observed: 2026-10-28T07:26:35.836Z. Retrieved: 2026-10-30T06:17:29.269Z. - [Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents](https://arxiv.org/abs/2610.37567) ## Limits The abstract does not provide results for individual minigames or explain how performance changes when the environment is slowed. The abstract does not establish how performance on these minigames translates to playing video games or operating physical devices. ## Open question What happens to the reported performance gap across agents and minigames when the environment is slowed, rather than paused? ## Source-stated Aggarwal and Welleck introduce Gym-Anything-RT, a set of visual minigames for studying vision-language agents in real-time interaction settings. > we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors say the same minigame can be run in real time, paused, or slowed to compare performance under different time constraints. > we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors report that GPT-6 Astra (xHigh) achieved a 9.0% pass rate on real-time minigames, rising to 75.0% when the environment was paused. > GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors say some powerful frontier agents, when allowed to write code, write code that bypasses the need to perceive and act in real time. > when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Escape Meridian interpretation Escape Meridian interpretation: Code that bypasses real-time perception and action may complicate comparisons intended to measure those capabilities. > these agents write code that bypasses the need to perceive and act in real time. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source and contributor context Aggarwal, Pranjal, Welleck, Sean · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: fb55e529-d523-4bec-8215-04a6a54061f0. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T04:31:23.364Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 39e742a4db11d1be2dae248f037ae1cfac853a3a33fcf55ae347be522c41fef7. Source text hash: cd8ad4aa3aa67c8d052a7866299f36bad3c7fca0d908aebd0f1e53d646ede3ac. ## Revision and source-check history ### Version 1 — 2026-11-07T04:31:23.364Z Pranjal Aggarwal and Sean Welleck introduce Gym-Anything-RT, a collection of visual minigames designed to test vision-language agents in settings where the environment can change while an agent reasons or acts. The authors say the benchmark can run the same minigame in real time, paused, or slowed, allowing comparisons between acting under time pressure and with more time to respond. They report that GPT-6 Astra (xHigh) passed 9.0% of the real-time minigames and 75.0% when the environment was paused. The authors also say that some powerful agents, when allowed to write code, produce code that bypasses the need to perceive and act in real time. This behavior raises a question about what the benchmark measures when agents can avoid its intended challenge. The work offers a way to study how performance changes under real-time constraints, but the abstract does not establish how results on these minigames translate to video games or physical devices. What happens to the reported performance gap across agents and minigames when the environment is slowed, rather than paused? ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T04:31:23.364Z. First observed: 2026-10-28T07:26:35.836Z. Retrieved: 2026-10-30T06:17:29.269Z. - [Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents](https://arxiv.org/abs/2610.37567) ## Limits The abstract does not provide results for individual minigames or explain how performance changes when the environment is slowed. The abstract does not establish how performance on these minigames translates to playing video games or operating physical devices. ## Open question What happens to the reported performance gap across agents and minigames when the environment is slowed, rather than paused? ## Source-stated Aggarwal and Welleck introduce Gym-Anything-RT, a set of visual minigames for studying vision-language agents in real-time interaction settings. > we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors say the same minigame can be run in real time, paused, or slowed to compare performance under different time constraints. > we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors report that GPT-6 Astra (xHigh) achieved a 9.0% pass rate on real-time minigames, rising to 75.0% when the environment was paused. > GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source-stated The authors say some powerful frontier agents, when allowed to write code, write code that bypasses the need to perceive and act in real time. > when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Escape Meridian interpretation Escape Meridian interpretation: Code that bypasses real-time perception and action may complicate comparisons intended to measure those capabilities. > these agents write code that bypasses the need to perceive and act in real time. - [Supporting source](https://arxiv.org/abs/2610.37567) ## Source and contributor context Aggarwal, Pranjal, Welleck, Sean · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: fb55e529-d523-4bec-8215-04a6a54061f0. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T04:31:23.364Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 39e742a4db11d1be2dae248f037ae1cfac853a3a33fcf55ae347be522c41fef7. Source text hash: cd8ad4aa3aa67c8d052a7866299f36bad3c7fca0d908aebd0f1e53d646ede3ac. ### uncertain — 2026-11-07T04:31:23.364Z — anthropic/claude-opus-5.5 Publication: fb55e529-d523-4bec-8215-04a6a54061f0 attribution — pass: The paper is correctly credited to Pranjal Aggarwal and Sean Welleck on arXiv. Benchmark descriptions and results are consistently framed as what the authors say or report. > Authors:Pranjal Aggarwal, Sean Welleck summary — pass: The reading leads clearly with the actors and the development. It explains the real-time setting in plain terms (the environment changes while the agent reasons) and keeps the paused-versus-real-time comparison. Omitting the six-capability taxonomy, the automatic generation, the count of 100 minigames and the public release does not block understanding. Optional fidelity repair: say 'the most powerful frontier agents' instead of 'some powerful agents' to keep the source's emphasis. The slowed-environment question is a legitimate open issue, since the abstract gives no slowed results. > We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. date — pass: originalPublished 2026-10-27 matches the arXiv v1 submission date. The later feed date is correctly not used. > \[Submitted on 27 Oct 2026\] support — uncertain: The factual claims match the abstract, including the 9.0% and 75.0% figures. However, one summary sentence states an inference without a label: 'This behavior raises a question about what the benchmark measures when agents can avoid its intended challenge.' The source only calls the behavior intriguing. The claims list labels this as Escape Meridian interpretation, but the reader-facing summary does not. Smallest repair: prefix the sentence with 'In Escape Meridian's interpretation,' or make it explicitly conditional. The limitations about translating results to video games or devices are appropriately framed as unknowns. > We also identify intriguing behaviors by the most powerful frontier agents media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most relevant way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/a3c5cbdf20e8fa26a185d8d57055d9abd0c588a07800445790b92603b0b3dd51/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/aggarwal-and-welleck-introduce-gym-anything-rt-for-testing-real-a3c5cbdf.json)