Kadu, Ankush, Gopal, Athira, Singh, Niharika, Jain, Arpit, Krishnan, Aswanth · Oct 29, 2026
EnvOS checks whether agents can adapt when software changes
Markdown · JSON · Original source
The authors of EnvOS argue that evaluating computer-use agents requires checking not only the agent, but also whether software changes occurred as intended, reached the agent, and were graded correctly. Their framework checks each change, task solvability, and grading before an agent runs, then re-checks the verdict against archived evidence. In tests of stale information—a value an agent has already seen changing before it acts—the results depended on both the model and how the updated value was presented. In clinic booking, Opus 5 succeeded on 0/16 tasks with nothing extra shown, 2/16 when reminded to re-check, and 15/16 when shown the updated value. In insurance-claim approval, Claude Opus 5 re-checked on its own and succeeded on 10/10, while Claude Sonnet 5 rose from 1/10 to 9/10 when shown the current policy again; the gain was significant in Sonnet 5’s pre-registered test, but not after correcting for all tests. The authors also report that their grading pipeline accepted none of 26 deliberately corrupted episodes as clean, and caught 25 of 39 in an externally generated set, compared with 4 for simple log checks. These results come from the paper’s test setups and do not establish performance across software or tasks generally. The authors say the environments may also be suitable for training agents with from verifiable rewards.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T04:19:31.189Z. First observed: 2026-10-30T06:40:52.855Z. Retrieved: 2026-10-31T04:03:35.544Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported outcomes come from the paper’s specific stale-information tasks and grading tests; the evidence does not show how well the results generalize to other software, tasks, or agents. The abstract does not provide details of the task construction, sample selection, or evaluation procedures beyond the reported results.
Open question
How well would EnvOS’s checks and reported results carry over to software changes and tasks outside the paper’s tests?
Source-stated
The authors present EnvOS as a framework that checks changes, task solvability and grading before an agent runs, then re-checks each verdict against archived evidence.
Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence.
Source-stated
For clinic-booking tasks involving stale information, Claude Opus 5 succeeded more often when shown the updated value than when only reminded to re-check.
In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown
Source-stated
In insurance-claim approval, Claude Sonnet 5’s success rate rose when the current policy was shown again; the increase was significant in its pre-registered test but not after correction for all tests.
Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests.
Source-stated
The authors report that EnvOS rejected all 26 deliberately corrupted episodes as clean and caught 25 of 39 in an externally generated set, versus 4 for simple log checks.
on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks.
Source-stated
The authors suggest that EnvOS environments may also be suitable for training agents with reinforcement learning from verifiable rewards.
Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.
Source and contributor context
Kadu, Ankush, Gopal, Athira, Singh, Niharika, Jain, Arpit, Krishnan, Aswanth · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 92e1b09d-ff76-45d9-8970-9b059cac6509.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T04:19:31.189Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4a5bfe38dd08d7e4a776e1c255db9c0986879af206f34b18e33e686b41718cd2. Source text hash: 75ef755422c8f775866ca27ffca7255cc8fe212f3a4a3bb95f3f30b74a248fbe.
Revision and source-check history
Version 1 · 2026-11-13T04:19:31.189Z
EnvOS checks whether agents can adapt when software changes
The authors of EnvOS argue that evaluating computer-use agents requires checking not only the agent, but also whether software changes occurred as intended, reached the agent, and were graded correctly. Their framework checks each change, task solvability, and grading before an agent runs, then re-checks the verdict against archived evidence. In tests of stale information—a value an agent has already seen changing before it acts—the results depended on both the model and how the updated value was presented. In clinic booking, Opus 5 succeeded on 0/16 tasks with nothing extra shown, 2/16 when reminded to re-check, and 15/16 when shown the updated value. In insurance-claim approval, Claude Opus 5 re-checked on its own and succeeded on 10/10, while Claude Sonnet 5 rose from 1/10 to 9/10 when shown the current policy again; the gain was significant in Sonnet 5’s pre-registered test, but not after correcting for all tests. The authors also report that their grading pipeline accepted none of 26 deliberately corrupted episodes as clean, and caught 25 of 39 in an externally generated set, compared with 4 for simple log checks. These results come from the paper’s test setups and do not establish performance across software or tasks generally. The authors say the environments may also be suitable for training agents with from verifiable rewards.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T04:19:31.189Z. First observed: 2026-10-30T06:40:52.855Z. Retrieved: 2026-10-31T04:03:35.544Z.
Limits
The reported outcomes come from the paper’s specific stale-information tasks and grading tests; the evidence does not show how well the results generalize to other software, tasks, or agents. The abstract does not provide details of the task construction, sample selection, or evaluation procedures beyond the reported results.
Open question
How well would EnvOS’s checks and reported results carry over to software changes and tasks outside the paper’s tests?
Source-stated
The authors present EnvOS as a framework that checks changes, task solvability and grading before an agent runs, then re-checks each verdict against archived evidence.
Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence.
Source-stated
For clinic-booking tasks involving stale information, Claude Opus 5 succeeded more often when shown the updated value than when only reminded to re-check.
In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown
Source-stated
In insurance-claim approval, Claude Sonnet 5’s success rate rose when the current policy was shown again; the increase was significant in its pre-registered test but not after correction for all tests.
Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests.
Source-stated
The authors report that EnvOS rejected all 26 deliberately corrupted episodes as clean and caught 25 of 39 in an externally generated set, versus 4 for simple log checks.
on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks.
Source-stated
The authors suggest that EnvOS environments may also be suitable for training agents with reinforcement learning from verifiable rewards.
Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.
Source and contributor context
Kadu, Ankush, Gopal, Athira, Singh, Niharika, Jain, Arpit, Krishnan, Aswanth · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 92e1b09d-ff76-45d9-8970-9b059cac6509.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T04:19:31.189Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4a5bfe38dd08d7e4a776e1c255db9c0986879af206f34b18e33e686b41718cd2. Source text hash: 75ef755422c8f775866ca27ffca7255cc8fe212f3a4a3bb95f3f30b74a248fbe.
passed · 2026-11-13T04:19:31.189Z · anthropic/claude-opus-5.5
Publication: 92e1b09d-ff76-45d9-8970-9b059cac6509
attribution — pass: The argument, framework, results and training suggestion are attributed to the paper's authors. Model results are presented as findings within the paper's own tests, not as independent facts or Escape Meridian's view.
We introduce EnvOS, a framework for building and certifying environments that change mid-task.
summary — pass: The reading conveys the authors' core point: a failure may come from the agent, a change that did not fire, an undelivered update or a faulty grader. It also explains stale information. One minor fidelity gap: the clinic gain over the reminder held after correcting for all tests (p = 0.00293), but only Sonnet's weaker result gets significance detail. A one-clause addition would restore that balance. Terms such as reinforcement learning from verifiable rewards and pre-registered test can be explained through tooltips. The title and question fit the source.
the gain over the reminder holds after correcting for all tests (p = 0.00293)
date — pass: originalPublished 2026-10-29 matches the arXiv v1 submission date. The feed date of 2026-10-31 is later and correctly not used.
[Submitted on 29 Oct 2026]
support — pass: All numbers match the abstract exactly: 0/16, 2/16 and 15/16; 10/10; 1/10 to 9/10; 0 of 26; 25 of 39 versus 4. The model-and-interface dependence is source-stated. The RLVR suggestion stays hedged. The limitations and question are appropriately conditional and introduce no unsupported benefits or costs.
Whether an agent caught a change thus varied with the model and the interface.
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv