Chu, Zhaoyang, Hu, Jiarui, Jiang, Xingyu, Zou, Pengyu, Li, Han, Peng, Chao, O'Hearn, Peter, Barr, Earl T., Harman, Mark, Sarro, Federica, Ye, He · May 21, 2026
arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks
Markdown · JSON · Original source
The authors introduce TerminalWorld, a built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: May 21, 2026 (2026-05-21).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T02:47:15.281Z. First observed: 2026-11-06T05:10:28.630Z. Retrieved: 2026-11-08T02:46:27.553Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract does not identify the eight or six agents tested, or describe the task validation process in detail. The abstract reports results but does not establish how scores translate to performance in deployment.
Open question
How will the keep tasks authentic as developer practices evolve?
Source-stated
The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings.
Source-stated
Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.
Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories
Source-stated
On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.
across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.
Source-stated
The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.
with only a weak correlation to their scores (Pearson r=0.20).
Source-stated
The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.
The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.
Source and contributor context
Chu, Zhaoyang, Hu, Jiarui, Jiang, Xingyu, Zou, Pengyu, Li, Han, Peng, Chao, O'Hearn, Peter, Barr, Earl T., Harman, Mark, Sarro, Federica, Ye, He · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 8252486b-51ec-4ae0-8643-8a442e2f55d6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T02:47:15.281Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a9570d76002e61db311a1d6e4bd73110d653259533e456cfd1b39acaa6dac2bb. Source text hash: ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e.
Revision and source-check history
Version 1 · 2026-11-08T02:47:15.281Z
arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks
The authors introduce TerminalWorld, a built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.
Original source
cs.AI · Article
Source details
Original source date: May 21, 2026 (2026-05-21).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T02:47:15.281Z. First observed: 2026-11-06T05:10:28.630Z. Retrieved: 2026-11-08T02:46:27.553Z.
Limits
The abstract does not identify the eight or six agents tested, or describe the task validation process in detail. The abstract reports results but does not establish how scores translate to performance in deployment.
Open question
How will the keep tasks authentic as developer practices evolve?
Source-stated
The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings.
Source-stated
Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.
Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories
Source-stated
On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.
across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.
Source-stated
The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.
with only a weak correlation to their scores (Pearson r=0.20).
Source-stated
The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.
The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.
Source and contributor context
Chu, Zhaoyang, Hu, Jiarui, Jiang, Xingyu, Zou, Pengyu, Li, Han, Peng, Chao, O'Hearn, Peter, Barr, Earl T., Harman, Mark, Sarro, Federica, Ye, He · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 8252486b-51ec-4ae0-8643-8a442e2f55d6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T02:47:15.281Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a9570d76002e61db311a1d6e4bd73110d653259533e456cfd1b39acaa6dac2bb. Source text hash: ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e.
passed · 2026-11-08T02:47:15.281Z · anthropic/claude-opus-5.5
Publication: 8252486b-51ec-4ae0-8643-8a442e2f55d6
attribution — pass: The reading attributes TerminalWorld, its methods, results and interpretive claims to the paper's authors on arXiv cs.AI. Claims such as 'current systems still struggle' and 'authentic and scalable' are framed as the authors' statements or arguments, not established fact.
Authors:Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye
summary — pass: The summary faithfully conveys the abstract: the data engine, 80,870 recordings, 1,530 tasks, 18 categories, 1,280 commands, the 200-task Verified subset, the 62.5% maximum pass rate, and the weak correlation with Terminal-Bench. 'Terminal', 'agents' and 'Pearson r' are left to term-library tooltips; 'weak correlation' is stated plainly. The question follows from the authors' scalability claim. One optional omission is the range from short operations to workflows exceeding 50 steps.
ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands
date — pass: originalPublished 2026-05-21 matches the v1 submission date. It is not the v2 revision date of 4 Nov 2026 or the November feed date.
[Submitted on 21 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)]
support — pass: Each claim is supported by a verbatim excerpt from the abstract. The significance sentence stays modest: the benchmark offers assessment on tasks drawn from actual usage. The reading explicitly declines to turn benchmark scores into evidence of real-world deployment reliability. Its limitations accurately note that the abstract does not name the models or agents tested.
TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20).
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant candidate.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference