{"id":"b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12","title":"arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks","argument":"The authors introduce TerminalWorld, a benchmark built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight frontier models and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not identify the eight frontier models or six agents tested, or describe the task validation process in detail. The abstract reports benchmark results but does not establish how scores translate to performance in deployment.","speaker":{"name":"Chu, Zhaoyang, Hu, Jiarui, Jiang, Xingyu, Zou, Pengyu, Li, Han, Peng, Chao, O'Hearn, Peter, Barr, Earl T., Harman, Mark, Sarro, Federica, Ye, He","roleAtSource":"Article author"},"source":{"title":"TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.22535","publishedAt":"2026-05-21","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.","excerpt":"We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from \"in-the-wild\" terminal recordings.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.","excerpt":"Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.","excerpt":"across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.","excerpt":"with only a weak correlation to their scores (Pearson r=0.20).","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.","excerpt":"The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.22535","checkedAt":"2026-11-08T03:18:06.699Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will the benchmark keep tasks authentic as developer practices evolve?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks","summary":"The authors introduce TerminalWorld, a benchmark built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight frontier models and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.","question":"How will the benchmark keep tasks authentic as developer practices evolve?","claims":[{"text":"The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.","excerpt":"We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from \"in-the-wild\" terminal recordings.","support":"source-stated"},{"text":"Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.","excerpt":"Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories","support":"source-stated"},{"text":"On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.","excerpt":"across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.","support":"source-stated"},{"text":"The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.","excerpt":"with only a weak correlation to their scores (Pearson r=0.20).","support":"source-stated"},{"text":"The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.","excerpt":"The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.","support":"source-stated"}],"limitations":["The abstract does not identify the eight frontier models or six agents tested, or describe the task validation process in detail.","The abstract reports benchmark results but does not establish how scores translate to performance in deployment."],"mediaId":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","correctionReason":null,"id":"b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12","publicationId":"8252486b-51ec-4ae0-8643-8a442e2f55d6","version":1,"sourceUrl":"https://arxiv.org/abs/2605.22535","sourceTitle":"TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-21","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:28.630Z","createdAt":"2026-11-08T02:47:15.281Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","kind":"article","url":"https://arxiv.org/abs/2605.22535","sourceUrl":"https://arxiv.org/abs/2605.22535","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T02:46:27.879Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.22535","checkedAt":"2026-11-08T03:18:06.699Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2c1848a0258ad9250e4d8c46565131368c9d2c0f4e2ae1cf1546453536364251","decidedAt":"2026-11-08T02:46:29.877Z"},"provenance":{"textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observationId":"a9570d76002e61db311a1d6e4bd73110d653259533e456cfd1b39acaa6dac2bb","retrievedAt":"2026-11-08T02:46:27.553Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"72f87504-696e-465e-8248-ffd8f6456c37"},"reviewer":{"model":"gpt-6-luna","runId":"65aff721-0dcd-40f7-8f23-41737686921d"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T02:47:15.281Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Chu, Zhaoyang","url":null,"kind":"unknown"},{"name":"Hu, Jiarui","url":null,"kind":"unknown"},{"name":"Jiang, Xingyu","url":null,"kind":"unknown"},{"name":"Zou, Pengyu","url":null,"kind":"unknown"},{"name":"Li, Han","url":null,"kind":"unknown"},{"name":"Peng, Chao","url":null,"kind":"unknown"},{"name":"O'Hearn, Peter","url":null,"kind":"unknown"},{"name":"Barr, Earl T.","url":null,"kind":"unknown"},{"name":"Harman, Mark","url":null,"kind":"unknown"},{"name":"Sarro, Federica","url":null,"kind":"unknown"},{"name":"Ye, He","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Chu, Zhaoyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chu, Zhaoyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Hu, Jiarui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Jiarui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Jiang, Xingyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Xingyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Zou, Pengyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zou, Pengyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Li, Han","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Han","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Peng, Chao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Peng, Chao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"O'Hearn, Peter","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"O'Hearn, Peter","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Barr, Earl T.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Barr, Earl T.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Harman, Mark","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Harman, Mark","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Sarro, Federica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sarro, Federica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Ye, He","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, He","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.22535","snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z","basis":"publisher-page-metadata","sourceLinks":51,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-terminalworld-benchmarks-agents-on-real-world-b993ebc3","formats":{"html":"/readings/arxiv-cs-ai-terminalworld-benchmarks-agents-on-real-world-b993ebc3","markdown":"/readings/arxiv-cs-ai-terminalworld-benchmarks-agents-on-real-world-b993ebc3.md","json":"/readings/arxiv-cs-ai-terminalworld-benchmarks-agents-on-real-world-b993ebc3.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12","revision":"8252486b-51ec-4ae0-8643-8a442e2f55d6","revisionKind":"publication-id","title":"arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks","url":"https://escapemeridian.com/readings/arxiv-cs-ai-terminalworld-benchmarks-agents-on-real-world-b993ebc3"},"source":{"type":"source","id":"https://arxiv.org/abs/2605.22535","revision":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","revisionKind":"sha256-source-text","title":"TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks","url":"https://arxiv.org/abs/2605.22535"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12","title":"arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks","argument":"The authors introduce TerminalWorld, a benchmark built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight frontier models and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not identify the eight frontier models or six agents tested, or describe the task validation process in detail. The abstract reports benchmark results but does not establish how scores translate to performance in deployment.","speaker":{"name":"Chu, Zhaoyang, Hu, Jiarui, Jiang, Xingyu, Zou, Pengyu, Li, Han, Peng, Chao, O'Hearn, Peter, Barr, Earl T., Harman, Mark, Sarro, Federica, Ye, He","roleAtSource":"Article author"},"source":{"title":"TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.22535","publishedAt":"2026-05-21","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.","excerpt":"We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from \"in-the-wild\" terminal recordings.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.","excerpt":"Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.","excerpt":"across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.","excerpt":"with only a weak correlation to their scores (Pearson r=0.20).","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"},{"text":"The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.","excerpt":"The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.","url":"https://arxiv.org/abs/2605.22535","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.22535","checkedAt":"2026-11-08T03:18:06.699Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will the benchmark keep tasks authentic as developer practices evolve?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: TerminalWorld benchmarks agents on real-world terminal tasks","summary":"The authors introduce TerminalWorld, a benchmark built from “in-the-wild” terminal recordings. Its data engine automatically reverse-engineers evaluation tasks from 80,870 recordings, producing 1,530 validated tasks across 18 categories and 1,280 commands. The authors also manually review a diverse subset of 200 tasks, called TerminalWorld-Verified. In tests on that subset, eight frontier models and six agents reach a maximum pass rate of 62.5%, which the authors say shows that current systems still struggle with authentic terminal workflows. Scores have only a weak correlation with scores on expert-curated benchmarks such as Terminal-Bench (Pearson r=0.20). The authors argue that these benchmarks capture distinct capabilities, and that building tasks from real-world recordings makes TerminalWorld authentic and scalable as developer practices evolve. The results offer a way to assess terminal agents on work drawn from actual usage, but the abstract does not establish how performance on this benchmark translates to success in deployment.","question":"How will the benchmark keep tasks authentic as developer practices evolve?","claims":[{"text":"The authors introduce TerminalWorld, a data engine that creates evaluation tasks from terminal recordings.","excerpt":"We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from \"in-the-wild\" terminal recordings.","support":"source-stated"},{"text":"Processing 80,870 recordings yields 1,530 validated tasks across 18 categories and 1,280 commands; the authors also curate a manually reviewed subset of 200 tasks.","excerpt":"Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories","support":"source-stated"},{"text":"On TerminalWorld-Verified, the tested models and agents achieve a maximum pass rate of 62.5%.","excerpt":"across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.","support":"source-stated"},{"text":"The authors report a weak correlation between TerminalWorld scores and scores on existing expert-curated benchmarks.","excerpt":"with only a weak correlation to their scores (Pearson r=0.20).","support":"source-stated"},{"text":"The authors argue that the automated engine makes TerminalWorld authentic and scalable as developer practices evolve.","excerpt":"The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve.","support":"source-stated"}],"limitations":["The abstract does not identify the eight frontier models or six agents tested, or describe the task validation process in detail.","The abstract reports benchmark results but does not establish how scores translate to performance in deployment."],"mediaId":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","correctionReason":null,"id":"b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12","publicationId":"8252486b-51ec-4ae0-8643-8a442e2f55d6","version":1,"sourceUrl":"https://arxiv.org/abs/2605.22535","sourceTitle":"TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-21","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:28.630Z","createdAt":"2026-11-08T02:47:15.281Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","kind":"article","url":"https://arxiv.org/abs/2605.22535","sourceUrl":"https://arxiv.org/abs/2605.22535","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T02:46:27.879Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.22535","checkedAt":"2026-11-08T03:18:06.699Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"d892620af982798265d87c1af764463ea5d8a9b79a6b0db0903375afb850fab0","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2c1848a0258ad9250e4d8c46565131368c9d2c0f4e2ae1cf1546453536364251","decidedAt":"2026-11-08T02:46:29.877Z"},"provenance":{"textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observationId":"a9570d76002e61db311a1d6e4bd73110d653259533e456cfd1b39acaa6dac2bb","retrievedAt":"2026-11-08T02:46:27.553Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"72f87504-696e-465e-8248-ffd8f6456c37"},"reviewer":{"model":"gpt-6-luna","runId":"65aff721-0dcd-40f7-8f23-41737686921d"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T02:47:15.281Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Chu, Zhaoyang","url":null,"kind":"unknown"},{"name":"Hu, Jiarui","url":null,"kind":"unknown"},{"name":"Jiang, Xingyu","url":null,"kind":"unknown"},{"name":"Zou, Pengyu","url":null,"kind":"unknown"},{"name":"Li, Han","url":null,"kind":"unknown"},{"name":"Peng, Chao","url":null,"kind":"unknown"},{"name":"O'Hearn, Peter","url":null,"kind":"unknown"},{"name":"Barr, Earl T.","url":null,"kind":"unknown"},{"name":"Harman, Mark","url":null,"kind":"unknown"},{"name":"Sarro, Federica","url":null,"kind":"unknown"},{"name":"Ye, He","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Chu, Zhaoyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chu, Zhaoyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Hu, Jiarui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Jiarui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Jiang, Xingyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Xingyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Zou, Pengyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zou, Pengyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Li, Han","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Han","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Peng, Chao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Peng, Chao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"O'Hearn, Peter","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"O'Hearn, Peter","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Barr, Earl T.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Barr, Earl T.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Harman, Mark","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Harman, Mark","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Sarro, Federica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sarro, Federica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}},{"actor":{"name":"Ye, He","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, He","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.22535","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.699Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","sourceUrl":"https://arxiv.org/abs/2605.22535","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.22535","snapshotId":"e8f2114c82ab26cf251afd7ff5d04152f28801f5565226e8581dee571bb56e83","contentSha256":"ff8bb302dddb3d71771b5ebb03621325bc9734d77de25093c629be8dea57adad","textHash":"ecc4060d035582c70a9cf67df706ec11ead501948289e3721d4cbd09c7672c1e","observedAt":"2026-11-08T03:18:06.699Z","basis":"publisher-page-metadata","sourceLinks":51,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b993ebc3822ec43ba3fe83c6cd107da8b295b5173dff9a1afe4a8b94e3b74a12/source"}}}],"audits":[{"publicationId":"8252486b-51ec-4ae0-8643-8a442e2f55d6","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes TerminalWorld, its methods, results and interpretive claims to the paper's authors on arXiv cs.AI. Claims such as 'current systems still struggle' and 'authentic and scalable' are framed as the authors' statements or arguments, not established fact.","excerpt":"Authors:Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye"},{"dimension":"summary","status":"pass","explanation":"The summary faithfully conveys the abstract: the data engine, 80,870 recordings, 1,530 tasks, 18 categories, 1,280 commands, the 200-task Verified subset, the 62.5% maximum pass rate, and the weak correlation with Terminal-Bench. 'Terminal', 'agents' and 'Pearson r' are left to term-library tooltips; 'weak correlation' is stated plainly. The question follows from the authors' scalability claim. One optional omission is the range from short operations to workflows exceeding 50 steps.","excerpt":"ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-05-21 matches the v1 submission date. It is not the v2 revision date of 4 Nov 2026 or the November feed date.","excerpt":"[Submitted on 21 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Each claim is supported by a verbatim excerpt from the abstract. The significance sentence stays modest: the benchmark offers assessment on tasks drawn from actual usage. The reading explicitly declines to turn benchmark scores into evidence of real-world deployment reliability. Its limitations accurately note that the abstract does not name the models or agents tested.","excerpt":"TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20)."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant candidate.","excerpt":null}],"createdAt":"2026-11-08T02:47:15.281Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}