Andrews, Austin Tudor David, Wilkinson, Liam, Heagerty, Jamie, Coppock, Harry, Foerster, Jakob Nicolaus, Costa, Rui Ponte · Sep 2, 2026
CivBench finds agents miss strategic checks and their own plans in Civilization VI
Markdown · JSON · Original source
The authors of CivBench present an open-source for testing language model agents in long-horizon environments where agents use tools to act. In Civilization VI, an episode can last over 300 turns and involve thousands of tool calls; the benchmark offers 76 tools and makes game state available as structured text. The authors report a pilot of 23 admissible runs across four model families, and stress that its aggregate outcomes do not reliably distinguish models at this scale. Instead, CivBench measures whether agents query strategically relevant state and follow through on near-term commitments in their planning reflections. Under a shared playbook protocol, agents checked victory progress every 30–75 turns, despite guidance to check every 20 turns. In 8 of 20 detectable defeats, they did not query within the 20-turn warning window before the game ended. Their follow-through scores ranged from 48.2% to 65.8% across models. The authors interpret these patterns as deviations under instruction, not proof that the agents lack the relevant capabilities. The benchmark makes a specific gap measurable: having tools and explicit guidance does not mean agents consistently monitor the situation or carry out their own stated plans. The pilot is too small to support a model ranking.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Sep 2, 2026 (2026-09-02).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T01:16:45.465Z. First observed: 2026-11-05T05:09:24.267Z. Retrieved: 2026-11-10T01:16:01.191Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The authors characterize the sample as a pilot and state that aggregate outcomes do not reliably discriminate models at this scale. The reported patterns are from runs under a shared playbook protocol; the supplied abstract does not establish whether they generalize to other protocols or settings.
Open question
Would these monitoring and follow-through patterns persist in larger samples or under protocols other than the shared playbook?
Source-stated
CivBench is an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through MCP.
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP).
Source-stated
The authors describe their results as a pilot and say aggregate outcomes do not reliably distinguish models at this scale.
The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale.
Source-stated
Under a shared playbook protocol, agents queried victory progress every 30–75 turns, although the playbook guided them to query every 20 turns.
despite playbook guidance to query victory progress every 20 turns, agents do so only every 30-75 turns
Source-stated
In 8 of 20 detectable defeats, agents failed to query within the warning window before the game ended.
in 8 of 20 detectable defeats they failed to query within the 20-turn warning window before game end.
Source-stated
The authors report that agents executed between 48.2% and 65.8% of near-term commitments in their planning reflections within ten turns.
RAG@10 between 48.2% and 65.8% across models
Source-stated
The authors interpret the observed patterns as deviations under instruction rather than evidence of absent capability.
we interpret them as deviations under instruction rather than absences of capability.
Source and contributor context
Andrews, Austin Tudor David, Wilkinson, Liam, Heagerty, Jamie, Coppock, Harry, Foerster, Jakob Nicolaus, Costa, Rui Ponte · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 3b8bc4c6-69fd-43a9-83b4-ebd8729fe40b.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T01:16:45.465Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d8076676080cc601afd2903e7e398e0536f5d265364bdc8a8416c047316ff8a5. Source text hash: ec94692c0029e3fb82ca7e4285240027d95b8f142b7f994e9a6e3ef56b015504.
Revision and source-check history
Version 1 · 2026-11-10T01:16:45.465Z
CivBench finds agents miss strategic checks and their own plans in Civilization VI
The authors of CivBench present an open-source for testing language model agents in long-horizon environments where agents use tools to act. In Civilization VI, an episode can last over 300 turns and involve thousands of tool calls; the benchmark offers 76 tools and makes game state available as structured text. The authors report a pilot of 23 admissible runs across four model families, and stress that its aggregate outcomes do not reliably distinguish models at this scale. Instead, CivBench measures whether agents query strategically relevant state and follow through on near-term commitments in their planning reflections. Under a shared playbook protocol, agents checked victory progress every 30–75 turns, despite guidance to check every 20 turns. In 8 of 20 detectable defeats, they did not query within the 20-turn warning window before the game ended. Their follow-through scores ranged from 48.2% to 65.8% across models. The authors interpret these patterns as deviations under instruction, not proof that the agents lack the relevant capabilities. The benchmark makes a specific gap measurable: having tools and explicit guidance does not mean agents consistently monitor the situation or carry out their own stated plans. The pilot is too small to support a model ranking.
Original source
cs.AI · Article
Source details
Original source date: Sep 2, 2026 (2026-09-02).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T01:16:45.465Z. First observed: 2026-11-05T05:09:24.267Z. Retrieved: 2026-11-10T01:16:01.191Z.
Limits
The authors characterize the sample as a pilot and state that aggregate outcomes do not reliably discriminate models at this scale. The reported patterns are from runs under a shared playbook protocol; the supplied abstract does not establish whether they generalize to other protocols or settings.
Open question
Would these monitoring and follow-through patterns persist in larger samples or under protocols other than the shared playbook?
Source-stated
CivBench is an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through MCP.
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP).
Source-stated
The authors describe their results as a pilot and say aggregate outcomes do not reliably distinguish models at this scale.
The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale.
Source-stated
Under a shared playbook protocol, agents queried victory progress every 30–75 turns, although the playbook guided them to query every 20 turns.
despite playbook guidance to query victory progress every 20 turns, agents do so only every 30-75 turns
Source-stated
In 8 of 20 detectable defeats, agents failed to query within the warning window before the game ended.
in 8 of 20 detectable defeats they failed to query within the 20-turn warning window before game end.
Source-stated
The authors report that agents executed between 48.2% and 65.8% of near-term commitments in their planning reflections within ten turns.
RAG@10 between 48.2% and 65.8% across models
Source-stated
The authors interpret the observed patterns as deviations under instruction rather than evidence of absent capability.
we interpret them as deviations under instruction rather than absences of capability.
Source and contributor context
Andrews, Austin Tudor David, Wilkinson, Liam, Heagerty, Jamie, Coppock, Harry, Foerster, Jakob Nicolaus, Costa, Rui Ponte · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 3b8bc4c6-69fd-43a9-83b4-ebd8729fe40b.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T01:16:45.465Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d8076676080cc601afd2903e7e398e0536f5d265364bdc8a8416c047316ff8a5. Source text hash: ec94692c0029e3fb82ca7e4285240027d95b8f142b7f994e9a6e3ef56b015504.
passed · 2026-11-10T01:16:45.465Z · anthropic/claude-opus-5.5
Publication: 3b8bc4c6-69fd-43a9-83b4-ebd8729fe40b
attribution — pass: The reading attributes CivBench, the pilot results, and the interpretation of the patterns to the paper's authors. It does not present their findings as Escape Meridian's own view or as independently verified facts.
we interpret them as deviations under instruction rather than absences of capability.
summary — pass: The reading states what was built and what was found, and keeps the pilot and no-ranking caveat. Terms are explained in plain language: tools as ways to act, follow-through on stated plans, and checking victory progress. The title and question follow from the evidence. Minor terms such as 'admissible runs' and 'long-horizon' do not block understanding. Optional term-library tooltips: MCP, RAG@10, long-horizon.
The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale.
date — pass: originalPublished 2026-09-02 matches the v1 submission date. It does not use the v2 revision date of 4 Nov 2026 or the feed date.
Submitted on 2 Sep 2026 (v1), last revised 4 Nov 2026 (this version, v2)
support — pass: Each number matches the abstract: 300+ turns, 76 tools, 23 runs, four model families, every 20 versus 30-75 turns, 8 of 20 defeats, and RAG@10 of 48.2-65.8%. The claim that tools and guidance do not ensure consistent monitoring or follow-through is grounded in the authors' statement. It is limited by the pilot caveat, and the question about larger samples and other protocols is appropriately open.
Both patterns arise despite tool access and explicit guidance
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct and relevant link to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv