# arXiv cs.AI: Kierston & KierstonBench test language models on simulated crime investigations [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-kierston-and-kierstonbench-test-language-models-on-dd69f323) Andrew Semenov and Svyatoslav Dorofeev present Kierston, a city-scale simulation of Allentown, Pennsylvania, and KierstonBench, which tests language models on simulated crime investigation and surveillance tasks. The simulation generates residents’ activities and the records investigative and surveillance systems collect, while models use a restricted records interface without access to the underlying simulation. The authors say this setup addresses a measurement problem: real deployments lack the ground truth needed to score models’ conclusions, and no benchmark evaluates these capabilities in a simulated environment. In evaluations of sixteen language models across 32 task families, most could not investigate records reliably and submitted identifications unsupported by the records. A small number of frontier models solved most tasks and abstained when evidence was insufficient, but still fell for some deceptions and assumptions. The authors conclude that none replaced an investigator. The benchmark offers a way to test performance against known simulated events; these results do not establish how models would perform in real investigations, where the authors say ground truth is lacking. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T18:01:04.022Z. First observed: 2026-11-04T05:18:22.194Z. Retrieved: 2026-11-11T17:49:05.608Z. - [Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks](https://arxiv.org/abs/2611.04785) ## Limits The benchmark evaluates models against known simulated events; the abstract does not establish how its results transfer to real investigations. The authors report that some models fell for common deceptions and assumptions, and that none replaced an investigator. ## Open question How well do results on KierstonBench predict language-model performance in real investigations, where the authors say ground truth is lacking? ## Source-stated The authors introduce Kierston, a city-scale simulation of Allentown, and KierstonBench, which evaluates language models on investigation and surveillance tasks using simulated events. > We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The simulation creates residents’ activities and records across several domains, with records distributed among separate providers. > Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated Models investigate through a restricted records interface and cannot access the underlying simulation. > Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The benchmark covers 32 task families, including record linkage, routine reconstruction, incident investigation and deceptive-account assessment. > The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated In an evaluation of sixteen language models, most submitted identifications unsupported by records; a small number of frontier models solved most tasks and abstained when evidence was insufficient. > In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The authors report that some frontier models still fell for common deceptions and assumptions, and none replaced an investigator. > In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The authors say current real deployments lack the ground truth needed to score models’ conclusions, motivating evaluation against known simulated events. > Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source and contributor context Semenov, Andrew, Dorofeev, Svyatoslav · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: b400d197-edef-44ad-8b2c-28b3a7ebc941. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T18:01:04.022Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 23c18ec633c0a8c889366ed90380d782e3bb8af853f589c67b7931a9d540cdea. Source text hash: ab3c4b21ad8f4dcaf44bf74ab7f546273e883de5ad7b94a15453f969785bba8e. ## Revision and source-check history ### Version 1 — 2026-11-11T18:01:04.022Z Andrew Semenov and Svyatoslav Dorofeev present Kierston, a city-scale simulation of Allentown, Pennsylvania, and KierstonBench, which tests language models on simulated crime investigation and surveillance tasks. The simulation generates residents’ activities and the records investigative and surveillance systems collect, while models use a restricted records interface without access to the underlying simulation. The authors say this setup addresses a measurement problem: real deployments lack the ground truth needed to score models’ conclusions, and no benchmark evaluates these capabilities in a simulated environment. In evaluations of sixteen language models across 32 task families, most could not investigate records reliably and submitted identifications unsupported by the records. A small number of frontier models solved most tasks and abstained when evidence was insufficient, but still fell for some deceptions and assumptions. The authors conclude that none replaced an investigator. The benchmark offers a way to test performance against known simulated events; these results do not establish how models would perform in real investigations, where the authors say ground truth is lacking. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T18:01:04.022Z. First observed: 2026-11-04T05:18:22.194Z. Retrieved: 2026-11-11T17:49:05.608Z. - [Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks](https://arxiv.org/abs/2611.04785) ## Limits The benchmark evaluates models against known simulated events; the abstract does not establish how its results transfer to real investigations. The authors report that some models fell for common deceptions and assumptions, and that none replaced an investigator. ## Open question How well do results on KierstonBench predict language-model performance in real investigations, where the authors say ground truth is lacking? ## Source-stated The authors introduce Kierston, a city-scale simulation of Allentown, and KierstonBench, which evaluates language models on investigation and surveillance tasks using simulated events. > We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The simulation creates residents’ activities and records across several domains, with records distributed among separate providers. > Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated Models investigate through a restricted records interface and cannot access the underlying simulation. > Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The benchmark covers 32 task families, including record linkage, routine reconstruction, incident investigation and deceptive-account assessment. > The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated In an evaluation of sixteen language models, most submitted identifications unsupported by records; a small number of frontier models solved most tasks and abstained when evidence was insufficient. > In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The authors report that some frontier models still fell for common deceptions and assumptions, and none replaced an investigator. > In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source-stated The authors say current real deployments lack the ground truth needed to score models’ conclusions, motivating evaluation against known simulated events. > Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. - [Supporting source](https://arxiv.org/abs/2611.04785) ## Source and contributor context Semenov, Andrew, Dorofeev, Svyatoslav · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: b400d197-edef-44ad-8b2c-28b3a7ebc941. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T18:01:04.022Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 23c18ec633c0a8c889366ed90380d782e3bb8af853f589c67b7931a9d540cdea. Source text hash: ab3c4b21ad8f4dcaf44bf74ab7f546273e883de5ad7b94a15453f969785bba8e. ### uncertain — 2026-11-11T18:01:04.022Z — anthropic/claude-opus-5.5 Publication: b400d197-edef-44ad-8b2c-28b3a7ebc941 attribution — pass: The paper is correctly attributed to Andrew Semenov and Svyatoslav Dorofeev, and its findings and claims are framed as the authors' rather than as established fact. > Authors:Andrew Semenov, Svyatoslav Dorofeev summary — uncertain: The summary softens a key finding. The source says most models could not perform records investigation, but the draft says they could not investigate records reliably. Repair: drop 'reliably'. It also omits the abstract's framing that models are candidates for criminal investigations and mass surveillance, the human stake behind the measurement problem. Repair: add one clause with that framing. 'Ground truth' and 'frontier models' are central but unexplained, so they need tooltips or brief plain glosses. > In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. date — pass: The original publication date of 2026-11-03 matches the v1 submission. The v2 revision on 10 Nov 2026 and the feed date of 11 Nov 2026 are correctly not used. > \[Submitted on 3 Nov 2026 (v1), last revised 10 Nov 2026 (this version, v2)\] support — pass: Each claim's excerpt matches the source verbatim and supports it. The limitation and the question about transfer to real investigations follow from the authors' own ground-truth statement and are not presented as findings. No benefits or costs are invented. > Real deployments lack the ground truth needed to score a model's conclusions media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the original source and the most direct way to read it. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/dd69f323b4888b114fd86923b690297f533a1a1a117446ccdd6b0cb3043b380d/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-kierston-and-kierstonbench-test-language-models-on-dd69f323.json)