Kozachek and Srivastava’s arXiv study tests LLM agents in public-service scenarios

Kozachek and Srivastava’s arXiv study tests LLM agents in public-service scenarios

In an arXiv paper, Diana Kozachek and Kinjalk Srivastava study what LLM agents do when instructions leave the task unclear. Their public-service scenarios involve decisions that can affect residents, patients, and students. In open-ended runs, agents read records in most cases, but produced an artifact or chose a task in fewer than one in five; with an explicit task, production was more common. The authors report that domain context did not change how many runs produced something, while reducing boundary-probing turns. Some artifacts contradicted the records. Comparing the scenarios with a context-free sandbox, the authors found that models’ production ranking largely carried over, but their ranking for whether they read records did not. The contrast suggests that producing an output and attending to case records are distinct behaviors in these tests. The authors propose a test institutions can run on their own records before procurement, using a rule-based taxonomy that needs no labeled data. The abstract does not explain why some artifacts contradict records or establish how well the proposed test predicts behavior in deployed public services.

Publication: b995c9ce-3e94-45f8-8348-3a0463bbaab6

attribution — pass: The draft correctly attributes the study to Diana Kozachek and Kinjalk Srivastava in an arXiv paper. Findings are framed as the authors' reports, and the proposed test is presented as their contribution.

Authors:Diana Kozachek, Kinjalk Srivastava

summary — uncertain: The figures and structure are faithful, but central terms are left unexplained for a general reader. 'Boundary-probing turns', 'artifact', 'context-free sandbox' and 'rule-based taxonomy' carry key findings and are never explained in plain language. A reader cannot tell what context reduced or what kind of output contradicted records. Smallest repair: briefly explain these terms or add tooltips without changing their wording. The draft also omits the scale of the study: ten LLMs, four prompt conditions and 2,400 trajectories. The question is a fair open issue.

Context cuts boundary-probing turns per run by $60\%$, and some artifacts contradict the records.

date — pass: The originalPublished date of 2026-11-11 matches the arXiv v1 submission date. It does not use the later feed date of 13 Nov.

[Submitted on 11 Nov 2026]

support — uncertain: The numeric claims match the abstract: 96.9% of runs read the records, 18.7% produced something, 86.0% produced with an explicit task, Spearman correlations were 0.79 and 0.19, and context cut boundary-probing by 60%. However, the sentence 'The contrast suggests that producing an output and attending to case records are distinct behaviors in these tests' is an inference. It is neither attributed to the authors nor labeled as Escape Meridian interpretation. Smallest repair: label it as EM interpretation conditioned on the ranking correlations, or remove it. The limitations sentence correctly avoids claiming the test predicts real-world deployment.

the production ranking of the models largely carries over (Spearman $0.79$), while the ranking by whether a model reads the records does not ($0.19$)

media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.