arXiv cs.AI paper introduces PersonaCodeBench for evaluating coding agents

arXiv cs.AI paper introduces PersonaCodeBench for evaluating coding agents

The authors introduce PersonaCodeBench, a benchmark for testing whether coding agents can meet users’ stated expectations as well as produce correct software. They argue that existing benchmarks increasingly include interaction, but give less attention to personalized requirements. PersonaCodeBench turns characteristics such as a user’s role, technical ability, interaction preferences and practical constraints into obligations that can be checked during multi-turn coding sessions. Requirements emerge over the interaction, while their content, disclosure conditions and acceptance criteria are fixed before evaluation. A controller manages the simulated user’s responses and when requirements are disclosed; online evaluation provides feedback and advances the interaction. Separate completion and personalization verifiers assess software correctness and whether the agent met the user’s requirements. A joint pass also requires adherence to a declared budget and compliance with validity conditions. The authors say they evaluate major frontier agents, create a leaderboard and analyze the experiments. The benchmark’s central idea is that delivering working code is not enough: an agent must also respect the particular user’s expectations and constraints. This makes personalization part of what the benchmark measures, rather than treating interaction as an add-on to a coding task.

Publication: 66c965e4-e5f3-43c9-85a6-5a602c774241

attribution — pass: The draft attributes the benchmark, its design and its evaluation claims to the paper's authors on arXiv cs.AI. It frames the evaluation and leaderboard as things the authors say they did, not as established results.

We introduce PersonaCodeBench, which turns user characteristics into verifiable obligations in persona-driven multi-turn coding sessions.

summary — uncertain: The summary is mostly faithful and readable. It explains coding agents, simulated users, progressive disclosure and the two verifiers. Three passages need repair. First, it drops 'all' from 'all major frontier agents'; restore it. Second, the closing contrast 'rather than treating interaction as an add-on' is not in the abstract; mark it as EM interpretation or cut it. Third, 'validity conditions' is a central term the source leaves undefined; add a term-library tooltip or note that the abstract does not define it.

We evaluate all major frontier agents, establish a comprehensive leaderboard, and conduct detailed experimental analyses to validate effectiveness.

date — pass: originalPublished, 2026-11-03, matches the arXiv v1 submission date. The later feed date of Nov 5 is correctly not used.

[Submitted on 3 Nov 2026]

support — uncertain: Claim 3 says the benchmark 'requires both' correctness and persona fulfillment for a joint pass, 'alongside budget and validity conditions.' In the source, 'both' refers to budget adherence and validity compliance. The abstract does not explicitly say a joint pass requires both verifiers to pass, although that is a plausible reading of 'joint.' Smallest repair: quote the source's wording, or label the two-verifier requirement as an inference. The limitations and the question appropriately avoid treating simulator results as real-world reliability.

Joint pass requires both adherence to the declared budget and compliance with validity conditions.

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most relevant way to access the source.