arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements

arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements

In their arXiv paper, Binhang Qi and coauthors introduce Req2Test-Bench to test whether LLM agents can generate tests that check required behavior when the implementation is withheld. The benchmark contains 5,086 instances drawn from developer-authored artifacts across 8 open-source projects, 5 programming languages and 4 BDD frameworks. The authors report a gap between tests that pass and tests that are behaviorally adequate: in the best-performing of 8 agent–model–reasoning configurations, 82.8% of generated tests passed, but 33.8% met the benchmark’s kill-completeness measure. That measure requires a generated test to kill every mutant killed by the ground-truth test. The authors argue that scaling up agents brings sharply diminishing returns in behavioral adequacy relative to generation cost. They identify under-specified requirements, including ambiguous setup and implicit expected outcomes, as sources of an information gap; they describe requirement-grounded harnesses as a promising direction, not a demonstrated solution. How would the reported gap change on requirements with clearer setup and expected outcomes?

Publication: 00516048-524f-4d9d-8684-70fe938e9f92

attribution — pass: The reading attributes the benchmark, results and arguments to Binhang Qi and coauthors in an arXiv paper, which matches the listed authors. Findings stay framed as what the authors report or argue.

Authors:Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong

summary — uncertain: The numbers and framing are faithful, but a non-specialist would struggle with the central finding. The adequacy measure is stated as killing every mutant killed by the ground-truth test, with no plain explanation of mutant, kill or ground-truth test. BDD is not expanded in the summary. The summary also omits the source's stated reason the task matters: tests act as validation constraints, or a harness, guiding agents that compile requirements into code. Smallest repair: add one plain sentence on why requirement-to-test generation matters and briefly gloss mutants and BDD, or supply tooltips for them.

requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation

date — pass: originalPublished is 2026-10-27, which matches the v1 submission date on the arXiv page. The later feed date is correctly not used.

[Submitted on 27 Oct 2026]

support — pass: All claims are directly supported by the abstract: 5,086 instances, 8 projects, 5 languages, 4 BDD frameworks, 82.8% passing versus 33.8% adequate, diminishing returns relative to cost, under-specified requirements as a cause, and harnesses presented as a promising direction rather than a proven fix. The question follows from the source's stated cause without asserting an answer.

produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate.

media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper. This is the most direct access to the source.