{"id":"b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1","title":"arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements","argument":"In their arXiv paper, Binhang Qi and coauthors introduce Req2Test-Bench to test whether LLM agents can generate tests that check required behavior when the implementation is withheld. The benchmark contains 5,086 instances drawn from developer-authored artifacts across 8 open-source projects, 5 programming languages and 4 BDD frameworks.\n\nThe authors report a gap between tests that pass and tests that are behaviorally adequate: in the best-performing of 8 agent–model–reasoning configurations, 82.8% of generated tests passed, but 33.8% met the benchmark’s kill-completeness measure. That measure requires a generated test to kill every mutant killed by the ground-truth test.\n\nThe authors argue that scaling up agents brings sharply diminishing returns in behavioral adequacy relative to generation cost. They identify under-specified requirements, including ambiguous setup and implicit expected outcomes, as sources of an information gap; they describe requirement-grounded harnesses as a promising direction, not a demonstrated solution. How would the reported gap change on requirements with clearer setup and expected outcomes?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide further detail about the evaluation or the benchmark instances. The reported adequacy measure is kill-completeness against the ground-truth test’s mutants; the abstract does not establish how the results translate to software testing in deployment.","speaker":{"name":"Qi, Binhang, Liu, Yiming, Lin, Yun, Zhou, Yangjie, Sun, Hailong, Dong, Jin Song","roleAtSource":"Article author"},"source":{"title":"Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36584","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"Qi and coauthors introduce Req2Test-Bench, a benchmark for evaluating agents that generate tests from software requirements.","excerpt":"We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The benchmark draws on developer-authored behavior-driven development artifacts from 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","excerpt":"recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The authors report that the best-performing configuration produced passing tests for 82.8% of instances, but behaviorally adequate tests for 33.8%.","excerpt":"produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate.","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The authors identify under-specified requirements as a source of the information gap and say requirement-grounded harnesses are a promising direction.","excerpt":"under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36584","checkedAt":"2026-11-14T04:34:09.136Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the reported gap change on requirements with clearer setup and expected outcomes?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements","summary":"In their arXiv paper, Binhang Qi and coauthors introduce Req2Test-Bench to test whether LLM agents can generate tests that check required behavior when the implementation is withheld. The benchmark contains 5,086 instances drawn from developer-authored artifacts across 8 open-source projects, 5 programming languages and 4 BDD frameworks.\n\nThe authors report a gap between tests that pass and tests that are behaviorally adequate: in the best-performing of 8 agent–model–reasoning configurations, 82.8% of generated tests passed, but 33.8% met the benchmark’s kill-completeness measure. That measure requires a generated test to kill every mutant killed by the ground-truth test.\n\nThe authors argue that scaling up agents brings sharply diminishing returns in behavioral adequacy relative to generation cost. They identify under-specified requirements, including ambiguous setup and implicit expected outcomes, as sources of an information gap; they describe requirement-grounded harnesses as a promising direction, not a demonstrated solution. How would the reported gap change on requirements with clearer setup and expected outcomes?","question":"How would the reported gap change on requirements with clearer setup and expected outcomes?","claims":[{"text":"Qi and coauthors introduce Req2Test-Bench, a benchmark for evaluating agents that generate tests from software requirements.","excerpt":"We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances","support":"source-stated"},{"text":"The benchmark draws on developer-authored behavior-driven development artifacts from 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","excerpt":"recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","support":"source-stated"},{"text":"The authors report that the best-performing configuration produced passing tests for 82.8% of instances, but behaviorally adequate tests for 33.8%.","excerpt":"produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate.","support":"source-stated"},{"text":"The authors identify under-specified requirements as a source of the information gap and say requirement-grounded harnesses are a promising direction.","excerpt":"under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide further detail about the evaluation or the benchmark instances.","The reported adequacy measure is kill-completeness against the ground-truth test’s mutants; the abstract does not establish how the results translate to software testing in deployment."],"mediaId":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","correctionReason":null,"id":"b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1","publicationId":"00516048-524f-4d9d-8684-70fe938e9f92","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36584","sourceTitle":"Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T07:12:41.239Z","createdAt":"2026-11-14T04:02:37.099Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","kind":"article","url":"https://arxiv.org/abs/2610.36584","sourceUrl":"https://arxiv.org/abs/2610.36584","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T20:16:45.029Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36584","checkedAt":"2026-11-14T04:34:09.136Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"56d48da8e326d6a98a4fa75c93f00eb0d4ff2d40b59e7244272ed7b9ba50db75","decidedAt":"2026-10-30T20:16:45.528Z"},"provenance":{"textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observationId":"6ae145f6d8ae52765b457c3c2b7b7ac8138806cde210c52ce8ca3138f0b4d335","retrievedAt":"2026-10-30T20:16:44.561Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"b46b78ba-2a49-448c-864f-e6c0659d0c73"},"reviewer":{"model":"gpt-6-luna","runId":"ed484e94-62bd-4dc4-845a-189cdba4c8de"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-14T04:02:37.099Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Qi, Binhang","url":null,"kind":"unknown"},{"name":"Liu, Yiming","url":null,"kind":"unknown"},{"name":"Lin, Yun","url":null,"kind":"unknown"},{"name":"Zhou, Yangjie","url":null,"kind":"unknown"},{"name":"Sun, Hailong","url":null,"kind":"unknown"},{"name":"Dong, Jin Song","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Qi, Binhang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qi, Binhang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Liu, Yiming","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Yiming","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Lin, Yun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lin, Yun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Zhou, Yangjie","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yangjie","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Sun, Hailong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Hailong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Dong, Jin Song","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dong, Jin Song","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36584","snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-qi-and-coauthors-benchmark-llm-agents-test-b395f86d","formats":{"html":"/readings/arxiv-cs-ai-qi-and-coauthors-benchmark-llm-agents-test-b395f86d","markdown":"/readings/arxiv-cs-ai-qi-and-coauthors-benchmark-llm-agents-test-b395f86d.md","json":"/readings/arxiv-cs-ai-qi-and-coauthors-benchmark-llm-agents-test-b395f86d.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1","revision":"00516048-524f-4d9d-8684-70fe938e9f92","revisionKind":"publication-id","title":"arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements","url":"https://escapemeridian.com/readings/arxiv-cs-ai-qi-and-coauthors-benchmark-llm-agents-test-b395f86d"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.36584","revision":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","revisionKind":"sha256-source-text","title":"Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements","url":"https://arxiv.org/abs/2610.36584"},"status":"partial","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1","title":"arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements","argument":"In their arXiv paper, Binhang Qi and coauthors introduce Req2Test-Bench to test whether LLM agents can generate tests that check required behavior when the implementation is withheld. The benchmark contains 5,086 instances drawn from developer-authored artifacts across 8 open-source projects, 5 programming languages and 4 BDD frameworks.\n\nThe authors report a gap between tests that pass and tests that are behaviorally adequate: in the best-performing of 8 agent–model–reasoning configurations, 82.8% of generated tests passed, but 33.8% met the benchmark’s kill-completeness measure. That measure requires a generated test to kill every mutant killed by the ground-truth test.\n\nThe authors argue that scaling up agents brings sharply diminishing returns in behavioral adequacy relative to generation cost. They identify under-specified requirements, including ambiguous setup and implicit expected outcomes, as sources of an information gap; they describe requirement-grounded harnesses as a promising direction, not a demonstrated solution. How would the reported gap change on requirements with clearer setup and expected outcomes?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide further detail about the evaluation or the benchmark instances. The reported adequacy measure is kill-completeness against the ground-truth test’s mutants; the abstract does not establish how the results translate to software testing in deployment.","speaker":{"name":"Qi, Binhang, Liu, Yiming, Lin, Yun, Zhou, Yangjie, Sun, Hailong, Dong, Jin Song","roleAtSource":"Article author"},"source":{"title":"Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36584","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"Qi and coauthors introduce Req2Test-Bench, a benchmark for evaluating agents that generate tests from software requirements.","excerpt":"We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The benchmark draws on developer-authored behavior-driven development artifacts from 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","excerpt":"recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The authors report that the best-performing configuration produced passing tests for 82.8% of instances, but behaviorally adequate tests for 33.8%.","excerpt":"produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate.","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"},{"text":"The authors identify under-specified requirements as a source of the information gap and say requirement-grounded harnesses are a promising direction.","excerpt":"under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes","url":"https://arxiv.org/abs/2610.36584","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36584","checkedAt":"2026-11-14T04:34:09.136Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the reported gap change on requirements with clearer setup and expected outcomes?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Qi and coauthors benchmark LLM agents’ test generation from software requirements","summary":"In their arXiv paper, Binhang Qi and coauthors introduce Req2Test-Bench to test whether LLM agents can generate tests that check required behavior when the implementation is withheld. The benchmark contains 5,086 instances drawn from developer-authored artifacts across 8 open-source projects, 5 programming languages and 4 BDD frameworks.\n\nThe authors report a gap between tests that pass and tests that are behaviorally adequate: in the best-performing of 8 agent–model–reasoning configurations, 82.8% of generated tests passed, but 33.8% met the benchmark’s kill-completeness measure. That measure requires a generated test to kill every mutant killed by the ground-truth test.\n\nThe authors argue that scaling up agents brings sharply diminishing returns in behavioral adequacy relative to generation cost. They identify under-specified requirements, including ambiguous setup and implicit expected outcomes, as sources of an information gap; they describe requirement-grounded harnesses as a promising direction, not a demonstrated solution. How would the reported gap change on requirements with clearer setup and expected outcomes?","question":"How would the reported gap change on requirements with clearer setup and expected outcomes?","claims":[{"text":"Qi and coauthors introduce Req2Test-Bench, a benchmark for evaluating agents that generate tests from software requirements.","excerpt":"We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances","support":"source-stated"},{"text":"The benchmark draws on developer-authored behavior-driven development artifacts from 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","excerpt":"recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks.","support":"source-stated"},{"text":"The authors report that the best-performing configuration produced passing tests for 82.8% of instances, but behaviorally adequate tests for 33.8%.","excerpt":"produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate.","support":"source-stated"},{"text":"The authors identify under-specified requirements as a source of the information gap and say requirement-grounded harnesses are a promising direction.","excerpt":"under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide further detail about the evaluation or the benchmark instances.","The reported adequacy measure is kill-completeness against the ground-truth test’s mutants; the abstract does not establish how the results translate to software testing in deployment."],"mediaId":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","correctionReason":null,"id":"b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1","publicationId":"00516048-524f-4d9d-8684-70fe938e9f92","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36584","sourceTitle":"Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T07:12:41.239Z","createdAt":"2026-11-14T04:02:37.099Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","kind":"article","url":"https://arxiv.org/abs/2610.36584","sourceUrl":"https://arxiv.org/abs/2610.36584","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T20:16:45.029Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36584","checkedAt":"2026-11-14T04:34:09.136Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3264b0e02f9a1bec9646123212e878f73f7e00aa849be4d3cee248f2f102e880","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"56d48da8e326d6a98a4fa75c93f00eb0d4ff2d40b59e7244272ed7b9ba50db75","decidedAt":"2026-10-30T20:16:45.528Z"},"provenance":{"textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observationId":"6ae145f6d8ae52765b457c3c2b7b7ac8138806cde210c52ce8ca3138f0b4d335","retrievedAt":"2026-10-30T20:16:44.561Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"b46b78ba-2a49-448c-864f-e6c0659d0c73"},"reviewer":{"model":"gpt-6-luna","runId":"ed484e94-62bd-4dc4-845a-189cdba4c8de"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-14T04:02:37.099Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Qi, Binhang","url":null,"kind":"unknown"},{"name":"Liu, Yiming","url":null,"kind":"unknown"},{"name":"Lin, Yun","url":null,"kind":"unknown"},{"name":"Zhou, Yangjie","url":null,"kind":"unknown"},{"name":"Sun, Hailong","url":null,"kind":"unknown"},{"name":"Dong, Jin Song","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Qi, Binhang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qi, Binhang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Liu, Yiming","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Yiming","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Lin, Yun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lin, Yun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Zhou, Yangjie","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yangjie","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Sun, Hailong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Hailong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}},{"actor":{"name":"Dong, Jin Song","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dong, Jin Song","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36584","locator":"head/meta[name=author|citation_author]"},"assertionId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T04:34:09.136Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","sourceUrl":"https://arxiv.org/abs/2610.36584","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36584","snapshotId":"569e612c506fa8f5294f5dcfd3f9623f2b7414306c5321efe63a616aa9ccc5bf","contentSha256":"c19980e77e2ef75a115816775b87eeeb4e8f2e345cc95c76cd675f7ecb449ac5","textHash":"96596e7bbe5bd0b6d6ad09a937aa40a321c96b62da2e4cb5406e9fb7b24b7500","observedAt":"2026-11-14T04:34:09.136Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b395f86dda33776059ef7bcb6aa6f50d3a05e1e99dd60fff573f24fa270de3a1/source"}}}],"audits":[{"publicationId":"00516048-524f-4d9d-8684-70fe938e9f92","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the benchmark, results and arguments to Binhang Qi and coauthors in an arXiv paper, which matches the listed authors. Findings stay framed as what the authors report or argue.","excerpt":"Authors:Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong"},{"dimension":"summary","status":"uncertain","explanation":"The numbers and framing are faithful, but a non-specialist would struggle with the central finding. The adequacy measure is stated as killing every mutant killed by the ground-truth test, with no plain explanation of mutant, kill or ground-truth test. BDD is not expanded in the summary. The summary also omits the source's stated reason the task matters: tests act as validation constraints, or a harness, guiding agents that compile requirements into code. Smallest repair: add one plain sentence on why requirement-to-test generation matters and briefly gloss mutants and BDD, or supply tooltips for them.","excerpt":"requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-27, which matches the v1 submission date on the arXiv page. The later feed date is correctly not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"All claims are directly supported by the abstract: 5,086 instances, 8 projects, 5 languages, 4 BDD frameworks, 82.8% passing versus 33.8% adequate, diminishing returns relative to cost, under-specified requirements as a cause, and harnesses presented as a promising direction rather than a proven fix. The question follows from the source's stated cause without asserting an answer.","excerpt":"produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page for this paper. This is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-14T04:02:37.099Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}