Al Mamun, Islam and Ameen test specialist contexts for software assurance

Al Mamun, Islam and Ameen test specialist contexts for software assurance

In a paper listed on arXiv, S M Abdulla Al Mamun, Akif Islam and Mohd. Ruhul Ameen examine whether adding specialist language-model contexts improves software assurance when deterministic tool evidence is available. Their five experiments cover Java warnings, OWASP Benchmark assessment, Defects4J test generation, SV-COMP verdicts and misleading evidence packages. The authors report mixed results. For eight development warnings, raising the context count from 1 to 50 increased token use 49-fold without improving unsafe clearance. On 32 SV-COMP tasks, raw verifier feedback increased mean correct model verdicts from 12.7 to 22.0, but incorrect verdicts also rose, from one to seven. A JBMC-plus-JVM validation baseline returned 26 correct verdicts and six unresolved outcomes. For two sets of 120 evidence packages, the models rejected 70–100% of applicable counterexamples, while construction-specific scope checks classified every package correctly. The authors say these descriptive findings support deterministic evidence admission in the tested settings, while pointing to limits in verdict aggregation, feedback interpretation and evaluation coverage. The results describe these experiments, not software assurance performance in all settings. An open question is whether these findings hold across broader software assurance tasks and evidence beyond the tested cases.

Publication: 0dcb13a8-bb4a-4def-845d-b4645d8c46a7

attribution — pass: The draft correctly attributes the paper to S M Abdulla Al Mamun, Akif Islam and Mohd. Ruhul Ameen on arXiv, and presents the findings as what the authors report.

Authors:S M Abdulla Al Mamun, Akif Islam, Mohd. Ruhul Ameen

summary — uncertain: The figures are faithful, but central terms are unexplained for non-specialists: specialist language-model contexts, deterministic tool evidence, unsafe clearance, SV-COMP, JBMC-plus-JVM, counterexamples and deterministic evidence admission. Readers may also misread 'rejected 70–100% of applicable counterexamples' as a success, when the contrast implies wrong rejections. The draft also omits a scope condition: the tested model was Qwen2.5-Coder-7B-Instruct, with a 1.5B check. Smallest repair: add plain-language glosses or tooltips, state that these rejections were errors, and name the model size.

The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests.

date — pass: originalPublished is 2026-10-27, which matches the arXiv v1 submission date. The feed date, 2026-10-29, is later and was correctly not used.

[Submitted on 27 Oct 2026]

support — pass: Each claim matches the abstract's numbers and wording, including the 49-fold token increase, 12.7 to 22.0 correct verdicts, one to seven incorrect verdicts, the 26-correct baseline and every package correctly classified. The limits, including the Defects4J result and the 'tested settings' scope, are preserved. The question follows from the authors' own stated limits and does not assert an unsupported conclusion.

These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access point to the source.