arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building

arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building

The authors of the arXiv cs.AI paper introduce ViBench 2.0 to test whether AI models can build working software through successive feature requests, rather than complete a single, fully specified coding task. They say existing coding benchmarks are nearing saturation and that their platform uses ViBench 2.0 to select models. The benchmark contains 17 full-stack web products modeled on widely used software. Together, the products grow over 241 feature requests and are judged when finished: a browser agent runs 544 test plans, and a build passes only if every plan passes. In evaluations of nine models, the authors report that the three strongest pass up to 98% of test plans but deliver a fully working product in at most 65% of builds; no open-weight model delivers one. This gap is the paper’s central finding: passing many checks does not necessarily mean the whole product works. Across three releases of one model family, the authors report that pass@1 on public products rose from 3% to 53%, while cost per build fell by more than half. They release eight products with the evaluation materials and grader, hold out nine, and report lessons from production use. The abstract does not give results for the held-out products or describe those lessons.

Publication: 936a7ddd-9c17-4ce7-87d0-1ce80b1d80dc

attribution — uncertain: The draft attributes claims to 'the authors' and refers to 'their platform' without naming Replit. The abstract identifies the platform, and the authors' stake matters because they use the benchmark to pick their own platform's models. Smallest repair: name Replit as the authors' agentic engineering platform, keeping the paper's claims attributed to its authors.

On Replit, our agentic engineering platform, users now build entire products without writing code

summary — uncertain: Mostly faithful, but a general reader is missing key meaning. The draft omits the motivating context that users now build whole products without writing code. It also leaves central terms unexplained: pass@1, open-weight, full-stack and long-horizon. It drops the authors' own framing that 'the gap is closing fast'. And it replaces the technical term 'harness' with 'evaluation materials'. Repair: restore the no-code and Replit context and the closing-fast framing, keep 'harness', and add tooltips for pass@1 and open-weight.

The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53%

date — pass: The originalPublished date of 2026-11-03 matches the arXiv v1 submission date. The later feed entry on 2026-11-05 is correctly not used.

[Submitted on 3 Nov 2026]

support — pass: The figures are all attributed and match the abstract: 17 products, 241 feature requests, 544 test plans, nine models, up to 98% of test plans vs at most 65% of builds, no open-weight model succeeding, 3% to 53%, cost more than halved, and eight released vs nine held out. The 'central finding' framing reflects the abstract's 'reliability gap'. The question and limitations follow from the evidence without overclaiming.

we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one

media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.