{"id":"e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8","title":"arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building","argument":"The authors of the arXiv cs.AI paper introduce ViBench 2.0 to test whether AI models can build working software through successive feature requests, rather than complete a single, fully specified coding task. They say existing coding benchmarks are nearing saturation and that their platform uses ViBench 2.0 to select models. The benchmark contains 17 full-stack web products modeled on widely used software. Together, the products grow over 241 feature requests and are judged when finished: a browser agent runs 544 test plans, and a build passes only if every plan passes. In evaluations of nine models, the authors report that the three strongest pass up to 98% of test plans but deliver a fully working product in at most 65% of builds; no open-weight model delivers one. This gap is the paper’s central finding: passing many checks does not necessarily mean the whole product works. Across three releases of one model family, the authors report that pass@1 on public products rose from 3% to 53%, while cost per build fell by more than half. They release eight products with the evaluation materials and grader, hold out nine, and report lessons from production use. The abstract does not give results for the held-out products or describe those lessons.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not identify the nine evaluated models. The abstract does not report results for the nine held-out products or compare them with the public products. The abstract says the authors report lessons from production use but does not describe those lessons.","speaker":{"name":"Furman, Daniel, Kirani, Preeya, Sheikh, Ibrahim, Li, Zhen, Zhong, Peter, Catasta, Michele","roleAtSource":"Article author"},"source":{"title":"ViBench 2.0: A Benchmark for Long-Horizon Software Building","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.05620","publishedAt":"2026-11-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present ViBench 2.0 as a benchmark for long-horizon software building and say they use it to select models for their platform.","excerpt":"We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models.","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The benchmark evaluates 17 web products after successive feature requests, using a strict requirement that every test plan pass.","excerpt":"Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"In the authors’ evaluation, strong performance on test plans did not always yield a fully working product.","excerpt":"the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The authors report improving pass@1 and lower build costs across three releases of one model family.","excerpt":"across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The authors release eight products and hold out nine.","excerpt":"We release eight products with the harness and grader, hold out nine","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05620","checkedAt":"2026-11-09T07:40:09.386Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do models perform on the nine held-out products compared with the eight public products?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building","summary":"The authors of the arXiv cs.AI paper introduce ViBench 2.0 to test whether AI models can build working software through successive feature requests, rather than complete a single, fully specified coding task. They say existing coding benchmarks are nearing saturation and that their platform uses ViBench 2.0 to select models. The benchmark contains 17 full-stack web products modeled on widely used software. Together, the products grow over 241 feature requests and are judged when finished: a browser agent runs 544 test plans, and a build passes only if every plan passes. In evaluations of nine models, the authors report that the three strongest pass up to 98% of test plans but deliver a fully working product in at most 65% of builds; no open-weight model delivers one. This gap is the paper’s central finding: passing many checks does not necessarily mean the whole product works. Across three releases of one model family, the authors report that pass@1 on public products rose from 3% to 53%, while cost per build fell by more than half. They release eight products with the evaluation materials and grader, hold out nine, and report lessons from production use. The abstract does not give results for the held-out products or describe those lessons.","question":"How do models perform on the nine held-out products compared with the eight public products?","claims":[{"text":"The authors present ViBench 2.0 as a benchmark for long-horizon software building and say they use it to select models for their platform.","excerpt":"We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models.","support":"source-stated"},{"text":"The benchmark evaluates 17 web products after successive feature requests, using a strict requirement that every test plan pass.","excerpt":"Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does","support":"source-stated"},{"text":"In the authors’ evaluation, strong performance on test plans did not always yield a fully working product.","excerpt":"the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds","support":"source-stated"},{"text":"The authors report improving pass@1 and lower build costs across three releases of one model family.","excerpt":"across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half","support":"source-stated"},{"text":"The authors release eight products and hold out nine.","excerpt":"We release eight products with the harness and grader, hold out nine","support":"source-stated"}],"limitations":["The abstract does not identify the nine evaluated models.","The abstract does not report results for the nine held-out products or compare them with the public products.","The abstract says the authors report lessons from production use but does not describe those lessons."],"mediaId":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","correctionReason":null,"id":"e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8","publicationId":"936a7ddd-9c17-4ce7-87d0-1ce80b1d80dc","version":1,"sourceUrl":"https://arxiv.org/abs/2611.05620","sourceTitle":"ViBench 2.0: A Benchmark for Long-Horizon Software Building","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:39.608Z","createdAt":"2026-11-09T07:25:20.954Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","kind":"article","url":"https://arxiv.org/abs/2611.05620","sourceUrl":"https://arxiv.org/abs/2611.05620","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T07:24:07.494Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05620","checkedAt":"2026-11-09T07:40:09.386Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","confidence":0.9400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"26ce4203bedef547880d802938307bff3285ab1545c96820b89c5719e84e52fb","decidedAt":"2026-11-09T07:24:09.018Z"},"provenance":{"textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observationId":"ba54b18495bb897323f07c7d31e6f9b29fa4906e801b899621c8dd8369e33ad9","retrievedAt":"2026-11-09T07:24:07.246Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"75d4462e-a92a-4a0f-88b1-c5190ecfb970"},"reviewer":{"model":"gpt-6-luna","runId":"547297d6-ca53-4f96-8a51-5f267d29c847"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-09T07:25:20.954Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Furman, Daniel","url":null,"kind":"unknown"},{"name":"Kirani, Preeya","url":null,"kind":"unknown"},{"name":"Sheikh, Ibrahim","url":null,"kind":"unknown"},{"name":"Li, Zhen","url":null,"kind":"unknown"},{"name":"Zhong, Peter","url":null,"kind":"unknown"},{"name":"Catasta, Michele","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Furman, Daniel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Furman, Daniel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Kirani, Preeya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kirani, Preeya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Sheikh, Ibrahim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sheikh, Ibrahim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Li, Zhen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Zhen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Zhong, Peter","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhong, Peter","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Catasta, Michele","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Catasta, Michele","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.05620","snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-vibench-2-0-a-benchmark-for-long-horizon-software-e306f8a0","formats":{"html":"/readings/arxiv-cs-ai-vibench-2-0-a-benchmark-for-long-horizon-software-e306f8a0","markdown":"/readings/arxiv-cs-ai-vibench-2-0-a-benchmark-for-long-horizon-software-e306f8a0.md","json":"/readings/arxiv-cs-ai-vibench-2-0-a-benchmark-for-long-horizon-software-e306f8a0.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-open-weights","name":"Open weights","explanation":"A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.","referenceUrl":"https://opensource.org/ai/open-weights"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8","revision":"936a7ddd-9c17-4ce7-87d0-1ce80b1d80dc","revisionKind":"publication-id","title":"arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building","url":"https://escapemeridian.com/readings/arxiv-cs-ai-vibench-2-0-a-benchmark-for-long-horizon-software-e306f8a0"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.05620","revision":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","revisionKind":"sha256-source-text","title":"ViBench 2.0: A Benchmark for Long-Horizon Software Building","url":"https://arxiv.org/abs/2611.05620"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8","title":"arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building","argument":"The authors of the arXiv cs.AI paper introduce ViBench 2.0 to test whether AI models can build working software through successive feature requests, rather than complete a single, fully specified coding task. They say existing coding benchmarks are nearing saturation and that their platform uses ViBench 2.0 to select models. The benchmark contains 17 full-stack web products modeled on widely used software. Together, the products grow over 241 feature requests and are judged when finished: a browser agent runs 544 test plans, and a build passes only if every plan passes. In evaluations of nine models, the authors report that the three strongest pass up to 98% of test plans but deliver a fully working product in at most 65% of builds; no open-weight model delivers one. This gap is the paper’s central finding: passing many checks does not necessarily mean the whole product works. Across three releases of one model family, the authors report that pass@1 on public products rose from 3% to 53%, while cost per build fell by more than half. They release eight products with the evaluation materials and grader, hold out nine, and report lessons from production use. The abstract does not give results for the held-out products or describe those lessons.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not identify the nine evaluated models. The abstract does not report results for the nine held-out products or compare them with the public products. The abstract says the authors report lessons from production use but does not describe those lessons.","speaker":{"name":"Furman, Daniel, Kirani, Preeya, Sheikh, Ibrahim, Li, Zhen, Zhong, Peter, Catasta, Michele","roleAtSource":"Article author"},"source":{"title":"ViBench 2.0: A Benchmark for Long-Horizon Software Building","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.05620","publishedAt":"2026-11-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present ViBench 2.0 as a benchmark for long-horizon software building and say they use it to select models for their platform.","excerpt":"We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models.","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The benchmark evaluates 17 web products after successive feature requests, using a strict requirement that every test plan pass.","excerpt":"Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"In the authors’ evaluation, strong performance on test plans did not always yield a fully working product.","excerpt":"the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The authors report improving pass@1 and lower build costs across three releases of one model family.","excerpt":"across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"},{"text":"The authors release eight products and hold out nine.","excerpt":"We release eight products with the harness and grader, hold out nine","url":"https://arxiv.org/abs/2611.05620","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05620","checkedAt":"2026-11-09T07:40:09.386Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do models perform on the nine held-out products compared with the eight public products?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: ViBench 2.0, a benchmark for long-horizon software building","summary":"The authors of the arXiv cs.AI paper introduce ViBench 2.0 to test whether AI models can build working software through successive feature requests, rather than complete a single, fully specified coding task. They say existing coding benchmarks are nearing saturation and that their platform uses ViBench 2.0 to select models. The benchmark contains 17 full-stack web products modeled on widely used software. Together, the products grow over 241 feature requests and are judged when finished: a browser agent runs 544 test plans, and a build passes only if every plan passes. In evaluations of nine models, the authors report that the three strongest pass up to 98% of test plans but deliver a fully working product in at most 65% of builds; no open-weight model delivers one. This gap is the paper’s central finding: passing many checks does not necessarily mean the whole product works. Across three releases of one model family, the authors report that pass@1 on public products rose from 3% to 53%, while cost per build fell by more than half. They release eight products with the evaluation materials and grader, hold out nine, and report lessons from production use. The abstract does not give results for the held-out products or describe those lessons.","question":"How do models perform on the nine held-out products compared with the eight public products?","claims":[{"text":"The authors present ViBench 2.0 as a benchmark for long-horizon software building and say they use it to select models for their platform.","excerpt":"We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models.","support":"source-stated"},{"text":"The benchmark evaluates 17 web products after successive feature requests, using a strict requirement that every test plan pass.","excerpt":"Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does","support":"source-stated"},{"text":"In the authors’ evaluation, strong performance on test plans did not always yield a fully working product.","excerpt":"the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds","support":"source-stated"},{"text":"The authors report improving pass@1 and lower build costs across three releases of one model family.","excerpt":"across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half","support":"source-stated"},{"text":"The authors release eight products and hold out nine.","excerpt":"We release eight products with the harness and grader, hold out nine","support":"source-stated"}],"limitations":["The abstract does not identify the nine evaluated models.","The abstract does not report results for the nine held-out products or compare them with the public products.","The abstract says the authors report lessons from production use but does not describe those lessons."],"mediaId":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","correctionReason":null,"id":"e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8","publicationId":"936a7ddd-9c17-4ce7-87d0-1ce80b1d80dc","version":1,"sourceUrl":"https://arxiv.org/abs/2611.05620","sourceTitle":"ViBench 2.0: A Benchmark for Long-Horizon Software Building","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:39.608Z","createdAt":"2026-11-09T07:25:20.954Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","kind":"article","url":"https://arxiv.org/abs/2611.05620","sourceUrl":"https://arxiv.org/abs/2611.05620","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T07:24:07.494Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05620","checkedAt":"2026-11-09T07:40:09.386Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"25294cd886a2e607d0e1d9f41ea73b8a7a224f08aff707d1d5d09ce5cb1b3714","confidence":0.9400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"26ce4203bedef547880d802938307bff3285ab1545c96820b89c5719e84e52fb","decidedAt":"2026-11-09T07:24:09.018Z"},"provenance":{"textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observationId":"ba54b18495bb897323f07c7d31e6f9b29fa4906e801b899621c8dd8369e33ad9","retrievedAt":"2026-11-09T07:24:07.246Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"75d4462e-a92a-4a0f-88b1-c5190ecfb970"},"reviewer":{"model":"gpt-6-luna","runId":"547297d6-ca53-4f96-8a51-5f267d29c847"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-09T07:25:20.954Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Furman, Daniel","url":null,"kind":"unknown"},{"name":"Kirani, Preeya","url":null,"kind":"unknown"},{"name":"Sheikh, Ibrahim","url":null,"kind":"unknown"},{"name":"Li, Zhen","url":null,"kind":"unknown"},{"name":"Zhong, Peter","url":null,"kind":"unknown"},{"name":"Catasta, Michele","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Furman, Daniel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Furman, Daniel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Kirani, Preeya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kirani, Preeya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Sheikh, Ibrahim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sheikh, Ibrahim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Li, Zhen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Zhen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Zhong, Peter","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhong, Peter","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}},{"actor":{"name":"Catasta, Michele","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Catasta, Michele","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05620","locator":"head/meta[name=author|citation_author]"},"assertionId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T07:40:09.386Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","sourceUrl":"https://arxiv.org/abs/2611.05620","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.05620","snapshotId":"79bcd2017847dd1bdd3bc026187baa5957e623ec70c30b65ebd11d15228ec326","contentSha256":"e2479b08156551fd59414e47fdce1d30fe0971c159ebbde6810f37b0f25bf0bb","textHash":"711127c2e8f7266f9dfb452bad4134c2fa5869457495d00a1ea03cff5acf2351","observedAt":"2026-11-09T07:40:09.386Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e306f8a02932032a182c307c9e7268a6833e7c34f5ac6045687b30be5a250de8/source"}}}],"audits":[{"publicationId":"936a7ddd-9c17-4ce7-87d0-1ce80b1d80dc","verdict":"uncertain","findings":[{"dimension":"attribution","status":"uncertain","explanation":"The draft attributes claims to 'the authors' and refers to 'their platform' without naming Replit. The abstract identifies the platform, and the authors' stake matters because they use the benchmark to pick their own platform's models. Smallest repair: name Replit as the authors' agentic engineering platform, keeping the paper's claims attributed to its authors.","excerpt":"On Replit, our agentic engineering platform, users now build entire products without writing code"},{"dimension":"summary","status":"uncertain","explanation":"Mostly faithful, but a general reader is missing key meaning. The draft omits the motivating context that users now build whole products without writing code. It also leaves central terms unexplained: pass@1, open-weight, full-stack and long-horizon. It drops the authors' own framing that 'the gap is closing fast'. And it replaces the technical term 'harness' with 'evaluation materials'. Repair: restore the no-code and Replit context and the closing-fast framing, keep 'harness', and add tooltips for pass@1 and open-weight.","excerpt":"The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53%"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-03 matches the arXiv v1 submission date. The later feed entry on 2026-11-05 is correctly not used.","excerpt":"[Submitted on 3 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"The figures are all attributed and match the abstract: 17 products, 241 feature requests, 544 test plans, nine models, up to 98% of test plans vs at most 65% of builds, no open-weight model succeeding, 3% to 53%, cost more than halved, and eight released vs nine held out. The 'central finding' framing reflects the abstract's 'reliability gap'. The question and limitations follow from the evidence without overclaiming.","excerpt":"we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page, which is the original source.","excerpt":null}],"createdAt":"2026-11-09T07:25:20.954Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}