{"id":"3b989f3a3017b9e57f257e400688dd1e718a9cc788f87272e47b6b90a444a237","title":"Wenting Zhao and coauthors present Commit0 in a Cohere paper","argument":"Published December 2, 2024, Wenting Zhao and coauthors introduce Commit0, a benchmark for testing whether AI agents can build software libraries from scratch. Agents receive a specification describing a library’s API and interactive unit tests, then try to implement the API; their code is checked by running the tests. The authors designed the benchmark to test more than one-shot code generation: agents must handle long specifications, feedback across stages and complex dependencies. Their experiments find that current agents can pass some tests but cannot yet reproduce full libraries. They also report that interactive feedback helps models pass more tests. The authors present this as support for benchmarks that let models use such feedback. The abstract gives no test scores or size of the improvement, so the scale of that reported benefit is unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 9, 2026; originally published December 2, 2024. The retrieved page provides the abstract, not the experiments’ test scores or a quantified comparison of results with and without interactive feedback.","speaker":{"name":"Cohere","roleAtSource":"Publisher"},"source":{"title":"Commit0: Library Generation from Scratch","publisher":"Cohere","format":"Article","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","publishedAt":"2024-12-02T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Wenting Zhao and coauthors introduce Commit0 as a benchmark in which AI agents attempt to implement software libraries from specifications and interactive tests.","excerpt":"we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch.","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The authors say current agents pass some unit tests but do not yet fully reproduce libraries.","excerpt":"current agents can pass some unit tests, none can yet fully reproduce full libraries.","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The authors report that interactive feedback helps models generate code that passes more unit tests.","excerpt":"interactive feedback is quite useful for models to generate code that passes more unit tests","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The source is dated December 2, 2024.","excerpt":"Dec 02, 2024","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much does interactive feedback improve test results, and how does that vary across libraries?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Wenting Zhao and coauthors present Commit0 in a Cohere paper","summary":"Published December 2, 2024, Wenting Zhao and coauthors introduce Commit0, a benchmark for testing whether AI agents can build software libraries from scratch. Agents receive a specification describing a library’s API and interactive unit tests, then try to implement the API; their code is checked by running the tests. The authors designed the benchmark to test more than one-shot code generation: agents must handle long specifications, feedback across stages and complex dependencies. Their experiments find that current agents can pass some tests but cannot yet reproduce full libraries. They also report that interactive feedback helps models pass more tests. The authors present this as support for benchmarks that let models use such feedback. The abstract gives no test scores or size of the improvement, so the scale of that reported benefit is unknown.","question":"How much does interactive feedback improve test results, and how does that vary across libraries?","claims":[{"text":"Wenting Zhao and coauthors introduce Commit0 as a benchmark in which AI agents attempt to implement software libraries from specifications and interactive tests.","excerpt":"we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch.","support":"source-stated"},{"text":"The authors say current agents pass some unit tests but do not yet fully reproduce libraries.","excerpt":"current agents can pass some unit tests, none can yet fully reproduce full libraries.","support":"source-stated"},{"text":"The authors report that interactive feedback helps models generate code that passes more unit tests.","excerpt":"interactive feedback is quite useful for models to generate code that passes more unit tests","support":"source-stated"},{"text":"The source is dated December 2, 2024.","excerpt":"Dec 02, 2024","support":"source-stated"}],"limitations":["Text as retrieved on November 9, 2026; originally published December 2, 2024.","The retrieved page provides the abstract, not the experiments’ test scores or a quantified comparison of results with and without interactive feedback."],"mediaId":null,"correctionReason":null,"id":"3b989f3a3017b9e57f257e400688dd1e718a9cc788f87272e47b6b90a444a237","publicationId":"d20b024e-4af6-7944-e478-46a6629d9a30","version":1,"sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","sourceTitle":"Commit0: Library Generation from Scratch","publisher":"Cohere","publishedAt":"2024-12-02T00:00:00.000Z","originalPublished":{"value":"2024-12-02T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-11-09T03:21:59.010Z","createdAt":"2026-11-16T08:13:23.723Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"b0b981cc6e18c3d2500691889fa6c4dc672e1f642069beb8802e9fa6bec17c99","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-11-09T03:21:56.838Z"},"provenance":{"textHash":"3f82722b3a8d3e30519dbc481d7ff5d7b72bdb578dc74ed6c003c9790fc184f4","observationId":"7b284e65e2a705a5bff8fe511f9052080060bae7359327a76ab780a5e65ed57f","retrievedAt":"2026-11-09T03:23:38.179Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"ee62d80b-106b-4dfa-357f-ae9882e974b7"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"51843368-3b7f-73ce-38b0-58eca34dcdb4"}},"audit":{"status":"passed","reviewedAt":"2026-11-16T08:08:51.906Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/wenting-zhao-and-coauthors-present-commit0-in-a-cohere-paper-3b989f3a","formats":{"html":"/readings/wenting-zhao-and-coauthors-present-commit0-in-a-cohere-paper-3b989f3a","markdown":"/readings/wenting-zhao-and-coauthors-present-commit0-in-a-cohere-paper-3b989f3a.md","json":"/readings/wenting-zhao-and-coauthors-present-commit0-in-a-cohere-paper-3b989f3a.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-api","name":"API","explanation":"An application programming interface: a defined way for software to request data or actions from another system.","referenceUrl":"https://developer.mozilla.org/en-US/docs/Glossary/API"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"3b989f3a3017b9e57f257e400688dd1e718a9cc788f87272e47b6b90a444a237","revision":"d20b024e-4af6-7944-e478-46a6629d9a30","revisionKind":"publication-id","title":"Wenting Zhao and coauthors present Commit0 in a Cohere paper","url":"https://escapemeridian.com/readings/wenting-zhao-and-coauthors-present-commit0-in-a-cohere-paper-3b989f3a"},"source":{"type":"source","id":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","revision":"3f82722b3a8d3e30519dbc481d7ff5d7b72bdb578dc74ed6c003c9790fc184f4","revisionKind":"sha256-source-text","title":"Commit0: Library Generation from Scratch","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"3b989f3a3017b9e57f257e400688dd1e718a9cc788f87272e47b6b90a444a237","title":"Wenting Zhao and coauthors present Commit0 in a Cohere paper","argument":"Published December 2, 2024, Wenting Zhao and coauthors introduce Commit0, a benchmark for testing whether AI agents can build software libraries from scratch. Agents receive a specification describing a library’s API and interactive unit tests, then try to implement the API; their code is checked by running the tests. The authors designed the benchmark to test more than one-shot code generation: agents must handle long specifications, feedback across stages and complex dependencies. Their experiments find that current agents can pass some tests but cannot yet reproduce full libraries. They also report that interactive feedback helps models pass more tests. The authors present this as support for benchmarks that let models use such feedback. The abstract gives no test scores or size of the improvement, so the scale of that reported benefit is unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 9, 2026; originally published December 2, 2024. The retrieved page provides the abstract, not the experiments’ test scores or a quantified comparison of results with and without interactive feedback.","speaker":{"name":"Cohere","roleAtSource":"Publisher"},"source":{"title":"Commit0: Library Generation from Scratch","publisher":"Cohere","format":"Article","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","publishedAt":"2024-12-02T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Wenting Zhao and coauthors introduce Commit0 as a benchmark in which AI agents attempt to implement software libraries from specifications and interactive tests.","excerpt":"we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch.","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The authors say current agents pass some unit tests but do not yet fully reproduce libraries.","excerpt":"current agents can pass some unit tests, none can yet fully reproduce full libraries.","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The authors report that interactive feedback helps models generate code that passes more unit tests.","excerpt":"interactive feedback is quite useful for models to generate code that passes more unit tests","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"},{"text":"The source is dated December 2, 2024.","excerpt":"Dec 02, 2024","url":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much does interactive feedback improve test results, and how does that vary across libraries?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Wenting Zhao and coauthors present Commit0 in a Cohere paper","summary":"Published December 2, 2024, Wenting Zhao and coauthors introduce Commit0, a benchmark for testing whether AI agents can build software libraries from scratch. Agents receive a specification describing a library’s API and interactive unit tests, then try to implement the API; their code is checked by running the tests. The authors designed the benchmark to test more than one-shot code generation: agents must handle long specifications, feedback across stages and complex dependencies. Their experiments find that current agents can pass some tests but cannot yet reproduce full libraries. They also report that interactive feedback helps models pass more tests. The authors present this as support for benchmarks that let models use such feedback. The abstract gives no test scores or size of the improvement, so the scale of that reported benefit is unknown.","question":"How much does interactive feedback improve test results, and how does that vary across libraries?","claims":[{"text":"Wenting Zhao and coauthors introduce Commit0 as a benchmark in which AI agents attempt to implement software libraries from specifications and interactive tests.","excerpt":"we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch.","support":"source-stated"},{"text":"The authors say current agents pass some unit tests but do not yet fully reproduce libraries.","excerpt":"current agents can pass some unit tests, none can yet fully reproduce full libraries.","support":"source-stated"},{"text":"The authors report that interactive feedback helps models generate code that passes more unit tests.","excerpt":"interactive feedback is quite useful for models to generate code that passes more unit tests","support":"source-stated"},{"text":"The source is dated December 2, 2024.","excerpt":"Dec 02, 2024","support":"source-stated"}],"limitations":["Text as retrieved on November 9, 2026; originally published December 2, 2024.","The retrieved page provides the abstract, not the experiments’ test scores or a quantified comparison of results with and without interactive feedback."],"mediaId":null,"correctionReason":null,"id":"3b989f3a3017b9e57f257e400688dd1e718a9cc788f87272e47b6b90a444a237","publicationId":"d20b024e-4af6-7944-e478-46a6629d9a30","version":1,"sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","sourceTitle":"Commit0: Library Generation from Scratch","publisher":"Cohere","publishedAt":"2024-12-02T00:00:00.000Z","originalPublished":{"value":"2024-12-02T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-11-09T03:21:59.010Z","createdAt":"2026-11-16T08:13:23.723Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/commit0-library-generation-from-scratch-2024-12-02","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"b0b981cc6e18c3d2500691889fa6c4dc672e1f642069beb8802e9fa6bec17c99","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-11-09T03:21:56.838Z"},"provenance":{"textHash":"3f82722b3a8d3e30519dbc481d7ff5d7b72bdb578dc74ed6c003c9790fc184f4","observationId":"7b284e65e2a705a5bff8fe511f9052080060bae7359327a76ab780a5e65ed57f","retrievedAt":"2026-11-09T03:23:38.179Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"ee62d80b-106b-4dfa-357f-ae9882e974b7"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"51843368-3b7f-73ce-38b0-58eca34dcdb4"}},"audit":{"status":"passed","reviewedAt":"2026-11-16T08:08:51.906Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"d20b024e-4af6-7944-e478-46a6629d9a30","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The page lists Wenting Zhao and six coauthors, and the paper is hosted on Cohere's research site. The reading attributes the benchmark and its findings to the authors throughout, so attribution is clear. The page is a single paper abstract, not a listing; the related-works links are a page sidebar.","excerpt":"Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, Alexander M Rush"},{"dimension":"summary","status":"pass","explanation":"The reading leads with the actor and the development: a benchmark testing whether AI agents can build libraries from a spec and tests. Benchmark, AI agent and API are defined beside the reading, and running the tests is explained in context. It faithfully covers the design goals, the mixed results and the feedback finding. The question follows from the abstract's lack of figures and is not leading.","excerpt":"Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests"},{"dimension":"date","status":"pass","explanation":"The source dateline reads Dec 02, 2024, which matches originalPublished (2024-12-02, the same UTC day). The modified date is the same day. The reading uses no later events. The related-works sidebar may be a later addition, but the reading does not use it. The source is a publisher page, so no archive disclosure is needed.","excerpt":"Dec 02, 2024"},{"dimension":"support","status":"pass","explanation":"Every claim is backed by the abstract: the benchmark's purpose, partial test passing without full library reproduction, and the benefit of interactive feedback. The reading correctly notes that the abstract gives no scores, so it does not overstate the size of the benefit or the real-world capability.","excerpt":"Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use."},{"dimension":"media","status":"pass","explanation":"No media candidates were supplied, and the reading selects none, consistent with the decision.","excerpt":null}],"createdAt":"2026-11-16T08:08:51.906Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}