{"id":"ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab","title":"Cohere’s authors examine how LLMs handle coding instructions across sessions","argument":"On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 6, 2026; originally published February 19, 2025. The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions.","speaker":{"name":"Cohere","roleAtSource":"Publisher"},"source":{"title":"From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","publisher":"Cohere","format":"Article","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","publishedAt":"2025-02-19T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors.","excerpt":"Feb 19, 2025\nFrom Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information.","excerpt":"we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o.","excerpt":"While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions.","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions.","excerpt":"Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Cohere’s authors examine how LLMs handle coding instructions across sessions","summary":"On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work.","question":"Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions?","claims":[{"text":"The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors.","excerpt":"Feb 19, 2025\nFrom Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","support":"source-stated"},{"text":"The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information.","excerpt":"we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information","support":"source-stated"},{"text":"The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o.","excerpt":"While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions.","support":"source-stated"},{"text":"The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions.","excerpt":"Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.","support":"source-stated"}],"limitations":["Text as retrieved on November 6, 2026; originally published February 19, 2025.","The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions."],"mediaId":null,"correctionReason":null,"id":"ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab","publicationId":"f77b027d-45fb-6c35-03e5-fb5ce1aae831","version":1,"sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","sourceTitle":"From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","publisher":"Cohere","publishedAt":"2025-02-19T00:00:00.000Z","originalPublished":{"value":"2025-02-19T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-11-06T03:17:58.523Z","createdAt":"2026-11-15T08:48:25.505Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"5493ceaf477843eaf0e322c5de866eea6c9b4f64725ec254924a3758b83c5e38","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-11-06T03:17:56.172Z"},"provenance":{"textHash":"ad31d9877910f79e51b2d6526ecaddd9364a974ae4fc4cbdf3d8486f1f050d99","observationId":"8a02dd67b8acadf46a3df4d4eb48eda3f8a731288152accaf93372ac1fc15de3","retrievedAt":"2026-11-06T03:19:38.148Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"a7494287-8cf7-0804-b3de-56dd6d1f77c9"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"ba3e96db-7020-8e76-5c46-5ad97e66f9c2"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T08:43:40.557Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8","formats":{"html":"/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8","markdown":"/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8.md","json":"/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab","revision":"f77b027d-45fb-6c35-03e5-fb5ce1aae831","revisionKind":"publication-id","title":"Cohere’s authors examine how LLMs handle coding instructions across sessions","url":"https://escapemeridian.com/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8"},"source":{"type":"source","id":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","revision":"ad31d9877910f79e51b2d6526ecaddd9364a974ae4fc4cbdf3d8486f1f050d99","revisionKind":"sha256-source-text","title":"From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab","title":"Cohere’s authors examine how LLMs handle coding instructions across sessions","argument":"On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 6, 2026; originally published February 19, 2025. The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions.","speaker":{"name":"Cohere","roleAtSource":"Publisher"},"source":{"title":"From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","publisher":"Cohere","format":"Article","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","publishedAt":"2025-02-19T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors.","excerpt":"Feb 19, 2025\nFrom Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information.","excerpt":"we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o.","excerpt":"While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions.","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"},{"text":"The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions.","excerpt":"Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.","url":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Cohere’s authors examine how LLMs handle coding instructions across sessions","summary":"On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work.","question":"Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions?","claims":[{"text":"The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors.","excerpt":"Feb 19, 2025\nFrom Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","support":"source-stated"},{"text":"The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information.","excerpt":"we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information","support":"source-stated"},{"text":"The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o.","excerpt":"While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions.","support":"source-stated"},{"text":"The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions.","excerpt":"Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.","support":"source-stated"}],"limitations":["Text as retrieved on November 6, 2026; originally published February 19, 2025.","The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions."],"mediaId":null,"correctionReason":null,"id":"ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab","publicationId":"f77b027d-45fb-6c35-03e5-fb5ce1aae831","version":1,"sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","sourceTitle":"From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions","publisher":"Cohere","publishedAt":"2025-02-19T00:00:00.000Z","originalPublished":{"value":"2025-02-19T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-11-06T03:17:58.523Z","createdAt":"2026-11-15T08:48:25.505Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"5493ceaf477843eaf0e322c5de866eea6c9b4f64725ec254924a3758b83c5e38","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-11-06T03:17:56.172Z"},"provenance":{"textHash":"ad31d9877910f79e51b2d6526ecaddd9364a974ae4fc4cbdf3d8486f1f050d99","observationId":"8a02dd67b8acadf46a3df4d4eb48eda3f8a731288152accaf93372ac1fc15de3","retrievedAt":"2026-11-06T03:19:38.148Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"a7494287-8cf7-0804-b3de-56dd6d1f77c9"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"ba3e96db-7020-8e76-5c46-5ad97e66f9c2"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T08:43:40.557Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"f77b027d-45fb-6c35-03e5-fb5ce1aae831","verdict":"uncertain","findings":[{"dimension":"attribution","status":"uncertain","explanation":"The page is published on Cohere's research site and lists eight authors, with Nathanaël Carraz Rakotonirina first. The title calls them \"Cohere’s authors,\" but the page does not state any author's affiliation, and some authors may belong to other institutions. Smallest repair: retitle along the lines of \"Paper on Cohere's site examines...\" or \"Researchers examine...\". The body's attribution to Rakotonirina and coauthors is accurate.","excerpt":"Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici"},{"dimension":"summary","status":"pass","explanation":"The summary faithfully states the question, the MemoryCode dataset, the finding that performance drops across sessions even for GPT-4o, the authors' suggested cause, and their framing of this as a fundamental limitation. It keeps the authors' force while noting the scope (synthetic data, simple coding instructions). LLM is defined on the page. The terms multi-session and isolated instructions are understandable in context. The question follows from the evidence and is not leading.","excerpt":"Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions."},{"dimension":"date","status":"pass","explanation":"The displayed date of February 19, 2025 matches the page dateline and the UTC calendar day of originalPublished. modifiedAt is the same day. The page is a publisher page, so no archive disclosure is needed. The reading does not use the related-works listings, which may have been added later, and it uses no hindsight framing.","excerpt":"Feb 19, 2025"},{"dimension":"support","status":"pass","explanation":"Each claim is directly stated in the abstract. The implication about carrying instructions across sessions is modest and hedged (\"may not translate\"), and it follows from the reported isolation-versus-sessions contrast. The limitations correctly note that the page gives no evidence on generalization.","excerpt":"While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions."},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified media candidates were supplied, so the empty selection is appropriate.","excerpt":null}],"createdAt":"2026-11-15T08:43:40.557Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}