# Cohere’s authors examine how LLMs handle coding instructions across sessions [Canonical reading](https://escapemeridian.com/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8) On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Cohere · Article Original source date: Feb 19, 2025 (2025-02-19T00:00:00.000Z). Date basis: Date stated on the publisher’s page. Added to Escape Meridian: 2026-11-15T08:48:25.505Z. First observed: 2026-11-06T03:17:58.523Z. Retrieved: 2026-11-06T03:19:38.148Z. - [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Limits Text as retrieved on November 6, 2026; originally published February 19, 2025. The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions. ## Open question Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions? ## Source-stated The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors. > Feb 19, 2025 > From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information. > we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o. > While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions. > Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions. - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source and contributor context Cohere · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Editorial provenance Version 1. Publication ID: f77b027d-45fb-6c35-03e5-fb5ce1aae831. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T08:43:40.557Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 8a02dd67b8acadf46a3df4d4eb48eda3f8a731288152accaf93372ac1fc15de3. Source text hash: ad31d9877910f79e51b2d6526ecaddd9364a974ae4fc4cbdf3d8486f1f050d99. ## Revision and source-check history ### Version 1 — 2026-11-15T08:48:25.505Z On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work. ## Original source Cohere · Article Original source date: Feb 19, 2025 (2025-02-19T00:00:00.000Z). Date basis: Date stated on the publisher’s page. Added to Escape Meridian: 2026-11-15T08:48:25.505Z. First observed: 2026-11-06T03:17:58.523Z. Retrieved: 2026-11-06T03:19:38.148Z. - [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Limits Text as retrieved on November 6, 2026; originally published February 19, 2025. The page provides an abstract but no detailed results or evidence about how the findings generalize beyond the synthetic dataset and simple coding instructions. ## Open question Do the authors’ findings on MemoryCode extend to longer interactions involving tasks beyond simple coding instructions? ## Source-stated The paper was published on February 19, 2025, and names Nathanaël Carraz Rakotonirina and coauthors as authors. > Feb 19, 2025 > From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors introduce MemoryCode, a synthetic multi-session dataset for testing whether models can follow simple coding instructions amid irrelevant information. > we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors report that tested models handle isolated instructions well, but performance deteriorates when instructions are spread across sessions, including for GPT-4o. > While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source-stated The authors suggest that difficulty retrieving and integrating information over long instruction chains accounts for the decline, which they describe as limiting effective collaboration in long interactions. > Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions. - [Supporting source](https://cohere.com/research/papers/from-tools-to-teammates-evaluating-llms-in-multi-session-coding-interactions-2025-02-19) ## Source and contributor context Cohere · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Editorial provenance Version 1. Publication ID: f77b027d-45fb-6c35-03e5-fb5ce1aae831. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T08:43:40.557Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 8a02dd67b8acadf46a3df4d4eb48eda3f8a731288152accaf93372ac1fc15de3. Source text hash: ad31d9877910f79e51b2d6526ecaddd9364a974ae4fc4cbdf3d8486f1f050d99. ### uncertain — 2026-11-15T08:43:40.557Z — anthropic/claude-opus-5.5 Publication: f77b027d-45fb-6c35-03e5-fb5ce1aae831 attribution — uncertain: The page is published on Cohere's research site and lists eight authors, with Nathanaël Carraz Rakotonirina first. The title calls them "Cohere’s authors," but the page does not state any author's affiliation, and some authors may belong to other institutions. Smallest repair: retitle along the lines of "Paper on Cohere's site examines..." or "Researchers examine...". The body's attribution to Rakotonirina and coauthors is accurate. > Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici summary — pass: The summary faithfully states the question, the MemoryCode dataset, the finding that performance drops across sessions even for GPT-4o, the authors' suggested cause, and their framing of this as a fundamental limitation. It keeps the authors' force while noting the scope (synthetic data, simple coding instructions). LLM is defined on the page. The terms multi-session and isolated instructions are understandable in context. The question follows from the evidence and is not leading. > Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions. date — pass: The displayed date of February 19, 2025 matches the page dateline and the UTC calendar day of originalPublished. modifiedAt is the same day. The page is a publisher page, so no archive disclosure is needed. The reading does not use the related-works listings, which may have been added later, and it uses no hindsight framing. > Feb 19, 2025 support — pass: Each claim is directly stated in the abstract. The implication about carrying instructions across sessions is modest and hedged ("may not translate"), and it follows from the reported isolation-versus-sessions contrast. The limitations correctly note that the page gives no evidence on generalization. > While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. media — pass: No media was selected and no verified media candidates were supplied, so the empty selection is appropriate. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/ed64e5e8199067ef8f9e8c5298ca98d6330783202f9f1cada8e5f8a3cc1078ab/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/cohere-s-authors-examine-how-llms-handle-coding-instructions-ed64e5e8.json)