Cohere’s authors examine how LLMs handle coding instructions across sessions

Cohere’s authors examine how LLMs handle coding instructions across sessions

On February 19, 2025, Nathanaël Carraz Rakotonirina and coauthors published a paper on whether LLMs can collaborate effectively over long interactions. They introduce MemoryCode, a synthetic multi-session dataset that tests whether models can follow simple coding instructions while filtering out irrelevant information. The authors report that tested models handle instructions presented in isolation, but even GPT-4o performs worse when instructions are spread across sessions. They suggest that models struggle to retrieve and integrate information across long instruction chains, and describe this as a fundamental limitation on effective collaboration in long interactions. The result points to a specific challenge for using LLMs in work that depends on carrying instructions forward across multiple sessions: doing well on isolated tasks may not translate to following a longer sequence of directions. The paper’s evidence here concerns a synthetic dataset and simple coding instructions; the page does not provide results showing how the findings carry over to other tasks or real-world work.

Publication: f77b027d-45fb-6c35-03e5-fb5ce1aae831

attribution — uncertain: The page is published on Cohere's research site and lists eight authors, with Nathanaël Carraz Rakotonirina first. The title calls them "Cohere’s authors," but the page does not state any author's affiliation, and some authors may belong to other institutions. Smallest repair: retitle along the lines of "Paper on Cohere's site examines..." or "Researchers examine...". The body's attribution to Rakotonirina and coauthors is accurate.

Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici

summary — pass: The summary faithfully states the question, the MemoryCode dataset, the finding that performance drops across sessions even for GPT-4o, the authors' suggested cause, and their framing of this as a fundamental limitation. It keeps the authors' force while noting the scope (synthetic data, simple coding instructions). LLM is defined on the page. The terms multi-session and isolated instructions are understandable in context. The question follows from the evidence and is not leading.

Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.

date — pass: The displayed date of February 19, 2025 matches the page dateline and the UTC calendar day of originalPublished. modifiedAt is the same day. The page is a publisher page, so no archive disclosure is needed. The reading does not use the related-works listings, which may have been added later, and it uses no hindsight framing.

Feb 19, 2025

support — pass: Each claim is directly stated in the abstract. The implication about carrying instructions across sessions is modest and hedged ("may not translate"), and it follows from the reported isolation-versus-sessions contrast. The limitations correctly note that the page gives no evidence on generalization.

While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions.

media — pass: No media was selected and no verified media candidates were supplied, so the empty selection is appropriate.