Escape Meridian

Passos, Patrick · Nov 14, 2026

One Run Is Not Enough: Reminding a small Gemma 4 coding agent of its last finding made no measurable difference in a paired test

Markdown · JSON · Original source

Patrick Passos reports a test of a simple memory aid for a coding agent: when earlier findings are cut as it runs out of room, the harness automatically puts the agent’s last finding back in front of it. Across 98 paired coding tasks using Gemma 4 31B (QAT, 4-bit), 47 tasks were solved with the reminder and 48 without; Passos reports no measurable difference. The results varied across the study: the first 48 pairs favored memory, while the later 50 pointed the other way, so the initial direction did not replicate. Passos also reports substantial run-to-run noise and says the study’s design could reliably detect only larger effects. The agent rarely wrote notes, and the reminder was injected in only some compactions, while the first request was larger. These details matter because the result concerns a specific harness mechanism, not every way an agent might retain or use memory. Passos explicitly says the treatment does not test agent-written memory. The paper’s conclusion is that single-run memory claims on task sets of this size sit inside run-to-run noise; it does not establish whether a different memory approach would have a different result.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXiv cs.AIArticle · arxiv.orgOpen original source

Original source

cs.AI · Article

Source details

Original source date: Nov 14, 2026 (2026-11-14).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T12:32:51.955Z. First observed: 2026-11-17T08:29:11.236Z. Retrieved: 2026-11-17T12:31:59.273Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The experiment used Gemma 4 31B (QAT, 4-bit) in the Gemma 4 Developer ; the results do not establish how other models or memory mechanisms would perform. The paper says the treatment does not test agent-written memory. The abstract reports that the reminder was injected at only 47% to 65% of compactions and that the model rarely wrote notes.

Open question

Would a test of agent-written memory produce a different result from the harness mechanism tested here?

Source-stated

Passos tested a harness mechanism that automatically put a coding agent’s last finding back in front of it after earlier findings were cut.

We tested a simple fix: automatically putting the agent's last finding back in front of it.

Source-stated

Across 98 paired tasks, the reminder condition solved 47 tasks and the no-reminder condition solved 48; Passos reports no measurable difference.

across all 98 tasks, 47 were solved with it and 48 without (-1.0 points, 95% interval [-8.3, +6.3], exact McNemar p=1.0).

Source-stated

The initial result favored memory, but the later extension pointed the other way, so the direction did not replicate.

the first 48 pairs, was 29/48 with memory against 25/48 without (+8.3 points, [-2.1, +18.3], p=0.22); the extension of 50 tasks, decided after that result, pointed the other way (18/50 against 23/50, -10.0 points, p=0.125), so the sign did not replicate.

Source-stated

Passos says the experiment tests a harness mechanism, not agent-written memory.

The treatment is a harness mechanism and does not test agent-written memory.

Source and contributor context

Passos, Patrick · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Agent harness

An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: 610a1f01-758b-4c97-805e-9cf6c36a06e1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T12:32:51.955Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 61653f85ea213fe77afb4988b46ca7c86300cf3fa4b3e284eb9af5914521f753. Source text hash: 67e07ad65f4e96d27df39ed4a5d8248cf8546b091a769212c0eaf88f789ba067.

Revision and source-check history

Version 1 · 2026-11-17T12:32:51.955Z

One Run Is Not Enough: Reminding a small Gemma 4 coding agent of its last finding made no measurable difference in a paired test

Patrick Passos reports a test of a simple memory aid for a coding agent: when earlier findings are cut as it runs out of room, the harness automatically puts the agent’s last finding back in front of it. Across 98 paired coding tasks using Gemma 4 31B (QAT, 4-bit), 47 tasks were solved with the reminder and 48 without; Passos reports no measurable difference. The results varied across the study: the first 48 pairs favored memory, while the later 50 pointed the other way, so the initial direction did not replicate. Passos also reports substantial run-to-run noise and says the study’s design could reliably detect only larger effects. The agent rarely wrote notes, and the reminder was injected in only some compactions, while the first request was larger. These details matter because the result concerns a specific harness mechanism, not every way an agent might retain or use memory. Passos explicitly says the treatment does not test agent-written memory. The paper’s conclusion is that single-run memory claims on task sets of this size sit inside run-to-run noise; it does not establish whether a different memory approach would have a different result.

Original source

cs.AI · Article

Source details

Original source date: Nov 14, 2026 (2026-11-14).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T12:32:51.955Z. First observed: 2026-11-17T08:29:11.236Z. Retrieved: 2026-11-17T12:31:59.273Z.

Limits

The experiment used Gemma 4 31B (QAT, 4-bit) in the Gemma 4 Developer ; the results do not establish how other models or memory mechanisms would perform. The paper says the treatment does not test agent-written memory. The abstract reports that the reminder was injected at only 47% to 65% of compactions and that the model rarely wrote notes.

Open question

Would a test of agent-written memory produce a different result from the harness mechanism tested here?

Source-stated

Passos tested a harness mechanism that automatically put a coding agent’s last finding back in front of it after earlier findings were cut.

We tested a simple fix: automatically putting the agent's last finding back in front of it.

Source-stated

Across 98 paired tasks, the reminder condition solved 47 tasks and the no-reminder condition solved 48; Passos reports no measurable difference.

across all 98 tasks, 47 were solved with it and 48 without (-1.0 points, 95% interval [-8.3, +6.3], exact McNemar p=1.0).

Source-stated

The initial result favored memory, but the later extension pointed the other way, so the direction did not replicate.

the first 48 pairs, was 29/48 with memory against 25/48 without (+8.3 points, [-2.1, +18.3], p=0.22); the extension of 50 tasks, decided after that result, pointed the other way (18/50 against 23/50, -10.0 points, p=0.125), so the sign did not replicate.

Source-stated

Passos says the experiment tests a harness mechanism, not agent-written memory.

The treatment is a harness mechanism and does not test agent-written memory.

Source and contributor context

Passos, Patrick · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Agent harness

An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: 610a1f01-758b-4c97-805e-9cf6c36a06e1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T12:32:51.955Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 61653f85ea213fe77afb4988b46ca7c86300cf3fa4b3e284eb9af5914521f753. Source text hash: 67e07ad65f4e96d27df39ed4a5d8248cf8546b091a769212c0eaf88f789ba067.

passed · 2026-11-17T12:32:51.955Z · anthropic/claude-opus-5.5

Publication: 610a1f01-758b-4c97-805e-9cf6c36a06e1

attribution — pass: The draft attributes the study and its conclusions to Patrick Passos, the sole listed author. It keeps the findings framed as what Passos reports rather than as established fact.

Authors:Patrick Passos

summary — pass: The title matches the source. The summary clearly explains the reminder mechanism, the 47 vs 48 result, the non-replicating split, run-to-run noise, limited power and the harness-only scope. Two optional repairs: say that the first 48 pairs were the primary outcome and that the extension was decided after that result, and add tooltips for 'compactions', 'harness' and 'QAT, 4-bit'. The question follows from the source's stated scope limit.

Together these results show that single-run memory claims on task sets of this size sit inside run-to-run noise.

date — pass: originalPublished is 2026-11-14, which matches the arXiv v1 submission date. The feed date of 2026-11-17 is later and correctly is not used.

[Submitted on 14 Nov 2026]

support — pass: Every claim and limitation is supported by verbatim excerpts. The noise, power and context-cost points are paraphrased faithfully. 'Reminder was injected' slightly narrows the source's 'something was injected', but the meaning is not materially altered. No unsupported benefits or costs are implied.

the first request is larger and something was injected at only 47% to 65% of compactions

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.