# Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores [Canonical reading](https://escapemeridian.com/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b) Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source OpenAI · Article Original source date: Jul 29, 2026 (2026-07-29T15:00:00.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-06T07:13:24.731Z. First observed: 2026-09-23T07:21:10.911Z. Retrieved: 2026-10-24T03:01:52.775Z. - [How enabling two settings tripled our scores on the ARC-AGI-3 benchmark](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Limits Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy. OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks. ## Open question How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3? ## Source-stated OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set. > With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source-stated OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action. > after each game action, all private reasoning was discarded. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source-stated OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens. > Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Escape Meridian interpretation Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability. > evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source and contributor context OpenAI · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Model tokens Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens. - [Definition reference](https://developers.google.com/machine-learning/glossary#token) ## API An application programming interface: a defined way for software to request data or actions from another system. - [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Editorial provenance Version 1. Publication ID: c4d9ad0d-6516-0fc2-1c07-076ea4106cfb. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T07:08:45.831Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_screen. Observation: 976b1f0741ee20dac5bada168e5050aaa2ebaa54f8fe1983d2e9c6910eddd810. Source text hash: 5d79c91ed6764a47fc2de6d649809be49443e22e14d4bbb6400e9993f772dcf7. ## Revision and source-check history ### Version 1 — 2026-11-06T07:13:24.731Z Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly. ## Original source OpenAI · Article Original source date: Jul 29, 2026 (2026-07-29T15:00:00.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-06T07:13:24.731Z. First observed: 2026-09-23T07:21:10.911Z. Retrieved: 2026-10-24T03:01:52.775Z. - [How enabling two settings tripled our scores on the ARC-AGI-3 benchmark](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Limits Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy. OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks. ## Open question How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3? ## Source-stated OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set. > With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source-stated OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action. > after each game action, all private reasoning was discarded. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source-stated OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens. > Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Escape Meridian interpretation Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability. > evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. - [Supporting source](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) ## Source and contributor context OpenAI · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Model tokens Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens. - [Definition reference](https://developers.google.com/machine-learning/glossary#token) ## API An application programming interface: a defined way for software to request data or actions from another system. - [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Editorial provenance Version 1. Publication ID: c4d9ad0d-6516-0fc2-1c07-076ea4106cfb. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T07:08:45.831Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_screen. Observation: 976b1f0741ee20dac5bada168e5050aaa2ebaa54f8fe1983d2e9c6910eddd810. Source text hash: 5d79c91ed6764a47fc2de6d649809be49443e22e14d4bbb6400e9993f772dcf7. ### uncertain — 2026-11-06T07:08:45.831Z — anthropic/claude-opus-5.5 Publication: c4d9ad0d-6516-0fc2-1c07-076ea4106cfb attribution — pass: The piece is a single OpenAI research publication credited to Ilan Bigio and Ted Sanders. The reading attributes the claims, scores and recommendations to them and to OpenAI, and labels the Escape Meridian inference as interpretation. The 'Keep reading' list at the end is a page sidebar, not the source itself. > Author Ilan Bigio, Ted Sanders summary — uncertain: The reading faithfully conveys the authors' argument, and harness is explained. However, two central terms lack plain-language context. Compaction is not defined in the reading or in the definitions list; the source describes it as summarizing a conversation that grows too long and then continuing. ARC-AGI-3 is also never described as a benchmark of unfamiliar 2D puzzle games. Smallest repair: add one clause for each, or a term-library tooltip for compaction. > If a conversation grows too long, we summarize it and continue. date — pass: The displayed date of July 29, 2026 matches the source dateline and the originalPublished instant (UTC day). The archive-copy status is disclosed in the limitations. The reading uses no later material or hindsight framing; the sidebar entries are not relied on. > July 29, 2026 ResearchPublication support — pass: All of the following are stated in the source: the scores of 13.3% and 38.3%, the discarded reasoning, the rolling truncation, the roughly 3x score with 6x fewer output tokens, and the recommendations. The interpretation claim is labeled and closely tracks the authors' own conclusion. The question and the limitations note real gaps without asserting unsupported benefits. > the harness used a rolling truncation window, causing older actions to become invisible as the history grew media — pass: No media was selected and no verified media candidates were supplied, so there is no identity or relevance issue. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b.json)