{"id":"7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2","title":"Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores","argument":"Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy. OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks.","speaker":{"name":"OpenAI","roleAtSource":"Publisher"},"source":{"title":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark","publisher":"OpenAI","format":"Article","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","publishedAt":"2026-07-29T15:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set.","excerpt":"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action.","excerpt":"after each game action, all private reasoning was discarded.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens.","excerpt":"Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability.","excerpt":"evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores","summary":"Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly.","question":"How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3?","claims":[{"text":"OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set.","excerpt":"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.","support":"source-stated"},{"text":"OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action.","excerpt":"after each game action, all private reasoning was discarded.","support":"source-stated"},{"text":"OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens.","excerpt":"Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.","support":"source-stated"},{"text":"Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability.","excerpt":"evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.","support":"interpretation"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy.","OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks."],"mediaId":null,"correctionReason":null,"id":"7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2","publicationId":"c4d9ad0d-6516-0fc2-1c07-076ea4106cfb","version":1,"sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","sourceTitle":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark","publisher":"OpenAI","publishedAt":"2026-07-29T15:00:00.000Z","originalPublished":{"value":"2026-07-29T15:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-09-23T07:21:10.911Z","createdAt":"2026-11-06T07:13:24.731Z","historical":true,"entityIds":["openai"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c5644015bbc29b6146fd8b98cd96b7d656e0143255b862a9ca85c75ca377fcda","reasonCodes":["bridge_screen"],"rubricVersion":"bridge-screen-v5","confidence":0.59,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-29T16:44:43.468Z"},"provenance":{"textHash":"5d79c91ed6764a47fc2de6d649809be49443e22e14d4bbb6400e9993f772dcf7","observationId":"976b1f0741ee20dac5bada168e5050aaa2ebaa54f8fe1983d2e9c6910eddd810","retrievedAt":"2026-10-24T03:01:52.775Z","kind":"archive-copy","preparer":{"model":"openai/gpt-6-luna","runId":"b27fe4f6-bfd5-62b2-2856-ef0530d99509"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"adc5b7f4-8f36-e9ab-c82a-d363e518be22"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T07:08:45.831Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b","formats":{"html":"/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b","markdown":"/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b.md","json":"/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-tokens","name":"Model tokens","explanation":"Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.","referenceUrl":"https://developers.google.com/machine-learning/glossary#token"},{"id":"term-api","name":"API","explanation":"An application programming interface: a defined way for software to request data or actions from another system.","referenceUrl":"https://developer.mozilla.org/en-US/docs/Glossary/API"}],"entities":[{"id":"openai","name":"OpenAI","kind":"organization","context":"OpenAI develops AI models and products and publishes research, release announcements and evaluations.","referenceUrls":["https://openai.com/news/"],"path":"/entities/openai","json":"/entities/openai.json","markdown":"/entities/openai.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2","revision":"c4d9ad0d-6516-0fc2-1c07-076ea4106cfb","revisionKind":"publication-id","title":"Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores","url":"https://escapemeridian.com/readings/ilan-bigio-and-ted-sanders-argue-in-openai-that-harness-7391d17b"},"source":{"type":"source","id":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","revision":"5d79c91ed6764a47fc2de6d649809be49443e22e14d4bbb6400e9993f772dcf7","revisionKind":"sha256-source-text","title":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2","title":"Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores","argument":"Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy. OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks.","speaker":{"name":"OpenAI","roleAtSource":"Publisher"},"source":{"title":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark","publisher":"OpenAI","format":"Article","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","publishedAt":"2026-07-29T15:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set.","excerpt":"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action.","excerpt":"after each game action, all private reasoning was discarded.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens.","excerpt":"Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Source-stated"},{"text":"Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability.","excerpt":"evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.","url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores","summary":"Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly.","question":"How would retaining reasoning and enabling compaction change scores for other models or on benchmarks beyond ARC-AGI-3?","claims":[{"text":"OpenAI’s authors report that GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% with retained reasoning and compaction on the ARC-AGI-3 public set.","excerpt":"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%.","support":"source-stated"},{"text":"OpenAI’s authors say the official harness discarded the model’s private reasoning after each game action.","excerpt":"after each game action, all private reasoning was discarded.","support":"source-stated"},{"text":"OpenAI’s authors report that retaining reasoning and enabling compaction let GPT‑5.6 Sol (max) achieve roughly three times the score with six times fewer output tokens.","excerpt":"Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.","support":"source-stated"},{"text":"Escape Meridian reads OpenAI’s account as showing that a benchmark result can reflect evaluation setup as well as model capability.","excerpt":"evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting.","support":"interpretation"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 29, 2026. This is an archived copy.","OpenAI reports results on the ARC-AGI-3 public task set; the account does not establish whether the settings have similar effects on other models or benchmarks."],"mediaId":null,"correctionReason":null,"id":"7391d17be2398389e30a72c577e562aed1a09e2b3c86b7d751cd013e277fb3c2","publicationId":"c4d9ad0d-6516-0fc2-1c07-076ea4106cfb","version":1,"sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","sourceTitle":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark","publisher":"OpenAI","publishedAt":"2026-07-29T15:00:00.000Z","originalPublished":{"value":"2026-07-29T15:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-09-23T07:21:10.911Z","createdAt":"2026-11-06T07:13:24.731Z","historical":true,"entityIds":["openai"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c5644015bbc29b6146fd8b98cd96b7d656e0143255b862a9ca85c75ca377fcda","reasonCodes":["bridge_screen"],"rubricVersion":"bridge-screen-v5","confidence":0.59,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-29T16:44:43.468Z"},"provenance":{"textHash":"5d79c91ed6764a47fc2de6d649809be49443e22e14d4bbb6400e9993f772dcf7","observationId":"976b1f0741ee20dac5bada168e5050aaa2ebaa54f8fe1983d2e9c6910eddd810","retrievedAt":"2026-10-24T03:01:52.775Z","kind":"archive-copy","preparer":{"model":"openai/gpt-6-luna","runId":"b27fe4f6-bfd5-62b2-2856-ef0530d99509"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"adc5b7f4-8f36-e9ab-c82a-d363e518be22"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T07:08:45.831Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"c4d9ad0d-6516-0fc2-1c07-076ea4106cfb","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The piece is a single OpenAI research publication credited to Ilan Bigio and Ted Sanders. The reading attributes the claims, scores and recommendations to them and to OpenAI, and labels the Escape Meridian inference as interpretation. The 'Keep reading' list at the end is a page sidebar, not the source itself.","excerpt":"Author\nIlan Bigio, Ted Sanders"},{"dimension":"summary","status":"uncertain","explanation":"The reading faithfully conveys the authors' argument, and harness is explained. However, two central terms lack plain-language context. Compaction is not defined in the reading or in the definitions list; the source describes it as summarizing a conversation that grows too long and then continuing. ARC-AGI-3 is also never described as a benchmark of unfamiliar 2D puzzle games. Smallest repair: add one clause for each, or a term-library tooltip for compaction.","excerpt":"If a conversation grows too long, we summarize it and continue."},{"dimension":"date","status":"pass","explanation":"The displayed date of July 29, 2026 matches the source dateline and the originalPublished instant (UTC day). The archive-copy status is disclosed in the limitations. The reading uses no later material or hindsight framing; the sidebar entries are not relied on.","excerpt":"July 29, 2026\nResearchPublication"},{"dimension":"support","status":"pass","explanation":"All of the following are stated in the source: the scores of 13.3% and 38.3%, the discarded reasoning, the rolling truncation, the roughly 3x score with 6x fewer output tokens, and the recommendations. The interpretation claim is labeled and closely tracks the authors' own conclusion. The question and the limitations note real gaps without asserting unsupported benefits.","excerpt":"the harness used a rolling truncation window, causing older actions to become invisible as the history grew"},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified media candidates were supplied, so there is no identity or relevance issue.","excerpt":null}],"createdAt":"2026-11-06T07:08:45.831Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}