Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores

Ilan Bigio and Ted Sanders argue in OpenAI that harness settings reshape benchmark scores

Ilan Bigio and Ted Sanders published this account on July 29, 2026. They argue that ARC-AGI-3 scores reflect not only a model’s abilities but also the harness—the setup that manages its interactions with the benchmark. In their tests, GPT‑5.6 Sol scored 13.3% with the official harness and 38.3% when reasoning was retained and compaction enabled. The authors say the official setup discarded the model’s private reasoning after each action and eventually removed older actions from its history. Retaining reasoning let the model carry forward prior thinking; compaction helped preserve what it had learned across longer runs. Together, the settings let GPT‑5.6 Sol (max) achieve roughly 3x the score with 6x fewer output tokens, according to OpenAI. The authors recommend these settings for API developers seeking performance and for model comparisons, because they say they better match use in ChatGPT and Codex. Their argument matters because a benchmark score can depend on evaluation design, not just the model being tested. The account does not establish how the results generalize beyond this public task set or whether other models would benefit similarly.

Publication: c4d9ad0d-6516-0fc2-1c07-076ea4106cfb

attribution — pass: The piece is a single OpenAI research publication credited to Ilan Bigio and Ted Sanders. The reading attributes the claims, scores and recommendations to them and to OpenAI, and labels the Escape Meridian inference as interpretation. The 'Keep reading' list at the end is a page sidebar, not the source itself.

Author Ilan Bigio, Ted Sanders

summary — uncertain: The reading faithfully conveys the authors' argument, and harness is explained. However, two central terms lack plain-language context. Compaction is not defined in the reading or in the definitions list; the source describes it as summarizing a conversation that grows too long and then continuing. ARC-AGI-3 is also never described as a benchmark of unfamiliar 2D puzzle games. Smallest repair: add one clause for each, or a term-library tooltip for compaction.

If a conversation grows too long, we summarize it and continue.

date — pass: The displayed date of July 29, 2026 matches the source dateline and the originalPublished instant (UTC day). The archive-copy status is disclosed in the limitations. The reading uses no later material or hindsight framing; the sidebar entries are not relied on.

July 29, 2026 ResearchPublication

support — pass: All of the following are stated in the source: the scores of 13.3% and 38.3%, the discarded reasoning, the rolling truncation, the roughly 3x score with 6x fewer output tokens, and the recommendations. The interpretation claim is labeled and closely tracks the authors' own conclusion. The question and the limitations note real gaps without asserting unsupported benefits.

the harness used a rolling truncation window, causing older actions to become invisible as the history grew

media — pass: No media was selected and no verified media candidates were supplied, so there is no identity or relevance issue.