# Melanie Mitchell argues that ARC scores may reward search over abstraction

[Canonical reading](https://escapemeridian.com/readings/melanie-mitchell-argues-that-arc-scores-may-reward-search-over-f2038a9f)

Melanie Mitchell’s post was published August 6, 2024. She explains that ARC tasks ask a solver to infer a rule from a few grid transformations and apply it to a new grid. Mitchell argues that Ryan Greenblatt’s method, which uses GPT-4o to generate, test and revise many Python programs, raises a question about whether extensive search can raise scores without demonstrating the abstraction ARC is meant to measure. She also describes the MindsAI team’s approach and says its rising score appears largely due to increased computation; whether that approach will support abstraction and analogical thinking in the real world remains open. Mitchell is excited that the ARC Prize draws attention to abstraction and analogy, but warns that making a benchmark a target can shift attention from its original purpose. She hopes a more dynamic private evaluation set could mitigate that risk. The reading does not establish how far either approach can scale or whether higher ARC scores will reflect general, humanlike abstraction.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Melanie Mitchell · Article

Original source date: Aug 6, 2024 (2024-08-06T21:05:45.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-17T14:09:01.224Z. First observed: 2026-10-23T14:16:07.802Z. Retrieved: 2026-10-23T14:17:41.015Z.

- [On the “ARC-AGI” $1 Million Reasoning Challenge](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Limits

Text as retrieved on October 23, 2026; originally published August 6, 2024. Mitchell says the MindsAI method’s details have not yet been published and describes its workings as her understanding based on interviews and podcasts. The post says Greenblatt’s blog post does not give the actual prompt or details on how promising programs and their votes are selected. The post does not establish whether increased scores on ARC indicate abstraction or analogical thinking in the real world.


## Open question

Would a more dynamic private evaluation set help distinguish progress in abstraction from higher scores produced by scaling up search?


## Source-stated

Mitchell argues that Greenblatt’s method raises the question of whether extensive computation and prompt engineering fit ARC’s aim of measuring abstraction.

> it brings up the question of how much brute-force computation can be used before the approach violates the spirit of the ARC challenge

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell reports that Greenblatt’s scores came from samples of the training and public evaluation sets, and that he warned the results might be slightly overoptimistic.

> I only ran my solution on a randomly selected set of 100 train and 100 test problems. The resulting standard error of the mean is reasonably tight (5%), but this means my results might be slightly overoptimistic.

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell says the increase in MindsAI’s score appears largely due to increased computation, and leaves open whether the approach will be relevant to abstraction and analogical thinking in the real world.

> the rising score of this method, from 34% to 43% over the course of several months, is largely due to increased computation use

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell warns that targeting ARC scores could draw attention away from the capacities the benchmark is intended to measure, and suggests changing the private evaluation tasks over time as a possible mitigation.

> One thing that could mitigate this kind of “overfitting” to a target is to make the private evaluation set more dynamic

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Editorial provenance

Version 1. Publication ID: 0f406435-3fc7-203b-0976-b75fef1bf7a2.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T14:05:06.405Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 6e92251c0ee606654e08bfdd99b0c104a9e6e60f16f4404cb946c1d49eafd388. Source text hash: 2fda5bbd93d18103a87e6eebe134f684bd264c323216e133925b2c2a63cbd0a9.


## Revision and source-check history

### Version 1 — 2026-11-17T14:09:01.224Z

Melanie Mitchell’s post was published August 6, 2024. She explains that ARC tasks ask a solver to infer a rule from a few grid transformations and apply it to a new grid. Mitchell argues that Ryan Greenblatt’s method, which uses GPT-4o to generate, test and revise many Python programs, raises a question about whether extensive search can raise scores without demonstrating the abstraction ARC is meant to measure. She also describes the MindsAI team’s approach and says its rising score appears largely due to increased computation; whether that approach will support abstraction and analogical thinking in the real world remains open. Mitchell is excited that the ARC Prize draws attention to abstraction and analogy, but warns that making a benchmark a target can shift attention from its original purpose. She hopes a more dynamic private evaluation set could mitigate that risk. The reading does not establish how far either approach can scale or whether higher ARC scores will reflect general, humanlike abstraction.

## Original source

Melanie Mitchell · Article

Original source date: Aug 6, 2024 (2024-08-06T21:05:45.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-17T14:09:01.224Z. First observed: 2026-10-23T14:16:07.802Z. Retrieved: 2026-10-23T14:17:41.015Z.

- [On the “ARC-AGI” $1 Million Reasoning Challenge](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Limits

Text as retrieved on October 23, 2026; originally published August 6, 2024. Mitchell says the MindsAI method’s details have not yet been published and describes its workings as her understanding based on interviews and podcasts. The post says Greenblatt’s blog post does not give the actual prompt or details on how promising programs and their votes are selected. The post does not establish whether increased scores on ARC indicate abstraction or analogical thinking in the real world.


## Open question

Would a more dynamic private evaluation set help distinguish progress in abstraction from higher scores produced by scaling up search?


## Source-stated

Mitchell argues that Greenblatt’s method raises the question of whether extensive computation and prompt engineering fit ARC’s aim of measuring abstraction.

> it brings up the question of how much brute-force computation can be used before the approach violates the spirit of the ARC challenge

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell reports that Greenblatt’s scores came from samples of the training and public evaluation sets, and that he warned the results might be slightly overoptimistic.

> I only ran my solution on a randomly selected set of 100 train and 100 test problems. The resulting standard error of the mean is reasonably tight (5%), but this means my results might be slightly overoptimistic.

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell says the increase in MindsAI’s score appears largely due to increased computation, and leaves open whether the approach will be relevant to abstraction and analogical thinking in the real world.

> the rising score of this method, from 34% to 43% over the course of several months, is largely due to increased computation use

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source-stated

Mitchell warns that targeting ARC scores could draw attention away from the capacities the benchmark is intended to measure, and suggests changing the private evaluation tasks over time as a possible mitigation.

> One thing that could mitigate this kind of “overfitting” to a target is to make the private evaluation set more dynamic

- [Supporting source](https://aiguide.substack.com/p/on-the-arc-agi-1-million-reasoning)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Editorial provenance

Version 1. Publication ID: 0f406435-3fc7-203b-0976-b75fef1bf7a2.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T14:05:06.405Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 6e92251c0ee606654e08bfdd99b0c104a9e6e60f16f4404cb946c1d49eafd388. Source text hash: 2fda5bbd93d18103a87e6eebe134f684bd264c323216e133925b2c2a63cbd0a9.


### passed — 2026-11-17T14:05:06.405Z — anthropic/claude-opus-5.5

Publication: 0f406435-3fc7-203b-0976-b75fef1bf7a2

attribution — pass: The page is a single Substack essay by Melanie Mitchell with its own argument. The reading attributes the arguments to Mitchell, and attributes Greenblatt's sampling caveat to Greenblatt as she quotes it.

> In this post I’m going to go into the weeds, describing how some people are trying to win a big $$$ prize

summary — pass: The reading leads with Mitchell's thesis and explains ARC tasks in plain language. It outlines Greenblatt's generate-test-revise method, MindsAI's compute-driven gains and her worry that a benchmark turned into a target loses its purpose. It also keeps her excitement and her stated hope for a dynamic evaluation set. 'Benchmark' is defined on the page. The question follows from the evidence. A brief gloss of 'private evaluation set' would help but is not essential.

> It would be disappointing, at least to me, if ARC can be solved largely via brute-force domain-specific search rather than via more general and humanlike abstraction.

date — pass: originalPublished is 2024-08-06T21:05:45Z, and the reading's date of August 6, 2024 matches the UTC day and the dateline. modifiedAt equals the publication time. The reading uses no hindsight framing and no later events. The page is a publisher page, so no archive disclosure is required. The retrieval date is disclosed as a limitation.

> Melanie Mitchell
Aug 06, 2024

support — pass: Each claim matches the source. Mitchell objects to the computation and prompt engineering involved. Greenblatt's scores came from 100-task samples, with his own caveat that they might be slightly overoptimistic. MindsAI's rise is attributed largely to computation, and the question of real-world relevance stays open. The dynamic evaluation set is offered as a possible mitigation. Limitations are grounded in the text, and no unsupported benefits are implied.

> not only because it requires so much computation and human engineering of prompts

media — pass: No media was selected and no verified media candidates were supplied, so nothing is misidentified.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/f2038a9ffedfb6f752920d2d6183d180fcb082fa62b5fe1cf47b1dc6b2fe5efc/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/melanie-mitchell-argues-that-arc-scores-may-reward-search-over-f2038a9f.json)
