Toby Ord · Jan 21, 2025
Toby Ord argues that inference-scaling charts can overstate progress
Markdown · JSON · Original source
On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force. For o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s score at human-level cost will take less than five years. Ord argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
Toby Ord · Article
Source details
Original source date: Jan 21, 2025 (2025-01-21T09:44:50+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-16T00:43:24.767Z. First observed: 2026-10-24T21:51:52.169Z. Retrieved: 2026-10-24T21:53:41.886Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 24, 2026; originally published January 21, 2025. Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force. Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes.
Open question
What would overlapping, like-for-like comparisons show about how o1 and o3 perform as rise?
Source-stated
Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.
What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.
Source-stated
Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.
the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.
Source-stated
Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.
The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.
Source-stated
Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.
some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark
Source-stated
Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.
My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.
Source-stated
Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.
What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.
Source and contributor context
Toby Ord · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Model inference
Using a trained model to produce a prediction or response to new input.
Editorial provenance
Version 1. Publication ID: cf7d04b5-ad87-c918-3c52-3ddf861dc9ba.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T00:40:34.462Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 572532fc1d5045dfb0b49a53b0e5e7da6d9ec9d9c18f473794b5a64a78c32cb0. Source text hash: 252424c62e85c6eb1b20e94506bf7ab880fe462990df91da1bd13f6d6c02d2e8.
Revision and source-check history
Version 1 · 2026-11-16T00:43:24.767Z
Toby Ord argues that inference-scaling charts can overstate progress
On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force. For o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s score at human-level cost will take less than five years. Ord argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.
Original source
Toby Ord · Article
Source details
Original source date: Jan 21, 2025 (2025-01-21T09:44:50+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-16T00:43:24.767Z. First observed: 2026-10-24T21:51:52.169Z. Retrieved: 2026-10-24T21:53:41.886Z.
Limits
Text as retrieved on October 24, 2026; originally published January 21, 2025. Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force. Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes.
Open question
What would overlapping, like-for-like comparisons show about how o1 and o3 perform as rise?
Source-stated
Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.
What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.
Source-stated
Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.
the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.
Source-stated
Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.
The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.
Source-stated
Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.
some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark
Source-stated
Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.
My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.
Source-stated
Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.
What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.
Source and contributor context
Toby Ord · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Model inference
Using a trained model to produce a prediction or response to new input.
Editorial provenance
Version 1. Publication ID: cf7d04b5-ad87-c918-3c52-3ddf861dc9ba.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T00:40:34.462Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 572532fc1d5045dfb0b49a53b0e5e7da6d9ec9d9c18f473794b5a64a78c32cb0. Source text hash: 252424c62e85c6eb1b20e94506bf7ab880fe462990df91da1bd13f6d6c02d2e8.
uncertain · 2026-11-16T00:40:34.462Z · anthropic/claude-opus-5.5
Publication: cf7d04b5-ad87-c918-3c52-3ddf861dc9ba
attribution — pass: This is a single essay by Toby Ord on his own site, and the reading consistently attributes the arguments, estimates and forecast to Ord.
Inference Scaling and the Log-x Chart January 21, 2025Toby Ord
summary — uncertain: The reading faithfully conveys Ord's argument and keeps his force. However, it never says that o1 and o3 are OpenAI's AI models, or that the charts came from OpenAI's own launch materials. It also leaves ARC-AGI unexplained; the source calls it a set of puzzles easy for humans but hard for LLMs. A non-specialist cannot tell whose models or test are being discussed. Smallest repair: add one clause identifying o1 and o3 as OpenAI models, and one plain-language gloss of ARC-AGI. A gloss of brute force (trying the basic approach over and over) would also help.
This is a set of puzzles designed to be very simple for humans, but beyond the reach of existing LLMs.
date — pass: January 21, 2025 matches the UTC day of originalPublished and the source dateline. The page was edited on 25 June 2025, adding an ARC-AGI leaderboard update. The reading neither uses nor relies on that added passage. Ord's under-five-years forecast is reported as his forecast, not as an outcome. The evidence kind is publisher-page, so no archive disclosure is required.
My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.
support — pass: Each claim cites text that is plainly part of the original January piece, which appears before the June addition. The claims cover logarithmic gains versus exponential costs, the lack of like-for-like cost comparisons, fine-tuning on ARC-AGI, the weak-evidence conclusion, the multi-year framing, and the criteria of higher slope or plateau. The reading adds no stakeholder or impact claims beyond the source.
the difficulty of determining whether o3 is an improvement on o1 from the publicly available data shows that this data is only weak evidence
media — pass: No media was selected, and no verified media candidates were supplied, so there is no media identity or relevance issue.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Further training of an existing model to adapt it to a particular task or use.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceUsing a trained model to produce a prediction or response to new input.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceUsing a trained model to produce a prediction or response to new input.
Definition reference