{"id":"328e02650c74df51474211fa317fd1163d5f59e503b62cfabbf2eeeb2d9478d2","title":"Toby Ord argues that inference-scaling charts can overstate progress","argument":"On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force.\n\nFor o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect fine-tuning on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s benchmark score at human-level cost will take less than five years.\n\nOrd argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.","topicIds":["investment-booms-and-busts","artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published January 21, 2025. Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force. Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes.","speaker":{"name":"Toby Ord","roleAtSource":"Publisher"},"source":{"title":"Inference Scaling and the Log-x Chart","publisher":"Toby Ord","format":"Article","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","publishedAt":"2025-01-21T09:44:50+00:00","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.","excerpt":"What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.","excerpt":"the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.","excerpt":"The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.","excerpt":"some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.","excerpt":"My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.","excerpt":"What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What would overlapping, like-for-like comparisons show about how o1 and o3 perform as inference costs rise?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Toby Ord argues that inference-scaling charts can overstate progress","summary":"On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force.\n\nFor o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect fine-tuning on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s benchmark score at human-level cost will take less than five years.\n\nOrd argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.","question":"What would overlapping, like-for-like comparisons show about how o1 and o3 perform as inference costs rise?","claims":[{"text":"Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.","excerpt":"What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.","support":"source-stated"},{"text":"Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.","excerpt":"the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.","support":"source-stated"},{"text":"Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.","excerpt":"The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.","support":"source-stated"},{"text":"Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.","excerpt":"some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark","support":"source-stated"},{"text":"Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.","excerpt":"My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.","support":"source-stated"},{"text":"Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.","excerpt":"What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published January 21, 2025.","Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force.","Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes."],"mediaId":null,"correctionReason":null,"id":"328e02650c74df51474211fa317fd1163d5f59e503b62cfabbf2eeeb2d9478d2","publicationId":"cf7d04b5-ad87-c918-3c52-3ddf861dc9ba","version":1,"sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","sourceTitle":"Inference Scaling and the Log-x Chart","publisher":"Toby Ord","publishedAt":"2025-01-21T09:44:50+00:00","originalPublished":{"value":"2025-01-21T09:44:50+00:00","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-24T21:51:52.169Z","createdAt":"2026-11-16T00:43:24.767Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts","artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"60b1616efe3d4a3095559b40efcb53fa6aaee39f1d4a8d2089016cf8edf72cc5","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T21:51:47.788Z"},"provenance":{"textHash":"252424c62e85c6eb1b20e94506bf7ab880fe462990df91da1bd13f6d6c02d2e8","observationId":"572532fc1d5045dfb0b49a53b0e5e7da6d9ec9d9c18f473794b5a64a78c32cb0","retrievedAt":"2026-10-24T21:53:41.886Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"ff396505-1454-b03d-2269-3edfe4f8bf67"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"1a5ac046-3b82-8141-7d31-df65327cbab5"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T00:40:34.462Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/toby-ord-argues-that-inference-scaling-charts-can-overstate-328e0265","formats":{"html":"/readings/toby-ord-argues-that-inference-scaling-charts-can-overstate-328e0265","markdown":"/readings/toby-ord-argues-that-inference-scaling-charts-can-overstate-328e0265.md","json":"/readings/toby-ord-argues-that-inference-scaling-charts-can-overstate-328e0265.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-inference","name":"Model inference","explanation":"Using a trained model to produce a prediction or response to new input.","referenceUrl":"https://developers.google.com/machine-learning/glossary#inference"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"328e02650c74df51474211fa317fd1163d5f59e503b62cfabbf2eeeb2d9478d2","revision":"cf7d04b5-ad87-c918-3c52-3ddf861dc9ba","revisionKind":"publication-id","title":"Toby Ord argues that inference-scaling charts can overstate progress","url":"https://escapemeridian.com/readings/toby-ord-argues-that-inference-scaling-charts-can-overstate-328e0265"},"source":{"type":"source","id":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","revision":"252424c62e85c6eb1b20e94506bf7ab880fe462990df91da1bd13f6d6c02d2e8","revisionKind":"sha256-source-text","title":"Inference Scaling and the Log-x Chart","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"328e02650c74df51474211fa317fd1163d5f59e503b62cfabbf2eeeb2d9478d2","title":"Toby Ord argues that inference-scaling charts can overstate progress","argument":"On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force.\n\nFor o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect fine-tuning on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s benchmark score at human-level cost will take less than five years.\n\nOrd argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.","topicIds":["investment-booms-and-busts","artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published January 21, 2025. Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force. Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes.","speaker":{"name":"Toby Ord","roleAtSource":"Publisher"},"source":{"title":"Inference Scaling and the Log-x Chart","publisher":"Toby Ord","format":"Article","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","publishedAt":"2025-01-21T09:44:50+00:00","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.","excerpt":"What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.","excerpt":"the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.","excerpt":"The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.","excerpt":"some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.","excerpt":"My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"},{"text":"Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.","excerpt":"What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.","url":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What would overlapping, like-for-like comparisons show about how o1 and o3 perform as inference costs rise?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Toby Ord argues that inference-scaling charts can overstate progress","summary":"On January 21, 2025, Toby Ord argues that charts of inference scaling can make progress look more impressive than it is. Their logarithmic x-axis compresses rising compute costs: a steady-looking line can represent capabilities increasing logarithmically while costs rise exponentially. Ord says this pattern is characteristic of brute force, but cautions that it does not establish that o1 or o3 use brute force.\n\nFor o1, Ord says the public data gives little evidence about how its scaling compares with brute force. The o3 chart does not compare systems at the same inference cost, and its results also reflect fine-tuning on ARC-AGI. Ord argues that the apparent leap therefore offers weak evidence of progress over the intervening months. His uncertain cost estimates frame the results as closer to what historical progress might make possible over years; he forecasts that reaching o3-high’s benchmark score at human-level cost will take less than five years.\n\nOrd argues that stronger evidence would compare performance across compute levels, against brute force and human performance, with overlapping cost ranges between models. He looks for a higher performance plateau or steeper curve, rather than only a shift toward lower costs. The public charts leave comparative scaling and the extent of improvement uncertain.","question":"What would overlapping, like-for-like comparisons show about how o1 and o3 perform as inference costs rise?","claims":[{"text":"Toby Ord argues that a logarithmic x-axis can make inference scaling look more impressive than it is, because capability gains may be logarithmic while costs rise exponentially.","excerpt":"What we really have is a disappointing logarithmic increase in capabilities as more compute is thrown at the problem.","support":"source-stated"},{"text":"Ord says the public data does not provide much evidence about whether o1 performs better than brute force or by how much.","excerpt":"the quantitative data they’ve shared doesn’t provide much evidence, nor allow us to see how much better than brute force it is.","support":"source-stated"},{"text":"Ord says the o1 and o3 results cannot be compared like for like because they were not tested at the same inference cost.","excerpt":"The systems are never tested at the same level of inference cost, so this chart doesn’t allow any like-for-like comparisons.","support":"source-stated"},{"text":"Ord says the o3 results include an unknown contribution from fine-tuning against the ARC-AGI benchmark.","excerpt":"some unknown fraction of the improvement in the yellow data series is from fine-tuning against the benchmark","support":"source-stated"},{"text":"Ord forecasts that it will take less than five years for AI systems to exceed o3-high’s benchmark score at human-level cost.","excerpt":"My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost.","support":"source-stated"},{"text":"Ord argues that stronger evidence of inference-scaling progress would show a higher curve slope or plateau, with comparisons across compute levels and overlapping ranges between models.","excerpt":"What substantial progress on inference scaling would look like on log-x charts is this kind of higher slope or higher plateau.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published January 21, 2025.","Ord says secrecy surrounding the launches and the lack of numbers on the x-axis make it hard to assess how o1 and o3 compare with brute force.","Ord describes his cost estimates and forecast as uncertain; the reading reports them as his estimates and forecast, not as outcomes."],"mediaId":null,"correctionReason":null,"id":"328e02650c74df51474211fa317fd1163d5f59e503b62cfabbf2eeeb2d9478d2","publicationId":"cf7d04b5-ad87-c918-3c52-3ddf861dc9ba","version":1,"sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","sourceTitle":"Inference Scaling and the Log-x Chart","publisher":"Toby Ord","publishedAt":"2025-01-21T09:44:50+00:00","originalPublished":{"value":"2025-01-21T09:44:50+00:00","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-24T21:51:52.169Z","createdAt":"2026-11-16T00:43:24.767Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts","artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"60b1616efe3d4a3095559b40efcb53fa6aaee39f1d4a8d2089016cf8edf72cc5","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T21:51:47.788Z"},"provenance":{"textHash":"252424c62e85c6eb1b20e94506bf7ab880fe462990df91da1bd13f6d6c02d2e8","observationId":"572532fc1d5045dfb0b49a53b0e5e7da6d9ec9d9c18f473794b5a64a78c32cb0","retrievedAt":"2026-10-24T21:53:41.886Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"ff396505-1454-b03d-2269-3edfe4f8bf67"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"1a5ac046-3b82-8141-7d31-df65327cbab5"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T00:40:34.462Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"cf7d04b5-ad87-c918-3c52-3ddf861dc9ba","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a single essay by Toby Ord on his own site, and the reading consistently attributes the arguments, estimates and forecast to Ord.","excerpt":"Inference Scaling and the Log-x Chart\nJanuary 21, 2025Toby Ord"},{"dimension":"summary","status":"uncertain","explanation":"The reading faithfully conveys Ord's argument and keeps his force. However, it never says that o1 and o3 are OpenAI's AI models, or that the charts came from OpenAI's own launch materials. It also leaves ARC-AGI unexplained; the source calls it a set of puzzles easy for humans but hard for LLMs. A non-specialist cannot tell whose models or test are being discussed. Smallest repair: add one clause identifying o1 and o3 as OpenAI models, and one plain-language gloss of ARC-AGI. A gloss of brute force (trying the basic approach over and over) would also help.","excerpt":"This is a set of puzzles designed to be very simple for humans, but beyond the reach of existing LLMs."},{"dimension":"date","status":"pass","explanation":"January 21, 2025 matches the UTC day of originalPublished and the source dateline. The page was edited on 25 June 2025, adding an ARC-AGI leaderboard update. The reading neither uses nor relies on that added passage. Ord's under-five-years forecast is reported as his forecast, not as an outcome. The evidence kind is publisher-page, so no archive disclosure is required.","excerpt":"My best guess is that it will be less than 5 years before AI systems are exceeding o3-high’s 88% on this benchmark at human-level cost."},{"dimension":"support","status":"pass","explanation":"Each claim cites text that is plainly part of the original January piece, which appears before the June addition. The claims cover logarithmic gains versus exponential costs, the lack of like-for-like cost comparisons, fine-tuning on ARC-AGI, the weak-evidence conclusion, the multi-year framing, and the criteria of higher slope or plateau. The reading adds no stakeholder or impact claims beyond the source.","excerpt":"the difficulty of determining whether o3 is an improvement on o1 from the publicly available data shows that this data is only weak evidence"},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied, so there is no media identity or relevance issue.","excerpt":null}],"createdAt":"2026-11-16T00:40:34.462Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}