Toby Ord asks whether AI agents’ task success has a half-life

Toby Ord asks whether AI agents’ task success has a half-life

Toby Ord’s article was published on May 7, 2025. Building on work by Kwa et al., Ord argues that a constant rate of failure during each minute a human would take to do a task could explain how AI agents’ success rates fall as tasks get longer. This model implies an exponentially declining success rate, which Ord says would give each agent a half-life: a task-length measure of the point where its success rate is 50%. Ord’s proposed mechanism is that longer tasks involve more subtasks, and failing any one can sink the whole task. That model could help estimate success at different task lengths and understand why an agent that can complete one day’s work may not reliably complete two days. But Ord stresses that the pattern is only suggestive, and that the task suite may not represent performance elsewhere. He calls for comparisons of the model with alternatives, analysis of human and AI performance, and tests on other task suites. Whether the model generalises beyond this suite, and whether it fits the data better than alternatives, remains unknown.

Publication: a6e539d8-66af-cf2b-38c3-bae0e50f87cb

attribution — pass: This is one essay by Toby Ord on his own site. The reading attributes the model and argument to Ord, and the underlying data to Kwa et al. Ord's claims stay framed as his arguments, not as established fact.

Building on the recent empirical work of Kwa et al. (2025), I show that within their suite of research-engineering tasks

summary — pass: The reading leads with Ord's argument and explains the constant failure rate, the half-life (the task length at which success is 50%) and the subtask mechanism in plain language. It keeps the one-day versus two-day illustration and Ord's hedges. The question names METR without introducing it, which is a minor gap; the core reading remains understandable. The question also follows directly from Ord's further-work section.

Ideally one would conduct a formal statistical analysis on how well an exponential decay curve fits METRs data compared to the log-logistic they use.

date — pass: originalPublished is 2025-05-07T11:54+01:00, which falls on May 7, 2025 in UTC and matches the dateline. The page was edited after publication with a Feb 2026 update. The reading explicitly excludes that update, and every cited excerpt comes from the original body. There is no hindsight framing, and the page is a publisher page, not an archive copy.

May 7, 2025Toby Ord

support — pass: Each claim matches its excerpt, and all cited text is plainly part of the original essay, not the later update. Two further statements are also supported in the source: that the pattern is only suggestive, and the one-day versus two-day illustration. The reading adds no invented benefits or costs.

So far these results and analysis are merely suggestive of a constant hazard rate.

media — pass: No media was selected and no verified candidates were supplied, so nothing needs checking.