# Solomon Shalom Lijo argues that measuring AI time savings requires counting errors [Canonical reading](https://escapemeridian.com/readings/solomon-shalom-lijo-argues-that-measuring-ai-time-savings-6ab64884) In a paper listed on arXiv, Solomon Shalom Lijo argues that AI time savings should be measured against errors, not just the clock. He calls the clock-based approach the “stopwatch view” and argues for a “ledger view” that weighs time saved against defects that escape checking. Lijo reports that reanalysis of METR’s developer study gives different estimates depending on the estimand: the same 246 issues support a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of them. He also reports that, in GDPval’s workflow of reviewing every output, review time alone caps GPT-5’s speedup at 3.6x. In a small pre-specified experiment, AI reviewers told that the drafter made errors less often checked fewer figures and let wrong figures through more often. The paper’s central practical point is that apparent time savings can come from reduced checking, with a cost in errors. How well do these findings apply to work outside the studies and experiment described in the abstract? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 23, 2026 (2026-10-23). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-06T13:31:24.606Z. First observed: 2026-10-28T04:30:30.467Z. Retrieved: 2026-10-30T01:35:18.512Z. - [Counting the Bandages: On measuring the time that AI saves](https://arxiv.org/abs/2610.35942) ## Limits The evidence provided is the paper's abstract, not its full methods or results; the experiment is described as small. The abstract reports a range of estimates for METR's developer study and says results depend on the estimand; it does not establish how broadly they apply beyond that study. The abstract describes a small experiment on AI reviewers checking AI drafts of a particular kind of brief, not a broad test across work settings. ## Open question How well do these findings apply to work outside the studies and experiment described in the abstract? ## Source-stated Lijo argues that time saved by AI should be assessed alongside errors that escape, rather than measured only by elapsed time. > I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated Lijo reports that estimates from the same 246 issues in METR's developer study vary by estimand: a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of issues. > the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them. - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated Lijo reports that review time alone caps GPT-5's speedup at 3.6x in GDPval's workflow, which reviews every output. > In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated In a small pre-specified experiment, AI reviewers told that the drafter's error rate was much lower checked fewer figures and let wrong figures through more often. > Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source and contributor context Lijo, Solomon Shalom · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## METR METR studies what AI systems can do and how reliably they complete tasks, including work with human developers. - [Explore this context](/entities/metr) - [Reference](https://metr.org/) ## Editorial provenance Version 1. Publication ID: c7d8551f-e434-4521-8044-4c356a89f632. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T13:31:24.606Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 034accac54d867be21556971efdadae218f877b72dc3e6d8186970342aaa3716. Source text hash: 57cb3c493ff4ccca936d393c4ab30ead6407c30c1e768088f31822ca56cd5da1. ## Revision and source-check history ### Version 1 — 2026-11-06T13:31:24.606Z In a paper listed on arXiv, Solomon Shalom Lijo argues that AI time savings should be measured against errors, not just the clock. He calls the clock-based approach the “stopwatch view” and argues for a “ledger view” that weighs time saved against defects that escape checking. Lijo reports that reanalysis of METR’s developer study gives different estimates depending on the estimand: the same 246 issues support a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of them. He also reports that, in GDPval’s workflow of reviewing every output, review time alone caps GPT-5’s speedup at 3.6x. In a small pre-specified experiment, AI reviewers told that the drafter made errors less often checked fewer figures and let wrong figures through more often. The paper’s central practical point is that apparent time savings can come from reduced checking, with a cost in errors. How well do these findings apply to work outside the studies and experiment described in the abstract? ## Original source arXiv cs.AI · Article Original source date: Oct 23, 2026 (2026-10-23). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-06T13:31:24.606Z. First observed: 2026-10-28T04:30:30.467Z. Retrieved: 2026-10-30T01:35:18.512Z. - [Counting the Bandages: On measuring the time that AI saves](https://arxiv.org/abs/2610.35942) ## Limits The evidence provided is the paper's abstract, not its full methods or results; the experiment is described as small. The abstract reports a range of estimates for METR's developer study and says results depend on the estimand; it does not establish how broadly they apply beyond that study. The abstract describes a small experiment on AI reviewers checking AI drafts of a particular kind of brief, not a broad test across work settings. ## Open question How well do these findings apply to work outside the studies and experiment described in the abstract? ## Source-stated Lijo argues that time saved by AI should be assessed alongside errors that escape, rather than measured only by elapsed time. > I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated Lijo reports that estimates from the same 246 issues in METR's developer study vary by estimand: a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of issues. > the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them. - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated Lijo reports that review time alone caps GPT-5's speedup at 3.6x in GDPval's workflow, which reviews every output. > In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source-stated In a small pre-specified experiment, AI reviewers told that the drafter's error rate was much lower checked fewer figures and let wrong figures through more often. > Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none - [Supporting source](https://arxiv.org/abs/2610.35942) ## Source and contributor context Lijo, Solomon Shalom · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## METR METR studies what AI systems can do and how reliably they complete tasks, including work with human developers. - [Explore this context](/entities/metr) - [Reference](https://metr.org/) ## Editorial provenance Version 1. Publication ID: c7d8551f-e434-4521-8044-4c356a89f632. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T13:31:24.606Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 034accac54d867be21556971efdadae218f877b72dc3e6d8186970342aaa3716. Source text hash: 57cb3c493ff4ccca936d393c4ab30ead6407c30c1e768088f31822ca56cd5da1. ### uncertain — 2026-11-06T13:31:24.606Z — anthropic/claude-opus-5.5 Publication: c7d8551f-e434-4521-8044-4c356a89f632 attribution — pass: The reading correctly attributes the paper and its arguments to Solomon Shalom Lijo, listed on arXiv cs.AI. Findings are framed as what Lijo argues or reports, not as established fact. > Authors:Solomon Shalom Lijo summary — uncertain: The core argument (stopwatch vs ledger view) is faithful, but central terms are not explained for a general reader. 'Estimand' (what quantity is being estimated) is the hinge of the METR finding. 'METR', 'GDPval' and 'pre-specified' are also unexplained. Smallest repairs: gloss 'estimand' or add a term-library tooltip, and briefly identify METR and GDPval. The draft also drops the abstract's own pointed conclusion ('faster and worse, entirely through reduced effort') and its claim that over-trust costs more total minutes. Restoring one of these would preserve the author's force. > faster and worse, entirely through reduced effort date — pass: originalPublished 2026-10-23 matches the arXiv v1 submission date. The feed date of 2026-10-29 is later and is correctly not used. > \[Submitted on 23 Oct 2026\] support — uncertain: Most claims are grounded. However, the summary and the METR claim present the 3%–82% range as part of how estimates 'vary by estimand'. The abstract ties only the 19% or 24% slowdown to the estimand. The 3%–82% figure is stated separately and appears to be a bound. Smallest repair: separate the two, e.g. 'and he bounds the share of issues AI helped on between 3% and 82%'. The draft also omits the 34% unadjusted figure; that omission is acceptable. > support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/6ab64884bec934fa8eb84a8025aa0287c9d06c43a7ceb87eff8b0c26fd4a45e6/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/solomon-shalom-lijo-argues-that-measuring-ai-time-savings-6ab64884.json)