Lijo, Solomon Shalom · Oct 23, 2026
Solomon Shalom Lijo argues that measuring AI time savings requires counting errors
Markdown · JSON · Original source
In a paper listed on , Solomon Shalom Lijo argues that AI savings should be measured against errors, not just the clock. He calls the clock-based approach the “stopwatch view” and argues for a “ledger view” that weighs time saved against defects that escape checking. Lijo reports that reanalysis of ’s developer study gives different estimates depending on the estimand: the same 246 issues support a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of them. He also reports that, in GDPval’s workflow of reviewing every output, review time alone caps GPT-5’s speedup at 3.6x. In a small pre-specified experiment, AI reviewers told that the drafter made errors less often checked fewer figures and let wrong figures through more often. The paper’s central practical point is that apparent time savings can come from reduced checking, with a cost in errors. How well do these findings apply to work outside the studies and experiment described in the abstract?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 23, 2026 (2026-10-23).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-06T13:31:24.606Z. First observed: 2026-10-28T04:30:30.467Z. Retrieved: 2026-10-30T01:35:18.512Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence provided is the paper's abstract, not its full methods or results; the experiment is described as small. The abstract reports a range of estimates for 's developer study and says results depend on the estimand; it does not establish how broadly they apply beyond that study. The abstract describes a small experiment on AI reviewers checking AI drafts of a particular kind of brief, not a broad test across work settings.
Open question
How well do these findings apply to work outside the studies and experiment described in the abstract?
Source-stated
Lijo argues that time saved by AI should be assessed alongside errors that escape, rather than measured only by elapsed time.
I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter.
Source-stated
Lijo reports that estimates from the same 246 issues in METR's developer study vary by estimand: a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of issues.
the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them.
Source-stated
Lijo reports that review time alone caps GPT-5's speedup at 3.6x in GDPval's workflow, which reviews every output.
In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x
Source-stated
In a small pre-specified experiment, AI reviewers told that the drafter's error rate was much lower checked fewer figures and let wrong figures through more often.
Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none
Source and contributor context
Lijo, Solomon Shalom · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: c7d8551f-e434-4521-8044-4c356a89f632.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T13:31:24.606Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 034accac54d867be21556971efdadae218f877b72dc3e6d8186970342aaa3716. Source text hash: 57cb3c493ff4ccca936d393c4ab30ead6407c30c1e768088f31822ca56cd5da1.
Revision and source-check history
Version 1 · 2026-11-06T13:31:24.606Z
Solomon Shalom Lijo argues that measuring AI time savings requires counting errors
In a paper listed on , Solomon Shalom Lijo argues that AI savings should be measured against errors, not just the clock. He calls the clock-based approach the “stopwatch view” and argues for a “ledger view” that weighs time saved against defects that escape checking. Lijo reports that reanalysis of ’s developer study gives different estimates depending on the estimand: the same 246 issues support a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of them. He also reports that, in GDPval’s workflow of reviewing every output, review time alone caps GPT-5’s speedup at 3.6x. In a small pre-specified experiment, AI reviewers told that the drafter made errors less often checked fewer figures and let wrong figures through more often. The paper’s central practical point is that apparent time savings can come from reduced checking, with a cost in errors. How well do these findings apply to work outside the studies and experiment described in the abstract?
Original source
cs.AI · Article
Source details
Original source date: Oct 23, 2026 (2026-10-23).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-06T13:31:24.606Z. First observed: 2026-10-28T04:30:30.467Z. Retrieved: 2026-10-30T01:35:18.512Z.
Limits
The evidence provided is the paper's abstract, not its full methods or results; the experiment is described as small. The abstract reports a range of estimates for 's developer study and says results depend on the estimand; it does not establish how broadly they apply beyond that study. The abstract describes a small experiment on AI reviewers checking AI drafts of a particular kind of brief, not a broad test across work settings.
Open question
How well do these findings apply to work outside the studies and experiment described in the abstract?
Source-stated
Lijo argues that time saved by AI should be assessed alongside errors that escape, rather than measured only by elapsed time.
I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter.
Source-stated
Lijo reports that estimates from the same 246 issues in METR's developer study vary by estimand: a slowdown of 19% or 24%, and AI helped on somewhere between 3% and 82% of issues.
the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them.
Source-stated
Lijo reports that review time alone caps GPT-5's speedup at 3.6x in GDPval's workflow, which reviews every output.
In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x
Source-stated
In a small pre-specified experiment, AI reviewers told that the drafter's error rate was much lower checked fewer figures and let wrong figures through more often.
Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none
Source and contributor context
Lijo, Solomon Shalom · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: c7d8551f-e434-4521-8044-4c356a89f632.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T13:31:24.606Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 034accac54d867be21556971efdadae218f877b72dc3e6d8186970342aaa3716. Source text hash: 57cb3c493ff4ccca936d393c4ab30ead6407c30c1e768088f31822ca56cd5da1.
uncertain · 2026-11-06T13:31:24.606Z · anthropic/claude-opus-5.5
Publication: c7d8551f-e434-4521-8044-4c356a89f632
attribution — pass: The reading correctly attributes the paper and its arguments to Solomon Shalom Lijo, listed on arXiv cs.AI. Findings are framed as what Lijo argues or reports, not as established fact.
Authors:Solomon Shalom Lijo
summary — uncertain: The core argument (stopwatch vs ledger view) is faithful, but central terms are not explained for a general reader. 'Estimand' (what quantity is being estimated) is the hinge of the METR finding. 'METR', 'GDPval' and 'pre-specified' are also unexplained. Smallest repairs: gloss 'estimand' or add a term-library tooltip, and briefly identify METR and GDPval. The draft also drops the abstract's own pointed conclusion ('faster and worse, entirely through reduced effort') and its claim that over-trust costs more total minutes. Restoring one of these would preserve the author's force.
faster and worse, entirely through reduced effort
date — pass: originalPublished 2026-10-23 matches the arXiv v1 submission date. The feed date of 2026-10-29 is later and is correctly not used.
[Submitted on 23 Oct 2026]
support — uncertain: Most claims are grounded. However, the summary and the METR claim present the 3%–82% range as part of how estimates 'vary by estimand'. The abstract ties only the 19% or 24% slowdown to the estimand. The 3%–82% figure is stated separately and appears to be a bound. Smallest repair: separate the two, e.g. 'and he bounds the share of issues AI helped on between 3% and 82%'. The draft also omits the 34% unadjusted figure; that omission is acceptable.
support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METR