Arvind Narayanan and Sayash Kapoor · Oct 19, 2026
Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation
Markdown · JSON · Original source
The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating means examining how they act, not just their final results or scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation. The authors warn that evidence at one level of evaluation may not apply to another. They cite a in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work. The authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
Arvind Narayanan and Sayash Kapoor · Article
Source details
Original source date: Oct 19, 2026 (2026-10-19T20:11:55.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-04T21:33:23.609Z. First observed: 2026-10-29T10:25:59.202Z. Retrieved: 2026-10-29T10:27:43.034Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 29, 2026; originally published October 19, 2026. The article refers to a but does not provide its methods or results in detail here. The article points to a linked paper, but the paper’s contents are not included in this evidence.
Open question
How well does evidence from laboratory evaluations of translate to their performance and effects in specific real-world contexts?
Source-stated
The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.
The evaluator must scrutinize the entire trajectory, not just the final outcome.
Source-stated
The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.
despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.
Source-stated
The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.
But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.
Source-stated
The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.
Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.
Source-stated
The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.
agent evaluation demands its own field of practice, one with higher standards for validity and rigor.
Source and contributor context
Arvind Narayanan and Sayash Kapoor · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Randomized controlled trial
A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Editorial provenance
Version 1. Publication ID: 8ddccfa0-4293-0474-458f-8c8e4086eda4.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-04T21:03:36.173Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 1081847df52616321edbe32109191a1fd25447088994bfe24023e9d2697d4941. Source text hash: f80113641a193476125c578fbf81329fbcd62494863685be6bf782532d1fc9c7.
Revision and source-check history
Version 1 · 2026-11-04T21:33:23.609Z
Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation
The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating means examining how they act, not just their final results or scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation. The authors warn that evidence at one level of evaluation may not apply to another. They cite a in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work. The authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.
Original source
Arvind Narayanan and Sayash Kapoor · Article
Source details
Original source date: Oct 19, 2026 (2026-10-19T20:11:55.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-04T21:33:23.609Z. First observed: 2026-10-29T10:25:59.202Z. Retrieved: 2026-10-29T10:27:43.034Z.
Limits
Text as retrieved on October 29, 2026; originally published October 19, 2026. The article refers to a but does not provide its methods or results in detail here. The article points to a linked paper, but the paper’s contents are not included in this evidence.
Open question
How well does evidence from laboratory evaluations of translate to their performance and effects in specific real-world contexts?
Source-stated
The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.
The evaluator must scrutinize the entire trajectory, not just the final outcome.
Source-stated
The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.
despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.
Source-stated
The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.
But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.
Source-stated
The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.
Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.
Source-stated
The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.
agent evaluation demands its own field of practice, one with higher standards for validity and rigor.
Source and contributor context
Arvind Narayanan and Sayash Kapoor · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Randomized controlled trial
A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Editorial provenance
Version 1. Publication ID: 8ddccfa0-4293-0474-458f-8c8e4086eda4.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-04T21:03:36.173Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 1081847df52616321edbe32109191a1fd25447088994bfe24023e9d2697d4941. Source text hash: f80113641a193476125c578fbf81329fbcd62494863685be6bf782532d1fc9c7.
passed · 2026-11-04T21:03:36.173Z · anthropic/claude-opus-5.5
Publication: 8ddccfa0-4293-0474-458f-8c8e4086eda4
attribution — pass: The source byline names Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan, and the post appears on the Normal Tech site (normaltech.ai). The reading attributes the arguments to these authors throughout. It is one argued piece, not a listing.
By Felix Chen, Stephan Rabanser, Sayash Kapoor, and Arvind Narayanan.
summary — pass: The summary covers the post's main points in plain language: trajectory-level scrutiny, cost, monitoring tools that also need evaluation, scope translation, the automation versus augmentation framing, independence, funding and access, and the call for broader participation. Central terms (AI agent, benchmark, randomized controlled trial) are defined on the reading page. The question follows directly from the scope-translation argument.
Crucially, the results of evaluation at one level of scope are not guaranteed to translate to another.
date — pass: originalPublished is 2026-10-19T20:11:55Z, which matches the source dateline of Oct 19, 2026. The reading states October 19, 2026 at day precision and uses no hindsight framing. The source is a publisher page, so no archive disclosure is needed. modifiedAt equals the publication time.
Oct 19, 2026
support — pass: Each claim and summary statement is grounded in source text, including trajectory scrutiny, rising cost, the trial's finding on completion time, the limits of an automation-only focus, independence and access, and the call for a distinct field. Claims remain attributed to the authors. The reading adds no unsupported benefits or costs.
Addressing these concerns requires significant engineering effort, and the work is getting increasingly expensive
media — pass: No media was selected, and no verified candidates were supplied. This is appropriate.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
An AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition reference