{"id":"5ca2474df293663bc45223cb48f953e797680fa88bfb92a3f66b4589ec33f613","title":"Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation","argument":"The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating AI agents means examining how they act, not just their final results or benchmark scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation.\n\nThe authors warn that evidence at one level of evaluation may not apply to another. They cite a randomized controlled trial in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work.\n\nThe authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation","artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 29, 2026; originally published October 19, 2026. The article refers to a randomized controlled trial but does not provide its methods or results in detail here. The article points to a linked paper, but the paper’s contents are not included in this evidence.","speaker":{"name":"Arvind Narayanan and Sayash Kapoor","roleAtSource":"Publisher"},"source":{"title":"New paper: Emerging trends in AI agent evaluation","publisher":"Arvind Narayanan and Sayash Kapoor","format":"Article","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","publishedAt":"2026-10-19T20:11:55.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.","excerpt":"The evaluator must scrutinize the entire trajectory, not just the final outcome.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.","excerpt":"despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.","excerpt":"But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.","excerpt":"Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.","excerpt":"agent evaluation demands its own field of practice, one with higher standards for validity and rigor.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does evidence from laboratory evaluations of AI agents translate to their performance and effects in specific real-world contexts?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation","summary":"The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating AI agents means examining how they act, not just their final results or benchmark scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation.\n\nThe authors warn that evidence at one level of evaluation may not apply to another. They cite a randomized controlled trial in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work.\n\nThe authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.","question":"How well does evidence from laboratory evaluations of AI agents translate to their performance and effects in specific real-world contexts?","claims":[{"text":"The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.","excerpt":"The evaluator must scrutinize the entire trajectory, not just the final outcome.","support":"source-stated"},{"text":"The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.","excerpt":"despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.","support":"source-stated"},{"text":"The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.","excerpt":"But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.","support":"source-stated"},{"text":"The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.","excerpt":"Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.","support":"source-stated"},{"text":"The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.","excerpt":"agent evaluation demands its own field of practice, one with higher standards for validity and rigor.","support":"source-stated"}],"limitations":["Text as retrieved on October 29, 2026; originally published October 19, 2026.","The article refers to a randomized controlled trial but does not provide its methods or results in detail here.","The article points to a linked paper, but the paper’s contents are not included in this evidence."],"mediaId":null,"correctionReason":null,"id":"5ca2474df293663bc45223cb48f953e797680fa88bfb92a3f66b4589ec33f613","publicationId":"8ddccfa0-4293-0474-458f-8c8e4086eda4","version":1,"sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","sourceTitle":"New paper: Emerging trends in AI agent evaluation","publisher":"Arvind Narayanan and Sayash Kapoor","publishedAt":"2026-10-19T20:11:55.000Z","originalPublished":{"value":"2026-10-19T20:11:55.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-29T10:25:59.202Z","createdAt":"2026-11-04T21:33:23.609Z","historical":true,"entityIds":[],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation","artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"731ac8b771cebcbc6af0338b09f2f7c127e3de7b33f16c8cadbdf13f1ba35265","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-29T10:25:56.538Z"},"provenance":{"textHash":"f80113641a193476125c578fbf81329fbcd62494863685be6bf782532d1fc9c7","observationId":"1081847df52616321edbe32109191a1fd25447088994bfe24023e9d2697d4941","retrievedAt":"2026-10-29T10:27:43.034Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"88b11ff2-cbf4-280c-8923-933105727268"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"8bffe3bf-de72-3e1b-b981-20718970429f"}},"audit":{"status":"passed","reviewedAt":"2026-11-04T21:03:36.173Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/felix-chen-stephan-rabanser-sayash-kapoor-and-arvind-narayanan-5ca2474d","formats":{"html":"/readings/felix-chen-stephan-rabanser-sayash-kapoor-and-arvind-narayanan-5ca2474d","markdown":"/readings/felix-chen-stephan-rabanser-sayash-kapoor-and-arvind-narayanan-5ca2474d.md","json":"/readings/felix-chen-stephan-rabanser-sayash-kapoor-and-arvind-narayanan-5ca2474d.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-rct","name":"Randomized controlled trial","explanation":"A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.","referenceUrl":"https://en.wikipedia.org/wiki/Randomized_controlled_trial"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"5ca2474df293663bc45223cb48f953e797680fa88bfb92a3f66b4589ec33f613","revision":"8ddccfa0-4293-0474-458f-8c8e4086eda4","revisionKind":"publication-id","title":"Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation","url":"https://escapemeridian.com/readings/felix-chen-stephan-rabanser-sayash-kapoor-and-arvind-narayanan-5ca2474d"},"source":{"type":"source","id":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","revision":"f80113641a193476125c578fbf81329fbcd62494863685be6bf782532d1fc9c7","revisionKind":"sha256-source-text","title":"New paper: Emerging trends in AI agent evaluation","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"5ca2474df293663bc45223cb48f953e797680fa88bfb92a3f66b4589ec33f613","title":"Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation","argument":"The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating AI agents means examining how they act, not just their final results or benchmark scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation.\n\nThe authors warn that evidence at one level of evaluation may not apply to another. They cite a randomized controlled trial in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work.\n\nThe authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation","artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 29, 2026; originally published October 19, 2026. The article refers to a randomized controlled trial but does not provide its methods or results in detail here. The article points to a linked paper, but the paper’s contents are not included in this evidence.","speaker":{"name":"Arvind Narayanan and Sayash Kapoor","roleAtSource":"Publisher"},"source":{"title":"New paper: Emerging trends in AI agent evaluation","publisher":"Arvind Narayanan and Sayash Kapoor","format":"Article","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","publishedAt":"2026-10-19T20:11:55.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.","excerpt":"The evaluator must scrutinize the entire trajectory, not just the final outcome.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.","excerpt":"despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.","excerpt":"But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.","excerpt":"Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"},{"text":"The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.","excerpt":"agent evaluation demands its own field of practice, one with higher standards for validity and rigor.","url":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does evidence from laboratory evaluations of AI agents translate to their performance and effects in specific real-world contexts?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue in Normal Tech for broader agent evaluation","summary":"The article was published on October 19, 2026. Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan argue that evaluating AI agents means examining how they act, not just their final results or benchmark scores. They say evaluators must check for shortcuts, assess agents in the settings where people use them, and contend with the cost and difficulty of monitoring their actions. Tools for inspecting agents also need evaluation.\n\nThe authors warn that evidence at one level of evaluation may not apply to another. They cite a randomized controlled trial in which using agent tools for software development increased overall task completion time, despite expectations. They argue that evaluation focused on automation can identify work agents might replace, but miss whether agents help people collaborate or improve their work.\n\nThe authors say who can evaluate, and with what independence, shapes the questions and methods; funding and access affect independent evaluators’ participation. They call for people from more disciplines to join the work and for agent evaluation to develop as a field with higher standards for validity and rigor. Their conclusion is that there is a window to bring in new participants, methods and ideas, and they urge prospective and current evaluators to contribute.","question":"How well does evidence from laboratory evaluations of AI agents translate to their performance and effects in specific real-world contexts?","claims":[{"text":"The authors argue that evaluating agents requires examining their full course of action, not only the final result, and that this work takes substantial engineering effort and is increasingly expensive.","excerpt":"The evaluator must scrutinize the entire trajectory, not just the final outcome.","support":"source-stated"},{"text":"The authors say evaluation evidence at one level of scope is not guaranteed to apply at another, and cite a trial where agent tools increased task completion time.","excerpt":"despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand.","support":"source-stated"},{"text":"The authors argue that focusing evaluation on automation can reveal where human work may be replaced, but does not explain how agents might help people collaborate or improve their work.","excerpt":"But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.","support":"source-stated"},{"text":"The authors say participation and independence shape evaluation questions and methods, and that funding and access affect independent evaluators’ ability to take part.","excerpt":"Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use.","support":"source-stated"},{"text":"The authors call for broader participation and a distinct field of practice for agent evaluation, with higher standards for validity and rigor.","excerpt":"agent evaluation demands its own field of practice, one with higher standards for validity and rigor.","support":"source-stated"}],"limitations":["Text as retrieved on October 29, 2026; originally published October 19, 2026.","The article refers to a randomized controlled trial but does not provide its methods or results in detail here.","The article points to a linked paper, but the paper’s contents are not included in this evidence."],"mediaId":null,"correctionReason":null,"id":"5ca2474df293663bc45223cb48f953e797680fa88bfb92a3f66b4589ec33f613","publicationId":"8ddccfa0-4293-0474-458f-8c8e4086eda4","version":1,"sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","sourceTitle":"New paper: Emerging trends in AI agent evaluation","publisher":"Arvind Narayanan and Sayash Kapoor","publishedAt":"2026-10-19T20:11:55.000Z","originalPublished":{"value":"2026-10-19T20:11:55.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-29T10:25:59.202Z","createdAt":"2026-11-04T21:33:23.609Z","historical":true,"entityIds":[],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation","artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.normaltech.ai/p/new-paper-emerging-trends-in-ai-agent","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"731ac8b771cebcbc6af0338b09f2f7c127e3de7b33f16c8cadbdf13f1ba35265","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-29T10:25:56.538Z"},"provenance":{"textHash":"f80113641a193476125c578fbf81329fbcd62494863685be6bf782532d1fc9c7","observationId":"1081847df52616321edbe32109191a1fd25447088994bfe24023e9d2697d4941","retrievedAt":"2026-10-29T10:27:43.034Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"88b11ff2-cbf4-280c-8923-933105727268"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"8bffe3bf-de72-3e1b-b981-20718970429f"}},"audit":{"status":"passed","reviewedAt":"2026-11-04T21:03:36.173Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"8ddccfa0-4293-0474-458f-8c8e4086eda4","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The source byline names Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan, and the post appears on the Normal Tech site (normaltech.ai). The reading attributes the arguments to these authors throughout. It is one argued piece, not a listing.","excerpt":"By Felix Chen, Stephan Rabanser, Sayash Kapoor, and Arvind Narayanan."},{"dimension":"summary","status":"pass","explanation":"The summary covers the post's main points in plain language: trajectory-level scrutiny, cost, monitoring tools that also need evaluation, scope translation, the automation versus augmentation framing, independence, funding and access, and the call for broader participation. Central terms (AI agent, benchmark, randomized controlled trial) are defined on the reading page. The question follows directly from the scope-translation argument.","excerpt":"Crucially, the results of evaluation at one level of scope are not guaranteed to translate to another."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-19T20:11:55Z, which matches the source dateline of Oct 19, 2026. The reading states October 19, 2026 at day precision and uses no hindsight framing. The source is a publisher page, so no archive disclosure is needed. modifiedAt equals the publication time.","excerpt":"Oct 19, 2026"},{"dimension":"support","status":"pass","explanation":"Each claim and summary statement is grounded in source text, including trajectory scrutiny, rising cost, the trial's finding on completion time, the limits of an automation-only focus, independence and access, and the call for a distinct field. Claims remain attributed to the authors. The reading adds no unsupported benefits or costs.","excerpt":"Addressing these concerns requires significant engineering effort, and the work is getting increasingly expensive"},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidates were supplied. This is appropriate.","excerpt":null}],"createdAt":"2026-11-04T21:03:36.173Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}