Zhang, Howard, Arvin, Chuck, Brown, Logan, Bhatia, Mihir, Choudhary, Neelu, Miao, Lily · Nov 10, 2026
arXiv cs.AI: Comparing Agent Evaluation Frameworks in Enterprise Analytics
Markdown · JSON · Original source
Howard Zhang and coauthors compare three evaluation frameworks for -based analytics agents, using frozen execution traces from two agents in a controlled research setting. They report that 97% of configurations met the study’s stability threshold. Across 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance (F1=0.90), while text-only evaluation matched its recall but produced five times more false alarms. In a separate phase, the study uncovered 24 novel failures, with no overlap between functional testing and security . The authors argue that evaluation quality depends on what information the can access, and that functional and security evaluation complement each other. They recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. The study’s findings come from two agents in a controlled setting, so their reach beyond that setting remains an open question.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T03:01:25.495Z. First observed: 2026-11-11T05:06:39.936Z. Retrieved: 2026-11-12T02:48:04.808Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract describes a controlled research setting using frozen execution traces and two analytics agents; it does not establish whether the findings generalize beyond this study. The supplied evidence is the paper's abstract, not its full methods or results.
Open question
Would the reported stability and differences in risk detection hold for analytics agents beyond the two studied in this controlled setting?
Source-stated
The authors compare three evaluation frameworks on two analytics agents in a controlled research setting using frozen execution traces.
applied to two analytics agents in a controlled research setting using frozen execution traces.
Source-stated
The study reports that 97% of configurations met its stability threshold, and that stability patterns were agent-independent.
97% of configurations meet the stability threshold and that stability patterns are agent-independent.
Source-stated
On 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance; text-only evaluation matched its recall but generated five times more false alarms.
the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms.
Source-stated
The discovery phase found 24 novel failures with no overlap between functional testing and security red-teaming.
uncovering 24 novel failures with zero overlap between functional testing and security red-teaming.
Source-stated
The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
Source and contributor context
Zhang, Howard, Arvin, Chuck, Brown, Logan, Bhatia, Mihir, Choudhary, Neelu, Miao, Lily · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Red-teaming
Deliberately attacking or tricking a system to find its weaknesses before others do.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b37fc706-f4bf-41de-89ab-8646ecba8957.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:01:25.495Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7df70a14d93fc284a907bf8d88f16468922eb43458b01a38a61e0a3bea71a8cf. Source text hash: f7aa99fe180391f05491c84b3cb6ccc00ba5c4a18b2a8532cebcac5ebc1b370e.
Revision and source-check history
Version 1 · 2026-11-12T03:01:25.495Z
arXiv cs.AI: Comparing Agent Evaluation Frameworks in Enterprise Analytics
Howard Zhang and coauthors compare three evaluation frameworks for -based analytics agents, using frozen execution traces from two agents in a controlled research setting. They report that 97% of configurations met the study’s stability threshold. Across 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance (F1=0.90), while text-only evaluation matched its recall but produced five times more false alarms. In a separate phase, the study uncovered 24 novel failures, with no overlap between functional testing and security . The authors argue that evaluation quality depends on what information the can access, and that functional and security evaluation complement each other. They recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. The study’s findings come from two agents in a controlled setting, so their reach beyond that setting remains an open question.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T03:01:25.495Z. First observed: 2026-11-11T05:06:39.936Z. Retrieved: 2026-11-12T02:48:04.808Z.
Limits
The abstract describes a controlled research setting using frozen execution traces and two analytics agents; it does not establish whether the findings generalize beyond this study. The supplied evidence is the paper's abstract, not its full methods or results.
Open question
Would the reported stability and differences in risk detection hold for analytics agents beyond the two studied in this controlled setting?
Source-stated
The authors compare three evaluation frameworks on two analytics agents in a controlled research setting using frozen execution traces.
applied to two analytics agents in a controlled research setting using frozen execution traces.
Source-stated
The study reports that 97% of configurations met its stability threshold, and that stability patterns were agent-independent.
97% of configurations meet the stability threshold and that stability patterns are agent-independent.
Source-stated
On 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance; text-only evaluation matched its recall but generated five times more false alarms.
the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms.
Source-stated
The discovery phase found 24 novel failures with no overlap between functional testing and security red-teaming.
uncovering 24 novel failures with zero overlap between functional testing and security red-teaming.
Source-stated
The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
Source and contributor context
Zhang, Howard, Arvin, Chuck, Brown, Logan, Bhatia, Mihir, Choudhary, Neelu, Miao, Lily · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Red-teaming
Deliberately attacking or tricking a system to find its weaknesses before others do.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b37fc706-f4bf-41de-89ab-8646ecba8957.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:01:25.495Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7df70a14d93fc284a907bf8d88f16468922eb43458b01a38a61e0a3bea71a8cf. Source text hash: f7aa99fe180391f05491c84b3cb6ccc00ba5c4a18b2a8532cebcac5ebc1b370e.
uncertain · 2026-11-12T03:01:25.495Z · anthropic/claude-opus-5.5
Publication: b37fc706-f4bf-41de-89ab-8646ecba8957
attribution — pass: The reading correctly attributes the study to Howard Zhang and coauthors on arXiv cs.AI. It keeps the reported results and recommendation as the authors' claims rather than established facts.
Authors:Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao
summary — uncertain: The numbers are faithful, but a non-specialist may struggle with the central comparison. The reading never says that two of the frameworks use an LLM as a judge, and it omits the third, deterministic-heuristics paradigm. It also leaves trace-based, red-teaming, F1, evaluation harness and frozen execution traces unexplained. It drops the authors' third conclusion, that LLM harnesses surface subtle errors human reviewers miss. Smallest repair: name the three paradigms, add tooltips for these terms, and restore the human-reviewer finding.
(trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics)
date — pass: The original publication date of 2026-11-10 matches the v1 submission date shown on the arXiv page.
[Submitted on 10 Nov 2026]
support — pass: Every claim is supported by an exact abstract excerpt: 97% stability, F1=0.90, 5x false alarms, 24 novel failures with zero overlap, and the two-framework recommendation. The open question and limitations accurately reflect the controlled, two-agent, frozen-trace scope. No benefits or costs beyond the source are invented.
applied to two analytics agents in a controlled research setting using frozen execution traces.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceDeliberately attacking or tricking a system to find its weaknesses before others do.
Definition referenceAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceDeliberately attacking or tricking a system to find its weaknesses before others do.
Definition referenceAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv