# arXiv cs.AI: Comparing Agent Evaluation Frameworks in Enterprise Analytics [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-comparing-agent-evaluation-frameworks-in-enterprise-42eea599) Howard Zhang and coauthors compare three evaluation frameworks for LLM-based analytics agents, using frozen execution traces from two agents in a controlled research setting. They report that 97% of configurations met the study’s stability threshold. Across 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance (F1=0.90), while text-only evaluation matched its recall but produced five times more false alarms. In a separate phase, the study uncovered 24 novel failures, with no overlap between functional testing and security red-teaming. The authors argue that evaluation quality depends on what information the evaluation harness can access, and that functional and security evaluation complement each other. They recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. The study’s findings come from two agents in a controlled setting, so their reach beyond that setting remains an open question. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T03:01:25.495Z. First observed: 2026-11-11T05:06:39.936Z. Retrieved: 2026-11-12T02:48:04.808Z. - [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](https://arxiv.org/abs/2611.14493) ## Limits The abstract describes a controlled research setting using frozen execution traces and two analytics agents; it does not establish whether the findings generalize beyond this study. The supplied evidence is the paper's abstract, not its full methods or results. ## Open question Would the reported stability and differences in risk detection hold for analytics agents beyond the two studied in this controlled setting? ## Source-stated The authors compare three evaluation frameworks on two analytics agents in a controlled research setting using frozen execution traces. > applied to two analytics agents in a controlled research setting using frozen execution traces. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The study reports that 97% of configurations met its stability threshold, and that stability patterns were agent-independent. > 97% of configurations meet the stability threshold and that stability patterns are agent-independent. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated On 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance; text-only evaluation matched its recall but generated five times more false alarms. > the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The discovery phase found 24 novel failures with no overlap between functional testing and security red-teaming. > uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. > We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source and contributor context Zhang, Howard, Arvin, Chuck, Brown, Logan, Bhatia, Mihir, Choudhary, Neelu, Miao, Lily · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Red-teaming Deliberately attacking or tricking a system to find its weaknesses before others do. - [Definition reference](https://en.wikipedia.org/wiki/Red_team) ## Agent harness An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs. - [Definition reference](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: b37fc706-f4bf-41de-89ab-8646ecba8957. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:01:25.495Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 7df70a14d93fc284a907bf8d88f16468922eb43458b01a38a61e0a3bea71a8cf. Source text hash: f7aa99fe180391f05491c84b3cb6ccc00ba5c4a18b2a8532cebcac5ebc1b370e. ## Revision and source-check history ### Version 1 — 2026-11-12T03:01:25.495Z Howard Zhang and coauthors compare three evaluation frameworks for LLM-based analytics agents, using frozen execution traces from two agents in a controlled research setting. They report that 97% of configurations met the study’s stability threshold. Across 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance (F1=0.90), while text-only evaluation matched its recall but produced five times more false alarms. In a separate phase, the study uncovered 24 novel failures, with no overlap between functional testing and security red-teaming. The authors argue that evaluation quality depends on what information the evaluation harness can access, and that functional and security evaluation complement each other. They recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. The study’s findings come from two agents in a controlled setting, so their reach beyond that setting remains an open question. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T03:01:25.495Z. First observed: 2026-11-11T05:06:39.936Z. Retrieved: 2026-11-12T02:48:04.808Z. - [Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents](https://arxiv.org/abs/2611.14493) ## Limits The abstract describes a controlled research setting using frozen execution traces and two analytics agents; it does not establish whether the findings generalize beyond this study. The supplied evidence is the paper's abstract, not its full methods or results. ## Open question Would the reported stability and differences in risk detection hold for analytics agents beyond the two studied in this controlled setting? ## Source-stated The authors compare three evaluation frameworks on two analytics agents in a controlled research setting using frozen execution traces. > applied to two analytics agents in a controlled research setting using frozen execution traces. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The study reports that 97% of configurations met its stability threshold, and that stability patterns were agent-independent. > 97% of configurations meet the stability threshold and that stability patterns are agent-independent. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated On 72 ground-truth test cases, trace-based evaluation had the best precision-recall balance; text-only evaluation matched its recall but generated five times more false alarms. > the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The discovery phase found 24 novel failures with no overlap between functional testing and security red-teaming. > uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source-stated The authors recommend trace-based evaluation for functional assessment and adversarial red-teaming for security validation. > We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation. - [Supporting source](https://arxiv.org/abs/2611.14493) ## Source and contributor context Zhang, Howard, Arvin, Chuck, Brown, Logan, Bhatia, Mihir, Choudhary, Neelu, Miao, Lily · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Red-teaming Deliberately attacking or tricking a system to find its weaknesses before others do. - [Definition reference](https://en.wikipedia.org/wiki/Red_team) ## Agent harness An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs. - [Definition reference](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: b37fc706-f4bf-41de-89ab-8646ecba8957. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:01:25.495Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 7df70a14d93fc284a907bf8d88f16468922eb43458b01a38a61e0a3bea71a8cf. Source text hash: f7aa99fe180391f05491c84b3cb6ccc00ba5c4a18b2a8532cebcac5ebc1b370e. ### uncertain — 2026-11-12T03:01:25.495Z — anthropic/claude-opus-5.5 Publication: b37fc706-f4bf-41de-89ab-8646ecba8957 attribution — pass: The reading correctly attributes the study to Howard Zhang and coauthors on arXiv cs.AI. It keeps the reported results and recommendation as the authors' claims rather than established facts. > Authors:Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao summary — uncertain: The numbers are faithful, but a non-specialist may struggle with the central comparison. The reading never says that two of the frameworks use an LLM as a judge, and it omits the third, deterministic-heuristics paradigm. It also leaves trace-based, red-teaming, F1, evaluation harness and frozen execution traces unexplained. It drops the authors' third conclusion, that LLM harnesses surface subtle errors human reviewers miss. Smallest repair: name the three paradigms, add tooltips for these terms, and restore the human-reviewer finding. > (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics) date — pass: The original publication date of 2026-11-10 matches the v1 submission date shown on the arXiv page. > \[Submitted on 10 Nov 2026\] support — pass: Every claim is supported by an exact abstract excerpt: 97% stability, F1=0.90, 5x false alarms, 24 novel failures with zero overlap, and the two-framework recommendation. The open question and limitations accurately reflect the controlled, two-agent, frozen-trace scope. No benefits or costs beyond the source are invented. > applied to two analytics agents in a controlled research setting using frozen execution traces. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/42eea599c5caa88faab55ee6dfc3731076981e9883f1b0a926eaa4e20a1fe710/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-comparing-agent-evaluation-frameworks-in-enterprise-42eea599.json)