# arXiv cs.AI: Signals: Continuous Diagnostics of AI Agents [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-signals-continuous-diagnostics-of-ai-agents-f6576067) The authors of Signals, Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra and Asif Mohammad, present a method for identifying problems in AI agents from agentic traces. They say agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally, and that nondeterministic behavior makes these problems difficult to reproduce and diagnose. Signals classifies problems using a taxonomy of 45 types across eight families, distinguishes realized failures, risk exposures and inefficiencies, and groups related findings into engineering issues called Signals. Each Signal includes actions a developer can take and checks for evaluating changes; further executions can add evidence to an existing Signal or reveal a new problem. The authors report 91.43% F1 when matching Signals to reference issues on public agent executions, about 18 percentage points higher than a one-pass LLM control using the same model and taxonomy. This reported result concerns matching Signals to reference issues; it does not establish performance across other agents or settings. The abstract does not describe the executions or reference issues in enough detail to assess how closely they represent other uses. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T00:16:14.955Z. First observed: 2026-11-04T05:15:17.022Z. Retrieved: 2026-11-09T17:08:38.590Z. - [Signals: Continuous Diagnostics of AI Agents](https://arxiv.org/abs/2611.04507) ## Limits The supplied evidence gives the abstract and publication-page information, not the paper's full methods or results; it does not specify the public agent executions or reference issues used for the reported evaluation. The abstract reports a matching result on public agent executions but does not establish performance across other agents or settings. ## Open question Which public agent executions and reference issues did the authors use to evaluate Signals, and how would the reported matching result vary across other agents or settings? ## Source-stated The authors say AI agents can make incorrect decisions or waste resources despite normally working tools and infrastructure, and that nondeterministic behavior makes such problems difficult to reproduce and diagnose. > AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated The authors developed Signals to identify problems from agentic traces and suggest remediations. > We developed Signals to identify such problems from agentic traces and suggest concrete remediations. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Signals uses a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. > a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Signals distinguishes realized failures, risk exposures and inefficiencies, groups related findings as engineering issues, and assesses severity from supporting evidence. > distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Each Signal includes developer actions and checks; further executions can add evidence to an existing Signal or reveal a new problem. > Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated The authors report 91.43% F1 when matching Signals to reference issues on public agent executions, about 18 percentage points higher than a one-pass LLM control using the same model and taxonomy. > achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source and contributor context Rambatla, Amey, Goel, Vatsal, Gaur, Sagar, Loya, Siddhant, Mishra, Anurag, Mohammad, Asif · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Agentic AI AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#agentic) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: a03f22c3-e217-402f-8f48-3dc099b3494d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T00:16:14.955Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ceae9a0aa73fb2dd1ed4ee8d3f022aade28ec5cff27e215d5c98f220dd58a65d. Source text hash: a379673be04dcc562e2dbf55ec5df6218210d98fc7687d3ff16f79c6c534e474. ## Revision and source-check history ### Version 1 — 2026-11-10T00:16:14.955Z The authors of Signals, Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra and Asif Mohammad, present a method for identifying problems in AI agents from agentic traces. They say agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally, and that nondeterministic behavior makes these problems difficult to reproduce and diagnose. Signals classifies problems using a taxonomy of 45 types across eight families, distinguishes realized failures, risk exposures and inefficiencies, and groups related findings into engineering issues called Signals. Each Signal includes actions a developer can take and checks for evaluating changes; further executions can add evidence to an existing Signal or reveal a new problem. The authors report 91.43% F1 when matching Signals to reference issues on public agent executions, about 18 percentage points higher than a one-pass LLM control using the same model and taxonomy. This reported result concerns matching Signals to reference issues; it does not establish performance across other agents or settings. The abstract does not describe the executions or reference issues in enough detail to assess how closely they represent other uses. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T00:16:14.955Z. First observed: 2026-11-04T05:15:17.022Z. Retrieved: 2026-11-09T17:08:38.590Z. - [Signals: Continuous Diagnostics of AI Agents](https://arxiv.org/abs/2611.04507) ## Limits The supplied evidence gives the abstract and publication-page information, not the paper's full methods or results; it does not specify the public agent executions or reference issues used for the reported evaluation. The abstract reports a matching result on public agent executions but does not establish performance across other agents or settings. ## Open question Which public agent executions and reference issues did the authors use to evaluate Signals, and how would the reported matching result vary across other agents or settings? ## Source-stated The authors say AI agents can make incorrect decisions or waste resources despite normally working tools and infrastructure, and that nondeterministic behavior makes such problems difficult to reproduce and diagnose. > AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated The authors developed Signals to identify problems from agentic traces and suggest remediations. > We developed Signals to identify such problems from agentic traces and suggest concrete remediations. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Signals uses a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. > a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Signals distinguishes realized failures, risk exposures and inefficiencies, groups related findings as engineering issues, and assesses severity from supporting evidence. > distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated Each Signal includes developer actions and checks; further executions can add evidence to an existing Signal or reveal a new problem. > Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source-stated The authors report 91.43% F1 when matching Signals to reference issues on public agent executions, about 18 percentage points higher than a one-pass LLM control using the same model and taxonomy. > achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy. - [Supporting source](https://arxiv.org/abs/2611.04507) ## Source and contributor context Rambatla, Amey, Goel, Vatsal, Gaur, Sagar, Loya, Siddhant, Mishra, Anurag, Mohammad, Asif · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Agentic AI AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#agentic) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: a03f22c3-e217-402f-8f48-3dc099b3494d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T00:16:14.955Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ceae9a0aa73fb2dd1ed4ee8d3f022aade28ec5cff27e215d5c98f220dd58a65d. Source text hash: a379673be04dcc562e2dbf55ec5df6218210d98fc7687d3ff16f79c6c534e474. ### passed — 2026-11-10T00:16:14.955Z — anthropic/claude-opus-5.5 Publication: a03f22c3-e217-402f-8f48-3dc099b3494d attribution — pass: The reading attributes the method and its claims to the six listed arXiv authors, matching the page. Problem claims and results are framed as what the authors say or report, not as established fact. > Authors:Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra, Asif Mohammad summary — pass: The summary leads with the actor and the development: a method for diagnosing AI agent problems from traces. It follows the abstract's structure, and the question raises a real open issue about the evaluation data. Technical terms are kept exactly. Suggested term-library tooltips: 'agentic traces' (records of an agent's steps), 'F1', 'nondeterministic' and 'one-pass LLM control'. Their absence does not block a general reader from grasping the core point that the method beat a simpler baseline. > We developed Signals to identify such problems from agentic traces and suggest concrete remediations. date — pass: The displayed original date, 2026-11-03, matches the v1 submission date. It correctly ignores the 4 Nov v2 revision, which only corrected author affiliation formatting, and the 5 Nov feed date. > \[Submitted on 3 Nov 2026 (v1), last revised 4 Nov 2026 (this version, v2)\] support — pass: Each claim matches a verbatim abstract excerpt. The 91.43% F1 result and the roughly 18-point gain are reported with their scope: matching Signals to reference issues on public executions. The limitations correctly say the result does not establish real-world reliability elsewhere. The draft adds no benefits, costs or forecasts beyond the source. > achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. No other media candidates were found. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/f6576067cac55f6876f143a51d09f577ecab8e3e214b2f4be47c163941622418/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-signals-continuous-diagnostics-of-ai-agents-f6576067.json)