Huang, Chenyi, Arokiaraj, Arun Josephraj, Wu, Bin, Bulathwela, Sahan · Oct 16, 2026
Annotrace: A Workbench for Step-Level Annotation of Agent Traces
Markdown · JSON · Original source
The authors of “Annotrace: A Workbench for Step-Level Annotation of Agent Traces” argue that studies of why agents fail need a shared way to label what happens during a run. They present Annotrace, which organizes traces into ordered, nested steps and lets researchers define a codebook, with a proposing an initial set of labels from sample traces. The authors describe annotation as a joint human-AI process: the model answers first and is then asked the same questions as the human annotator. Annotrace also measures agreement between annotators. In a multi-agent question-answering corpus, the authors report reproducing 235 annotation units from a previous manual effort; two models labeling the same traces agreed 91% of the at κ = 0.82. These results describe that corpus and agreement, not whether the labels are correct or whether the approach generalizes to other studies. The paper is under review at the AAAI 2027 Demonstration Program. How well does Annotrace support consistent, useful labels on agent traces beyond the multi-agent question-answering corpus described by the authors?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 16, 2026 (2026-10-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T05:03:04.436Z. First observed: 2026-10-28T08:26:45.288Z. Retrieved: 2026-10-29T19:17:33.320Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported evaluation is on a multi-agent question-answering corpus; the evidence does not establish generalization to other tasks or settings. The paper is listed as under review at the AAAI 2027 Demonstration Program. Agreement between models, as reported here, does not by itself establish that their labels are accurate or useful.
Open question
How well does Annotrace support consistent, useful labels on agent traces beyond the multi-agent question-answering corpus described by the authors?
Source-stated
The authors say research into why LLM agents fail often depends on manually labeling traces, while existing tools do not combine structured labeling with an understanding of steps, delegation, and tool calls.
Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool.
Source-stated
Annotrace reads six trace formats and organizes each run as an ordered, nested list of steps.
Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps.
Source-stated
The authors report reproducing 235 annotation units from an earlier manual effort and 91% agreement between two models on the same traces, at κ = 0.82.
On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $\kappa = 0.82$.
Escape Meridian interpretation
The reported evaluation does not establish whether labels are correct or whether the results generalize beyond the corpus studied.
On a multi-agent question-answering corpus
Source and contributor context
Huang, Chenyi, Arokiaraj, Arun Josephraj, Wu, Bin, Bulathwela, Sahan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0d983343-ce30-4b1e-8d59-00c826355fa0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T05:03:04.436Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9b25fba92643f23669b48a814edf614c84fb8b65def5175779fddd286dd301e2. Source text hash: 158e9ca3625e831f1e0d7648d22ef8f0076e2df52fb5cdf38c424684e5fa3bbc.
Revision and source-check history
Version 1 · 2026-11-14T05:03:04.436Z
Annotrace: A Workbench for Step-Level Annotation of Agent Traces
The authors of “Annotrace: A Workbench for Step-Level Annotation of Agent Traces” argue that studies of why agents fail need a shared way to label what happens during a run. They present Annotrace, which organizes traces into ordered, nested steps and lets researchers define a codebook, with a proposing an initial set of labels from sample traces. The authors describe annotation as a joint human-AI process: the model answers first and is then asked the same questions as the human annotator. Annotrace also measures agreement between annotators. In a multi-agent question-answering corpus, the authors report reproducing 235 annotation units from a previous manual effort; two models labeling the same traces agreed 91% of the at κ = 0.82. These results describe that corpus and agreement, not whether the labels are correct or whether the approach generalizes to other studies. The paper is under review at the AAAI 2027 Demonstration Program. How well does Annotrace support consistent, useful labels on agent traces beyond the multi-agent question-answering corpus described by the authors?
Original source
cs.AI · Article
Source details
Original source date: Oct 16, 2026 (2026-10-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T05:03:04.436Z. First observed: 2026-10-28T08:26:45.288Z. Retrieved: 2026-10-29T19:17:33.320Z.
Limits
The reported evaluation is on a multi-agent question-answering corpus; the evidence does not establish generalization to other tasks or settings. The paper is listed as under review at the AAAI 2027 Demonstration Program. Agreement between models, as reported here, does not by itself establish that their labels are accurate or useful.
Open question
How well does Annotrace support consistent, useful labels on agent traces beyond the multi-agent question-answering corpus described by the authors?
Source-stated
The authors say research into why LLM agents fail often depends on manually labeling traces, while existing tools do not combine structured labeling with an understanding of steps, delegation, and tool calls.
Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool.
Source-stated
Annotrace reads six trace formats and organizes each run as an ordered, nested list of steps.
Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps.
Source-stated
The authors report reproducing 235 annotation units from an earlier manual effort and 91% agreement between two models on the same traces, at κ = 0.82.
On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $\kappa = 0.82$.
Escape Meridian interpretation
The reported evaluation does not establish whether labels are correct or whether the results generalize beyond the corpus studied.
On a multi-agent question-answering corpus
Source and contributor context
Huang, Chenyi, Arokiaraj, Arun Josephraj, Wu, Bin, Bulathwela, Sahan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0d983343-ce30-4b1e-8d59-00c826355fa0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T05:03:04.436Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9b25fba92643f23669b48a814edf614c84fb8b65def5175779fddd286dd301e2. Source text hash: 158e9ca3625e831f1e0d7648d22ef8f0076e2df52fb5cdf38c424684e5fa3bbc.
uncertain · 2026-11-14T05:03:04.436Z · anthropic/claude-opus-5.5
Publication: 0d983343-ce30-4b1e-8d59-00c826355fa0
attribution — pass: The tool, motivation and results are attributed to the paper's authors, matching the arXiv listing. The reported figures are framed as author reports, not established facts.
Authors:Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu, Sahan Bulathwela
summary — uncertain: The reading tracks the abstract, but central terms go unexplained. 'Agent traces' (step-by-step records of what an AI agent did), 'codebook', 'frontier model' and 'κ' (a chance-corrected agreement statistic) are not glossed or tooltipped, so a general reader may not grasp what is being labelled or what 0.82 means. It also drops 'exactly' from the 235-unit result and leaves unclear that this means matching a prior study's segmentation. Smallest repair: explain or tooltip 'trace' and 'κ', and restore 'exactly'. The question is also repeated inside the summary text.
it reproduces a previous manual effort's 235 annotation units exactly
date — pass: originalPublished 2026-10-16 matches the v1 submission date. It is not a later revision (v3, 27 Oct) or the feed date.
[Submitted on 16 Oct 2026 (v1), last revised 27 Oct 2026 (this version, v3)]
support — pass: The source-stated claims match the abstract. The interpretation that model agreement does not establish correctness or generalization is labeled as interpretation and stays appropriately cautious. No unsupported benefits or costs are implied. Note: 'answers first and is then asked' slightly sequences what the source states in parallel, but this is not material.
two models labelling the same traces agree 91% of the time at $\kappa = 0.82$
media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv