Wong, Dowling · Nov 18, 2026
arXiv cs.AI: Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics
Markdown · JSON · Original source
Dowling Wong argues that judging autonomous agents that run experiments only by their best score misses how they got there: whether they stayed within permitted files, retained failed trials, and made keep-or-discard decisions traceable. The paper proposes evaluating experimental process alongside outcome. Its governed harness gives a control plane—not the proposing changes—responsibility for each trial, including checking edit scope, deciding whether to keep or discard it, and recording it in an append-only ledger. Wong reports applying the same contract across six ML tasks and 1,445 trials. The ledger surfaced failures obscured by task scores, including a no-memory arm where all 120 trials failed, exhausted -adapter proposals, and a reset failure detected by reproducibility hashes; affected campaigns were rerun. In a matched , a loop logging only successful trials omitted 43 of 50 trials, while the governed ledger recorded all 50. The paper also reports that memory’s effect on repeated bad proposals varies by task. Wong says the trial ledgers are released so reported numbers can be recomputed. The abstract describes results from these tasks and trials; it does not establish how the approach performs beyond them.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 18, 2026 (2026-11-18).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T07:18:38.580Z. First observed: 2026-11-19T05:01:30.346Z. Retrieved: 2026-11-19T07:17:49.912Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract reports results across six tasks and 1,445 trials; it does not establish how the approach performs on other tasks. The supplied evidence is the paper abstract, not the full paper; it does not provide full metric results or detailed methods.
Open question
How would the same contract perform on tasks beyond those evaluated in the paper?
Source-stated
Wong argues that best-score-only evaluation misses whether experiments stayed within permitted files, retained failed trials, and made decisions traceable.
That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence.
Source-stated
The proposed harness gives a control plane responsibility for trial lifecycle, edit-scope checks, keep-or-discard decisions, and recording each trial in an append-only ledger.
A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger.
Source-stated
The paper reports using the same contract across six ML tasks and 1,445 trials, with ledger records exposing failures that task scores hide.
Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides
Source-stated
In a matched counterfactual, logging only successful trials omitted 43 of 50 trials, whereas the governed ledger recorded all 50.
In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50.
Source-stated
The reported effect of memory on repeated bad proposals depends on the task.
Memory ablations show that memory's effect on repeated bad proposals depends on the task.
Source-stated
Wong says releasing the trial ledgers allows the reported numbers to be recomputed.
We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.
Source and contributor context
Wong, Dowling · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Counterfactual
An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 76d2a8d7-c966-448a-824a-72d5d5c2185c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T07:18:38.580Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 311c36647d2be2d6b55be6db4b54686c269c046a10a6253a9f35eef40db873d6. Source text hash: 940aa3f53eedff0b3cf3cee7ed4c29a1d240c3d0947b5c39d36b04d91791df52.
Revision and source-check history
Version 1 · 2026-11-19T07:18:38.580Z
arXiv cs.AI: Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics
Dowling Wong argues that judging autonomous agents that run experiments only by their best score misses how they got there: whether they stayed within permitted files, retained failed trials, and made keep-or-discard decisions traceable. The paper proposes evaluating experimental process alongside outcome. Its governed harness gives a control plane—not the proposing changes—responsibility for each trial, including checking edit scope, deciding whether to keep or discard it, and recording it in an append-only ledger. Wong reports applying the same contract across six ML tasks and 1,445 trials. The ledger surfaced failures obscured by task scores, including a no-memory arm where all 120 trials failed, exhausted -adapter proposals, and a reset failure detected by reproducibility hashes; affected campaigns were rerun. In a matched , a loop logging only successful trials omitted 43 of 50 trials, while the governed ledger recorded all 50. The paper also reports that memory’s effect on repeated bad proposals varies by task. Wong says the trial ledgers are released so reported numbers can be recomputed. The abstract describes results from these tasks and trials; it does not establish how the approach performs beyond them.
Original source
cs.AI · Article
Source details
Original source date: Nov 18, 2026 (2026-11-18).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T07:18:38.580Z. First observed: 2026-11-19T05:01:30.346Z. Retrieved: 2026-11-19T07:17:49.912Z.
Limits
The abstract reports results across six tasks and 1,445 trials; it does not establish how the approach performs on other tasks. The supplied evidence is the paper abstract, not the full paper; it does not provide full metric results or detailed methods.
Open question
How would the same contract perform on tasks beyond those evaluated in the paper?
Source-stated
Wong argues that best-score-only evaluation misses whether experiments stayed within permitted files, retained failed trials, and made decisions traceable.
That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence.
Source-stated
The proposed harness gives a control plane responsibility for trial lifecycle, edit-scope checks, keep-or-discard decisions, and recording each trial in an append-only ledger.
A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger.
Source-stated
The paper reports using the same contract across six ML tasks and 1,445 trials, with ledger records exposing failures that task scores hide.
Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides
Source-stated
In a matched counterfactual, logging only successful trials omitted 43 of 50 trials, whereas the governed ledger recorded all 50.
In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50.
Source-stated
The reported effect of memory on repeated bad proposals depends on the task.
Memory ablations show that memory's effect on repeated bad proposals depends on the task.
Source-stated
Wong says releasing the trial ledgers allows the reported numbers to be recomputed.
We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.
Source and contributor context
Wong, Dowling · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Counterfactual
An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 76d2a8d7-c966-448a-824a-72d5d5c2185c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T07:18:38.580Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 311c36647d2be2d6b55be6db4b54686c269c046a10a6253a9f35eef40db873d6. Source text hash: 940aa3f53eedff0b3cf3cee7ed4c29a1d240c3d0947b5c39d36b04d91791df52.
uncertain · 2026-11-19T07:18:38.580Z · anthropic/claude-opus-5.5
Publication: 76d2a8d7-c966-448a-824a-72d5d5c2185c
attribution — pass: The arXiv page lists Dowling Wong as the sole author. Attributing the abstract's 'we' claims and arguments to Wong is accurate, and the reading keeps them as reported claims rather than established facts.
Authors:Dowling Wong
summary — uncertain: The reading is faithful but has two gaps. First, it omits the paper's named contribution: governance metrics (invalid-action rate, repeated-bad rate, provenance completeness) computed from the ledger. The title highlights these as 'Failure Metrics'. Smallest repair: add one sentence naming them. Second, central terms are left unexplained for a general reader: control plane, contract, no-memory arm, matched counterfactual and memory ablations. The question relies on 'the same contract' without saying what it is. Smallest repair: add plain glosses or tooltips, e.g. a control plane as separate supervising software.
Governance metrics such as the invalid-action rate, the repeated-bad rate and provenance completeness are computed from that ledger.
date — pass: originalPublished is 2026-11-18. This matches the v1 submission date on arXiv. The feed date of Nov 19 is later and is correctly not used.
[Submitted on 18 Nov 2026]
support — pass: Each claim and summary detail matches the abstract: the 1,445 trials, six tasks, 120 failed no-memory trials, 43 of 50 unlogged trials, reruns and the ledger release. The limitations correctly note that the evidence is the abstract only and that results may not generalize. The draft states no unsupported benefits or costs.
In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50.
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way for a reader to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Methods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition reference