Escape Meridian

Du, Gaoyuan, Liu, Xiaoyang, Zhou, Rex, Jiang, Dengdu, Fu, Rong, Ding, Meng, Cao, Defu · Nov 14, 2026

The authors argue that a critic can make AutoML search worse

Markdown · JSON · Original source

The authors of “The Critic Can Hurt: On the Safety of Critics in -Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a for detecting harmful critic feedback in AutoML.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXiv cs.AIArticle · arxiv.orgOpen original source

Original source

cs.AI · Article

Source details

Original source date: Nov 14, 2026 (2026-11-14).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:46:13.569Z. First observed: 2026-11-17T08:31:33.970Z. Retrieved: 2026-11-17T09:32:51.752Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite. The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline.

Open question

Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?

Source-stated

The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.

a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case

Source-stated

The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.

stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold

Source-stated

The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.

non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities

Source-stated

The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.

a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline

Source-stated

The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.

We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.

Source and contributor context

Du, Gaoyuan, Liu, Xiaoyang, Zhou, Rex, Jiang, Dengdu, Fu, Rong, Ding, Meng, Cao, Defu · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: 11a20f82-9b57-404b-87e4-44dae2a6fef5.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T09:46:13.569Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: d954bd67c961debba41f935736fe1ff28a15f9492f976d1fd2269cdc59116f4b. Source text hash: 2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a.

Revision and source-check history

Version 1 · 2026-11-17T09:46:13.569Z

The authors argue that a critic can make AutoML search worse

The authors of “The Critic Can Hurt: On the Safety of Critics in -Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a for detecting harmful critic feedback in AutoML.

Original source

cs.AI · Article

Source details

Original source date: Nov 14, 2026 (2026-11-14).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:46:13.569Z. First observed: 2026-11-17T08:31:33.970Z. Retrieved: 2026-11-17T09:32:51.752Z.

Limits

The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite. The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline.

Open question

Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?

Source-stated

The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.

a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case

Source-stated

The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.

stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold

Source-stated

The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.

non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities

Source-stated

The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.

a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline

Source-stated

The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.

We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.

Source and contributor context

Du, Gaoyuan, Liu, Xiaoyang, Zhou, Rex, Jiang, Dengdu, Fu, Rong, Ding, Meng, Cao, Defu · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: 11a20f82-9b57-404b-87e4-44dae2a6fef5.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T09:46:13.569Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: d954bd67c961debba41f935736fe1ff28a15f9492f976d1fd2269cdc59116f4b. Source text hash: 2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a.

passed · 2026-11-17T09:46:13.569Z · anthropic/claude-opus-5.5

Publication: 11a20f82-9b57-404b-87e4-44dae2a6fef5

attribution — pass: The reading attributes every finding and recommendation to the paper's authors, matching the arXiv abstract. Results are framed as reported claims, not established fact. The title correctly frames the critic-harm thesis as the authors' argument.

Authors:Gaoyuan Du, Xiaoyang Liu, Rex Zhou, Dengdu Jiang, Rong Fu, Meng Ding, Defu Cao

summary — pass: The reading leads with the thesis that a critic can hurt rather than help. It keeps the abstract's when/why/how structure and its scope conditions, and closes with the authors' 'audit before trusting' recommendation, preserving their force. Technical terms such as AutoML, AUC, 8B, frontier and best-so-far search are kept exactly and suit term-library tooltips; suggested additions are AutoML and AUC. Omitting the idealized necessary condition q(m-1) greater than beta is a minor loss and does not distort the meaning. The question follows from a stated gap.

A critic is not a free addition: audit before trusting, and gate when capability or verification is insufficient.

date — pass: The originalPublished value of 2026-11-14 matches the v1 submission date on the arXiv page. The later feed date of 2026-11-17 was correctly not used.

[v1] Sat, 14 Nov 2026 00:24:12 UTC (140 KB)

support — pass: All five claims quote exact abstract text and keep their qualifiers. These include 'on the evaluated suite', 'not guaranteed' and 'will release'. The significance sentence, that critic feedback can alter which candidates get explored, follows from the source's description of critics steering iterative search and of errors consuming search opportunities. The limitations accurately note what the abstract does not establish.

LLM critics increasingly steer iterative search in agentic AutoML, prompt optimization, and self-refinement--evaluated for whether they help, but rarely for whether they can hurt.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct way to reach the source.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.