{"id":"6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5","title":"The authors argue that a critic can make AutoML search worse","argument":"The authors of “The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite. The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline.","speaker":{"name":"Du, Gaoyuan, Liu, Xiaoyang, Zhou, Rex, Jiang, Dengdu, Fu, Rong, Ding, Meng, Cao, Defu","roleAtSource":"Article author"},"source":{"title":"The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.19037","publishedAt":"2026-11-14","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.","excerpt":"a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.","excerpt":"stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.","excerpt":"non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.","excerpt":"a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.","excerpt":"We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.19037","checkedAt":"2026-11-17T10:50:07.423Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"The authors argue that a critic can make AutoML search worse","summary":"The authors of “The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.","question":"Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?","claims":[{"text":"The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.","excerpt":"a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case","support":"source-stated"},{"text":"The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.","excerpt":"stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold","support":"source-stated"},{"text":"The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.","excerpt":"non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities","support":"source-stated"},{"text":"The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.","excerpt":"a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline","support":"source-stated"},{"text":"The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.","excerpt":"We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.","support":"source-stated"}],"limitations":["The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite.","The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline."],"mediaId":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","correctionReason":null,"id":"6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5","publicationId":"11a20f82-9b57-404b-87e4-44dae2a6fef5","version":1,"sourceUrl":"https://arxiv.org/abs/2611.19037","sourceTitle":"The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-14","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:31:33.970Z","createdAt":"2026-11-17T09:46:13.569Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","kind":"article","url":"https://arxiv.org/abs/2611.19037","sourceUrl":"https://arxiv.org/abs/2611.19037","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-17T09:32:51.807Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.19037","checkedAt":"2026-11-17T10:50:07.423Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d207122733f0b4f5496a8a99a388ac45bba712f6ca2aeef5e8b348cca6eb5113","decidedAt":"2026-11-17T09:32:52.310Z"},"provenance":{"textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observationId":"d954bd67c961debba41f935736fe1ff28a15f9492f976d1fd2269cdc59116f4b","retrievedAt":"2026-11-17T09:32:51.752Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fad6e7d2-fc9f-43e1-8419-193306344111"},"reviewer":{"model":"gpt-6-luna","runId":"261723b2-eb26-4fa7-8337-86c018986d8d"}},"audit":{"status":"passed","reviewedAt":"2026-11-17T09:46:13.569Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Du, Gaoyuan","url":null,"kind":"unknown"},{"name":"Liu, Xiaoyang","url":null,"kind":"unknown"},{"name":"Zhou, Rex","url":null,"kind":"unknown"},{"name":"Jiang, Dengdu","url":null,"kind":"unknown"},{"name":"Fu, Rong","url":null,"kind":"unknown"},{"name":"Ding, Meng","url":null,"kind":"unknown"},{"name":"Cao, Defu","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Du, Gaoyuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Du, Gaoyuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Liu, Xiaoyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Xiaoyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Zhou, Rex","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Rex","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Jiang, Dengdu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Dengdu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Fu, Rong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Rong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Ding, Meng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ding, Meng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Cao, Defu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Cao, Defu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.19037","snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5/source"}},"canonicalUrl":"https://escapemeridian.com/readings/the-authors-argue-that-a-critic-can-make-automl-search-worse-6ac013d7","formats":{"html":"/readings/the-authors-argue-that-a-critic-can-make-automl-search-worse-6ac013d7","markdown":"/readings/the-authors-argue-that-a-critic-can-make-automl-search-worse-6ac013d7.md","json":"/readings/the-authors-argue-that-a-critic-can-make-automl-search-worse-6ac013d7.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5","revision":"11a20f82-9b57-404b-87e4-44dae2a6fef5","revisionKind":"publication-id","title":"The authors argue that a critic can make AutoML search worse","url":"https://escapemeridian.com/readings/the-authors-argue-that-a-critic-can-make-automl-search-worse-6ac013d7"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.19037","revision":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","revisionKind":"sha256-source-text","title":"The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search","url":"https://arxiv.org/abs/2611.19037"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5","title":"The authors argue that a critic can make AutoML search worse","argument":"The authors of “The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite. The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline.","speaker":{"name":"Du, Gaoyuan, Liu, Xiaoyang, Zhou, Rex, Jiang, Dengdu, Fu, Rong, Ding, Meng, Cao, Defu","roleAtSource":"Article author"},"source":{"title":"The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.19037","publishedAt":"2026-11-14","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.","excerpt":"a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.","excerpt":"stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.","excerpt":"non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.","excerpt":"a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"},{"text":"The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.","excerpt":"We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.","url":"https://arxiv.org/abs/2611.19037","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.19037","checkedAt":"2026-11-17T10:50:07.423Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"The authors argue that a critic can make AutoML search worse","summary":"The authors of “The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search” argue that adding an LLM critic to guide search can reduce performance, rather than simply help. In their audit, a naive 8B critic pushed results below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC. They report that the size of the loss varies with the dataset, frontier and budget, and that stronger critics reduced the loss on the evaluated suite without establishing a universal capability threshold. Their explanation is about how errors spend limited search opportunities: non-directional errors were nearly costless under best-so-far search, while directional errors consumed opportunities. The authors say a simple validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. That distinction matters for anyone using a critic to steer automated experiments: its feedback may alter which candidates get explored, and adding one is not automatically beneficial. The authors call for auditing critics before trusting them and gating feedback when capability or verification is insufficient. They also say they will release CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.","question":"Under what conditions, if any, can the validation gate or learned detector preserve the full-budget no-critic baseline?","claims":[{"text":"The authors report that a naive 8B critic drove search below the no-critic random-search floor on 7 of 8 tabular datasets, with a worst-case loss of 0.245 AUC.","excerpt":"a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case","support":"source-stated"},{"text":"The authors say the loss varies with dataset, frontier and budget, and that stronger critics reduced it on the evaluated suite without establishing a universal capability threshold.","excerpt":"stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold","support":"source-stated"},{"text":"The authors attribute the difference in cost to error direction: non-directional errors were nearly costless under best-so-far search, while directional errors consumed finite search opportunities.","excerpt":"non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities","support":"source-stated"},{"text":"The authors report that a validation gate and a learned detector can reduce realized loss, but are not guaranteed to preserve the full-budget no-critic baseline.","excerpt":"a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline","support":"source-stated"},{"text":"The authors say they will release CRITFAULT as a benchmark for detecting harmful critic feedback in AutoML.","excerpt":"We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML.","support":"source-stated"}],"limitations":["The reported dataset result covers 8 tabular datasets; the abstract does not establish that the same outcome holds beyond the evaluated suite.","The abstract does not give details of the validation gate or learned detector, or state conditions under which either preserves the full-budget no-critic baseline."],"mediaId":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","correctionReason":null,"id":"6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5","publicationId":"11a20f82-9b57-404b-87e4-44dae2a6fef5","version":1,"sourceUrl":"https://arxiv.org/abs/2611.19037","sourceTitle":"The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-14","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:31:33.970Z","createdAt":"2026-11-17T09:46:13.569Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","kind":"article","url":"https://arxiv.org/abs/2611.19037","sourceUrl":"https://arxiv.org/abs/2611.19037","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-17T09:32:51.807Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.19037","checkedAt":"2026-11-17T10:50:07.423Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"73caa4557bfa8c24cf01971949e709d7c276cbf7371b57dc6ae2548f4769f76a","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d207122733f0b4f5496a8a99a388ac45bba712f6ca2aeef5e8b348cca6eb5113","decidedAt":"2026-11-17T09:32:52.310Z"},"provenance":{"textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observationId":"d954bd67c961debba41f935736fe1ff28a15f9492f976d1fd2269cdc59116f4b","retrievedAt":"2026-11-17T09:32:51.752Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fad6e7d2-fc9f-43e1-8419-193306344111"},"reviewer":{"model":"gpt-6-luna","runId":"261723b2-eb26-4fa7-8337-86c018986d8d"}},"audit":{"status":"passed","reviewedAt":"2026-11-17T09:46:13.569Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Du, Gaoyuan","url":null,"kind":"unknown"},{"name":"Liu, Xiaoyang","url":null,"kind":"unknown"},{"name":"Zhou, Rex","url":null,"kind":"unknown"},{"name":"Jiang, Dengdu","url":null,"kind":"unknown"},{"name":"Fu, Rong","url":null,"kind":"unknown"},{"name":"Ding, Meng","url":null,"kind":"unknown"},{"name":"Cao, Defu","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Du, Gaoyuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Du, Gaoyuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Liu, Xiaoyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Xiaoyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Zhou, Rex","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Rex","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Jiang, Dengdu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Dengdu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Fu, Rong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Rong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Ding, Meng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ding, Meng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}},{"actor":{"name":"Cao, Defu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Cao, Defu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.19037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:50:07.423Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","sourceUrl":"https://arxiv.org/abs/2611.19037","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.19037","snapshotId":"ffbe9eb9485f682b17eca4774055ed0751b9bffc370257b6f54aa2cbc6ae0ac2","contentSha256":"d7a98962d5114d2ac3d13a226757a7b4f08db12ddca033ece9d04a007bf6582e","textHash":"2fe67b2d50989e33c0bb928f2fc67a873f0db8b048a4b06992d960ae4979205a","observedAt":"2026-11-17T10:50:07.423Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6ac013d78f5ff35d4b6c593aebdf77fbe6eeefc5291bc0504672afa2370639a5/source"}}}],"audits":[{"publicationId":"11a20f82-9b57-404b-87e4-44dae2a6fef5","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes every finding and recommendation to the paper's authors, matching the arXiv abstract. Results are framed as reported claims, not established fact. The title correctly frames the critic-harm thesis as the authors' argument.","excerpt":"Authors:Gaoyuan Du, Xiaoyang Liu, Rex Zhou, Dengdu Jiang, Rong Fu, Meng Ding, Defu Cao"},{"dimension":"summary","status":"pass","explanation":"The reading leads with the thesis that a critic can hurt rather than help. It keeps the abstract's when/why/how structure and its scope conditions, and closes with the authors' 'audit before trusting' recommendation, preserving their force. Technical terms such as AutoML, AUC, 8B, frontier and best-so-far search are kept exactly and suit term-library tooltips; suggested additions are AutoML and AUC. Omitting the idealized necessary condition q(m-1) greater than beta is a minor loss and does not distort the meaning. The question follows from a stated gap.","excerpt":"A critic is not a free addition: audit before trusting, and gate when capability or verification is insufficient."},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-11-14 matches the v1 submission date on the arXiv page. The later feed date of 2026-11-17 was correctly not used.","excerpt":"[v1]\nSat, 14 Nov 2026 00:24:12 UTC (140 KB)"},{"dimension":"support","status":"pass","explanation":"All five claims quote exact abstract text and keep their qualifiers. These include 'on the evaluated suite', 'not guaranteed' and 'will release'. The significance sentence, that critic feedback can alter which candidates get explored, follows from the source's description of critics steering iterative search and of errors consuming search opportunities. The limitations accurately note what the abstract does not establish.","excerpt":"LLM critics increasingly steer iterative search in agentic AutoML, prompt optimization, and self-refinement--evaluated for whether they help, but rarely for whether they can hurt."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-17T09:46:13.569Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}