· Nov 3, 2026
Hugging Face: Same Model, Different Harness—When to Pick CUGA?
Markdown · JSON · Original source
The IBM Research authors argue that choosing an can substantially affect task completion, even when the model stays the same, and that the right choice depends on the work. In their AppWorld comparison using GLM-5.3-flash, CUGA passed 73 of 90 tasks, Hermes 42, and Deep Agents 27. They describe AppWorld as a proxy: its consumer apps test some capabilities also needed for enterprise automation, but it is not a direct enterprise . The authors say the comparison covered 90 tasks on one benchmark, used setups that were not identical, and did not test policy enforcement because the runs used no policies. They present CUGA as a harness for business work across , apps, and web UIs, with policies that can be enforced by the runtime. Their examples include pausing for a person’s approval before a payment tool runs and blocking a payment over a configured limit. They also describe deployment inside an organization’s own environment, including air-gapped deployments. The authors invite readers to submit a Hermes or Deep Agents configuration for evaluation on the same tasks, and say they will publish the rerun results whichever harness wins.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T15:57:39.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T06:08:57.006Z. First observed: 2026-11-03T16:01:44.564Z. Retrieved: 2026-11-03T16:02:04.340Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The comparison covers 90 tasks on one , and the authors say the results may not generalize to other tasks. The harness setups were not identical; CUGA used an AppWorld configuration the other harnesses did not. The runs did not use policies, so they do not measure policy enforcement. The source reports a benchmark comparison, not evidence that the same task-completion rates will occur in enterprise deployments.
Open question
How would Hermes or Deep Agents perform on the same tasks if configured for AppWorld, and would a rerun change the comparison?
Source-stated
The IBM Research authors argue that agent harness choice can substantially affect task completion even when the model is held constant.
we kept the model the same and changed only the harness.
Source-stated
In the authors’ AppWorld comparison using GLM-5.3-flash, CUGA passed 73 of 90 tasks, Hermes 42, and Deep Agents 27.
CUGA 81.1 (73/90) 66.7 (20/30) Hermes 46.7 (42/90) 26.7 (8/30) Deep Agents 30.0 (27/90) 6.7 (2/30)
Source-stated
The authors describe AppWorld as a proxy that tests a subset of capabilities also needed for enterprise automation, rather than as a direct enterprise benchmark.
The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs
Source-stated
The authors say the comparison did not test policy enforcement, because the runs did not use policies.
And the runs did not use policies, so they do not measure policy enforcement.
Source-stated
The authors distinguish policy guidance from runtime enforcement: approvals and ToolGuards can stop tool calls regardless of the model’s decision.
approvals and ToolGuards are enforced by the runtime.
Source-stated
The authors invite readers to submit a Hermes or Deep Agents configuration for evaluation on the same tasks and say they will publish the rerun results whichever harness wins.
We will rerun it on the same tasks and publish the results, whichever harness wins.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
API
An application programming interface: a defined way for software to request data or actions from another system.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: 8205e233-f6ef-4727-8740-7be62d3c4d6a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T06:08:57.006Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 6d25cccc9c9e051aa347ab34c5678170ac4d02c8baf2e5932482feb26421054e. Source text hash: 6f89f46ed476eb3e60346ce38977bbc2665d2e00398265aca7511800662ae266.
Revision and source-check history
Version 1 · 2026-11-10T06:08:57.006Z
Hugging Face: Same Model, Different Harness—When to Pick CUGA?
The IBM Research authors argue that choosing an can substantially affect task completion, even when the model stays the same, and that the right choice depends on the work. In their AppWorld comparison using GLM-5.3-flash, CUGA passed 73 of 90 tasks, Hermes 42, and Deep Agents 27. They describe AppWorld as a proxy: its consumer apps test some capabilities also needed for enterprise automation, but it is not a direct enterprise . The authors say the comparison covered 90 tasks on one benchmark, used setups that were not identical, and did not test policy enforcement because the runs used no policies. They present CUGA as a harness for business work across , apps, and web UIs, with policies that can be enforced by the runtime. Their examples include pausing for a person’s approval before a payment tool runs and blocking a payment over a configured limit. They also describe deployment inside an organization’s own environment, including air-gapped deployments. The authors invite readers to submit a Hermes or Deep Agents configuration for evaluation on the same tasks, and say they will publish the rerun results whichever harness wins.
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T15:57:39.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T06:08:57.006Z. First observed: 2026-11-03T16:01:44.564Z. Retrieved: 2026-11-03T16:02:04.340Z.
Limits
The comparison covers 90 tasks on one , and the authors say the results may not generalize to other tasks. The harness setups were not identical; CUGA used an AppWorld configuration the other harnesses did not. The runs did not use policies, so they do not measure policy enforcement. The source reports a benchmark comparison, not evidence that the same task-completion rates will occur in enterprise deployments.
Open question
How would Hermes or Deep Agents perform on the same tasks if configured for AppWorld, and would a rerun change the comparison?
Source-stated
The IBM Research authors argue that agent harness choice can substantially affect task completion even when the model is held constant.
we kept the model the same and changed only the harness.
Source-stated
In the authors’ AppWorld comparison using GLM-5.3-flash, CUGA passed 73 of 90 tasks, Hermes 42, and Deep Agents 27.
CUGA 81.1 (73/90) 66.7 (20/30) Hermes 46.7 (42/90) 26.7 (8/30) Deep Agents 30.0 (27/90) 6.7 (2/30)
Source-stated
The authors describe AppWorld as a proxy that tests a subset of capabilities also needed for enterprise automation, rather than as a direct enterprise benchmark.
The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs
Source-stated
The authors say the comparison did not test policy enforcement, because the runs did not use policies.
And the runs did not use policies, so they do not measure policy enforcement.
Source-stated
The authors distinguish policy guidance from runtime enforcement: approvals and ToolGuards can stop tool calls regardless of the model’s decision.
approvals and ToolGuards are enforced by the runtime.
Source-stated
The authors invite readers to submit a Hermes or Deep Agents configuration for evaluation on the same tasks and say they will publish the rerun results whichever harness wins.
We will rerun it on the same tasks and publish the results, whichever harness wins.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
API
An application programming interface: a defined way for software to request data or actions from another system.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: 8205e233-f6ef-4727-8740-7be62d3c4d6a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T06:08:57.006Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 6d25cccc9c9e051aa347ab34c5678170ac4d02c8baf2e5932482feb26421054e. Source text hash: 6f89f46ed476eb3e60346ce38977bbc2665d2e00398265aca7511800662ae266.
uncertain · 2026-11-10T06:08:57.006Z · anthropic/claude-opus-5.5
Publication: 8205e233-f6ef-4727-8740-7be62d3c4d6a
attribution — pass: The reading attributes the argument and results to the IBM Research authors, with Hugging Face as the publisher. It keeps benchmark results and CUGA's capabilities as the authors' claims. The title's 'Hugging Face:' prefix names the publisher, and the summary correctly identifies the authors.
CUGA (ConfigUrable Generalist Agent) is IBM's open-source harness built for that job
summary — uncertain: The summary never explains the central term 'agent harness': the software around a model that plans steps, calls tools, and checks or enforces rules. A non-specialist cannot readily see why swapping it changes results. It also leaves 'open-weight model' and 'air-gapped' unexplained. Smallest repair: add one plain-language line defining a harness, ideally anchored to the source's prepare, call the model, execute loop. The rest is faithful, including the stated limits.
Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat.
date — pass: The originalPublished value, 2026-11-03, matches the article's stated publication date of November 3, 2026.
Published November 3, 2026
support — pass: The scores, the AppWorld proxy caveat, the absence of policies, runtime enforcement, the payment approval and limit examples, air-gapped deployment, and the rerun invitation are all stated in the source. The question follows from the authors' own disclosure that CUGA used a tailored configuration. No unsupported benefits are implied.
The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not.
media — pass: The selected media is the verified direct link to the original Hugging Face article, which is the most relevant way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn application programming interface: a defined way for software to request data or actions from another system.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn application programming interface: a defined way for software to request data or actions from another system.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference