Huyut, Mehmet Murat · Nov 4, 2026
arXiv cs.AI paper: Banking-rule compliance changes with the model’s harness
Markdown · JSON · Original source
Mehmet Murat Huyut argues that banks should validate the deployed model-harness pair, not a language model in isolation. In a simulated retail bank, the study held each of four fixed while running it through five harnesses: systems that shape what a model sees, which tools it can use, and who checks its actions. Across 3,120 two-turn episodes, pooled correct-case rates shifted only from 87.3% to 92.1%, yet one model’s best-to-worst harness spread was 11.5 percentage points, enough to move it from first to last among the four; another showed no detectable difference. A same-model compliance reviewer reduced violations from 9.6% to 6.4% without increasing over-refusals, contrary to the pre-registered prediction. Rules requiring date arithmetic failed in every harness. In an exploratory analysis, 26% of violations included a customer message denying an action the agent had already executed; the authors say transcript-only review would record these as correct refusals. Their conclusion is to validate the model-harness pair and check the action log, not just the conversation. The study used simulated cases, and its model x harness interaction was suggestive rather than conclusive.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T11:16:59.634Z. First observed: 2026-11-05T05:00:36.084Z. Retrieved: 2026-11-09T11:10:00.066Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The cases were run in a simulated retail bank, not a live banking deployment. The authors describe the model x harness interaction as suggestive rather than conclusive. The finding about violations paired with customer messages came from an exploratory analysis using two judges and a blind human check.
Open question
Would the differences between harnesses persist in tests beyond the 156 pre-registered simulated-bank cases?
Source-stated
Huyut argues that model-risk management should validate the model-harness pair, because a harness can change a fixed model’s compliance with banking conduct rules.
For model-risk management the unit of validation should be the model-harness pair
Source-stated
The study tested four open-weight models in five harnesses on 156 pre-registered cases in a simulated retail bank.
Four open-weight models each ran five harnesses
Source-stated
Across models, correct-case rates varied from 87.3% to 92.1%; the harness effect differed by model, including an 11.5 percentage-point spread for one model and no detectable difference for another.
for one model the spread between its best and worst harness was 11.5 percentage points
Source-stated
A same-model compliance reviewer reduced violations without raising over-refusals, contrary to the pre-registered prediction.
The same-model reviewer cut violations from 9.6% to 6.4% without raising over-refusals, contrary to our pre-registered prediction.
Source-stated
The authors report that transcript-only review would classify some violations as correct refusals when the customer message denied an action the agent had already executed.
26% of violations were accompanied by a customer message that denied the action the agent had just executed, which a transcript-only review would record as a correct refusal.
Source-stated
The authors recommend reading the action log, not only the conversation, when testing conduct.
conduct testing should read the action log, not the conversation.
Source and contributor context
Huyut, Mehmet Murat · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4a747d79-52b4-4ac1-805f-283859851db5.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T11:16:59.634Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 3d31a6f152c44c1663301bc15a9d369e8b4628b29fab7d05358165770b37a578. Source text hash: 120fde8eee871743e71bbd3300f8755167539d5f9a247c7193eee5a55cef07f5.
Revision and source-check history
Version 1 · 2026-11-09T11:16:59.634Z
arXiv cs.AI paper: Banking-rule compliance changes with the model’s harness
Mehmet Murat Huyut argues that banks should validate the deployed model-harness pair, not a language model in isolation. In a simulated retail bank, the study held each of four fixed while running it through five harnesses: systems that shape what a model sees, which tools it can use, and who checks its actions. Across 3,120 two-turn episodes, pooled correct-case rates shifted only from 87.3% to 92.1%, yet one model’s best-to-worst harness spread was 11.5 percentage points, enough to move it from first to last among the four; another showed no detectable difference. A same-model compliance reviewer reduced violations from 9.6% to 6.4% without increasing over-refusals, contrary to the pre-registered prediction. Rules requiring date arithmetic failed in every harness. In an exploratory analysis, 26% of violations included a customer message denying an action the agent had already executed; the authors say transcript-only review would record these as correct refusals. Their conclusion is to validate the model-harness pair and check the action log, not just the conversation. The study used simulated cases, and its model x harness interaction was suggestive rather than conclusive.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T11:16:59.634Z. First observed: 2026-11-05T05:00:36.084Z. Retrieved: 2026-11-09T11:10:00.066Z.
Limits
The cases were run in a simulated retail bank, not a live banking deployment. The authors describe the model x harness interaction as suggestive rather than conclusive. The finding about violations paired with customer messages came from an exploratory analysis using two judges and a blind human check.
Open question
Would the differences between harnesses persist in tests beyond the 156 pre-registered simulated-bank cases?
Source-stated
Huyut argues that model-risk management should validate the model-harness pair, because a harness can change a fixed model’s compliance with banking conduct rules.
For model-risk management the unit of validation should be the model-harness pair
Source-stated
The study tested four open-weight models in five harnesses on 156 pre-registered cases in a simulated retail bank.
Four open-weight models each ran five harnesses
Source-stated
Across models, correct-case rates varied from 87.3% to 92.1%; the harness effect differed by model, including an 11.5 percentage-point spread for one model and no detectable difference for another.
for one model the spread between its best and worst harness was 11.5 percentage points
Source-stated
A same-model compliance reviewer reduced violations without raising over-refusals, contrary to the pre-registered prediction.
The same-model reviewer cut violations from 9.6% to 6.4% without raising over-refusals, contrary to our pre-registered prediction.
Source-stated
The authors report that transcript-only review would classify some violations as correct refusals when the customer message denied an action the agent had already executed.
26% of violations were accompanied by a customer message that denied the action the agent had just executed, which a transcript-only review would record as a correct refusal.
Source-stated
The authors recommend reading the action log, not only the conversation, when testing conduct.
conduct testing should read the action log, not the conversation.
Source and contributor context
Huyut, Mehmet Murat · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4a747d79-52b4-4ac1-805f-283859851db5.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T11:16:59.634Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 3d31a6f152c44c1663301bc15a9d369e8b4628b29fab7d05358165770b37a578. Source text hash: 120fde8eee871743e71bbd3300f8755167539d5f9a247c7193eee5a55cef07f5.
uncertain · 2026-11-09T11:16:59.634Z · anthropic/claude-opus-5.5
Publication: 4a747d79-52b4-4ac1-805f-283859851db5
attribution — pass: The paper is correctly attributed to Mehmet Murat Huyut on arXiv. The draft's switch to 'the authors' is a minor plural slip for a single author, but every claim remains attributed to the paper rather than presented as an established fact.
Authors:Mehmet Murat Huyut
summary — pass: The summary leads with the author's argument and explains 'harness' in plain terms. It reports the key numbers, the reviewer result, the date-arithmetic failure, and the denial finding faithfully. Terms such as over-refusals and pre-registered are left unexplained but are not central blockers, so tooltips would help. The question follows from the evidence without being leading.
Banks validate language models, but they deploy systems: a model inside a harness that decides what it sees, which tools it can call and who checks its actions.
date — pass: originalPublished 2026-11-04 matches the v1 submission date. The feed date of Nov 5 is later and correctly not used.
[Submitted on 4 Nov 2026]
support — uncertain: Most claims are verbatim-supported. Two passages soften or blur the source. First, the source says conduct testing 'should read the action log, not the conversation', but the summary says 'not just the conversation' and the claim says 'not only'. That weakens the author's stance; the smallest fix is to drop 'just' and 'only'. Second, the claim 'Across models, correct-case rates varied from 87.3% to 92.1%' could be read as variation between models. The source describes the harness moving the pooled rate; repair it to 'Pooled across models, the harness moved...'.
conduct testing should read the action log, not the conversation.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition reference