Kireyev, Pavel · Sep 25, 2026
arXiv cs.AI: RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs
Markdown · JSON · Original source
Pavel Kireyev introduces RAW, a method for estimating whether a refusal changes a user’s likelihood of returning to a deployed . The method treats variation across near-identical prompts—where the same model sometimes refuses and sometimes does not—as a natural experiment, using existing conversation logs rather than testing refusal directly. Kireyev says direct A/B-testing is ethically and operationally constrained for products and unavailable to third-party platforms using a fixed . Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports that, for about half the models it could measure precisely, refusal reduced subsequent re-engagement by 3 to 17 percentage points; the rest showed no distinguishable effect. One deployment’s positive point estimate is attributed to persistent users retrying the refused request. Kireyev presents the measurement method as his contribution, says it can run from logs alone for about $0.30 per 1,000 conversations, and reports releasing labels, moderation scores, prompt , and pipeline code. The abstract reports results from these datasets and models; it does not establish that the measured effects generalize to other deployments or settings.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Sep 25, 2026 (2026-09-25).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T16:33:00.546Z. First observed: 2026-11-18T05:04:07.505Z. Retrieved: 2026-11-18T16:31:21.566Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide detail here on how the estimates were validated. The reported findings concern the 15 deployed in LMSYS-Chat-1M and WildChat described in the abstract; generalization beyond those models and datasets is not established in the supplied text.
Open question
How well does RAW’s estimate of refusal’s effect on re-engagement hold across deployments beyond LMSYS-Chat-1M and WildChat?
Source-stated
Kireyev introduces RAW, which uses variation in whether a model refuses near-identical prompts to estimate refusal’s causal effect on subsequent user behavior from existing logs.
at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.
Source-stated
Kireyev says direct A/B-testing of refusal is constrained for frontier-lab products and unavailable to third-party platforms using a fixed API.
A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.
Source-stated
Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports a measurable reduction in subsequent re-engagement for about half of the models it could measure precisely; the others showed no distinguishable effect.
Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect
Source-stated
The paper attributes one deployment’s positive point estimate to persistent users retrying the refused request.
among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.
Source-stated
The paper says RAW runs from logs alone for about $0.30 per 1,000 conversations and reports releasing labels, moderation scores, prompt embeddings, and pipeline code.
Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.
Source and contributor context
Kireyev, Pavel · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
API
An application programming interface: a defined way for software to request data or actions from another system.
Embedding vector
A list of numbers that represents a piece of text, an image or another item so software can compare meanings.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:33:00.546Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 10b34b465164d6e22718e68a3f5b5cf14f4df2d5c3cd8a7d8c54147399812b84. Source text hash: 7447c0992193fde3abcedeb2d0b7ba8045027a3cf961c7e5bba85e448237589c.
Correction reason: Changed “The authors present” to “Kireyev presents” because the supplied source lists one author, correcting the attribution.
Revision and source-check history
Version 1 · 2026-11-18T16:33:00.546Z · Changed “The authors present” to “Kireyev presents” because the supplied source lists one author, correcting the attribution.
arXiv cs.AI: RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs
Pavel Kireyev introduces RAW, a method for estimating whether a refusal changes a user’s likelihood of returning to a deployed . The method treats variation across near-identical prompts—where the same model sometimes refuses and sometimes does not—as a natural experiment, using existing conversation logs rather than testing refusal directly. Kireyev says direct A/B-testing is ethically and operationally constrained for products and unavailable to third-party platforms using a fixed . Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports that, for about half the models it could measure precisely, refusal reduced subsequent re-engagement by 3 to 17 percentage points; the rest showed no distinguishable effect. One deployment’s positive point estimate is attributed to persistent users retrying the refused request. Kireyev presents the measurement method as his contribution, says it can run from logs alone for about $0.30 per 1,000 conversations, and reports releasing labels, moderation scores, prompt , and pipeline code. The abstract reports results from these datasets and models; it does not establish that the measured effects generalize to other deployments or settings.
Original source
cs.AI · Article
Source details
Original source date: Sep 25, 2026 (2026-09-25).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T16:33:00.546Z. First observed: 2026-11-18T05:04:07.505Z. Retrieved: 2026-11-18T16:31:21.566Z.
Limits
The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide detail here on how the estimates were validated. The reported findings concern the 15 deployed in LMSYS-Chat-1M and WildChat described in the abstract; generalization beyond those models and datasets is not established in the supplied text.
Open question
How well does RAW’s estimate of refusal’s effect on re-engagement hold across deployments beyond LMSYS-Chat-1M and WildChat?
Source-stated
Kireyev introduces RAW, which uses variation in whether a model refuses near-identical prompts to estimate refusal’s causal effect on subsequent user behavior from existing logs.
at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.
Source-stated
Kireyev says direct A/B-testing of refusal is constrained for frontier-lab products and unavailable to third-party platforms using a fixed API.
A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.
Source-stated
Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports a measurable reduction in subsequent re-engagement for about half of the models it could measure precisely; the others showed no distinguishable effect.
Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect
Source-stated
The paper attributes one deployment’s positive point estimate to persistent users retrying the refused request.
among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.
Source-stated
The paper says RAW runs from logs alone for about $0.30 per 1,000 conversations and reports releasing labels, moderation scores, prompt embeddings, and pipeline code.
Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.
Source and contributor context
Kireyev, Pavel · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
API
An application programming interface: a defined way for software to request data or actions from another system.
Embedding vector
A list of numbers that represents a piece of text, an image or another item so software can compare meanings.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:33:00.546Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 10b34b465164d6e22718e68a3f5b5cf14f4df2d5c3cd8a7d8c54147399812b84. Source text hash: 7447c0992193fde3abcedeb2d0b7ba8045027a3cf961c7e5bba85e448237589c.
Correction reason: Changed “The authors present” to “Kireyev presents” because the supplied source lists one author, correcting the attribution.
uncertain · 2026-11-18T16:33:00.546Z · anthropic/claude-opus-5.5
Publication: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6
attribution — pass: The source page lists one author, Pavel Kireyev, so attributing the method and findings to Kireyev or to 'the paper' is correct. Findings remain framed as the paper's reports rather than established facts.
Authors:Pavel Kireyev
summary — uncertain: The summary is faithful, but key terms that carry its meaning are left unexplained for a general reader: 'A/B-testing' (randomly showing users different versions), 'pp' (percentage points), 'positive point estimate' (refusal appearing to raise re-engagement), and 'LMSYS-Chat-1M and WildChat' (public chat-log datasets). It also opens by saying refusal changes a user's 'likelihood of returning', while the source only says 'subsequent re-engagement', which may not mean returning. The source's stated stakes are also dropped: it calls the unmeasured cost a blind spot for consumer-facing platforms. Smallest repair: briefly gloss these terms, say 're-engagement', and restore the blind-spot sentence.
is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them
date — pass: The originalPublished date of 2026-09-25 matches the arXiv v1 submission date. The 2026-11-18 feed date is correctly not used as the publication date.
[Submitted on 25 Sep 2026]
support — pass: Each claim is directly supported by verbatim abstract text: the method, the A/B-testing constraint, the 3 to 17 pp finding for about half of the precisely measured models, the retry explanation, and the cost and released artifacts. The limitations and the generalization question are appropriately hedged. No unsupported benefits or costs are introduced.
a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect
media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceAn application programming interface: a defined way for software to request data or actions from another system.
Definition referenceA list of numbers that represents a piece of text, an image or another item so software can compare meanings.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceAn application programming interface: a defined way for software to request data or actions from another system.
Definition referenceA list of numbers that represents a piece of text, an image or another item so software can compare meanings.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition reference