# arXiv cs.AI: RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-raw-refusal-aversion-in-the-wild-a-causal-9314c7c7)

Pavel Kireyev introduces RAW, a method for estimating whether a refusal changes a user’s likelihood of returning to a deployed LLM. The method treats variation across near-identical prompts—where the same model sometimes refuses and sometimes does not—as a natural experiment, using existing conversation logs rather than testing refusal directly. Kireyev says direct A/B-testing is ethically and operationally constrained for frontier-lab products and unavailable to third-party platforms using a fixed API. Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports that, for about half the models it could measure precisely, refusal reduced subsequent re-engagement by 3 to 17 percentage points; the rest showed no distinguishable effect. One deployment’s positive point estimate is attributed to persistent users retrying the refused request. Kireyev presents the measurement method as his contribution, says it can run from logs alone for about $0.30 per 1,000 conversations, and reports releasing labels, moderation scores, prompt embeddings, and pipeline code. The abstract reports results from these datasets and models; it does not establish that the measured effects generalize to other deployments or settings.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Sep 25, 2026 (2026-09-25).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T16:33:00.546Z. First observed: 2026-11-18T05:04:07.505Z. Retrieved: 2026-11-18T16:31:21.566Z.

- [RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs](https://arxiv.org/abs/2611.21641)

## Limits

The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide detail here on how the estimates were validated. The reported findings concern the 15 deployed LLMs in LMSYS-Chat-1M and WildChat described in the abstract; generalization beyond those models and datasets is not established in the supplied text.


## Open question

How well does RAW’s estimate of refusal’s effect on re-engagement hold across deployments beyond LMSYS-Chat-1M and WildChat?


## Source-stated

Kireyev introduces RAW, which uses variation in whether a model refuses near-identical prompts to estimate refusal’s causal effect on subsequent user behavior from existing logs.

> at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

Kireyev says direct A/B-testing of refusal is constrained for frontier-lab products and unavailable to third-party platforms using a fixed API.

> A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports a measurable reduction in subsequent re-engagement for about half of the models it could measure precisely; the others showed no distinguishable effect.

> Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

The paper attributes one deployment’s positive point estimate to persistent users retrying the refused request.

> among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

The paper says RAW runs from logs alone for about $0.30 per 1,000 conversations and reports releasing labels, moderation scores, prompt embeddings, and pipeline code.

> Our contribution is the measurement method, which runs on any deployed model from logs alone for about \\$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source and contributor context

Kireyev, Pavel · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## API

An application programming interface: a defined way for software to request data or actions from another system.

- [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API)

## Embedding vector

A list of numbers that represents a piece of text, an image or another item so software can compare meanings.

- [Definition reference](https://developers.google.com/machine-learning/glossary#embedding-vector)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:33:00.546Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 10b34b465164d6e22718e68a3f5b5cf14f4df2d5c3cd8a7d8c54147399812b84. Source text hash: 7447c0992193fde3abcedeb2d0b7ba8045027a3cf961c7e5bba85e448237589c.

Correction reason: Changed “The authors present” to “Kireyev presents” because the supplied source lists one author, correcting the attribution.


## Revision and source-check history

### Version 1 — 2026-11-18T16:33:00.546Z

Pavel Kireyev introduces RAW, a method for estimating whether a refusal changes a user’s likelihood of returning to a deployed LLM. The method treats variation across near-identical prompts—where the same model sometimes refuses and sometimes does not—as a natural experiment, using existing conversation logs rather than testing refusal directly. Kireyev says direct A/B-testing is ethically and operationally constrained for frontier-lab products and unavailable to third-party platforms using a fixed API. Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports that, for about half the models it could measure precisely, refusal reduced subsequent re-engagement by 3 to 17 percentage points; the rest showed no distinguishable effect. One deployment’s positive point estimate is attributed to persistent users retrying the refused request. Kireyev presents the measurement method as his contribution, says it can run from logs alone for about $0.30 per 1,000 conversations, and reports releasing labels, moderation scores, prompt embeddings, and pipeline code. The abstract reports results from these datasets and models; it does not establish that the measured effects generalize to other deployments or settings.

## Original source

arXiv cs.AI · Article

Original source date: Sep 25, 2026 (2026-09-25).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T16:33:00.546Z. First observed: 2026-11-18T05:04:07.505Z. Retrieved: 2026-11-18T16:31:21.566Z.

- [RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs](https://arxiv.org/abs/2611.21641)

## Limits

The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide detail here on how the estimates were validated. The reported findings concern the 15 deployed LLMs in LMSYS-Chat-1M and WildChat described in the abstract; generalization beyond those models and datasets is not established in the supplied text.


## Open question

How well does RAW’s estimate of refusal’s effect on re-engagement hold across deployments beyond LMSYS-Chat-1M and WildChat?


## Source-stated

Kireyev introduces RAW, which uses variation in whether a model refuses near-identical prompts to estimate refusal’s causal effect on subsequent user behavior from existing logs.

> at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

Kireyev says direct A/B-testing of refusal is constrained for frontier-lab products and unavailable to third-party platforms using a fixed API.

> A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

Applying RAW to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, the paper reports a measurable reduction in subsequent re-engagement for about half of the models it could measure precisely; the others showed no distinguishable effect.

> Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

The paper attributes one deployment’s positive point estimate to persistent users retrying the refused request.

> among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source-stated

The paper says RAW runs from logs alone for about $0.30 per 1,000 conversations and reports releasing labels, moderation scores, prompt embeddings, and pipeline code.

> Our contribution is the measurement method, which runs on any deployed model from logs alone for about \\$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.

- [Supporting source](https://arxiv.org/abs/2611.21641)

## Source and contributor context

Kireyev, Pavel · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## API

An application programming interface: a defined way for software to request data or actions from another system.

- [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API)

## Embedding vector

A list of numbers that represents a piece of text, an image or another item so software can compare meanings.

- [Definition reference](https://developers.google.com/machine-learning/glossary#embedding-vector)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:33:00.546Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 10b34b465164d6e22718e68a3f5b5cf14f4df2d5c3cd8a7d8c54147399812b84. Source text hash: 7447c0992193fde3abcedeb2d0b7ba8045027a3cf961c7e5bba85e448237589c.

Correction reason: Changed “The authors present” to “Kireyev presents” because the supplied source lists one author, correcting the attribution.


### uncertain — 2026-11-18T16:33:00.546Z — anthropic/claude-opus-5.5

Publication: 595f2e33-ed6b-4346-81e2-07cdaaed6fb6

attribution — pass: The source page lists one author, Pavel Kireyev, so attributing the method and findings to Kireyev or to 'the paper' is correct. Findings remain framed as the paper's reports rather than established facts.

> Authors:Pavel Kireyev

summary — uncertain: The summary is faithful, but key terms that carry its meaning are left unexplained for a general reader: 'A/B-testing' (randomly showing users different versions), 'pp' (percentage points), 'positive point estimate' (refusal appearing to raise re-engagement), and 'LMSYS-Chat-1M and WildChat' (public chat-log datasets). It also opens by saying refusal changes a user's 'likelihood of returning', while the source only says 'subsequent re-engagement', which may not mean returning. The source's stated stakes are also dropped: it calls the unmeasured cost a blind spot for consumer-facing platforms. Smallest repair: briefly gloss these terms, say 're-engagement', and restore the blind-spot sentence.

> is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them

date — pass: The originalPublished date of 2026-09-25 matches the arXiv v1 submission date. The 2026-11-18 feed date is correctly not used as the publication date.

> \[Submitted on 25 Sep 2026\]

support — pass: Each claim is directly supported by verbatim abstract text: the method, the A/B-testing constraint, the 3 to 17 pp finding for about half of the precisely measured models, the retry explanation, and the cost and released artifacts. The limitations and the generalization question are appropriately hedged. No unsupported benefits or costs are introduced.

> a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect

media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/9314c7c766dba35c68689f679b8cf5303ffac7f29b59d0bca8e86d67fbd8ae4a/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-raw-refusal-aversion-in-the-wild-a-causal-9314c7c7.json)
