Davidson, Thomas · Aug 27, 2025
Thomas Davidson: Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas
Markdown · JSON · Original source
Thomas Davidson reports an experiment comparing six large with 1,854 people on content moderation decisions. The task varied how offensive two posts were relative to each other. Davidson says both people and models devoted more effort to cases where the two posts had equal offensiveness: people took longer to decide, while the models produced longer (CoT). The relationship was weak for individual decisions but much stronger when cases were grouped by types of offensive language. In qualitative analysis, summaries of Gemini 2.5 Pro’s CoT mentioned secondary factors more often when offensiveness was equal. Davidson argues that reasoning traces could help identify difficult cases and make model reasoning more interpretable. The findings concern this experimental task; they do not establish that reasoning traces improve moderation in practice.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Aug 27, 2025 (2025-08-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T14:03:56.156Z. First observed: 2026-11-18T05:16:18.059Z. Retrieved: 2026-11-18T14:02:59.476Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the abstract of an experimental content moderation task; it does not describe real-world moderation deployment or show that improve moderation outcomes. The reported correlation is low for individual decisions, limiting what the grouped relationship alone establishes about identifying a particular difficult case.
Open question
Given the low correlation for individual decisions, how useful would be for prioritizing individual difficult cases?
Source-stated
The study compared six large reasoning models with 1,854 human subjects on a content moderation task that varied the relative offensiveness of two options.
This study addresses this issue through an experiment using a content moderation task, where task difficulty is manipulated by varying the relative offensiveness of two different options. Comparing six large reasoning models with 1,854 human subjects
Source-stated
When the two posts had equal offensiveness, people took longer to adjudicate and models produced longer chain-of-thought (CoT).
People took 31\% longer and models produced 51 to 306\% longer CoTs when offensiveness was held constant across the two posts.
Source-stated
The association between human response time and model reasoning effort was low for individual decisions, but strong for tasks grouped by types of offensive language.
The correlation is relatively low for individual decisions ($r = 0.15$--$0.19$), but average response times and average reasoning effort are closely correlated when tasks are grouped by the types of offensive language ($r = 0.85$--$0.93$).
Source-stated
The study presents reasoning traces as a potential way to prioritize difficult cases and enhance interpretability, rather than demonstrating those uses in practice.
The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.
Source and contributor context
Davidson, Thomas · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reasoning model
A language model trained to work through a problem in written steps before it answers.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c8a49c03-3331-4459-8b20-0533d7dbd8e1.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T14:03:56.156Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a42a24ca294c77c49b4d8f3053c975d55f24043932c8e0f5f49cad4227993a93. Source text hash: dbdc47eedd265bd2b7070f5a05ff21d97c1b787c79be2d41d0be5101ec1a5565.
Revision and source-check history
Version 1 · 2026-11-18T14:03:56.156Z
Thomas Davidson: Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas
Thomas Davidson reports an experiment comparing six large with 1,854 people on content moderation decisions. The task varied how offensive two posts were relative to each other. Davidson says both people and models devoted more effort to cases where the two posts had equal offensiveness: people took longer to decide, while the models produced longer (CoT). The relationship was weak for individual decisions but much stronger when cases were grouped by types of offensive language. In qualitative analysis, summaries of Gemini 2.5 Pro’s CoT mentioned secondary factors more often when offensiveness was equal. Davidson argues that reasoning traces could help identify difficult cases and make model reasoning more interpretable. The findings concern this experimental task; they do not establish that reasoning traces improve moderation in practice.
Original source
cs.AI · Article
Source details
Original source date: Aug 27, 2025 (2025-08-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T14:03:56.156Z. First observed: 2026-11-18T05:16:18.059Z. Retrieved: 2026-11-18T14:02:59.476Z.
Limits
The evidence is the abstract of an experimental content moderation task; it does not describe real-world moderation deployment or show that improve moderation outcomes. The reported correlation is low for individual decisions, limiting what the grouped relationship alone establishes about identifying a particular difficult case.
Open question
Given the low correlation for individual decisions, how useful would be for prioritizing individual difficult cases?
Source-stated
The study compared six large reasoning models with 1,854 human subjects on a content moderation task that varied the relative offensiveness of two options.
This study addresses this issue through an experiment using a content moderation task, where task difficulty is manipulated by varying the relative offensiveness of two different options. Comparing six large reasoning models with 1,854 human subjects
Source-stated
When the two posts had equal offensiveness, people took longer to adjudicate and models produced longer chain-of-thought (CoT).
People took 31\% longer and models produced 51 to 306\% longer CoTs when offensiveness was held constant across the two posts.
Source-stated
The association between human response time and model reasoning effort was low for individual decisions, but strong for tasks grouped by types of offensive language.
The correlation is relatively low for individual decisions ($r = 0.15$--$0.19$), but average response times and average reasoning effort are closely correlated when tasks are grouped by the types of offensive language ($r = 0.85$--$0.93$).
Source-stated
The study presents reasoning traces as a potential way to prioritize difficult cases and enhance interpretability, rather than demonstrating those uses in practice.
The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.
Source and contributor context
Davidson, Thomas · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reasoning model
A language model trained to work through a problem in written steps before it answers.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c8a49c03-3331-4459-8b20-0533d7dbd8e1.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T14:03:56.156Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a42a24ca294c77c49b4d8f3053c975d55f24043932c8e0f5f49cad4227993a93. Source text hash: dbdc47eedd265bd2b7070f5a05ff21d97c1b787c79be2d41d0be5101ec1a5565.
passed · 2026-11-18T14:03:56.156Z · anthropic/claude-opus-5.5
Publication: c8a49c03-3331-4459-8b20-0533d7dbd8e1
attribution — pass: The page lists Thomas Davidson as the sole author. The reading attributes the experiment, findings and suggested uses to him rather than presenting them as established fact.
Authors:Thomas Davidson
summary — pass: The title matches the source. The summary accurately conveys the design, main finding, the weak individual versus strong grouped correlation, the Gemini 2.5 Pro analysis and the hedged suggested uses. The question follows from the evidence. Optional improvements: add a tooltip or brief gloss for 'large reasoning models' and 'chain-of-thought (CoT)', and note the paper's framing that this is a subjective task without verifiable answers. These are refinements, not fidelity failures.
it is unclear whether these relationships hold on more subjective tasks without verifiable answers
date — pass: originalPublished is 2025-08-27, which matches the v1 submission date. Later revisions (v2 in Dec 2025, v3 in Nov 2026) and the 2026 feed date are correctly not used.
[Submitted on 27 Aug 2025 (v1), last revised 17 Nov 2026 (this version, v3)]
support — pass: Each claim matches its verbatim excerpt. 'Equal offensiveness' fairly renders 'held constant across the two posts.' The suggested uses remain framed as potential, consistent with the source. The limitations and the question about individual-case prioritization are grounded in the reported low individual correlation and do not invent benefits or costs.
The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model trained to work through a problem in written steps before it answers.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivThe step-by-step working a model writes out before giving its final answer.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referenceA language model trained to work through a problem in written steps before it answers.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivThe step-by-step working a model writes out before giving its final answer.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition reference