# Thomas Davidson: Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas

[Canonical reading](https://escapemeridian.com/readings/thomas-davidson-humans-and-ai-respond-similarly-to-task-0a9ab97f)

Thomas Davidson reports an experiment comparing six large reasoning models with 1,854 people on content moderation decisions. The task varied how offensive two posts were relative to each other. Davidson says both people and models devoted more effort to cases where the two posts had equal offensiveness: people took longer to decide, while the models produced longer chain-of-thought (CoT). The relationship was weak for individual decisions but much stronger when cases were grouped by types of offensive language. In qualitative analysis, summaries of Gemini 2.5 Pro’s CoT mentioned secondary factors more often when offensiveness was equal. Davidson argues that reasoning traces could help identify difficult cases and make model reasoning more interpretable. The findings concern this experimental task; they do not establish that reasoning traces improve moderation in practice.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Aug 27, 2025 (2025-08-27).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T14:03:56.156Z. First observed: 2026-11-18T05:16:18.059Z. Retrieved: 2026-11-18T14:02:59.476Z.

- [Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas](https://arxiv.org/abs/2508.20262)

## Limits

The evidence is the abstract of an experimental content moderation task; it does not describe real-world moderation deployment or show that reasoning traces improve moderation outcomes. The reported correlation is low for individual decisions, limiting what the grouped relationship alone establishes about identifying a particular difficult case.


## Open question

Given the low correlation for individual decisions, how useful would reasoning traces be for prioritizing individual difficult cases?


## Source-stated

The study compared six large reasoning models with 1,854 human subjects on a content moderation task that varied the relative offensiveness of two options.

> This study addresses this issue through an experiment using a content moderation task, where task difficulty is manipulated by varying the relative offensiveness of two different options. Comparing six large reasoning models with 1,854 human subjects

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

When the two posts had equal offensiveness, people took longer to adjudicate and models produced longer chain-of-thought (CoT).

> People took 31\\% longer and models produced 51 to 306\\% longer CoTs when offensiveness was held constant across the two posts.

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

The association between human response time and model reasoning effort was low for individual decisions, but strong for tasks grouped by types of offensive language.

> The correlation is relatively low for individual decisions ($r = 0.15$--$0.19$), but average response times and average reasoning effort are closely correlated when tasks are grouped by the types of offensive language ($r = 0.85$--$0.93$).

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

The study presents reasoning traces as a potential way to prioritize difficult cases and enhance interpretability, rather than demonstrating those uses in practice.

> The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source and contributor context

Davidson, Thomas · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Reasoning model

A language model trained to work through a problem in written steps before it answers.

- [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model)

## Chain of thought

The step-by-step working a model writes out before giving its final answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#chain-of-thought-prompting)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: c8a49c03-3331-4459-8b20-0533d7dbd8e1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T14:03:56.156Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a42a24ca294c77c49b4d8f3053c975d55f24043932c8e0f5f49cad4227993a93. Source text hash: dbdc47eedd265bd2b7070f5a05ff21d97c1b787c79be2d41d0be5101ec1a5565.


## Revision and source-check history

### Version 1 — 2026-11-18T14:03:56.156Z

Thomas Davidson reports an experiment comparing six large reasoning models with 1,854 people on content moderation decisions. The task varied how offensive two posts were relative to each other. Davidson says both people and models devoted more effort to cases where the two posts had equal offensiveness: people took longer to decide, while the models produced longer chain-of-thought (CoT). The relationship was weak for individual decisions but much stronger when cases were grouped by types of offensive language. In qualitative analysis, summaries of Gemini 2.5 Pro’s CoT mentioned secondary factors more often when offensiveness was equal. Davidson argues that reasoning traces could help identify difficult cases and make model reasoning more interpretable. The findings concern this experimental task; they do not establish that reasoning traces improve moderation in practice.

## Original source

arXiv cs.AI · Article

Original source date: Aug 27, 2025 (2025-08-27).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T14:03:56.156Z. First observed: 2026-11-18T05:16:18.059Z. Retrieved: 2026-11-18T14:02:59.476Z.

- [Humans and AI respond similarly to task difficulty when reasoning about content moderation dilemmas](https://arxiv.org/abs/2508.20262)

## Limits

The evidence is the abstract of an experimental content moderation task; it does not describe real-world moderation deployment or show that reasoning traces improve moderation outcomes. The reported correlation is low for individual decisions, limiting what the grouped relationship alone establishes about identifying a particular difficult case.


## Open question

Given the low correlation for individual decisions, how useful would reasoning traces be for prioritizing individual difficult cases?


## Source-stated

The study compared six large reasoning models with 1,854 human subjects on a content moderation task that varied the relative offensiveness of two options.

> This study addresses this issue through an experiment using a content moderation task, where task difficulty is manipulated by varying the relative offensiveness of two different options. Comparing six large reasoning models with 1,854 human subjects

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

When the two posts had equal offensiveness, people took longer to adjudicate and models produced longer chain-of-thought (CoT).

> People took 31\\% longer and models produced 51 to 306\\% longer CoTs when offensiveness was held constant across the two posts.

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

The association between human response time and model reasoning effort was low for individual decisions, but strong for tasks grouped by types of offensive language.

> The correlation is relatively low for individual decisions ($r = 0.15$--$0.19$), but average response times and average reasoning effort are closely correlated when tasks are grouped by the types of offensive language ($r = 0.85$--$0.93$).

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source-stated

The study presents reasoning traces as a potential way to prioritize difficult cases and enhance interpretability, rather than demonstrating those uses in practice.

> The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.

- [Supporting source](https://arxiv.org/abs/2508.20262)

## Source and contributor context

Davidson, Thomas · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Reasoning model

A language model trained to work through a problem in written steps before it answers.

- [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model)

## Chain of thought

The step-by-step working a model writes out before giving its final answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#chain-of-thought-prompting)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: c8a49c03-3331-4459-8b20-0533d7dbd8e1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T14:03:56.156Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a42a24ca294c77c49b4d8f3053c975d55f24043932c8e0f5f49cad4227993a93. Source text hash: dbdc47eedd265bd2b7070f5a05ff21d97c1b787c79be2d41d0be5101ec1a5565.


### passed — 2026-11-18T14:03:56.156Z — anthropic/claude-opus-5.5

Publication: c8a49c03-3331-4459-8b20-0533d7dbd8e1

attribution — pass: The page lists Thomas Davidson as the sole author. The reading attributes the experiment, findings and suggested uses to him rather than presenting them as established fact.

> Authors:Thomas Davidson

summary — pass: The title matches the source. The summary accurately conveys the design, main finding, the weak individual versus strong grouped correlation, the Gemini 2.5 Pro analysis and the hedged suggested uses. The question follows from the evidence. Optional improvements: add a tooltip or brief gloss for 'large reasoning models' and 'chain-of-thought (CoT)', and note the paper's framing that this is a subjective task without verifiable answers. These are refinements, not fidelity failures.

> it is unclear whether these relationships hold on more subjective tasks without verifiable answers

date — pass: originalPublished is 2025-08-27, which matches the v1 submission date. Later revisions (v2 in Dec 2025, v3 in Nov 2026) and the 2026 feed date are correctly not used.

> \[Submitted on 27 Aug 2025 (v1), last revised 17 Nov 2026 (this version, v3)\]

support — pass: Each claim matches its verbatim excerpt. 'Equal offensiveness' fairly renders 'held constant across the two posts.' The suggested uses remain framed as potential, consistent with the source. The limitations and the question about individual-case prioritization are grounded in the reported low individual correlation and do not invent benefits or costs.

> The results point to the potential for using reasoning traces to prioritize difficult cases and enhance interpretability.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/0a9ab97f995e366c1dac1996aff817cc30da4b4b616e3daff91584d63ad913f4/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/thomas-davidson-humans-and-ai-respond-similarly-to-task-0a9ab97f.json)
