# Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations [Canonical reading](https://escapemeridian.com/readings/imperfectly-imperfect-evaluating-graded-cognitive-diversity-in-1bf136d6) Authors Sion Weatherhead, Ben Newell, Aaron Belbasis and Flora Salim argue that LLM agents can reproduce some human trait distributions in questionnaires without reproducing the differences in reasoning and behaviour those traits are meant to represent. They compare simulated agents conditioned on synthetic or human-derived profiles with a cohort of 473 adults, using cognitive tests, trait questionnaires and decision scenarios. Across seven simulation runs of four models, agents matched Need for Cognition and Big Five distributions on self-report questionnaires, but often performed unlike people on the cognitive tasks. In four Claude Sonnet 4.6 runs, every agent earned full marks on both CRTs regardless of assigned reflection level; in the other runs, full-mark rates still far exceeded the human sample. The BRT scenarios also showed overproduction of many coded behaviours, while links between reflection and behaviour carried over only in part. The authors’ point is that assigning a persona or matching its questionnaire scores is not enough: a useful synthetic population must also preserve the structured variation in reasoning, errors and behaviour associated with those traits. These results describe the tested models and tasks; they do not establish how well other simulations or research settings will perform. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T11:46:57.286Z. First observed: 2026-11-10T08:39:26.779Z. Retrieved: 2026-11-12T11:34:10.431Z. - [Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations](https://arxiv.org/abs/2611.10508) ## Limits The abstract reports results from seven full-battery simulation runs of four models; it does not identify all four models by name. The abstract describes performance on the stated tests and scenarios; it does not establish how the results generalize to other research settings. ## Open question What approach would allow simulated populations to reproduce the human variation in reasoning, error type and behaviour that the authors say their trait profiles are meant to organise? ## Source-stated The authors evaluate whether simulated populations reproduce human variation in reasoning, traits, associations and downstream behaviour. > tests whether simulated populations reproduce human variation in reasoning, self-reported traits, their associations, and downstream behaviour. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated Across seven simulation runs of four models, agents recovered Need for Cognition and Big Five distributions on questionnaires but fell short on cognitive-task performance and downstream behaviours. > Across seven full-battery simulation runs of four models, agents recover Need for Cognition and Big Five trait distributions on self-report questionnaires, but fall short on simulating diversity of cognitive task performance and downstream behaviours. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated In the four Claude Sonnet 4.6 runs, every agent scored full marks on both CRTs regardless of assigned reflection level. > In the four Claude Sonnet 4.6 runs, every agent scores full marks on both CRTs, whatever reflection level its persona was assigned. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated Across the BRT scenarios, every run over-produced many coded behaviours, and human links between reflection and behaviour carried over only in part. > In the BRT scenarios, every run over-produces 8 to 11 of 15 coded behaviours by at least 20 percentage points, and people's links between reflection and behaviour carry over only in part. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated The authors argue that useful synthetic populations must preserve structured variation in reasoning, errors and behaviour, not only trait descriptors. > Useful synthetic populations must preserve not only trait descriptors but the structured variation in reasoning, error type, and behaviour those descriptors are meant to organise. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source and contributor context Weatherhead, Sion, Newell, Ben, Belbasis, Aaron, Salim, Flora · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: fef6eea9-f02f-4206-8fe5-89b14e0c761b. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T11:46:57.286Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 718d1333fa8d938281c0bfd86e721cffa6e88861801b4d980c488713c301f5c9. Source text hash: 43ccf80c6622cd9e5aaf66b1fed1d958ddc3014747140c6e8ed8308ec57bd170. ## Revision and source-check history ### Version 1 — 2026-11-12T11:46:57.286Z Authors Sion Weatherhead, Ben Newell, Aaron Belbasis and Flora Salim argue that LLM agents can reproduce some human trait distributions in questionnaires without reproducing the differences in reasoning and behaviour those traits are meant to represent. They compare simulated agents conditioned on synthetic or human-derived profiles with a cohort of 473 adults, using cognitive tests, trait questionnaires and decision scenarios. Across seven simulation runs of four models, agents matched Need for Cognition and Big Five distributions on self-report questionnaires, but often performed unlike people on the cognitive tasks. In four Claude Sonnet 4.6 runs, every agent earned full marks on both CRTs regardless of assigned reflection level; in the other runs, full-mark rates still far exceeded the human sample. The BRT scenarios also showed overproduction of many coded behaviours, while links between reflection and behaviour carried over only in part. The authors’ point is that assigning a persona or matching its questionnaire scores is not enough: a useful synthetic population must also preserve the structured variation in reasoning, errors and behaviour associated with those traits. These results describe the tested models and tasks; they do not establish how well other simulations or research settings will perform. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T11:46:57.286Z. First observed: 2026-11-10T08:39:26.779Z. Retrieved: 2026-11-12T11:34:10.431Z. - [Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations](https://arxiv.org/abs/2611.10508) ## Limits The abstract reports results from seven full-battery simulation runs of four models; it does not identify all four models by name. The abstract describes performance on the stated tests and scenarios; it does not establish how the results generalize to other research settings. ## Open question What approach would allow simulated populations to reproduce the human variation in reasoning, error type and behaviour that the authors say their trait profiles are meant to organise? ## Source-stated The authors evaluate whether simulated populations reproduce human variation in reasoning, traits, associations and downstream behaviour. > tests whether simulated populations reproduce human variation in reasoning, self-reported traits, their associations, and downstream behaviour. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated Across seven simulation runs of four models, agents recovered Need for Cognition and Big Five distributions on questionnaires but fell short on cognitive-task performance and downstream behaviours. > Across seven full-battery simulation runs of four models, agents recover Need for Cognition and Big Five trait distributions on self-report questionnaires, but fall short on simulating diversity of cognitive task performance and downstream behaviours. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated In the four Claude Sonnet 4.6 runs, every agent scored full marks on both CRTs regardless of assigned reflection level. > In the four Claude Sonnet 4.6 runs, every agent scores full marks on both CRTs, whatever reflection level its persona was assigned. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated Across the BRT scenarios, every run over-produced many coded behaviours, and human links between reflection and behaviour carried over only in part. > In the BRT scenarios, every run over-produces 8 to 11 of 15 coded behaviours by at least 20 percentage points, and people's links between reflection and behaviour carry over only in part. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source-stated The authors argue that useful synthetic populations must preserve structured variation in reasoning, errors and behaviour, not only trait descriptors. > Useful synthetic populations must preserve not only trait descriptors but the structured variation in reasoning, error type, and behaviour those descriptors are meant to organise. - [Supporting source](https://arxiv.org/abs/2611.10508) ## Source and contributor context Weatherhead, Sion, Newell, Ben, Belbasis, Aaron, Salim, Flora · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: fef6eea9-f02f-4206-8fe5-89b14e0c761b. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T11:46:57.286Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 718d1333fa8d938281c0bfd86e721cffa6e88861801b4d980c488713c301f5c9. Source text hash: 43ccf80c6622cd9e5aaf66b1fed1d958ddc3014747140c6e8ed8308ec57bd170. ### uncertain — 2026-11-12T11:46:57.286Z — anthropic/claude-opus-5.5 Publication: fef6eea9-f02f-4206-8fe5-89b14e0c761b attribution — pass: The reading correctly attributes the paper and its argument to Sion Weatherhead, Ben Newell, Aaron Belbasis and Flora Salim. It frames the conclusion as the authors' point, not as established fact. > Authors:Sion Weatherhead, Ben Newell, Aaron Belbasis, Flora Salim summary — uncertain: The summary is faithful and keeps the authors' central thesis. However, it uses 'CRTs' and 'BRT scenarios' without defining them (Cognitive Reflection Test; Behavioural Reflection Test). It also does not explain 'Need for Cognition', 'Big Five' or what the reflection levels mean. A non-specialist cannot tell what full marks or overproduced behaviours signify. The smallest repair is to spell out CRT and BRT and gloss these terms with tooltips. The reading also omits the 'intuitive-lure' explanation and the detail that Claude Sonnet 5 came closest to the human lure pattern, which makes the full-mark finding harder to interpret. > a Cognitive Reflection Test (CRT) and a Behavioural Reflection Test (BRT) whose decision scenarios measure reflection in behaviour date — pass: originalPublished is 2026-11-07, which matches the v1 submission date. The later feed date of 2026-11-10 is correctly not used. > \[Submitted on 7 Nov 2026\] support — pass: The claims and excerpts match the abstract, including the 473-adult cohort, the seven runs of four models, the Sonnet 4.6 full marks and the BRT overproduction. 'Far exceeded the human sample' is supported by the 49 to 85% versus under 10% figures. The limitations and the generalization caveat are appropriately cautious. The question follows from the authors' stated thesis. > but 49 to 85% score full marks, against under 10% in the human sample media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/1bf136d6d812e1f8483beb925af6ec2894a55984b7f5af170197a98cbbb97a4d/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/imperfectly-imperfect-evaluating-graded-cognitive-diversity-in-1bf136d6.json)