# arXiv cs.AI: AI Isn't Getting Dumber, We're Getting Pickier [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-ai-isn-t-getting-dumber-we-re-getting-pickier-a1a291dd) Authors Zihan Chen, Lei Nico Zheng and Di Zhu argue that complaints about AI assistants getting worse may reflect changing human standards, not just changes to the models. They examine blind votes on Chatbot Arena, where people compare answers from two anonymous large language models (LLMs), using pairs whose model versions were fixed. For the same unchanged pairs, votes saying “both are bad” increased, while the stronger model’s share of decisive votes stayed flat. The authors report that the pattern holds within the same voter and model pair, across language and model types, and with prompt and interface changes accounted for. They interpret this split—worsening absolute verdicts but stable relative comparisons—as an “evaluation drift”: as the category improves, people may judge a product that stands still more harshly. That distinction matters, they argue, for reading LLM leaderboards and for human feedback used to train models. The evidence comes from 1,260,967 votes on Chatbot Arena between May 2023 and August 2024; the paper describes itself as a working paper. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T01:47:48.213Z. First observed: 2026-11-11T05:06:35.111Z. Retrieved: 2026-11-12T01:47:02.164Z. - [AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models](https://arxiv.org/abs/2611.14333) ## Limits The evidence described here is from Chatbot Arena votes cast between May 2023 and August 2024, not other evaluation settings. The study compares pairs whose model versions were fixed; the abstract does not establish how the findings apply to changing model versions. The source identifies the paper as a working paper. ## Open question Would the rise in “both are bad” votes also appear in human evaluations beyond the Chatbot Arena votes studied here? ## Source-stated The authors study blind human votes on Chatbot Arena for pairs of models whose versions were fixed. > We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated For unchanged model pairs, the share of votes saying “both are bad” rose over the study period. > For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The stronger model’s share of decisive votes did not trend in 2024, even as “both are bad” votes rose. > The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The authors interpret the pattern as evaluation drift: standards may rise with progress in the category, making a stationary product seem to decline. > The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The authors say this evaluation drift matters for LLM leaderboards and human feedback used to train models. > an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source and contributor context Chen, Zihan, Zheng, Lei Nico, Zhu, Di · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 0190af6e-0964-4778-8a1c-8eb0cefd21a9. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T01:47:48.213Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 630a1fff1adf714461c930d34a644fa6ee07990a5c0fbd8afd644b1764eaeb99. Source text hash: e3f02d3074e141ded1d8f32717f8975ecdee864b49ae60489e89860e0bb2877b. ## Revision and source-check history ### Version 1 — 2026-11-12T01:47:48.213Z Authors Zihan Chen, Lei Nico Zheng and Di Zhu argue that complaints about AI assistants getting worse may reflect changing human standards, not just changes to the models. They examine blind votes on Chatbot Arena, where people compare answers from two anonymous large language models (LLMs), using pairs whose model versions were fixed. For the same unchanged pairs, votes saying “both are bad” increased, while the stronger model’s share of decisive votes stayed flat. The authors report that the pattern holds within the same voter and model pair, across language and model types, and with prompt and interface changes accounted for. They interpret this split—worsening absolute verdicts but stable relative comparisons—as an “evaluation drift”: as the category improves, people may judge a product that stands still more harshly. That distinction matters, they argue, for reading LLM leaderboards and for human feedback used to train models. The evidence comes from 1,260,967 votes on Chatbot Arena between May 2023 and August 2024; the paper describes itself as a working paper. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T01:47:48.213Z. First observed: 2026-11-11T05:06:35.111Z. Retrieved: 2026-11-12T01:47:02.164Z. - [AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models](https://arxiv.org/abs/2611.14333) ## Limits The evidence described here is from Chatbot Arena votes cast between May 2023 and August 2024, not other evaluation settings. The study compares pairs whose model versions were fixed; the abstract does not establish how the findings apply to changing model versions. The source identifies the paper as a working paper. ## Open question Would the rise in “both are bad” votes also appear in human evaluations beyond the Chatbot Arena votes studied here? ## Source-stated The authors study blind human votes on Chatbot Arena for pairs of models whose versions were fixed. > We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated For unchanged model pairs, the share of votes saying “both are bad” rose over the study period. > For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The stronger model’s share of decisive votes did not trend in 2024, even as “both are bad” votes rose. > The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The authors interpret the pattern as evaluation drift: standards may rise with progress in the category, making a stationary product seem to decline. > The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source-stated The authors say this evaluation drift matters for LLM leaderboards and human feedback used to train models. > an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models. - [Supporting source](https://arxiv.org/abs/2611.14333) ## Source and contributor context Chen, Zihan, Zheng, Lei Nico, Zhu, Di · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 0190af6e-0964-4778-8a1c-8eb0cefd21a9. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T01:47:48.213Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 630a1fff1adf714461c930d34a644fa6ee07990a5c0fbd8afd644b1764eaeb99. Source text hash: e3f02d3074e141ded1d8f32717f8975ecdee864b49ae60489e89860e0bb2877b. ### passed — 2026-11-12T01:47:48.213Z — anthropic/claude-opus-5.5 Publication: 0190af6e-0964-4778-8a1c-8eb0cefd21a9 attribution — pass: The reading correctly credits Zihan Chen, Lei Nico Zheng and Di Zhu with the paper on arXiv. It keeps the findings and the evaluation-drift interpretation attributed to the authors, using phrases like 'the authors report' and 'they argue'. > Authors:Zihan Chen, Lei Nico Zheng, Di Zhu summary — pass: The reading leads with the authors' thesis and explains Chatbot Arena and LLMs in plain terms. It conveys the central split between rising absolute complaints and stable relative rankings, and keeps the hedged 'may' framing. The term 'decisive votes' is understandable in context. The title and question are faithful, and the question raises a genuine unresolved issue about generalization. > Absolute verdicts on AI drifted while relative comparisons held steady date — pass: originalPublished is 2026-11-10, which matches the arXiv v1 submission date. The later feed entry is dated 2026-11-11, but the earlier submission date is the correct original publication date. > \[Submitted on 10 Nov 2026\] support — pass: Each claim is directly supported by the abstract. The 'flat' ranking claim correctly keeps the 2024 scope. Robustness claims about voters, pairs, languages, model types, prompt controls and interface periods are stated in the source. The significance for leaderboards and training feedback is attributed to the authors. The limitations are accurate, and no unsupported benefits or costs are added. > It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. media — pass: The selected media item is the verified direct link to the arXiv abstract page for this paper. It is the most direct way for a reader to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/a1a291dd64bc087cb5075a27b8f7588775b49ec025e8ff00b4298822d33c8db9/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-ai-isn-t-getting-dumber-we-re-getting-pickier-a1a291dd.json)