# Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs — arXiv cs.AI [Canonical reading](https://escapemeridian.com/readings/bouba-beats-kiki-and-other-regularities-in-the-subjective-56c0f372) Katy Blumer, Jon Kleinberg, Ravi Kumar and Andrew Tomkins argue that evaluating a language model’s subjective choices needs methods different from checking whether it gives a factually correct answer. They study how models choose among short-form alternatives—numbers, baby names and fictional cities—across 29 aspects such as length, popularity and spoken sound. The authors report reproducible preferences for some aspects, but also find that how a preference is elicited can change which aspects affect a selection. Seemingly minor changes in a question can radically alter the overall distribution of pairwise preferences, which they say raises questions about manipulation. They also report unexpected interactions when a model both generates alternatives and ranks them. The paper’s point is not that a model’s preferences identify the best choice: it examines how those preferences vary when there is no clear winner. This matters for anyone using model rankings in subjective tasks, because the wording of the comparison and the creation of the options may shape the result. The abstract does not specify which question variations cause the largest shifts or how broadly the findings extend beyond the short-form examples studied. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 21, 2026 (2026-10-21). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T12:37:46.119Z. First observed: 2026-10-30T06:45:40.674Z. Retrieved: 2026-10-31T06:33:15.443Z. - [Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs](https://arxiv.org/abs/2610.37692) ## Limits The supplied evidence is the paper’s abstract, not its full methods or results; it does not detail which question variations cause the largest shifts. The study described here is restricted to short-form comparisons involving numbers, baby names and fictional cities, so the abstract does not establish how broadly the findings extend. ## Open question Which seemingly minor changes in how a preference question is posed produce the largest changes in models’ pairwise preference distributions? ## Source-stated The authors frame subjective evaluation as examining how models choose among viable responses when there is no clear winner, rather than measuring correctness. > we instead ask, how do models make choices among multiple viable responses without a clear winner? - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The study compares short-form numbers, baby names and fictional cities across 29 aspects. > consider comparison tasks among three such content types: numbers, baby names, and fictional cities. We identify 29 aspects - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report strong, reproducible model preferences for or against some aspects of the examples. > models tend to have strong and reproducible preferences for or against some of these aspects. - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report that changing how a preference is elicited may activate or deactivate aspects in a model’s selection. > variations in how the model is asked to express a preference may cause some of these aspects to activate or deactivate - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors say seemingly unimportant changes to a question can radically alter pairwise preference distributions, raising questions about manipulation. > overall preference distributions among pairs may vary radically based on seemingly-unimportant variations in the nature of the question posed to the model, raising new questions about manipulation. - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report unexpected interactions when models both create alternatives and rank them. > if models are used both to create lists of alternatives and to rank them, there are unexpected interactions between the two stages - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source and contributor context Blumer, Katy, Kleinberg, Jon, Kumar, Ravi, Tomkins, Andrew · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5b25c22b-a7d6-4242-8917-369057f88aca. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T12:37:46.119Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 28757653db3d9440f7bbfbcf55ce5c46041397934b6fff97d8587b1165533ed6. Source text hash: 1a79c07ea16920653fc067160a88f950a8470ede21746e44cf119e6342e8cf44. ## Revision and source-check history ### Version 1 — 2026-11-11T12:37:46.119Z Katy Blumer, Jon Kleinberg, Ravi Kumar and Andrew Tomkins argue that evaluating a language model’s subjective choices needs methods different from checking whether it gives a factually correct answer. They study how models choose among short-form alternatives—numbers, baby names and fictional cities—across 29 aspects such as length, popularity and spoken sound. The authors report reproducible preferences for some aspects, but also find that how a preference is elicited can change which aspects affect a selection. Seemingly minor changes in a question can radically alter the overall distribution of pairwise preferences, which they say raises questions about manipulation. They also report unexpected interactions when a model both generates alternatives and ranks them. The paper’s point is not that a model’s preferences identify the best choice: it examines how those preferences vary when there is no clear winner. This matters for anyone using model rankings in subjective tasks, because the wording of the comparison and the creation of the options may shape the result. The abstract does not specify which question variations cause the largest shifts or how broadly the findings extend beyond the short-form examples studied. ## Original source arXiv cs.AI · Article Original source date: Oct 21, 2026 (2026-10-21). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T12:37:46.119Z. First observed: 2026-10-30T06:45:40.674Z. Retrieved: 2026-10-31T06:33:15.443Z. - [Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs](https://arxiv.org/abs/2610.37692) ## Limits The supplied evidence is the paper’s abstract, not its full methods or results; it does not detail which question variations cause the largest shifts. The study described here is restricted to short-form comparisons involving numbers, baby names and fictional cities, so the abstract does not establish how broadly the findings extend. ## Open question Which seemingly minor changes in how a preference question is posed produce the largest changes in models’ pairwise preference distributions? ## Source-stated The authors frame subjective evaluation as examining how models choose among viable responses when there is no clear winner, rather than measuring correctness. > we instead ask, how do models make choices among multiple viable responses without a clear winner? - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The study compares short-form numbers, baby names and fictional cities across 29 aspects. > consider comparison tasks among three such content types: numbers, baby names, and fictional cities. We identify 29 aspects - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report strong, reproducible model preferences for or against some aspects of the examples. > models tend to have strong and reproducible preferences for or against some of these aspects. - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report that changing how a preference is elicited may activate or deactivate aspects in a model’s selection. > variations in how the model is asked to express a preference may cause some of these aspects to activate or deactivate - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors say seemingly unimportant changes to a question can radically alter pairwise preference distributions, raising questions about manipulation. > overall preference distributions among pairs may vary radically based on seemingly-unimportant variations in the nature of the question posed to the model, raising new questions about manipulation. - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source-stated The authors report unexpected interactions when models both create alternatives and rank them. > if models are used both to create lists of alternatives and to rank them, there are unexpected interactions between the two stages - [Supporting source](https://arxiv.org/abs/2610.37692) ## Source and contributor context Blumer, Katy, Kleinberg, Jon, Kumar, Ravi, Tomkins, Andrew · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5b25c22b-a7d6-4242-8917-369057f88aca. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T12:37:46.119Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 28757653db3d9440f7bbfbcf55ce5c46041397934b6fff97d8587b1165533ed6. Source text hash: 1a79c07ea16920653fc067160a88f950a8470ede21746e44cf119e6342e8cf44. ### passed — 2026-11-11T12:37:46.119Z — anthropic/claude-opus-5.5 Publication: 5b25c22b-a7d6-4242-8917-369057f88aca attribution — pass: The reading credits the four listed authors (Blumer, Kleinberg, Kumar, Tomkins) and keeps findings attributed with phrases like 'the authors report' and 'they say'. It does not present them as established fact. > Authors:Katy Blumer, Jon Kleinberg, Ravi Kumar, Andrew Tomkins summary — pass: The summary leads with the authors' argument that subjective evaluation needs different methods. It plainly lists all four findings, the content types and the 29 aspects, and keeps the hedged significance. A general reader can follow it. Minor note: the title's 'Bouba Beats Kiki' allusion is unexplained, but the abstract also does not define it, and it is not needed to understand the reading. The question follows from a gap the abstract leaves open. > how do models make choices among multiple viable responses without a clear winner? date — pass: originalPublished 2026-10-21 matches the arXiv v1 submission date. It is not the later feed date of 2026-10-31. > \[Submitted on 21 Oct 2026\] support — pass: Each claim is backed by a verbatim abstract excerpt. The impact statement is conditional ('may shape the result') and grounded in the abstract's findings on question wording and the generate-then-rank stages. One small drift: 'can radically alter' slightly firms up the source's 'may vary radically', but not materially. The limitations correctly note that the evidence is an abstract restricted to short-form content. > overall preference distributions among pairs may vary radically based on seemingly-unimportant variations in the nature of the question posed to the model media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source. No other media was found. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/56c0f3728cb90092d6110fbf76c6bab40dbc7b54d051b551fffa7e6497ee66fc/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/bouba-beats-kiki-and-other-regularities-in-the-subjective-56c0f372.json)