arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities

arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities

Authors Aaron Broukhim, Nadir Weibel and Eshin Jolly argue that preference evaluations designed for text should not simply be transferred to speech. In a controlled study, they compared human and synthetic preference annotations for text and audio carrying identical semantic content across 100 prompts. They report that reaching good agreement for audio and near good agreement for text required about 9 raters, while agreement between the modalities was near chance. They also report differences in how people judged responses: audio raters had narrower decision thresholds, less length bias and more user-oriented criteria. The authors say synthetic ratings predicted inter-rater agreement, which they propose as an early signal for selecting stimuli and as a proxy for human annotations. Their conclusion is that audio preference data needs evaluation protocols designed for its modality, rather than direct adaptation from text. The supplied abstract does not establish how well these findings generalize beyond the study.

Publication: 2aaf1621-1c78-4a25-86f9-6ad7d8f1e790

attribution — pass: The findings and conclusions are attributed to Aaron Broukhim, Nadir Weibel and Eshin Jolly, matching the arXiv listing. The title prefix 'arXiv cs.AI' matches the publisher metadata, although the paper's primary category is cs.SD (Sound). This is not a misattribution.

Authors:Aaron Broukhim, Nadir Weibel, Eshin Jolly

summary — uncertain: The summary is faithful but leaves out the context needed for a general reader. It never says that preference-based reinforcement learning (PbRL) is the source's 'dominant framework for aligning AI systems', so it is unclear why rater agreement matters. Central terms are unexplained: synthetic ratings, preference annotations, modalities, inter-rater agreement, ICC, length bias and decision thresholds. Smallest repair: add one opening sentence giving the PbRL alignment context and plain glosses or tooltips for these terms, keeping the terms exact.

Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences.

date — pass: originalPublished is 2026-02-26, which matches the v1 submission date. The 5 Nov 2026 date is a later revision (v5) and the feed date, not the first publication.

[Submitted on 26 Feb 2026 (v1), last revised 5 Nov 2026 (this version, v5)]

support — pass: Every claim matches the abstract and is presented as the authors' report. The roughly 9 raters, near-chance cross-modality agreement, audio rater differences, the synthetic-rating proxy and the modality-specific conclusion all appear in the source. The limitation and the question about synthetic-rating validity are appropriately open. No benefits are invented.

synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations.

media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most direct way to reach the source.