{"id":"19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5","title":"arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities","argument":"Authors Aaron Broukhim, Nadir Weibel and Eshin Jolly argue that preference evaluations designed for text should not simply be transferred to speech. In a controlled study, they compared human and synthetic preference annotations for text and audio carrying identical semantic content across 100 prompts. They report that reaching good agreement for audio and near good agreement for text required about 9 raters, while agreement between the modalities was near chance. They also report differences in how people judged responses: audio raters had narrower decision thresholds, less length bias and more user-oriented criteria. The authors say synthetic ratings predicted inter-rater agreement, which they propose as an early signal for selecting stimuli and as a proxy for human annotations. Their conclusion is that audio preference data needs evaluation protocols designed for its modality, rather than direct adaptation from text. The supplied abstract does not establish how well these findings generalize beyond the study.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract and does not detail the study methods or how well the reported findings generalize beyond the 100 prompts.","speaker":{"name":"Broukhim, Aaron, Weibel, Nadir, Jolly, Eshin","roleAtSource":"Article author"},"source":{"title":"Same Words, Different Judgments: How Preferences Vary Across Modalities","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2602.22710","publishedAt":"2026-02-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that evaluation protocols for audio preference data need modality-specific design rather than direct adaptation from text.","excerpt":"these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The study compared human and synthetic preference annotations for text and audio with identical semantic content across 100 prompts.","excerpt":"comparing text and audio evaluations of identical semantic content across 100 prompts.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors report that approximately 9 raters were needed for good agreement in audio and near good agreement in text.","excerpt":"achieving \\textit{good} (audio) to \\textit{near good} (text) agreement within either modality (ICC(2,$k$) $\\approx$ .81 audio, .74 text) requires $\\sim$9 raters","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors report near-chance agreement between modalities and differences in audio raters’ decision thresholds, length bias and evaluation criteria.","excerpt":"audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors say synthetic ratings can predict inter-rater agreement and may serve as an early signal for stimulus selection and a proxy for human annotations.","excerpt":"synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2602.22710","checkedAt":"2026-11-09T13:48:10.612Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do synthetic ratings predict inter-rater agreement when used to select stimuli or as a proxy for human annotations?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities","summary":"Authors Aaron Broukhim, Nadir Weibel and Eshin Jolly argue that preference evaluations designed for text should not simply be transferred to speech. In a controlled study, they compared human and synthetic preference annotations for text and audio carrying identical semantic content across 100 prompts. They report that reaching good agreement for audio and near good agreement for text required about 9 raters, while agreement between the modalities was near chance. They also report differences in how people judged responses: audio raters had narrower decision thresholds, less length bias and more user-oriented criteria. The authors say synthetic ratings predicted inter-rater agreement, which they propose as an early signal for selecting stimuli and as a proxy for human annotations. Their conclusion is that audio preference data needs evaluation protocols designed for its modality, rather than direct adaptation from text. The supplied abstract does not establish how well these findings generalize beyond the study.","question":"How well do synthetic ratings predict inter-rater agreement when used to select stimuli or as a proxy for human annotations?","claims":[{"text":"The authors argue that evaluation protocols for audio preference data need modality-specific design rather than direct adaptation from text.","excerpt":"these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.","support":"source-stated"},{"text":"The study compared human and synthetic preference annotations for text and audio with identical semantic content across 100 prompts.","excerpt":"comparing text and audio evaluations of identical semantic content across 100 prompts.","support":"source-stated"},{"text":"The authors report that approximately 9 raters were needed for good agreement in audio and near good agreement in text.","excerpt":"achieving \\textit{good} (audio) to \\textit{near good} (text) agreement within either modality (ICC(2,$k$) $\\approx$ .81 audio, .74 text) requires $\\sim$9 raters","support":"source-stated"},{"text":"The authors report near-chance agreement between modalities and differences in audio raters’ decision thresholds, length bias and evaluation criteria.","excerpt":"audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement.","support":"source-stated"},{"text":"The authors say synthetic ratings can predict inter-rater agreement and may serve as an early signal for stimulus selection and a proxy for human annotations.","excerpt":"synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations.","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract and does not detail the study methods or how well the reported findings generalize beyond the 100 prompts."],"mediaId":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","correctionReason":null,"id":"19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5","publicationId":"2aaf1621-1c78-4a25-86f9-6ad7d8f1e790","version":1,"sourceUrl":"https://arxiv.org/abs/2602.22710","sourceTitle":"Same Words, Different Judgments: How Preferences Vary Across Modalities","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-02-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:00:36.504Z","createdAt":"2026-11-09T13:17:10.042Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","kind":"article","url":"https://arxiv.org/abs/2602.22710","sourceUrl":"https://arxiv.org/abs/2602.22710","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T13:08:33.468Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2602.22710","checkedAt":"2026-11-09T13:48:10.612Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","confidence":0.74,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e70811bee6a635fb830b18e013b8521c6ebda965c5f98ca9d56b05fec256163a","decidedAt":"2026-11-09T13:08:35.278Z"},"provenance":{"textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observationId":"3fedca936cb7f2d97e8a31e83e1f8b41af49b814ce944d914d002cc5137476d6","retrievedAt":"2026-11-09T13:08:33.149Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7af8007d-3407-4c68-89a2-ff488546b012"},"reviewer":{"model":"gpt-6-luna","runId":"58215f09-8868-4e30-8260-feefd768997b"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-09T13:17:10.042Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Broukhim, Aaron","url":null,"kind":"unknown"},{"name":"Weibel, Nadir","url":null,"kind":"unknown"},{"name":"Jolly, Eshin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Broukhim, Aaron","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Broukhim, Aaron","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}},{"actor":{"name":"Weibel, Nadir","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weibel, Nadir","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}},{"actor":{"name":"Jolly, Eshin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jolly, Eshin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2602.22710","snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-same-words-different-judgments-how-preferences-vary-19cd0f66","formats":{"html":"/readings/arxiv-cs-ai-same-words-different-judgments-how-preferences-vary-19cd0f66","markdown":"/readings/arxiv-cs-ai-same-words-different-judgments-how-preferences-vary-19cd0f66.md","json":"/readings/arxiv-cs-ai-same-words-different-judgments-how-preferences-vary-19cd0f66.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5","revision":"2aaf1621-1c78-4a25-86f9-6ad7d8f1e790","revisionKind":"publication-id","title":"arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities","url":"https://escapemeridian.com/readings/arxiv-cs-ai-same-words-different-judgments-how-preferences-vary-19cd0f66"},"source":{"type":"source","id":"https://arxiv.org/abs/2602.22710","revision":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","revisionKind":"sha256-source-text","title":"Same Words, Different Judgments: How Preferences Vary Across Modalities","url":"https://arxiv.org/abs/2602.22710"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5","title":"arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities","argument":"Authors Aaron Broukhim, Nadir Weibel and Eshin Jolly argue that preference evaluations designed for text should not simply be transferred to speech. In a controlled study, they compared human and synthetic preference annotations for text and audio carrying identical semantic content across 100 prompts. They report that reaching good agreement for audio and near good agreement for text required about 9 raters, while agreement between the modalities was near chance. They also report differences in how people judged responses: audio raters had narrower decision thresholds, less length bias and more user-oriented criteria. The authors say synthetic ratings predicted inter-rater agreement, which they propose as an early signal for selecting stimuli and as a proxy for human annotations. Their conclusion is that audio preference data needs evaluation protocols designed for its modality, rather than direct adaptation from text. The supplied abstract does not establish how well these findings generalize beyond the study.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract and does not detail the study methods or how well the reported findings generalize beyond the 100 prompts.","speaker":{"name":"Broukhim, Aaron, Weibel, Nadir, Jolly, Eshin","roleAtSource":"Article author"},"source":{"title":"Same Words, Different Judgments: How Preferences Vary Across Modalities","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2602.22710","publishedAt":"2026-02-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that evaluation protocols for audio preference data need modality-specific design rather than direct adaptation from text.","excerpt":"these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The study compared human and synthetic preference annotations for text and audio with identical semantic content across 100 prompts.","excerpt":"comparing text and audio evaluations of identical semantic content across 100 prompts.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors report that approximately 9 raters were needed for good agreement in audio and near good agreement in text.","excerpt":"achieving \\textit{good} (audio) to \\textit{near good} (text) agreement within either modality (ICC(2,$k$) $\\approx$ .81 audio, .74 text) requires $\\sim$9 raters","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors report near-chance agreement between modalities and differences in audio raters’ decision thresholds, length bias and evaluation criteria.","excerpt":"audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"},{"text":"The authors say synthetic ratings can predict inter-rater agreement and may serve as an early signal for stimulus selection and a proxy for human annotations.","excerpt":"synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations.","url":"https://arxiv.org/abs/2602.22710","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2602.22710","checkedAt":"2026-11-09T13:48:10.612Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do synthetic ratings predict inter-rater agreement when used to select stimuli or as a proxy for human annotations?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Same Words, Different Judgments: How Preferences Vary Across Modalities","summary":"Authors Aaron Broukhim, Nadir Weibel and Eshin Jolly argue that preference evaluations designed for text should not simply be transferred to speech. In a controlled study, they compared human and synthetic preference annotations for text and audio carrying identical semantic content across 100 prompts. They report that reaching good agreement for audio and near good agreement for text required about 9 raters, while agreement between the modalities was near chance. They also report differences in how people judged responses: audio raters had narrower decision thresholds, less length bias and more user-oriented criteria. The authors say synthetic ratings predicted inter-rater agreement, which they propose as an early signal for selecting stimuli and as a proxy for human annotations. Their conclusion is that audio preference data needs evaluation protocols designed for its modality, rather than direct adaptation from text. The supplied abstract does not establish how well these findings generalize beyond the study.","question":"How well do synthetic ratings predict inter-rater agreement when used to select stimuli or as a proxy for human annotations?","claims":[{"text":"The authors argue that evaluation protocols for audio preference data need modality-specific design rather than direct adaptation from text.","excerpt":"these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.","support":"source-stated"},{"text":"The study compared human and synthetic preference annotations for text and audio with identical semantic content across 100 prompts.","excerpt":"comparing text and audio evaluations of identical semantic content across 100 prompts.","support":"source-stated"},{"text":"The authors report that approximately 9 raters were needed for good agreement in audio and near good agreement in text.","excerpt":"achieving \\textit{good} (audio) to \\textit{near good} (text) agreement within either modality (ICC(2,$k$) $\\approx$ .81 audio, .74 text) requires $\\sim$9 raters","support":"source-stated"},{"text":"The authors report near-chance agreement between modalities and differences in audio raters’ decision thresholds, length bias and evaluation criteria.","excerpt":"audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement.","support":"source-stated"},{"text":"The authors say synthetic ratings can predict inter-rater agreement and may serve as an early signal for stimulus selection and a proxy for human annotations.","excerpt":"synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations.","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract and does not detail the study methods or how well the reported findings generalize beyond the 100 prompts."],"mediaId":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","correctionReason":null,"id":"19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5","publicationId":"2aaf1621-1c78-4a25-86f9-6ad7d8f1e790","version":1,"sourceUrl":"https://arxiv.org/abs/2602.22710","sourceTitle":"Same Words, Different Judgments: How Preferences Vary Across Modalities","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-02-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:00:36.504Z","createdAt":"2026-11-09T13:17:10.042Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","kind":"article","url":"https://arxiv.org/abs/2602.22710","sourceUrl":"https://arxiv.org/abs/2602.22710","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T13:08:33.468Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2602.22710","checkedAt":"2026-11-09T13:48:10.612Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"dda7224312d5979c68fc7d439986fa357077f2c8c99d3688b4a11fe31ee767bf","confidence":0.74,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e70811bee6a635fb830b18e013b8521c6ebda965c5f98ca9d56b05fec256163a","decidedAt":"2026-11-09T13:08:35.278Z"},"provenance":{"textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observationId":"3fedca936cb7f2d97e8a31e83e1f8b41af49b814ce944d914d002cc5137476d6","retrievedAt":"2026-11-09T13:08:33.149Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7af8007d-3407-4c68-89a2-ff488546b012"},"reviewer":{"model":"gpt-6-luna","runId":"58215f09-8868-4e30-8260-feefd768997b"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-09T13:17:10.042Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Broukhim, Aaron","url":null,"kind":"unknown"},{"name":"Weibel, Nadir","url":null,"kind":"unknown"},{"name":"Jolly, Eshin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Broukhim, Aaron","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Broukhim, Aaron","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}},{"actor":{"name":"Weibel, Nadir","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weibel, Nadir","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}},{"actor":{"name":"Jolly, Eshin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jolly, Eshin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2602.22710","locator":"head/meta[name=author|citation_author]"},"assertionId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T13:48:10.612Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","sourceUrl":"https://arxiv.org/abs/2602.22710","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2602.22710","snapshotId":"8cd0821bd186d4445b49407cccba54368097c7b8dd98ce18bacee5195d39a853","contentSha256":"9b2c405e0aa33a07fd6fa26bbb0306dab00d5b3c148d55de77ee31c20c70a47a","textHash":"cdf0d7a6b4de64527f699b1c46f57dd6b018cff7ce0bb1c87070bcc022d65aba","observedAt":"2026-11-09T13:48:10.612Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/19cd0f6698186d55d454101fe3442e32a60cf007c03a42336d5546a968d573e5/source"}}}],"audits":[{"publicationId":"2aaf1621-1c78-4a25-86f9-6ad7d8f1e790","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The findings and conclusions are attributed to Aaron Broukhim, Nadir Weibel and Eshin Jolly, matching the arXiv listing. The title prefix 'arXiv cs.AI' matches the publisher metadata, although the paper's primary category is cs.SD (Sound). This is not a misattribution.","excerpt":"Authors:Aaron Broukhim, Nadir Weibel, Eshin Jolly"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful but leaves out the context needed for a general reader. It never says that preference-based reinforcement learning (PbRL) is the source's 'dominant framework for aligning AI systems', so it is unclear why rater agreement matters. Central terms are unexplained: synthetic ratings, preference annotations, modalities, inter-rater agreement, ICC, length bias and decision thresholds. Smallest repair: add one opening sentence giving the PbRL alignment context and plain glosses or tooltips for these terms, keeping the terms exact.","excerpt":"Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-02-26, which matches the v1 submission date. The 5 Nov 2026 date is a later revision (v5) and the feed date, not the first publication.","excerpt":"[Submitted on 26 Feb 2026 (v1), last revised 5 Nov 2026 (this version, v5)]"},{"dimension":"support","status":"pass","explanation":"Every claim matches the abstract and is presented as the authors' report. The roughly 9 raters, near-chance cross-modality agreement, audio rater differences, the synthetic-rating proxy and the modality-specific conclusion all appear in the source. The limitation and the question about synthetic-rating validity are appropriately open. No benefits are invented.","excerpt":"synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page, which is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-09T13:17:10.042Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}