{"id":"806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af","title":"arXiv cs.AI paper: Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","argument":"The paper’s authors investigate whether soft prompt tuning can bring lightweight open-source LLM judges closer to human annotation policy. They train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization. The authors report that hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations compared with untuned baselines. They also report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions. The work addresses a practical tradeoff described by the authors: proprietary models can achieve high agreement with human annotators, but are costly and difficult to deploy. Their results suggest that adapting open-source judges may offer another route to closer agreement on the tested benchmark and dimensions. The abstract does not establish how well the adaptation works beyond that scope.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available source text is the paper’s abstract; it does not provide quantitative agreement results or details of the training and evaluation setup. The reported findings concern FairBotBench and the stated dimensions of bias, toxicity, and personalization; the abstract does not report results outside that scope.","speaker":{"name":"Janusko, Tamás, Kleinkop, Sven, Bochnia, Ricardo, Hempel, Gunnar, Tomschke, Steffen, Anke, Jürgen, Thiele, Maik","roleAtSource":"Article author"},"source":{"title":"Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07126","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors investigate using soft prompt tuning to align open-source LLM judges more closely with human annotation policy.","excerpt":"we investigate whether soft prompt tuning can align open-source LLM judges more closely with human annotation policy.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization.","excerpt":"We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors report that hybrid soft-prompt tuning consistently improves agreement with human annotations over untuned baselines.","excerpt":"hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations over the corresponding untuned baselines.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions.","excerpt":"For Qwen 3.5 9B, the adapted judges surpass both hosted reference judges across all three evaluation dimensions.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors describe proprietary models as costly and difficult to deploy in practical settings.","excerpt":"they remain costly and difficult to deploy in practical settings.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07126","checkedAt":"2026-11-08T20:12:09.222Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Does the reported improvement in agreement also hold when the adapted judges are evaluated outside FairBotBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper: Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","summary":"The paper’s authors investigate whether soft prompt tuning can bring lightweight open-source LLM judges closer to human annotation policy. They train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization. The authors report that hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations compared with untuned baselines. They also report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions. The work addresses a practical tradeoff described by the authors: proprietary models can achieve high agreement with human annotators, but are costly and difficult to deploy. Their results suggest that adapting open-source judges may offer another route to closer agreement on the tested benchmark and dimensions. The abstract does not establish how well the adaptation works beyond that scope.","question":"Does the reported improvement in agreement also hold when the adapted judges are evaluated outside FairBotBench?","claims":[{"text":"The authors investigate using soft prompt tuning to align open-source LLM judges more closely with human annotation policy.","excerpt":"we investigate whether soft prompt tuning can align open-source LLM judges more closely with human annotation policy.","support":"source-stated"},{"text":"The authors train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization.","excerpt":"We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization.","support":"source-stated"},{"text":"The authors report that hybrid soft-prompt tuning consistently improves agreement with human annotations over untuned baselines.","excerpt":"hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations over the corresponding untuned baselines.","support":"source-stated"},{"text":"The authors report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions.","excerpt":"For Qwen 3.5 9B, the adapted judges surpass both hosted reference judges across all three evaluation dimensions.","support":"source-stated"},{"text":"The authors describe proprietary models as costly and difficult to deploy in practical settings.","excerpt":"they remain costly and difficult to deploy in practical settings.","support":"source-stated"}],"limitations":["The available source text is the paper’s abstract; it does not provide quantitative agreement results or details of the training and evaluation setup.","The reported findings concern FairBotBench and the stated dimensions of bias, toxicity, and personalization; the abstract does not report results outside that scope."],"mediaId":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","correctionReason":null,"id":"806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af","publicationId":"c0d96bc5-aa60-45b4-8dc9-b15ef9ee857f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07126","sourceTitle":"Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:12:38.656Z","createdAt":"2026-11-08T19:08:23.725Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","kind":"article","url":"https://arxiv.org/abs/2611.07126","sourceUrl":"https://arxiv.org/abs/2611.07126","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T19:07:29.347Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07126","checkedAt":"2026-11-08T20:12:09.222Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","confidence":0.79,"provider":"typesafe","model":"jev-1.13.0","inputHash":"6156d1df52cf5ed20ed620a73a7325ed92a08a668109bb82ee1ab3ec203dd67a","decidedAt":"2026-11-08T19:07:30.298Z"},"provenance":{"textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observationId":"ed29653feb8b50ab2545759944e42b1bb5853cdea04b89e6b93f59366968f805","retrievedAt":"2026-11-08T19:07:29.162Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"639ef56a-a2bd-46ea-8aa3-5d2315462f83"},"reviewer":{"model":"gpt-6-luna","runId":"8fd628fa-6038-43c2-8d95-f08fa62b0683"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T19:08:23.725Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Janusko, Tamás","url":null,"kind":"unknown"},{"name":"Kleinkop, Sven","url":null,"kind":"unknown"},{"name":"Bochnia, Ricardo","url":null,"kind":"unknown"},{"name":"Hempel, Gunnar","url":null,"kind":"unknown"},{"name":"Tomschke, Steffen","url":null,"kind":"unknown"},{"name":"Anke, Jürgen","url":null,"kind":"unknown"},{"name":"Thiele, Maik","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Janusko, Tamás","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Janusko, Tamás","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Kleinkop, Sven","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kleinkop, Sven","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Bochnia, Ricardo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bochnia, Ricardo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Hempel, Gunnar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hempel, Gunnar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Tomschke, Steffen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tomschke, Steffen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Anke, Jürgen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anke, Jürgen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Thiele, Maik","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Thiele, Maik","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07126","snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-adapting-llm-judges-to-observed-human-806c05f4","formats":{"html":"/readings/arxiv-cs-ai-paper-adapting-llm-judges-to-observed-human-806c05f4","markdown":"/readings/arxiv-cs-ai-paper-adapting-llm-judges-to-observed-human-806c05f4.md","json":"/readings/arxiv-cs-ai-paper-adapting-llm-judges-to-observed-human-806c05f4.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-embedding-vector","name":"Embedding vector","explanation":"A list of numbers that represents a piece of text, an image or another item so software can compare meanings.","referenceUrl":"https://developers.google.com/machine-learning/glossary#embedding-vector"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af","revision":"c0d96bc5-aa60-45b4-8dc9-b15ef9ee857f","revisionKind":"publication-id","title":"arXiv cs.AI paper: Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-adapting-llm-judges-to-observed-human-806c05f4"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.07126","revision":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","revisionKind":"sha256-source-text","title":"Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","url":"https://arxiv.org/abs/2611.07126"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af","title":"arXiv cs.AI paper: Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","argument":"The paper’s authors investigate whether soft prompt tuning can bring lightweight open-source LLM judges closer to human annotation policy. They train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization. The authors report that hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations compared with untuned baselines. They also report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions. The work addresses a practical tradeoff described by the authors: proprietary models can achieve high agreement with human annotators, but are costly and difficult to deploy. Their results suggest that adapting open-source judges may offer another route to closer agreement on the tested benchmark and dimensions. The abstract does not establish how well the adaptation works beyond that scope.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available source text is the paper’s abstract; it does not provide quantitative agreement results or details of the training and evaluation setup. The reported findings concern FairBotBench and the stated dimensions of bias, toxicity, and personalization; the abstract does not report results outside that scope.","speaker":{"name":"Janusko, Tamás, Kleinkop, Sven, Bochnia, Ricardo, Hempel, Gunnar, Tomschke, Steffen, Anke, Jürgen, Thiele, Maik","roleAtSource":"Article author"},"source":{"title":"Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07126","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors investigate using soft prompt tuning to align open-source LLM judges more closely with human annotation policy.","excerpt":"we investigate whether soft prompt tuning can align open-source LLM judges more closely with human annotation policy.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization.","excerpt":"We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors report that hybrid soft-prompt tuning consistently improves agreement with human annotations over untuned baselines.","excerpt":"hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations over the corresponding untuned baselines.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions.","excerpt":"For Qwen 3.5 9B, the adapted judges surpass both hosted reference judges across all three evaluation dimensions.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"},{"text":"The authors describe proprietary models as costly and difficult to deploy in practical settings.","excerpt":"they remain costly and difficult to deploy in practical settings.","url":"https://arxiv.org/abs/2611.07126","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07126","checkedAt":"2026-11-08T20:12:09.222Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Does the reported improvement in agreement also hold when the adapted judges are evaluated outside FairBotBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper: Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","summary":"The paper’s authors investigate whether soft prompt tuning can bring lightweight open-source LLM judges closer to human annotation policy. They train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization. The authors report that hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations compared with untuned baselines. They also report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions. The work addresses a practical tradeoff described by the authors: proprietary models can achieve high agreement with human annotators, but are costly and difficult to deploy. Their results suggest that adapting open-source judges may offer another route to closer agreement on the tested benchmark and dimensions. The abstract does not establish how well the adaptation works beyond that scope.","question":"Does the reported improvement in agreement also hold when the adapted judges are evaluated outside FairBotBench?","claims":[{"text":"The authors investigate using soft prompt tuning to align open-source LLM judges more closely with human annotation policy.","excerpt":"we investigate whether soft prompt tuning can align open-source LLM judges more closely with human annotation policy.","support":"source-stated"},{"text":"The authors train soft prompts on FairBotBench and evaluate adapted judges on bias, toxicity, and personalization.","excerpt":"We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization.","support":"source-stated"},{"text":"The authors report that hybrid soft-prompt tuning consistently improves agreement with human annotations over untuned baselines.","excerpt":"hybrid soft-prompt tuning, which combines textual prompts with learned continuous prompt embeddings, consistently improves agreement with human annotations over the corresponding untuned baselines.","support":"source-stated"},{"text":"The authors report that adapted Qwen 3.5 9B judges surpass both hosted reference judges on all three evaluation dimensions.","excerpt":"For Qwen 3.5 9B, the adapted judges surpass both hosted reference judges across all three evaluation dimensions.","support":"source-stated"},{"text":"The authors describe proprietary models as costly and difficult to deploy in practical settings.","excerpt":"they remain costly and difficult to deploy in practical settings.","support":"source-stated"}],"limitations":["The available source text is the paper’s abstract; it does not provide quantitative agreement results or details of the training and evaluation setup.","The reported findings concern FairBotBench and the stated dimensions of bias, toxicity, and personalization; the abstract does not report results outside that scope."],"mediaId":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","correctionReason":null,"id":"806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af","publicationId":"c0d96bc5-aa60-45b4-8dc9-b15ef9ee857f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07126","sourceTitle":"Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:12:38.656Z","createdAt":"2026-11-08T19:08:23.725Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","kind":"article","url":"https://arxiv.org/abs/2611.07126","sourceUrl":"https://arxiv.org/abs/2611.07126","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T19:07:29.347Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07126","checkedAt":"2026-11-08T20:12:09.222Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"aa5ce212578215f70127442cf19ec4e76f3e490f4b601fae60a4d0c4752df1af","confidence":0.79,"provider":"typesafe","model":"jev-1.13.0","inputHash":"6156d1df52cf5ed20ed620a73a7325ed92a08a668109bb82ee1ab3ec203dd67a","decidedAt":"2026-11-08T19:07:30.298Z"},"provenance":{"textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observationId":"ed29653feb8b50ab2545759944e42b1bb5853cdea04b89e6b93f59366968f805","retrievedAt":"2026-11-08T19:07:29.162Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"639ef56a-a2bd-46ea-8aa3-5d2315462f83"},"reviewer":{"model":"gpt-6-luna","runId":"8fd628fa-6038-43c2-8d95-f08fa62b0683"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T19:08:23.725Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Janusko, Tamás","url":null,"kind":"unknown"},{"name":"Kleinkop, Sven","url":null,"kind":"unknown"},{"name":"Bochnia, Ricardo","url":null,"kind":"unknown"},{"name":"Hempel, Gunnar","url":null,"kind":"unknown"},{"name":"Tomschke, Steffen","url":null,"kind":"unknown"},{"name":"Anke, Jürgen","url":null,"kind":"unknown"},{"name":"Thiele, Maik","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Janusko, Tamás","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Janusko, Tamás","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Kleinkop, Sven","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kleinkop, Sven","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Bochnia, Ricardo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bochnia, Ricardo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Hempel, Gunnar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hempel, Gunnar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Tomschke, Steffen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tomschke, Steffen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Anke, Jürgen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anke, Jürgen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}},{"actor":{"name":"Thiele, Maik","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Thiele, Maik","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07126","locator":"head/meta[name=author|citation_author]"},"assertionId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:12:09.222Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","sourceUrl":"https://arxiv.org/abs/2611.07126","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07126","snapshotId":"ad11a6588d297642fb15214183c8510ae35672520639e202de39ee4baa45181f","contentSha256":"713f3232ad4fbe24b2198362ac84b380dde9cf67db8359b3b5e214ef33d6ed2c","textHash":"dd4d9cf5f6d135f8b2a4ef57a406d6397af22a50f261169460c7c85c1c2f6fe4","observedAt":"2026-11-08T20:12:09.222Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/806c05f44d7e9e2bf620deae40dab8874d8a042e1717a6b93375bb0c88ddb4af/source"}}}],"audits":[{"publicationId":"c0d96bc5-aa60-45b4-8dc9-b15ef9ee857f","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The findings are consistently attributed to the paper's authors as reported results, not stated as established fact. The arXiv listing and author list support this. The paper's primary category is cs.CL, but it is cross-listed in cs.AI, so the publisher label is acceptable.","excerpt":"Authors:Tamás Janusko, Sven Kleinkop, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, Jürgen Anke, Maik Thiele"},{"dimension":"summary","status":"uncertain","explanation":"The summary never says what an LLM judge does: an AI model that scores chatbot responses. A non-specialist cannot tell what is being judged or why agreement with human raters matters. Soft prompt tuning, human annotation policy and the unnamed hosted reference judges also lack any plain explanation or tooltip. Smallest repair: in the first sentence, say these judges are LLMs used to automatically evaluate chatbot responses, and add tooltips for soft prompt tuning and LLM judge. The title only restates the paper title, but that is acceptable.","excerpt":"Large Language Models (LLMs) are increasingly employed as automated judges for evaluating chatbot responses."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-04 matches the v1 submission date. The later feed date of 2026-11-06 is correctly not used.","excerpt":"[Submitted on 4 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches the abstract verbatim, and the cost and deployment tradeoff is attributed to the authors. The significance sentence is hedged and scoped to the tested benchmark and dimensions. The limitations correctly note there are no quantitative results. Minor point: the abstract says the prompts were trained on FairBotBench but does not explicitly say evaluation was on FairBotBench. The question's framing is therefore a reasonable inference rather than a stated fact.","excerpt":"We train soft prompts on the FairBotBench benchmark and evaluate the adapted judges across the dimensions of bias, toxicity, and personalization."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-08T19:08:23.725Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}