{"id":"8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f","title":"arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","argument":"Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here.","speaker":{"name":"Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua","roleAtSource":"Article author"},"source":{"title":"Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09457","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.","excerpt":"We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.","excerpt":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.","excerpt":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.","excerpt":"On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","excerpt":"Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.","excerpt":"Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09457","checkedAt":"2026-11-13T00:34:08.769Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","summary":"Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.","question":"Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?","claims":[{"text":"The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.","excerpt":"We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.","support":"source-stated"},{"text":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.","excerpt":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.","support":"source-stated"},{"text":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.","excerpt":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.","support":"source-stated"},{"text":"On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.","excerpt":"On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.","support":"source-stated"},{"text":"Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","excerpt":"Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","support":"source-stated"},{"text":"The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.","excerpt":"Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.","support":"source-stated"}],"limitations":["The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings.","The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here."],"mediaId":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","correctionReason":null,"id":"8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f","publicationId":"ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09457","sourceTitle":"Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:06:42.404Z","createdAt":"2026-11-12T23:31:25.726Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","kind":"article","url":"https://arxiv.org/abs/2611.09457","sourceUrl":"https://arxiv.org/abs/2611.09457","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T23:05:11.101Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09457","checkedAt":"2026-11-13T00:34:08.769Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","confidence":0.6000000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"4327f3fd6f9358053c80427d1ed4bfd26b670d61ecc4f7cb48df6091b1e80f40","decidedAt":"2026-11-12T23:05:13.512Z"},"provenance":{"textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observationId":"a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249","retrievedAt":"2026-11-12T23:05:10.784Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0958ba11-a089-4944-816b-5c5003d241a1"},"reviewer":{"model":"gpt-6-luna","runId":"5135384a-f751-414b-8b6b-1c4639f84221"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T23:31:25.726Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mishra, Viraj","url":null,"kind":"unknown"},{"name":"Gupta, Shubham","url":null,"kind":"unknown"},{"name":"Gupta, Dhruv","url":null,"kind":"unknown"},{"name":"Tai, Yuehong Cassandra","url":null,"kind":"unknown"},{"name":"Li, Qunhua","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mishra, Viraj","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mishra, Viraj","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Gupta, Shubham","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Shubham","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Gupta, Dhruv","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Dhruv","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Tai, Yuehong Cassandra","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tai, Yuehong Cassandra","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Li, Qunhua","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Qunhua","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09457","snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27","formats":{"html":"/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27","markdown":"/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27.md","json":"/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-counterfactual","name":"Counterfactual","explanation":"An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.","referenceUrl":"https://dictionary.cambridge.org/dictionary/english/counterfactual"},{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f","revision":"ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab","revisionKind":"publication-id","title":"arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","url":"https://escapemeridian.com/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.09457","revision":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","revisionKind":"sha256-source-text","title":"Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","url":"https://arxiv.org/abs/2611.09457"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f","title":"arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","argument":"Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here.","speaker":{"name":"Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua","roleAtSource":"Article author"},"source":{"title":"Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09457","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.","excerpt":"We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.","excerpt":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.","excerpt":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.","excerpt":"On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","excerpt":"Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"},{"text":"The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.","excerpt":"Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.","url":"https://arxiv.org/abs/2611.09457","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09457","checkedAt":"2026-11-13T00:34:08.769Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","summary":"Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.","question":"Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?","claims":[{"text":"The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.","excerpt":"We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.","support":"source-stated"},{"text":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.","excerpt":"Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.","support":"source-stated"},{"text":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.","excerpt":"Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.","support":"source-stated"},{"text":"On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.","excerpt":"On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.","support":"source-stated"},{"text":"Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","excerpt":"Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.","support":"source-stated"},{"text":"The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.","excerpt":"Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.","support":"source-stated"}],"limitations":["The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings.","The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here."],"mediaId":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","correctionReason":null,"id":"8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f","publicationId":"ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09457","sourceTitle":"Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:06:42.404Z","createdAt":"2026-11-12T23:31:25.726Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","kind":"article","url":"https://arxiv.org/abs/2611.09457","sourceUrl":"https://arxiv.org/abs/2611.09457","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T23:05:11.101Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09457","checkedAt":"2026-11-13T00:34:08.769Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"06dd8cdb6601b77ecb3c6241821109b3cf98d7be5194bf1a4691ee48ed3bd840","confidence":0.6000000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"4327f3fd6f9358053c80427d1ed4bfd26b670d61ecc4f7cb48df6091b1e80f40","decidedAt":"2026-11-12T23:05:13.512Z"},"provenance":{"textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observationId":"a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249","retrievedAt":"2026-11-12T23:05:10.784Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0958ba11-a089-4944-816b-5c5003d241a1"},"reviewer":{"model":"gpt-6-luna","runId":"5135384a-f751-414b-8b6b-1c4639f84221"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T23:31:25.726Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mishra, Viraj","url":null,"kind":"unknown"},{"name":"Gupta, Shubham","url":null,"kind":"unknown"},{"name":"Gupta, Dhruv","url":null,"kind":"unknown"},{"name":"Tai, Yuehong Cassandra","url":null,"kind":"unknown"},{"name":"Li, Qunhua","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mishra, Viraj","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mishra, Viraj","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Gupta, Shubham","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Shubham","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Gupta, Dhruv","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Dhruv","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Tai, Yuehong Cassandra","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tai, Yuehong Cassandra","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}},{"actor":{"name":"Li, Qunhua","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Qunhua","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09457","locator":"head/meta[name=author|citation_author]"},"assertionId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T00:34:08.769Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","sourceUrl":"https://arxiv.org/abs/2611.09457","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09457","snapshotId":"3951b7261231846f92b33ca5552eeb8b7a2294b965568aef4b802735a40d4753","contentSha256":"841481fc81e44ecca2507ef966acc9f52c3f9e95ffc5bd837bab1e0e11a90c28","textHash":"69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24","observedAt":"2026-11-13T00:34:08.769Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f/source"}}}],"audits":[{"publicationId":"ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft names all five listed authors and attributes the findings, interpretation and recommendation to them. It correctly frames the k-nearest-neighbor interpretation as what the authors say the result indicates.","excerpt":"Authors:Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful and keeps the authors' central caution. A general reader may still struggle with several unexplained terms: 'original mapping' (the dataset's established label-to-category pairing), retrieval-augmented annotation, fine-tuning, and k-nearest-neighbor classification and 'neighborhood labels.' The audit itself is not explained in plain terms. A smallest repair would add one plain sentence: swapping which definition goes with which label shows whether the model follows the written definition or the familiar label and examples. The draft also does not state the practical stake that social-science coding could look accurate without following the codebook.","excerpt":"high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-11-06 matches the arXiv v1 submission date. The feed date of Nov 9 is correctly not used.","excerpt":"[v1]\nFri, 6 Nov 2026 13:28:18 UTC (110 KB)"},{"dimension":"support","status":"pass","explanation":"Each claim closely matches the abstract, including the 74–93% figure, the 'two tasks' scope and the joint-validation recommendation. The question about generalizing beyond the three datasets follows from the stated scope and is not leading. The limitations accurately note that only the abstract is available. No benefits or costs are invented.","excerpt":"Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified, direct arXiv abstract page for this paper. It matches the source URL and is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-12T23:31:25.726Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}