# arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation [Canonical reading](https://escapemeridian.com/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27) Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T23:31:25.726Z. First observed: 2026-11-09T05:06:42.404Z. Retrieved: 2026-11-12T23:05:10.784Z. - [Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation](https://arxiv.org/abs/2611.09457) ## Limits The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here. ## Open question Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied? ## Source-stated The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping. > We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. > Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. > Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal. > On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. > Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly. > Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source and contributor context Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Counterfactual An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have. - [Definition reference](https://dictionary.cambridge.org/dictionary/english/counterfactual) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T23:31:25.726Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249. Source text hash: 69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24. ## Revision and source-check history ### Version 1 — 2026-11-12T23:31:25.726Z Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether large language models annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their counterfactual audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T23:31:25.726Z. First observed: 2026-11-09T05:06:42.404Z. Retrieved: 2026-11-12T23:05:10.784Z. - [Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation](https://arxiv.org/abs/2611.09457) ## Limits The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here. ## Open question Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied? ## Source-stated The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping. > We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions. > Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. > Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal. > On the two tasks with substantial gains, direct \\(k\\)-nearest-neighbor classification recovers 74--93\\% of the improvement, indicating that neighborhood labels provide much of the predictive signal. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. > Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source-stated The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly. > Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly. - [Supporting source](https://arxiv.org/abs/2611.09457) ## Source and contributor context Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Counterfactual An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have. - [Definition reference](https://dictionary.cambridge.org/dictionary/english/counterfactual) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T23:31:25.726Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249. Source text hash: 69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24. ### uncertain — 2026-11-12T23:31:25.726Z — anthropic/claude-opus-5.5 Publication: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab attribution — pass: The draft names all five listed authors and attributes the findings, interpretation and recommendation to them. It correctly frames the k-nearest-neighbor interpretation as what the authors say the result indicates. > Authors:Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li summary — uncertain: The summary is faithful and keeps the authors' central caution. A general reader may still struggle with several unexplained terms: 'original mapping' (the dataset's established label-to-category pairing), retrieval-augmented annotation, fine-tuning, and k-nearest-neighbor classification and 'neighborhood labels.' The audit itself is not explained in plain terms. A smallest repair would add one plain sentence: swapping which definition goes with which label shows whether the model follows the written definition or the familiar label and examples. The draft also does not state the practical stake that social-science coding could look accurate without following the codebook. > high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations date — pass: originalPublished 2026-11-06 matches the arXiv v1 submission date. The feed date of Nov 9 is correctly not used. > \[v1\] Fri, 6 Nov 2026 13:28:18 UTC (110 KB) support — pass: Each claim closely matches the abstract, including the 74–93% figure, the 'two tasks' scope and the joint-validation recommendation. The question about generalizing beyond the three datasets follows from the stated scope and is not leading. The limitations accurately note that only the abstract is available. No benefits or costs are invented. > Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly. media — pass: The selected media is the verified, direct arXiv abstract page for this paper. It matches the source URL and is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/8a379c279f28f4ccfd725cfe29a4d135c69526c38b5e0ff4d2d719871d61fa7f/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-definitions-label-names-or-demonstrations-auditing-what-8a379c27.json)