Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua · Nov 6, 2026
arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
Markdown · JSON · Original source
Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T23:31:25.726Z. First observed: 2026-11-09T05:06:42.404Z. Retrieved: 2026-11-12T23:05:10.784Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here.
Open question
Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?
Source-stated
The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.
We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.
Source-stated
Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.
Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.
Source-stated
Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.
Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.
Source-stated
On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.
On the two tasks with substantial gains, direct \(k\)-nearest-neighbor classification recovers 74--93\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.
Source-stated
Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.
Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.
Source-stated
The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.
Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Source and contributor context
Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Counterfactual
An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T23:31:25.726Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249. Source text hash: 69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24.
Revision and source-check history
Version 1 · 2026-11-12T23:31:25.726Z
arXiv: Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai and Qunhua Li examine whether annotating social-science texts follow researchers’ category definitions, or instead rely on category labels and labeled examples. Their audit reassigns definitions among output labels to distinguish adherence to the new definitions from preserving a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions; standard strengthens the original mapping without reliably increasing adherence to definitions. In retrieval-augmented annotation, retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations. On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement, which the authors say indicates that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. The authors’ central caution is that better accuracy need not mean the model is following the researcher’s definitions: definitions, labels, demonstrations, retrieval and training data should be aligned and validated jointly.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T23:31:25.726Z. First observed: 2026-11-09T05:06:42.404Z. Retrieved: 2026-11-12T23:05:10.784Z.
Limits
The reported audit covers three political-text datasets; the abstract does not establish whether the findings generalize to other datasets or annotation settings. The source page provides the paper’s abstract, not the full methods or results, so details behind the reported comparisons and gains are unavailable here.
Open question
Do the findings about definition adherence and alignment also hold on political-text datasets beyond the three studied?
Source-stated
The authors introduce a counterfactual audit that reassigns definitions among output labels to distinguish adherence to supplied definitions from preserving a dataset’s original mapping.
We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping.
Source-stated
Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing adherence to definitions.
Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.
Source-stated
Retrieving the nearest training examples generally outperforms retrieving examples separately within each category, despite using fewer demonstrations.
Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations.
Source-stated
On two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74–93% of the improvement; the authors say this indicates neighborhood labels provide much of the predictive signal.
On the two tasks with substantial gains, direct \(k\)-nearest-neighbor classification recovers 74--93\% of the improvement, indicating that neighborhood labels provide much of the predictive signal.
Source-stated
Relabeling fixed demonstrations redirects predictions, while aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.
Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping.
Source-stated
The authors argue that accuracy-improving interventions need not increase adherence to the researcher’s category definitions, and recommend aligning and validating the relevant elements jointly.
Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Source and contributor context
Mishra, Viraj, Gupta, Shubham, Gupta, Dhruv, Tai, Yuehong Cassandra, Li, Qunhua · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Counterfactual
An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T23:31:25.726Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a45a1cbf445657e7f53641e0f9ef47278c54eefdd42a8f7a10760e0d52149249. Source text hash: 69b39d36df6defaf7b9d0a26e2ffcafc9e9b319a41a9b76413c204278c21ca24.
uncertain · 2026-11-12T23:31:25.726Z · anthropic/claude-opus-5.5
Publication: ce7a38d8-2f1d-4b9a-8422-7756f3ee6fab
attribution — pass: The draft names all five listed authors and attributes the findings, interpretation and recommendation to them. It correctly frames the k-nearest-neighbor interpretation as what the authors say the result indicates.
Authors:Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li
summary — uncertain: The summary is faithful and keeps the authors' central caution. A general reader may still struggle with several unexplained terms: 'original mapping' (the dataset's established label-to-category pairing), retrieval-augmented annotation, fine-tuning, and k-nearest-neighbor classification and 'neighborhood labels.' The audit itself is not explained in plain terms. A smallest repair would add one plain sentence: swapping which definition goes with which label shows whether the model follows the written definition or the familiar label and examples. The draft also does not state the practical stake that social-science coding could look accurate without following the codebook.
high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations
date — pass: originalPublished 2026-11-06 matches the arXiv v1 submission date. The feed date of Nov 9 is correctly not used.
[v1] Fri, 6 Nov 2026 13:28:18 UTC (110 KB)
support — pass: Each claim closely matches the abstract, including the 74–93% figure, the 'two tasks' scope and the joint-validation recommendation. The question about generalizing beyond the three datasets follows from the stated scope and is not leading. The limitations accurately note that only the abstract is available. No benefits or costs are invented.
Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
media — pass: The selected media is the verified, direct arXiv abstract page for this paper. It matches the source URL and is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceAn argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceAn argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv