# Khan and Misri test how many LLM-labelled rows a decision encoder needs [Canonical reading](https://escapemeridian.com/readings/khan-and-misri-test-how-many-llm-labelled-rows-a-decision-f5973fe0) Mohammed Nayeem Khan and Aliyas Ahmad Misri test how training-set size and label errors affect Laya, a decision encoder that selects among named options without generating text. In 19 fine-tunes, accuracy on held-out items from the same generator rose from 69.4% before fine-tuning to 86.9% using all 2,818 training rows, but a saturating fit with bootstrap intervals could not pin down the curve’s plateau. The authors report that a single run with 10% symmetric label noise showed no detectable change, while 20% and 40% cost 3.4 and 8.7 points. Confusable-pair noise at 20% cost 7.4 points, and under noise the model shifted from over-confident to under-confident. The comparison with a TF-IDF classifier was much closer at full size than with 5% of the rows; at full size, Laya’s 0.9-point lead had an interval that included zero. These results describe a synthetic setup, with LLM-labelled data, one backbone, one trainer and mostly one seed; the held-out items were in-distribution, and no out-of-family test was run. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 12, 2026 (2026-11-12). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T09:31:36.609Z. First observed: 2026-11-13T07:43:09.674Z. Retrieved: 2026-11-13T09:19:04.753Z. - [How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?](https://arxiv.org/abs/2611.16498) ## Limits The evaluation used synthetic, LLM-labelled data, one backbone and one trainer, with mostly one seed; the abstract does not establish how results transfer beyond this setup. All 1,065 held-out items were in-distribution, and the authors report that no out-of-family test was run. The reported 10% symmetric label noise result comes from a single run. The source describes the plateau as unresolved: its saturating fit with bootstrap intervals could not pin it down. ## Open question What additional data or runs, if any, would let the authors pin down the accuracy curve’s plateau? ## Source-stated The authors describe Laya as a decision encoder that selects a named option from a question and text without generating text. > A decision encoder such as Laya reads a question with named options (for example, which department should handle a ticket) and a text, and picks an option. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated Across the reported fine-tunes, accuracy on held-out items from the same generator increased with training-set size, from 69.4% before fine-tuning to 86.9% using all 2,818 rows. > We ran 19 fine-tunes of Laya (ModernBERT-large) on nested subsets of a 2,818-row pool and scored them on 1,065 held-out items from the same generator, so every test item is in-distribution and no out-of-family test was run. Accuracy rises from 69.4% before fine-tuning to 74.3% with 5% of the rows, 8 - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors say their saturating fit with bootstrap intervals could not determine the accuracy curve’s plateau. > The curve flattens, and a saturating fit with bootstrap intervals cannot pin down its plateau. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated In a single run, 10% symmetric label noise showed no detectable change; the authors report lower accuracy with 20% and 40% noise. > A single run with 10% symmetric label noise shows no detectable change (-0.2 points, 95% interval -1.7 to +1.4); 20% costs 3.4 points and 40% costs 8.7 points. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors report that 20% confusable-pair noise cost 7.4 points and that the model shifted from over-confident to under-confident under noise. > Confusable-pair noise at 20% costs 7.4 points, and under noise the model moves from over-confident to under-confident. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated At full size, Laya’s reported 0.9-point lead over the TF-IDF classifier had an interval including zero; at 5% of the rows, the reported lead was 21.1 points. > Laya is +0.9 points ahead of a TF-IDF classifier trained on the same rows at full size (the interval includes zero) and +21.1 points ahead at 5% of the rows. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors limit the study’s scope to synthetic, LLM-labelled data and one backbone and trainer, with mostly one seed. > Everything is synthetic, LLM-labelled and from one backbone, one trainer and mostly one seed. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source and contributor context Khan, Mohammed Nayeem, Misri, Aliyas Ahmad · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5473cb5e-c05a-4d5f-8e8c-c5337455fef7. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T09:31:36.609Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ab204b9fe732910360019a274525ca34ce5feb82bac6e55d61fe04629989d702. Source text hash: 6c1e3ea362f4e73a81e92ac38cf8bf5fd370e6ce8585fdb2a63ca8c9630ef009. ## Revision and source-check history ### Version 1 — 2026-11-13T09:31:36.609Z Mohammed Nayeem Khan and Aliyas Ahmad Misri test how training-set size and label errors affect Laya, a decision encoder that selects among named options without generating text. In 19 fine-tunes, accuracy on held-out items from the same generator rose from 69.4% before fine-tuning to 86.9% using all 2,818 training rows, but a saturating fit with bootstrap intervals could not pin down the curve’s plateau. The authors report that a single run with 10% symmetric label noise showed no detectable change, while 20% and 40% cost 3.4 and 8.7 points. Confusable-pair noise at 20% cost 7.4 points, and under noise the model shifted from over-confident to under-confident. The comparison with a TF-IDF classifier was much closer at full size than with 5% of the rows; at full size, Laya’s 0.9-point lead had an interval that included zero. These results describe a synthetic setup, with LLM-labelled data, one backbone, one trainer and mostly one seed; the held-out items were in-distribution, and no out-of-family test was run. ## Original source arXiv cs.AI · Article Original source date: Nov 12, 2026 (2026-11-12). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T09:31:36.609Z. First observed: 2026-11-13T07:43:09.674Z. Retrieved: 2026-11-13T09:19:04.753Z. - [How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?](https://arxiv.org/abs/2611.16498) ## Limits The evaluation used synthetic, LLM-labelled data, one backbone and one trainer, with mostly one seed; the abstract does not establish how results transfer beyond this setup. All 1,065 held-out items were in-distribution, and the authors report that no out-of-family test was run. The reported 10% symmetric label noise result comes from a single run. The source describes the plateau as unresolved: its saturating fit with bootstrap intervals could not pin it down. ## Open question What additional data or runs, if any, would let the authors pin down the accuracy curve’s plateau? ## Source-stated The authors describe Laya as a decision encoder that selects a named option from a question and text without generating text. > A decision encoder such as Laya reads a question with named options (for example, which department should handle a ticket) and a text, and picks an option. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated Across the reported fine-tunes, accuracy on held-out items from the same generator increased with training-set size, from 69.4% before fine-tuning to 86.9% using all 2,818 rows. > We ran 19 fine-tunes of Laya (ModernBERT-large) on nested subsets of a 2,818-row pool and scored them on 1,065 held-out items from the same generator, so every test item is in-distribution and no out-of-family test was run. Accuracy rises from 69.4% before fine-tuning to 74.3% with 5% of the rows, 8 - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors say their saturating fit with bootstrap intervals could not determine the accuracy curve’s plateau. > The curve flattens, and a saturating fit with bootstrap intervals cannot pin down its plateau. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated In a single run, 10% symmetric label noise showed no detectable change; the authors report lower accuracy with 20% and 40% noise. > A single run with 10% symmetric label noise shows no detectable change (-0.2 points, 95% interval -1.7 to +1.4); 20% costs 3.4 points and 40% costs 8.7 points. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors report that 20% confusable-pair noise cost 7.4 points and that the model shifted from over-confident to under-confident under noise. > Confusable-pair noise at 20% costs 7.4 points, and under noise the model moves from over-confident to under-confident. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated At full size, Laya’s reported 0.9-point lead over the TF-IDF classifier had an interval including zero; at 5% of the rows, the reported lead was 21.1 points. > Laya is +0.9 points ahead of a TF-IDF classifier trained on the same rows at full size (the interval includes zero) and +21.1 points ahead at 5% of the rows. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source-stated The authors limit the study’s scope to synthetic, LLM-labelled data and one backbone and trainer, with mostly one seed. > Everything is synthetic, LLM-labelled and from one backbone, one trainer and mostly one seed. - [Supporting source](https://arxiv.org/abs/2611.16498) ## Source and contributor context Khan, Mohammed Nayeem, Misri, Aliyas Ahmad · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5473cb5e-c05a-4d5f-8e8c-c5337455fef7. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T09:31:36.609Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ab204b9fe732910360019a274525ca34ce5feb82bac6e55d61fe04629989d702. Source text hash: 6c1e3ea362f4e73a81e92ac38cf8bf5fd370e6ce8585fdb2a63ca8c9630ef009. ### passed — 2026-11-13T09:31:36.609Z — anthropic/claude-opus-5.5 Publication: 5473cb5e-c05a-4d5f-8e8c-c5337455fef7 attribution — pass: The reading correctly names Mohammed Nayeem Khan and Aliyas Ahmad Misri as the authors and keeps the results attributed to them with phrases such as 'the authors report'. It does not present them as established facts. > Authors:Mohammed Nayeem Khan, Aliyas Ahmad Misri summary — pass: The title, summary and question match the abstract: Laya, data-size scaling, the unresolved plateau, the noise effects, the confidence shift, the TF-IDF comparison and the scope limits. 'Decision encoder' is explained in plain language. Terms such as TF-IDF, symmetric and confusable-pair noise, and seed are technical terms suited to tooltips. Optional, non-blocking omissions: the seed spread (s.d. 1.6) exceeding some size steps, and the independent-labeller result. The plateau question follows directly from the evidence. > Seed spread at 20% of the rows (s.d. 1.6 points) exceeds some of the steps between sizes. date — pass: The originalPublished date, 2026-11-12, matches the arXiv v1 submission date. The feed date of 2026-11-13 is later and is correctly not used. > \[Submitted on 12 Nov 2026\] support — pass: Every figure is grounded verbatim in the abstract: 19 fine-tunes, 2,818 rows, 69.4% to 86.9%, the 10/20/40% noise results, 7.4 points, +0.9 with an interval including zero, +21.1 at 5% of rows, and the in-distribution and single-backbone limits. The reading does not overstate the results as real-world reliability, and it introduces no unsupported benefits or costs. > Everything is synthetic, LLM-labelled and from one backbone, one trainer and mostly one seed. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/f5973fe04840960beff72921112a333c5098f2211e15d3509fc463a3b7461fb1/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/khan-and-misri-test-how-many-llm-labelled-rows-a-decision-f5973fe0.json)