# Pavlovic, Paun and Poesio argue human soft-labels matter more for calibration than accuracy [Canonical reading](https://escapemeridian.com/readings/pavlovic-paun-and-poesio-argue-human-soft-labels-matter-more-0179e71f) Maja Pavlovic, Silviu Paun and Massimo Poesio argue that human soft-labels—labels that retain uncertainty rather than giving only one answer—may matter less for improving accuracy than for improving calibration: whether a model’s confidence reflects its uncertainty. Their paper reports a controlled audit on MNIST and a synthetic variant, designed to separate the effect of soft-labels from mode shifts, or changes in which label is treated as the main answer. The authors say human soft-labels improved accuracy, but had a larger value as a regularizer, helping calibration on difficult samples and stable convergence across training runs. They also report that dataset cartography showed models trained on human soft-labels mirrored human uncertainty, while models trained on synthetic labels did not align with humans. The distinction matters because earlier studies, they argue, conflated benefits of soft-labels with correction of mislabeled data. The authors present their work as a diagnostic testbed for human-AI uncertainty alignment, not evidence here of performance across other datasets or settings. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: May 18, 2026 (2026-05-18). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T20:33:19.882Z. First observed: 2026-11-06T05:10:00.855Z. Retrieved: 2026-11-08T20:32:29.210Z. - [An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration](https://arxiv.org/abs/2605.18648) ## Limits The supplied evidence is the paper’s abstract and publisher page; it does not provide detailed methods, measurements, or results beyond the summary presented there. The reported audit covers MNIST and a synthetic variant, so the supplied evidence does not establish whether the findings extend to other datasets or settings. ## Open question Do human soft-labels produce the reported calibration and uncertainty-alignment patterns beyond MNIST and the synthetic variant? ## Source-stated The authors argue earlier work conflated benefits of soft-labels with correction of mislabeled data, obscuring the effects of soft-label supervision itself. > prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated In an audit using MNIST and a synthetic variant, the authors report that human soft-labels improved accuracy. > across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors say human soft-labels’ larger value was improved calibration on difficult samples and more stable convergence across training runs. > their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors report that models trained on human soft-labels mirrored human uncertainty, unlike models trained on synthetic labels. > models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors describe the work as a diagnostic testbed for human-AI uncertainty alignment. > Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source and contributor context Pavlovic, Maja, Paun, Silviu, Poesio, Massimo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 8289903d-91a0-4bf1-8119-fbb425bb4407. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T20:33:19.882Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 8ffe823e229e7025b68b2b90a53a1c2e6538dcfcbdc3ad5e3814ebd093213694. Source text hash: 2841df4b028bebad1af317e4b066ffc2ef5b496343152cb83d7f79559e85e9a2. ## Revision and source-check history ### Version 1 — 2026-11-08T20:33:19.882Z Maja Pavlovic, Silviu Paun and Massimo Poesio argue that human soft-labels—labels that retain uncertainty rather than giving only one answer—may matter less for improving accuracy than for improving calibration: whether a model’s confidence reflects its uncertainty. Their paper reports a controlled audit on MNIST and a synthetic variant, designed to separate the effect of soft-labels from mode shifts, or changes in which label is treated as the main answer. The authors say human soft-labels improved accuracy, but had a larger value as a regularizer, helping calibration on difficult samples and stable convergence across training runs. They also report that dataset cartography showed models trained on human soft-labels mirrored human uncertainty, while models trained on synthetic labels did not align with humans. The distinction matters because earlier studies, they argue, conflated benefits of soft-labels with correction of mislabeled data. The authors present their work as a diagnostic testbed for human-AI uncertainty alignment, not evidence here of performance across other datasets or settings. ## Original source arXiv cs.AI · Article Original source date: May 18, 2026 (2026-05-18). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T20:33:19.882Z. First observed: 2026-11-06T05:10:00.855Z. Retrieved: 2026-11-08T20:32:29.210Z. - [An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration](https://arxiv.org/abs/2605.18648) ## Limits The supplied evidence is the paper’s abstract and publisher page; it does not provide detailed methods, measurements, or results beyond the summary presented there. The reported audit covers MNIST and a synthetic variant, so the supplied evidence does not establish whether the findings extend to other datasets or settings. ## Open question Do human soft-labels produce the reported calibration and uncertainty-alignment patterns beyond MNIST and the synthetic variant? ## Source-stated The authors argue earlier work conflated benefits of soft-labels with correction of mislabeled data, obscuring the effects of soft-label supervision itself. > prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated In an audit using MNIST and a synthetic variant, the authors report that human soft-labels improved accuracy. > across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors say human soft-labels’ larger value was improved calibration on difficult samples and more stable convergence across training runs. > their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors report that models trained on human soft-labels mirrored human uncertainty, unlike models trained on synthetic labels. > models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source-stated The authors describe the work as a diagnostic testbed for human-AI uncertainty alignment. > Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment. - [Supporting source](https://arxiv.org/abs/2605.18648) ## Source and contributor context Pavlovic, Maja, Paun, Silviu, Poesio, Massimo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 8289903d-91a0-4bf1-8119-fbb425bb4407. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T20:33:19.882Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 8ffe823e229e7025b68b2b90a53a1c2e6538dcfcbdc3ad5e3814ebd093213694. Source text hash: 2841df4b028bebad1af317e4b066ffc2ef5b496343152cb83d7f79559e85e9a2. ### passed — 2026-11-08T20:33:19.882Z — anthropic/claude-opus-5.5 Publication: 8289903d-91a0-4bf1-8119-fbb425bb4407 attribution — pass: The draft names the listed authors, Maja Pavlovic, Silviu Paun and Massimo Poesio, and attributes each finding to them as reported or argued claims, not as established facts. > Authors:Maja Pavlovic, Silviu Paun, Massimo Poesio summary — pass: The summary keeps the abstract's central argument: soft-labels give some accuracy gain, but their larger value is regularization that improves calibration and stable convergence. It explains soft-labels, calibration and mode shifts in plain language. Regularizer, dataset cartography and MNIST are not explained inline but suit term-library tooltips. The title and the question about generalization beyond MNIST follow from the evidence. > their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. date — pass: originalPublished is 2026-05-18, matching the v1 submission date. It correctly ignores the later v3 revision of 5 Nov 2026 and the November feed date. > \[Submitted on 18 May 2026 (v1), last revised 5 Nov 2026 (this version, v3)\] support — pass: Every claim's excerpt is an exact quotation from the abstract, and the claims are source-stated. The limitations correctly note that the evidence is only the abstract and covers MNIST and a synthetic variant. The draft does not inflate the findings into real-world reliability claims. > Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the original source and the most relevant link. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/0179e71f4d5d7f5362533c8adc9e0aba7d3f62136af881944c16233ab1232488/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/pavlovic-paun-and-poesio-argue-human-soft-labels-matter-more-0179e71f.json)