Fortuna, Carolina, Bertalanic, Blaz · Nov 12, 2026
Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation
Markdown · JSON · Original source
Authors Carolina Fortuna and Blaz Bertalanic report that, in the tested Fermi estimation tasks, combining answers from multiple language models delivered little improvement because the models shared item-level error: a common error on a particular question. The paper tests the usual averaging argument that independent, unbiased errors shrink as the number of answers grows. Across 2.51M answers from 15 , teammates’ errors were uncorrelated given the item, but the item-level component accounted for a median 0.906 of error variance over items. Teams of thirty reduced a single agent’s error by a median 1.7%, while draws unbiased on each item would have reduced it by 74–79%. The authors report that three rounds of debate reduced within-team dispersion but left the item-level component unchanged; revision without peers achieved the same small gain. Mixing five models of similar accuracy reduced error relative to their homogeneous teams, but no mixture outperformed the most accurate model. The authors say the findings are conditional on the models, , prompts and settings tested.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:32:02.541Z. First observed: 2026-11-13T07:39:33.241Z. Retrieved: 2026-11-13T11:17:41.478Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported findings are conditional on the models, , prompts and settings tested; the evidence does not establish whether they generalize beyond those conditions.
Open question
Do the reported limits on aggregation persist with models, , prompts or settings beyond those tested?
Source-stated
The authors report that teams of thirty reduced a single agent’s error by a median 1.7%, while draws unbiased on each item would have reduced it by 74–79%.
Teams of thirty therefore reduce a single agent's error by a median 1.7% (at most 5.4%), whereas draws that were also unbiased on each item would reduce it by 74-79%.
Source-stated
In the tested answers, the item-level component accounted for a median 0.906 of error variance over items.
the item-level component accounts for a median 0.906 of the error variance over items.
Source-stated
The authors report that debate reduced within-team dispersion but did not change the item-level component, and that revision without peers achieved the same small gain.
Three rounds of debate reduce within-team dispersion by 93% and leave the item-level component unchanged; revision without peers attains the same small gain.
Source-stated
The authors report that mixing five models of similar accuracy reduced error relative to their homogeneous teams, but did not outperform the most accurate model.
Mixing five models of similar accuracy reduces error by 6.6% relative to their own homogeneous teams, but no mixture improves on the most accurate model.
Source-stated
The paper’s findings are conditional on the evaluated models, benchmarks, prompts and settings.
The results are conditional on the models, benchmarks, prompts and settings tested.
Source and contributor context
Fortuna, Carolina, Bertalanic, Blaz · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 0a1560c8-e133-4d95-80f0-68198afd0691.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:32:02.541Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 106e022c543d79cebed3b6349890288e1456987702dac9ab025207b5a4e2e066. Source text hash: e68fb33600a9d002c3d227bc98cb2f9b872f586da90e5f17a3bc20f9987a5630.
Revision and source-check history
Version 1 · 2026-11-13T11:32:02.541Z
Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation
Authors Carolina Fortuna and Blaz Bertalanic report that, in the tested Fermi estimation tasks, combining answers from multiple language models delivered little improvement because the models shared item-level error: a common error on a particular question. The paper tests the usual averaging argument that independent, unbiased errors shrink as the number of answers grows. Across 2.51M answers from 15 , teammates’ errors were uncorrelated given the item, but the item-level component accounted for a median 0.906 of error variance over items. Teams of thirty reduced a single agent’s error by a median 1.7%, while draws unbiased on each item would have reduced it by 74–79%. The authors report that three rounds of debate reduced within-team dispersion but left the item-level component unchanged; revision without peers achieved the same small gain. Mixing five models of similar accuracy reduced error relative to their homogeneous teams, but no mixture outperformed the most accurate model. The authors say the findings are conditional on the models, , prompts and settings tested.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:32:02.541Z. First observed: 2026-11-13T07:39:33.241Z. Retrieved: 2026-11-13T11:17:41.478Z.
Limits
The reported findings are conditional on the models, , prompts and settings tested; the evidence does not establish whether they generalize beyond those conditions.
Open question
Do the reported limits on aggregation persist with models, , prompts or settings beyond those tested?
Source-stated
The authors report that teams of thirty reduced a single agent’s error by a median 1.7%, while draws unbiased on each item would have reduced it by 74–79%.
Teams of thirty therefore reduce a single agent's error by a median 1.7% (at most 5.4%), whereas draws that were also unbiased on each item would reduce it by 74-79%.
Source-stated
In the tested answers, the item-level component accounted for a median 0.906 of error variance over items.
the item-level component accounts for a median 0.906 of the error variance over items.
Source-stated
The authors report that debate reduced within-team dispersion but did not change the item-level component, and that revision without peers achieved the same small gain.
Three rounds of debate reduce within-team dispersion by 93% and leave the item-level component unchanged; revision without peers attains the same small gain.
Source-stated
The authors report that mixing five models of similar accuracy reduced error relative to their homogeneous teams, but did not outperform the most accurate model.
Mixing five models of similar accuracy reduces error by 6.6% relative to their own homogeneous teams, but no mixture improves on the most accurate model.
Source-stated
The paper’s findings are conditional on the evaluated models, benchmarks, prompts and settings.
The results are conditional on the models, benchmarks, prompts and settings tested.
Source and contributor context
Fortuna, Carolina, Bertalanic, Blaz · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 0a1560c8-e133-4d95-80f0-68198afd0691.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:32:02.541Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 106e022c543d79cebed3b6349890288e1456987702dac9ab025207b5a4e2e066. Source text hash: e68fb33600a9d002c3d227bc98cb2f9b872f586da90e5f17a3bc20f9987a5630.
uncertain · 2026-11-13T11:32:02.541Z · anthropic/claude-opus-5.5
Publication: 0a1560c8-e133-4d95-80f0-68198afd0691
attribution — pass: The reading correctly credits the findings to Carolina Fortuna and Blaz Bertalanic and keeps results framed as what the authors report. The work is identified as an arXiv paper.
Authors:Carolina Fortuna, Blaz Bertalanic
summary — uncertain: Mostly faithful, but the central term Fermi estimation is never explained (rough order-of-magnitude guesses at quantities). Add a term-library tooltip or a short gloss. Open-weight also needs a tooltip. The lead, combining answers from multiple language models, blurs the design: teams were mainly repeated draws from one model, and the shared error is each model's mean error on an item. Smallest repair: say teams of answers from the same model, with mixed-model teams tested separately. The summary could also note that multi-agent systems are often justified by this averaging argument; that is what makes the result matter.
We decompose the log10 error of each answer into the mean error of its model on the item, common to all its draws on the item, and a residual
date — pass: originalPublished 2026-11-12 matches the v1 submission date. The feed date of 2026-11-13 is correctly not used.
[Submitted on 12 Nov 2026]
support — pass: All figures (2.51M answers, 15 models, 0.906, 1.7%, 74-79%, debate, mixing) and the conditional limitation match the abstract verbatim. The question follows from the authors' own stated scope limit. No unsupported benefits or costs are implied.
The results are conditional on the models, benchmarks, prompts and settings tested.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference