# arXiv cs.AI: Measuring deep learning performance across architectures and tasks [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-measuring-deep-learning-performance-across-5ea67a25) Kevin L Coakley and Odd Erik Gundersen argue that deep learning performance is not a single value: variation in algorithm design and implementation can produce a distribution of results. They report 186 experiments across image classification and time series forecasting architectures, each run 100 times with different random seeds. The authors report that performance distributions are frequently non-Gaussian, particularly in time series forecasting. Mean performance did not predict robustness in image classification and was only moderately correlated with it in time series forecasting. Model size did not systematically affect robustness. In their experiments, time series models produced nearly three times more underperforming outliers than image classification models. The paper argues that average performance can miss long tails and extreme outliers, which matter when consistent results are critical. The authors say model assessment should consider spread, symmetry, and tail risk, not just the mean. The abstract does not establish whether these findings extend to other architectures or tasks. Do the reported patterns hold beyond image classification and time series forecasting? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:25:47.370Z. First observed: 2026-10-30T06:42:55.233Z. Retrieved: 2026-10-31T06:48:05.899Z. - [Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks](https://arxiv.org/abs/2610.38037) ## Limits The evidence provided is the paper's abstract; it does not include detailed experimental methods or results needed to assess how the reported patterns depend on specific architectures, datasets, or measures. The reported experiments cover image classification and time series forecasting; the abstract does not establish whether the findings extend to other tasks. ## Open question Do the reported patterns hold beyond image classification and time series forecasting? ## Source-stated Coakley and Gundersen report testing performance variation across image classification and time series forecasting architectures, repeating each experiment with different random seeds. > We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors report that performance distributions were frequently non-Gaussian, particularly in time series forecasting, and that mean performance did not predict robustness for image classification. > Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors report that time series models produced nearly three times more underperforming outliers than image classification models in their experiments. > Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors argue that mean performance alone is insufficient for model selection where consistent performance matters, and call for assessing spread, symmetry, and tail risk. > Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical. - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source and contributor context Coakley, Kevin L, Gundersen, Odd Erik · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Deep learning Machine learning that uses many layers of simple calculations; the basis of most current AI. - [Definition reference](https://en.wikipedia.org/wiki/Deep_learning) ## Time-series Observations arranged over time, such as monthly sales; forecasting uses their patterns to estimate later values. - [Definition reference](https://developers.google.com/machine-learning/glossary#time-series-analysis) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 34d1f791-588d-436e-8416-bd8acbce7122. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T08:25:47.370Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: f2ab04d1b22c9700ae534f54c250f1afb89b32f58d68638a687544eaeeadfd12. Source text hash: b4339eb1e3c332a6bb2d305d6bf607d87f35c339c3226a134c8fa870f681b412. ## Revision and source-check history ### Version 1 — 2026-11-11T08:25:47.370Z Kevin L Coakley and Odd Erik Gundersen argue that deep learning performance is not a single value: variation in algorithm design and implementation can produce a distribution of results. They report 186 experiments across image classification and time series forecasting architectures, each run 100 times with different random seeds. The authors report that performance distributions are frequently non-Gaussian, particularly in time series forecasting. Mean performance did not predict robustness in image classification and was only moderately correlated with it in time series forecasting. Model size did not systematically affect robustness. In their experiments, time series models produced nearly three times more underperforming outliers than image classification models. The paper argues that average performance can miss long tails and extreme outliers, which matter when consistent results are critical. The authors say model assessment should consider spread, symmetry, and tail risk, not just the mean. The abstract does not establish whether these findings extend to other architectures or tasks. Do the reported patterns hold beyond image classification and time series forecasting? ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:25:47.370Z. First observed: 2026-10-30T06:42:55.233Z. Retrieved: 2026-10-31T06:48:05.899Z. - [Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks](https://arxiv.org/abs/2610.38037) ## Limits The evidence provided is the paper's abstract; it does not include detailed experimental methods or results needed to assess how the reported patterns depend on specific architectures, datasets, or measures. The reported experiments cover image classification and time series forecasting; the abstract does not establish whether the findings extend to other tasks. ## Open question Do the reported patterns hold beyond image classification and time series forecasting? ## Source-stated Coakley and Gundersen report testing performance variation across image classification and time series forecasting architectures, repeating each experiment with different random seeds. > We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors report that performance distributions were frequently non-Gaussian, particularly in time series forecasting, and that mean performance did not predict robustness for image classification. > Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors report that time series models produced nearly three times more underperforming outliers than image classification models in their experiments. > Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source-stated The authors argue that mean performance alone is insufficient for model selection where consistent performance matters, and call for assessing spread, symmetry, and tail risk. > Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical. - [Supporting source](https://arxiv.org/abs/2610.38037) ## Source and contributor context Coakley, Kevin L, Gundersen, Odd Erik · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Deep learning Machine learning that uses many layers of simple calculations; the basis of most current AI. - [Definition reference](https://en.wikipedia.org/wiki/Deep_learning) ## Time-series Observations arranged over time, such as monthly sales; forecasting uses their patterns to estimate later values. - [Definition reference](https://developers.google.com/machine-learning/glossary#time-series-analysis) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 34d1f791-588d-436e-8416-bd8acbce7122. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T08:25:47.370Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: f2ab04d1b22c9700ae534f54c250f1afb89b32f58d68638a687544eaeeadfd12. Source text hash: b4339eb1e3c332a6bb2d305d6bf607d87f35c339c3226a134c8fa870f681b412. ### uncertain — 2026-11-11T08:25:47.370Z — anthropic/claude-opus-5.5 Publication: 34d1f791-588d-436e-8416-bd8acbce7122 attribution — pass: The reading correctly names Kevin L Coakley and Odd Erik Gundersen as the authors of this arXiv paper. It presents the findings and recommendations as their reports and arguments rather than as established fact. > Authors:Kevin L Coakley, Odd Erik Gundersen summary — pass: The summary is faithful and readable. It covers the method, the main results and the authors' recommendation, and the question follows from the scope limit. Small fidelity repair: the source attributes variation to non-determinism in design and implementation, but the draft says only variation. The draft also omits the early-stopping result and the 'Trustworthy AI in high-stakes applications' framing. Terms such as random seeds, non-Gaussian and tail risk should get term-library tooltips. Neither gap prevents understanding. > Non-determinism in deep learning algorithm design and implementation leads to performance variation date — pass: The originalPublished date, 2026-10-27, matches the arXiv v1 submission date. The source also lists a Sci. Rep. journal reference, but the frozen evidence gives no earlier date for it. > \[Submitted on 27 Oct 2026\] support — uncertain: Claim 2 says mean performance did not predict robustness for image classification. That is source-stated, but its supporting excerpt quotes the model-size sentence instead. Smallest repair: replace the excerpt with the quoted sentence below. All other claims, limitations and the conditional framing are supported, and no unsupported benefits or costs are implied. > Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/5ea67a254e4c9601e7a00816fd4cdee657a5604eb5a2898172875d1fa6cec709/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-measuring-deep-learning-performance-across-5ea67a25.json)