Biegert, Tobias, Allen, Sam, Zhang, Zhongwei, Lerch, Sebastian · Oct 22, 2026
Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
Markdown · JSON · Original source
The paper’s authors argue that evaluation choices can change how AI weather models compare with physics-based models on extreme weather forecasts. In a common experimental design, the considered physics-based model tends to rank best when evaluation is restricted to observed extreme events, while AI models generally do better with weighted scoring functions. The authors report that AI models underpredict extremes, so their scores are generally dominated by misses; the physics-based model forecasts extremes more frequently, and its scores are dominated by false alarms. Because different evaluation metrics penalise these errors differently, rankings can shift. The authors say score differences arise primarily from miscalibration rather than discrimination ability. Their findings offer a reason that studies may reach conflicting conclusions, and highlight the importance of evaluation metrics. The abstract does not identify the models, datasets or specific metrics used, so how these findings apply to other forecasting comparisons remains unknown. What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
physics.ao-ph · Article
Source details
Original source date: Oct 22, 2026 (2026-10-22).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T05:01:20.442Z. First observed: 2026-10-23T04:01:19.247Z. Retrieved: 2026-10-24T23:01:08.894Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract, which does not identify the models, datasets, specific evaluation metrics or definitions of extreme events used. The abstract describes results within a common experimental design; it does not establish that the reported rankings apply to other models or evaluation settings.
Open question
What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies?
Source-stated
The authors report that model rankings are highly sensitive to evaluation metrics: the considered physics-based model tends to perform best on observed extreme events, while AI models generally do better with weighted scoring functions.
model rankings are highly sensitive to the choice of evaluation metrics: the considered physics-based model tends to perform best when evaluation is restricted to observed extreme events, whereas AI models generally perform better when evaluated using weighted scoring functions.
Source-stated
The authors report that AI models underpredict extreme events, with scores generally dominated by misses, while the considered physics-based model forecasts extremes more frequently, with scores dominated by false alarms.
AI models underpredict the frequency of extremes, so that their scores are generally dominated by misses, while the physics-based model forecasts extremes more frequently and its scores are dominated by false alarms.
Source-stated
The authors say different evaluation metrics penalise misses and false alarms differently, and that score differences arise primarily from miscalibration rather than discrimination ability.
Different evaluation metrics penalise these errors differently. Further score decompositions confirm that the score differences arise primarily from miscalibration rather than discrimination ability.
Source-stated
The authors present the evaluation-metric findings as relevant to conflicting conclusions in prior studies of AI forecasts for extreme weather.
This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models.
Source-stated
The abstract says AI weather models are already used operationally and perform competitively with physics-based models at a fraction of the computational cost.
AI weather models perform competitively with traditional physics-based models at a fraction of the computational cost, and they are thus already used operationally.
Source and contributor context
Biegert, Tobias, Allen, Sam, Zhang, Zhongwei, Lerch, Sebastian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 765ad1fb-de9e-4fc6-8d97-13449d970c47.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T05:01:20.442Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 3be77ec02bd74f92e0999d83fc7cd8959b31ad4199594944a452183c6a388057. Source text hash: cf200c153343da15df1121b3a25acb45a7576bd96458a791a8a21633975e53ec.
Revision and source-check history
Version 1 · 2026-11-07T05:01:20.442Z
Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
The paper’s authors argue that evaluation choices can change how AI weather models compare with physics-based models on extreme weather forecasts. In a common experimental design, the considered physics-based model tends to rank best when evaluation is restricted to observed extreme events, while AI models generally do better with weighted scoring functions. The authors report that AI models underpredict extremes, so their scores are generally dominated by misses; the physics-based model forecasts extremes more frequently, and its scores are dominated by false alarms. Because different evaluation metrics penalise these errors differently, rankings can shift. The authors say score differences arise primarily from miscalibration rather than discrimination ability. Their findings offer a reason that studies may reach conflicting conclusions, and highlight the importance of evaluation metrics. The abstract does not identify the models, datasets or specific metrics used, so how these findings apply to other forecasting comparisons remains unknown. What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies?
Original source
physics.ao-ph · Article
Source details
Original source date: Oct 22, 2026 (2026-10-22).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T05:01:20.442Z. First observed: 2026-10-23T04:01:19.247Z. Retrieved: 2026-10-24T23:01:08.894Z.
Limits
The supplied evidence is the abstract, which does not identify the models, datasets, specific evaluation metrics or definitions of extreme events used. The abstract describes results within a common experimental design; it does not establish that the reported rankings apply to other models or evaluation settings.
Open question
What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies?
Source-stated
The authors report that model rankings are highly sensitive to evaluation metrics: the considered physics-based model tends to perform best on observed extreme events, while AI models generally do better with weighted scoring functions.
model rankings are highly sensitive to the choice of evaluation metrics: the considered physics-based model tends to perform best when evaluation is restricted to observed extreme events, whereas AI models generally perform better when evaluated using weighted scoring functions.
Source-stated
The authors report that AI models underpredict extreme events, with scores generally dominated by misses, while the considered physics-based model forecasts extremes more frequently, with scores dominated by false alarms.
AI models underpredict the frequency of extremes, so that their scores are generally dominated by misses, while the physics-based model forecasts extremes more frequently and its scores are dominated by false alarms.
Source-stated
The authors say different evaluation metrics penalise misses and false alarms differently, and that score differences arise primarily from miscalibration rather than discrimination ability.
Different evaluation metrics penalise these errors differently. Further score decompositions confirm that the score differences arise primarily from miscalibration rather than discrimination ability.
Source-stated
The authors present the evaluation-metric findings as relevant to conflicting conclusions in prior studies of AI forecasts for extreme weather.
This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models.
Source-stated
The abstract says AI weather models are already used operationally and perform competitively with physics-based models at a fraction of the computational cost.
AI weather models perform competitively with traditional physics-based models at a fraction of the computational cost, and they are thus already used operationally.
Source and contributor context
Biegert, Tobias, Allen, Sam, Zhang, Zhongwei, Lerch, Sebastian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 765ad1fb-de9e-4fc6-8d97-13449d970c47.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T05:01:20.442Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 3be77ec02bd74f92e0999d83fc7cd8959b31ad4199594944a452183c6a388057. Source text hash: cf200c153343da15df1121b3a25acb45a7576bd96458a791a8a21633975e53ec.
passed · 2026-11-07T05:01:20.442Z · anthropic/claude-opus-5.5
Publication: 765ad1fb-de9e-4fc6-8d97-13449d970c47
attribution — pass: Findings are consistently attributed to the paper's authors (Biegert, Allen, Zhang, Lerch). Model rankings, error patterns and the miscalibration explanation remain the authors' reported results, not established fact or EM opinion.
Authors:Tobias Biegert, Sam Allen, Zhongwei Zhang, Sebastian Lerch
summary — pass: The title matches the source. The summary leads with the argument: evaluation choices change how AI and physics-based models rank on extremes. It explains the mechanism in plain terms: AI models miss more, the physics-based model gives more false alarms, and metrics penalise these errors differently. The question follows from the stated gap. Optional repairs: add tooltips for miscalibration, discrimination ability and weighted scoring functions, and mention in the summary that AI models are already used operationally. That stake currently appears only in the claims.
Different evaluation metrics penalise these errors differently.
date — pass: The original publication date, 2026-10-22, matches the arXiv v1 submission date. The feed date of Oct 23 is correctly not used.
[Submitted on 22 Oct 2026]
support — pass: Each claim closely tracks the abstract. Every claim keeps its scope: the considered physics-based model, a common experimental design, and the hedges generally and tends to. Saying the findings offer a reason studies may conflict matches the authors' sheds light framing. The limitations accurately note that the abstract does not name the models, datasets or metrics. There are no invented benefits, costs or forecasts.
This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models.
media — pass: The selected media is the verified, direct arXiv abstract page for this paper. It is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv