# Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings [Canonical reading](https://escapemeridian.com/readings/can-we-trust-evaluations-of-ai-weather-forecasts-for-extremes-4305c07e) The paper’s authors argue that evaluation choices can change how AI weather models compare with physics-based models on extreme weather forecasts. In a common experimental design, the considered physics-based model tends to rank best when evaluation is restricted to observed extreme events, while AI models generally do better with weighted scoring functions. The authors report that AI models underpredict extremes, so their scores are generally dominated by misses; the physics-based model forecasts extremes more frequently, and its scores are dominated by false alarms. Because different evaluation metrics penalise these errors differently, rankings can shift. The authors say score differences arise primarily from miscalibration rather than discrimination ability. Their findings offer a reason that studies may reach conflicting conclusions, and highlight the importance of evaluation metrics. The abstract does not identify the models, datasets or specific metrics used, so how these findings apply to other forecasting comparisons remains unknown. What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv physics.ao-ph · Article Original source date: Oct 22, 2026 (2026-10-22). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T05:01:20.442Z. First observed: 2026-10-23T04:01:19.247Z. Retrieved: 2026-10-24T23:01:08.894Z. - [Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings](https://arxiv.org/abs/2610.31237) ## Limits The supplied evidence is the abstract, which does not identify the models, datasets, specific evaluation metrics or definitions of extreme events used. The abstract describes results within a common experimental design; it does not establish that the reported rankings apply to other models or evaluation settings. ## Open question What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies? ## Source-stated The authors report that model rankings are highly sensitive to evaluation metrics: the considered physics-based model tends to perform best on observed extreme events, while AI models generally do better with weighted scoring functions. > model rankings are highly sensitive to the choice of evaluation metrics: the considered physics-based model tends to perform best when evaluation is restricted to observed extreme events, whereas AI models generally perform better when evaluated using weighted scoring functions. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors report that AI models underpredict extreme events, with scores generally dominated by misses, while the considered physics-based model forecasts extremes more frequently, with scores dominated by false alarms. > AI models underpredict the frequency of extremes, so that their scores are generally dominated by misses, while the physics-based model forecasts extremes more frequently and its scores are dominated by false alarms. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors say different evaluation metrics penalise misses and false alarms differently, and that score differences arise primarily from miscalibration rather than discrimination ability. > Different evaluation metrics penalise these errors differently. Further score decompositions confirm that the score differences arise primarily from miscalibration rather than discrimination ability. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors present the evaluation-metric findings as relevant to conflicting conclusions in prior studies of AI forecasts for extreme weather. > This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The abstract says AI weather models are already used operationally and perform competitively with physics-based models at a fraction of the computational cost. > AI weather models perform competitively with traditional physics-based models at a fraction of the computational cost, and they are thus already used operationally. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source and contributor context Biegert, Tobias, Allen, Sam, Zhang, Zhongwei, Lerch, Sebastian · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 765ad1fb-de9e-4fc6-8d97-13449d970c47. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T05:01:20.442Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 3be77ec02bd74f92e0999d83fc7cd8959b31ad4199594944a452183c6a388057. Source text hash: cf200c153343da15df1121b3a25acb45a7576bd96458a791a8a21633975e53ec. ## Revision and source-check history ### Version 1 — 2026-11-07T05:01:20.442Z The paper’s authors argue that evaluation choices can change how AI weather models compare with physics-based models on extreme weather forecasts. In a common experimental design, the considered physics-based model tends to rank best when evaluation is restricted to observed extreme events, while AI models generally do better with weighted scoring functions. The authors report that AI models underpredict extremes, so their scores are generally dominated by misses; the physics-based model forecasts extremes more frequently, and its scores are dominated by false alarms. Because different evaluation metrics penalise these errors differently, rankings can shift. The authors say score differences arise primarily from miscalibration rather than discrimination ability. Their findings offer a reason that studies may reach conflicting conclusions, and highlight the importance of evaluation metrics. The abstract does not identify the models, datasets or specific metrics used, so how these findings apply to other forecasting comparisons remains unknown. What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies? ## Original source arXiv physics.ao-ph · Article Original source date: Oct 22, 2026 (2026-10-22). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T05:01:20.442Z. First observed: 2026-10-23T04:01:19.247Z. Retrieved: 2026-10-24T23:01:08.894Z. - [Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings](https://arxiv.org/abs/2610.31237) ## Limits The supplied evidence is the abstract, which does not identify the models, datasets, specific evaluation metrics or definitions of extreme events used. The abstract describes results within a common experimental design; it does not establish that the reported rankings apply to other models or evaluation settings. ## Open question What rankings result when the same models are evaluated across the specific metrics and extreme-event definitions used in different studies? ## Source-stated The authors report that model rankings are highly sensitive to evaluation metrics: the considered physics-based model tends to perform best on observed extreme events, while AI models generally do better with weighted scoring functions. > model rankings are highly sensitive to the choice of evaluation metrics: the considered physics-based model tends to perform best when evaluation is restricted to observed extreme events, whereas AI models generally perform better when evaluated using weighted scoring functions. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors report that AI models underpredict extreme events, with scores generally dominated by misses, while the considered physics-based model forecasts extremes more frequently, with scores dominated by false alarms. > AI models underpredict the frequency of extremes, so that their scores are generally dominated by misses, while the physics-based model forecasts extremes more frequently and its scores are dominated by false alarms. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors say different evaluation metrics penalise misses and false alarms differently, and that score differences arise primarily from miscalibration rather than discrimination ability. > Different evaluation metrics penalise these errors differently. Further score decompositions confirm that the score differences arise primarily from miscalibration rather than discrimination ability. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The authors present the evaluation-metric findings as relevant to conflicting conclusions in prior studies of AI forecasts for extreme weather. > This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source-stated The abstract says AI weather models are already used operationally and perform competitively with physics-based models at a fraction of the computational cost. > AI weather models perform competitively with traditional physics-based models at a fraction of the computational cost, and they are thus already used operationally. - [Supporting source](https://arxiv.org/abs/2610.31237) ## Source and contributor context Biegert, Tobias, Allen, Sam, Zhang, Zhongwei, Lerch, Sebastian · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 765ad1fb-de9e-4fc6-8d97-13449d970c47. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-07T05:01:20.442Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 3be77ec02bd74f92e0999d83fc7cd8959b31ad4199594944a452183c6a388057. Source text hash: cf200c153343da15df1121b3a25acb45a7576bd96458a791a8a21633975e53ec. ### passed — 2026-11-07T05:01:20.442Z — anthropic/claude-opus-5.5 Publication: 765ad1fb-de9e-4fc6-8d97-13449d970c47 attribution — pass: Findings are consistently attributed to the paper's authors (Biegert, Allen, Zhang, Lerch). Model rankings, error patterns and the miscalibration explanation remain the authors' reported results, not established fact or EM opinion. > Authors:Tobias Biegert, Sam Allen, Zhongwei Zhang, Sebastian Lerch summary — pass: The title matches the source. The summary leads with the argument: evaluation choices change how AI and physics-based models rank on extremes. It explains the mechanism in plain terms: AI models miss more, the physics-based model gives more false alarms, and metrics penalise these errors differently. The question follows from the stated gap. Optional repairs: add tooltips for miscalibration, discrimination ability and weighted scoring functions, and mention in the summary that AI models are already used operationally. That stake currently appears only in the claims. > Different evaluation metrics penalise these errors differently. date — pass: The original publication date, 2026-10-22, matches the arXiv v1 submission date. The feed date of Oct 23 is correctly not used. > \[Submitted on 22 Oct 2026\] support — pass: Each claim closely tracks the abstract. Every claim keeps its scope: the considered physics-based model, a common experimental design, and the hedges generally and tends to. Saying the findings offer a reason studies may conflict matches the authors' sheds light framing. The limitations accurately note that the abstract does not name the models, datasets or metrics. There are no invented benefits, costs or forecasts. > This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models. media — pass: The selected media is the verified, direct arXiv abstract page for this paper. It is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/4305c07e018d08c644e31788c18372af5eaf4f15a4a4a1c18d03fdde3f80a0c8/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/can-we-trust-evaluations-of-ai-weather-forecasts-for-extremes-4305c07e.json)