Short- to Medium-Term Weather Forecast Skill of the AI-Based Pangu-Weather Model Using Automatic Weather Stations in China
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis paper evaluated Pangu's short- to medium-term weather forecasting skill over various meteorological parameters using data from over 2,000 weather stations in China. The work is generally meaningful, however, from a scientific perspective, this work is rather unremarkable. In fact, since 2023, Chinese meteorological agencies have been conducting daily evaluation of Pangu and comparing them with EC, NCEP and other numerical models. The conclusions from these comparisons are already quite evident. Therefore, I believe that this article is not yet ready for publication. Here are some specific comments:
1 Pangu cannot be initialized with ERA5 in real forecast, otherwise the comparison will be meaningless.
2 It is necessary to conduct comparative evaluations of the forecast results based on representative cases. Generally, it is believed that Pangu has certain advantages in predicting typhoon paths and large-scale circulation patterns. Therefore, it is suggested to select at least one typhoon process and one heavy rain process as representative case analysis.
3 As previously mentioned, the scientific concept underlying this article remains weak, with conclusions that are widely known. Hence, I recommend that the authors incorporate additional comparative analysis between different AI model such as FUXI or FengWu, and assess the forecasting outcomes of various models from an AI model design perspective.
Author Response
Comments1:
[- This paper evaluated Pangu's short- to medium-term weather forecasting skill over various meteorological parameters using data from over 2,000 weather stations in China. The work is generally meaningful, however, from a scientific perspective, this work is rather unremarkable. In fact, since 2023, Chinese meteorological agencies have been conducting daily evaluation of Pangu and comparing them with EC, NCEP and other numerical models. The conclusions from these comparisons are already quite evident. Therefore, I believe that this article is not yet ready for publication.]
Response1:
[Thanks a lot for your review of our study. Your professional comments are of great help in improving our manuscript. Also, we provide the point-to-point responses to your comments. We hope that our changes are adequate and are willing to make additional revisions if necessary.]
Comments2:
[- Pangu cannot be initialized with ERA5 in real forecast, otherwise the comparison will be meaningless.]
Response2:
[We agree that the ERA5 dataset is released with a delay of approximately five days, making it unsuitable for real-time short-term forecasting. However, we believe using ERA5 as the initialization field for Pangu still holds research value for the following reasons. Firstly, the high precision of ERA5 reanalysis data allows Pangu to achieve optimal accuracy in its meteorological outputs, providing a benchmark for evaluating the model’s maximum potential. While this delay renders ERA5 unsuitable for short-term forecasts, it is still highly relevant for medium-term forecasts where precision is paramount. Secondly, for short-term forecasts, although ERA5 initial fields are unavailable in real time, alternatives such as GFS initial fields, which have only a 3–5 hour delay, can be effectively utilized. Experimental results in Section 3.3 further demonstrate that forecasts initialized with GFS exhibit accuracy comparable to those using ERA5 in most scenarios. Consequently, using ERA5 not only helps explore the upper limits of Pangu’s capabilities but also offers valuable insights into its practical performance in real-time forecasting contexts.]
Comments3:
[- It is necessary to conduct comparative evaluations of the forecast results based on representative cases. Generally, it is believed that Pangu has certain advantages in predicting typhoon paths and large-scale circulation patterns. Therefore, it is suggested to select at least one typhoon process and one heavy rain process as representative case analysis.]
Response3:
[We appreciate the reviewer’s insightful suggestion regarding the inclusion of representative case analyses, such as typhoon processes and heavy rain events, to evaluate Pangu's forecasting performance. We have incorporated this recommendation and added a new Section 3.5, which focuses on assessing Pangu's forecast skill in tracking typhoon paths, using Typhoon Bebinca as a representative case study.]
Comments4:
[- As previously mentioned, the scientific concept underlying this article remains weak, with conclusions that are widely known. Hence, I recommend that the authors incorporate additional comparative analysis between different AI model such as FUXI or FengWu, and assess the forecasting outcomes of various models from an AI model design perspective.]
Response4:
[We appreciate the reviewer’s insightful suggestion to include more AI-based models in the accuracy evaluation. We agree that this is both meaningful and necessary. As a result, we have incorporated comparative analyses about FuXi and FengWu in Sections 3.2 and the newly added Section 3.5, where such comparisons are most relevant. However, we would like to emphasize that the main focus of this paper remains on the forecast skill of the Pangu model relative to traditional NWP models, as well as the impact of Pangu’s unique model architecture on its forecasting performance.]
Reviewer 2 Report
Comments and Suggestions for AuthorsOverall, this paper addresses a cutting-edge topic by investigating the performance of the AI-based model Pangu in short- to medium-term weather forecasting. The research is well-designed, with a sound methodology and data-supported conclusions. It has significant academic and practical value. The manuscript is logically structured and presents a clear narrative.
I recommend the paper for publication after minor revisions.
Below are specific suggestions for improvement:
1. The current title does not fully reflect the scope of the study, as the verification is conducted solely on surface variables observed at weather stations. To make the title more specific and accurate, consider revising it to reflect the focus on surface-level parameter validation. For example: Short- to medium-term weather forecast skill of the AI-based Pangu-Weather model using automatic weather stations in China
2. In Fig.2, change Pangu(b) to Pangu(c)
3. The color bars in Figures 3 and 4 make it challenging to distinguish between positive and negative values clearly. It is recommended to modify the color scheme to improve readability.
Author Response
Comments1:
[- Overall, this paper addresses a cutting-edge topic by investigating the performance of the AI-based model Pangu in short- to medium-term weather forecasting. The research is well-designed, with a sound methodology and data-supported conclusions. It has significant academic and practical value. The manuscript is logically structured and presents a clear narrative.]
Response1:[Thanks a lot for your professional comments and acknowledging our study. In this revision, we've done a polish to this paper. We hope that our changes are adequate and are willing to make additional revisions if necessary. Also, we provide the point-to-point responses (in red) to your comments (in black). We hope that our changes are adequate and are willing to make additional revisions if necessary.]
Comments2:
[- The current title does not fully reflect the scope of the study, as the verification is conducted solely on surface variables observed at weather stations. To make the title more specific and accurate, consider revising it to reflect the focus on surface-level parameter validation. For example: Short- to medium-term weather forecast skill of the AI-based Pangu-Weather model using automatic weather stations in China.]
Response2:
[We fully agree that the original title did not adequately capture the focus of the study, particularly its emphasis on surface-level parameter validation based on weather station observations. Following your recommendation, we have revised the title to:
“Short- to medium-term weather forecast skill of the AI-based Pangu-Weather model using automatic weather stations in China.”]
Comments3:
[- In Fig.2, change Pangu(b) to Pangu(c).]
Response3:
[Since Pangu has a higher temporal resolution (1 hour) compared to GFS and ECMWF (3 hours), it would be meaningless to evaluate accuracy if only Pangu’s forecast results are considered without corresponding GFS and ECMWF data for comparison. Therefore, this study aligns the timing and frequency of the initialization fields for Pangu with those of GFS, which has a larger dataset available. We believe this approach ensures the comparison is as fair as possible. To further address any potential misunderstanding highlighted in the review, we have updated Figure 2, changing "GFS(b) and Pangu(b)" to "GFS and Pangu(b)."]
Comments4:
[- The color bars in Figures 3 and 4 make it challenging to distinguish between positive and negative values clearly. It is recommended to modify the color scheme to improve readability.]
Response4:
[We have taken note of the issue regarding the difficulty in distinguishing between positive and negative values in the color bars of Figures 3 and 4. To address this, we have changed the color scheme from "viridis" to "RdBu," which provides a clearer contrast between positive and negative values. We believe this adjustment enhances the readability and clarity of the figures.]
Reviewer 3 Report
Comments and Suggestions for AuthorsAll comments are considered minor.
General comment 1: It would be very interesting to place a comment regarding precipitation with in the prospects of this analysis.
General comment 2: It would be very interesting to place a comment regarding the computational cost of the effort for a whole year.
General comment 3: It will be very interesting to place results for specific stations, especially for very short lead times, in order to get an idea of the value of the method regarding nowcasting.
Line 40: The words "so on" are not necessary.
Line 121: Please replace "study" with "studies".
Figure 2: Please check if the figure for Pangu is missing.
Lines 201 and 202: Please drop the dashes before the words "both" and "it".
Line 246: Please replace "Figure" with Figures".
Author Response
Comments1:
[- All comments are considered minor.]
Response1:
[Thanks a lot for your professional comments and acknowledging our study. In this revision, we've done a polish to this paper. We hope that our changes are adequate and are willing to make additional revisions if necessary. Also, we provide the point-to-point responses (in red) to your comments (in black). We hope that our changes are adequate and are willing to make additional revisions if necessary.]
Comments2:
[- It would be very interesting to place a comment regarding precipitation with in the prospects of this analysis.]
Response2:
[We have revised as suggested. The related content has been added in lines 577–583.]
Comments3:
[- It would be very interesting to place a comment regarding the computational cost of the effort for a whole year.]
Response3:
[The reviewer's suggestion is valuable. However, the authors believe that the computational cost of ensemble forecasting is more relevant to discuss than the total cost for a whole year, as forecasting is a real-time operation. We have incorporated this consideration in lines 590–593.]
Comments4:
[- It will be very interesting to place results for specific stations, especially for very short lead times, in order to get an idea of the value of the method regarding nowcasting.]
Response4:
[We appreciate the reviewer’s suggestion to include results for specific stations to evaluate the method’s value for nowcasting. While we acknowledge the potential interest in such an analysis, we believe that the accuracy metrics presented in Section 3.2, which assess Pangu’s forecast skill across different lead times, along with the overall bias distribution in Section 3.1, already provide a comprehensive reflection of the method’s value regarding nowcasting in both temporal and spatial dimensions. Therefore, we have chosen to focus on these analyses in this study.]
Comments5:
[- Line 40: The words "so on" are not necessary.]
Response5:
[ We have revised as suggested.]
Comments6:
[- Line 121: Please replace "study" with "studies".]
Response6:
[We have revised as suggested.]
Comments7:
[- Figure 2: Please check if the figure for Pangu is missing.]
Response7:
[The figure for Pangu is not missing. Since Pangu has a higher temporal resolution (1 hour) compared to GFS and ECMWF (3 hours), it would be meaningless to evaluate accuracy if only Pangu’s forecast results are considered without corresponding GFS and ECMWF data for comparison. Therefore, this study aligns the timing and frequency of the initialization fields for Pangu with those of GFS, which has a larger dataset available. We believe this approach ensures the comparison is as fair as possible. To further address any potential misunderstanding highlighted in the review, we have updated Figure 2, changing "GFS(b) and Pangu(b)" to "GFS and Pangu(b)."]
Comments8:
[- Lines 201 and 202: Please drop the dashes before the words "both" and "it".]
Response8:
[We have revised as suggested.]
Comments9:
[- Line 246: Please replace "Figure" with Figures".]
Response9:
[We have revised as suggested.]
Round 2
Reviewer 1 Report
Comments and Suggestions for Authorssee the attachment
Comments for author File: 
Comments.pdf
Author Response
Please see the attachment.
Author Response File: 
Author Response.docx
