Comparative Analysis of SCMOC and Models Rainstorm Forecasting Performance in Qinling Mountains and Their Surrounding Areas

Abstract

Taking CMPA (CMA Multi-source Merged Precipitation Analysis System) analysis data as a reference, the research analyzes the forecast performance of ECMWF, CMA-Meso, and SCMOC (National Meteorological Center grid precipitation forecast guidance product) in 74 rainstorm cases in 2020 and 2021 in Qinling Mountains and their surrounding areas by using the dichotomy classical verification score comprehensive diagram and the object-oriented MODE spatial verification method, based on the circulation classification in rainstorm weather. The research conclusions are as follows: (1) based on the high- and low-altitude circulation situation and focused on the direct impact system, rainstorms in the Qinling Mountains and their surrounding areas can be divided into five patterns. (2) Point-to-point verification shows that SCMOC has obvious advantages in rainstorm forecast, but the disadvantage is that the Bias is relatively high. CMA-Meso has advantages in RST (weak weather system) decentralized rainstorm forecast. (3) MODE verification shows that the number of ECMWF and SCMOC independent objects is significantly lower than that of observation, the forecast area of regional rainstorm objects of SCMOC is significantly larger, the SCMOC scattered rainstorm objects are missed, and the number of independent precipitation objects of CMA-Meso is higher than that of the other two precipitation products. (4) The forecast object area and intensity of SCMOC and observation match best in the XFC (westerly trough) circulation situation, while ECMWF has the best results for the forecast of FGXFC (subtropical high westerly trough) rainstorms.

1. Introduction

The Qinling Mountains, which combine the North and the South, are China’s central water tower, the ancestral vein of the Chinese nation, and an important symbol of Chinese culture. Ecological environment protection and natural disaster prevention and control in the Qinling Mountains have great and far-reaching significance to the Qinling Mountains region and even the whole of China. The Qinling Mountains and their surrounding areas have a complex terrain, structure, unique terrain, and frequent rainstorms in summer, which make it very easy for geological disasters such as mountain floods and debris flows to be formed. Therefore, strengthening the study of the rainstorm forecast and early warning capacity of the Qinling Mountains and their surrounding areas is particularly important for ecological environment protection, disaster prevention, and mitigation. Many scholars have studied the formation mechanism, evolution characteristics, and influence factors of rainstorms in the Qinling Mountains and their surrounding areas [1]. Bi et al. [2] elaborated the correlation between Qinling topography and precipitation. Mu et al. [3] summarized the impact of the Qinling Mountains on precipitation in local circulation formation and airflow convergence and so on. Pan et al. [4] analyzed the contribution of the intersection of cold and warm air around the Qinling Mountains to precipitation based on mesoscale methods. Wang and Ren [5] analyzed the differences in precipitation changes in the Qinling Mountains in the past 50 years. Fang et al. [6] analyzed the climate characteristics of autumn precipitation in the neighboring areas of Qinling Mountains. Other studies include the inhibitory effect of aerosols on topographic cloud precipitation in Qinling Mountains [7], interannual and chronological temperature and precipitation changes in Qinling Mountains [8], etc. These studies expounded the spatial form and circulation characteristics of heavy precipitation over the complex terrain of the Qinling Mountains and their surrounding areas, and significantly improved people’s scientific understanding of the occurrence law of heavy precipitation. However, few studies have used numerical models and objective forecast products to analyze the rainstorm forecast in Qinling Mountains and their surrounding areas.

In fact, the modern weather forecast is a secondary process and application of numerical model forecast results [9,10]. The dependence of the forecast of meteorological elements on the numerical model is higher and the modern weather forecast without a numerical model cannot be imagined. Through the fine evaluation of numerical model forecast results, obtaining forecast indexes and improving the element forecast performance are important means to improve forecast ability [11,12,13]. In terms of evaluating and verifying the forecast for precipitation, the dichotomy contingency table is the most common and universal method [14,15]. By classifying events through the contingency table, we can calculate a series of scoring indices such as the POD (probability of detection), FAR (false alarm rate), CSI (critical success index), TSS (true skill score), Bias (frequency bias), GSS or ETS (Gilbert Skill Score), OR (odds ratio), and ACC (accuracy) to describe the forecast performance of the model. The main advantage of traditional dichotomous scoring is that the verification conclusion is intuitive and easy to understand, but these scoring indices reflect only one aspect of precipitation forecast performance. For example, POD, FAR, and other indexes in some cases cannot reflect the quality of the forecast. The score of CSI for small probability events tends to climate probability. The TSS score can truly reflect the forecast capability and verify the significance of forecast skills, but for small probability events, TSS tends to POD. Although the improved scoring index HSS (Heidke Skill Score) of TSS (True Skill Score) has better performance, it is still insufficient for small probability events. It is worth noting that the comprehensive diagram of forecast score can overcome the deficiencies of a single verification index and express multiple scores in a graph, comprehensively depicting the forecast performance of different objective precipitation products from a multidimensional perspective, which can make up for the defects of a single forecast score to a certain extent.

Another major problem in the verification and evaluation of the rainstorm forecast performance is the position error of the forecast relative to the observation [16]. The contingency table precipitation verification method mainly compares the precipitation intensity of the corresponding positions of forecast and observation, so there may be double penalties due to the position shift of the precipitation forecast [17,18,19,20]. In addition, the dichotomy contingency table is a point-to-point verification, which cannot describe the error of the spatial position of precipitation. In the case of precipitation position offset, the same verification results will be obtained regardless of the offset distance, which makes the verification conclusion inconsistent with the subjective analysis of the forecaster.

MODE (Method for Object-based Diagnostic Evaluation) is a spatial diagnostic verification method developed by Davis [21] on the basis of tropical cyclone evaluation; it is a typical feature-displacement discrimination method. Based on the specified precipitation threshold and convolution radius, the MODE method performs spatial convolution on the precipitation field, just like the artificial extraction and abstraction of spatial precipitation objects by the forecasters’ subjective analysis. In some way, it can show the visual analysis ability of forecasters or researchers. By setting a precipitation threshold and a spatial convolution function, the MODE method evaluates the degree of similarity between forecast and observation objects, which can reflect the subjective discrimination results of the forecasters; thus, it has unique advantages [22].

Considering the respective characteristics of comprehensive diagram and MODE method, this paper analyzes the precipitation forecast performance in Qinling Mountains and their surrounding areas based on comprehensive graph and object-oriented MODE method. In addition, in order to identify the possible precipitation forecast performance differences under different circulation situations and summarize the model forecast rules, the article calculates overall performance for all the precipitation cases. On the other hand, using the European Center for Medium-Range Weather Forecasts ERA5 reanalysis data classifies the rainstorm processes according to the circulation situation and then verifies them, respectively, to obtain a more targeted discrimination index and provide references for the fine forecast and early warning.

2. Data and Methods

2.1. Data

The circulation situation is analyzed by ECMWF ERA5 reanalysis data and the observed precipitation is from the CMA multisource CMPA (merged precipitation analysis system) of the National Meteorological Information Center. The study has shown that this data can well represent the actual distribution of observed precipitation. Data periods range from 00:00 on 1 May 2020 to 00:00 on 30 September 2021, with hourly temporal resolution and spatial resolution of 0.05° × 0.05°. The model’s data are the ECMWF high-resolution precipitation forecast issued by the European Center for Medium-Range Weather Forecasts, CMA-Meso independently developed by China Meteorological Administration, and SCMOC (the refined grid precipitation forecast guidance product of the National Meteorological Center during the same period) every day at 12 UTC during the same period. It should be noted that the ECMWF precipitation forecast resolution is 0.125° × 0.125°, the CMA-Meso precipitation forecast data have a resolution of 0.1° × 0.1°, and the SCMOC horizontal resolution is the same as that of observation. In addition, the time resolution of ECMWF is 3 h, the time resolution of CMA-Meso and SCMOC is hour by hour, and the maximum available time range of all precipitation forecast products is 240 h. Considering the availability of the model’s forecast data, ECMWF and CMA-Meso data are truncated for 12 h, which is the model’s 12–36 h precipitation forecast. SCMOC is the objective product of the precipitation forecast in the next 24 h starting from 00 UTC every day. Furthermore, the inverse distance interpolation method is used to interpolate the model’s data in a spatial resolution consistent with the CMPA analysis data (0.05° × 0.05°) to facilitate comparison.

The study scope is the Qinling Mountains and their surrounding areas (Figure 1), with a latitude and longitude range of 31°–40° N and 103°–113° E. All times in the text are in universal time.

2.2. Methods

Based on the reanalysis data of ERA5, the circulation situation of the rainstorm weather process that occurred in Qinling Mountains and their surrounding areas from 1 May 2020 to 30 September 2021 is classified, and the classification method is forecaster discrimination analysis of the rainstorm weather process one by one.

The comprehensive diagram can depict the traditional forecast score performance of precipitation in many aspects. This article uses the comprehensive diagram to verify the forecast score of different precipitation products. The scoring indicators of the comprehensive diagram mainly include POD, SCR (success rate), Bias, and TS (threat score). The formula is as follows:

$$\mathrm{POD}=\frac{A}{A+C}$$

(1)

$$\mathrm{SCR}=\frac{A}{A+B}$$

(2)

$$\mathrm{Bias}=\frac{A+B}{A+C}$$

(3)

$$\mathrm{TS}=\frac{A}{A+B+C}$$

(4)

where A is the number of correctly forecast precipitation events exceeding the threshold, B is the number of false alarms, and C is the number of missed alarms. The threshold range of POD and SCR is 0–1, and the larger the value, the better the forecast effect. Bias and TS scores have ideal values of 1, and Bias above 1 or below 1 indicates excessive or insufficient forecast frequency.

MODE spatial verification and evaluation technique is mainly from Davis et al. [20]. Different from traditional point-to-point verification, the MODE verification method can reflect the spatial structure and scale changes of model forecasting; its verification results are more consistent with the subjective judgment results of forecasters and have more practical significance. The calculation includes four steps. ① Use convolution thresholding to identify independent precipitation objects in the forecast and observed precipitation fields. ② Compute and output the independent precipitation object attributes of the forecast and observed fields, respectively, and cluster the independent precipitation objects based on the thresholds such as centroid distance and object area to output the clustered object attributes. The discrimination method of clustered objects is such that the distance between the centroids of two independent objects is less than the sum of their scales. ③ Based on the identification of object properties, compute the similarity of the forecast and observed independent objects using fuzzy logic algorithms, and match the independent precipitation objects in the observed and forecast fields. ④ Output various properties such as the similarity of clustered matching objects.

The process used for resolving objects in a raw data field is called convolution thresholding. The raw data field is first convolved with a simple filter function as follows:

$$C\left(x,y\right)={\displaystyle \sum }_{u,v}\varnothing \left(u,v\right)f\left(x-u,y-v\right)$$

(5)

In this formula, f is the raw data field, $\varnothing$ is the filter function, and C is the resulting convolved field. The variables (x, y) and (u, v) are grid and displacement coordinates, respectively. The filter function $\varnothing$ is a simple circular filter determined by a radius of influence R and a height H.

$$\varnothing \left(x,y\right)=\left\{\begin{array}{c}H if x^2+y^2\le R^2\\ 0\hspace{1em}otherwise\end{array}\right.$$

(6)

The parameters R and H are not independent. They are related by the requirement that the integral of $\varnothing$ over the grid be unified:

$$\pi R^{2}H=1$$

(7)

Thus, the radius of influence R is the only tunable parameter in the convolution process. Once R is chosen, H is determined by the above equation. Once the convolved field C is in hand, it is thresholded to create a mask field M:

$$M\left(x,y\right)=\left\{\begin{array}{c}1 if C\left(x,y\right)\ge T\\ 0\hspace{1em}otherwise\end{array}\right.$$

(8)

where T is the precipitation threshold. The objects are the connected regions where M = 1. Finally, the raw data are restored to object interiors to obtain the object.

The similarity calculation formula for the MODE method is as follows:

$$T(a)=\frac{{\displaystyle \sum _i{w}_{i}C(a_i)I_i(a_i)}}{{\displaystyle \sum _i{w}_{i}C(a_i)}}$$

(9)

where a_i represents the ith attribute of the object, / represents the weight coefficient of the ith attribute of the object, and C(a_i) represents the confidence level of the ith attribute. I_i(a_i) represents the interest function of the ith property of the forecast field object, and is a function of a_i. Statistically, T(a) indicates the similarity between the matched forecast and observed precipitation objects. In other words, T(a) signifies whether the comprehensive attributes of the forecast object are consistent with the observed object. If the value of T(a) is larger, the two objects are more similar; that is, the better the forecasting ability.

3. Results

3.1. Rainstorm Weather Circulation Situation Classification

The rainstorm disaster level adopts the national standard of the People’s Republic of China (GB/T 33680-2017), and the rainstorm at a single station is defined as the precipitation of the station from 00 UTC on the current day to 00 UTC on the next day (or 12 UTC on the current day to 12 UTC on the next day) greater than or equal to 50 mm. In the past, in the definition of rainstorm weather processes in Qinling Mountains and their surrounding areas, the regional rainstorm is defined as the rainstorm with more than five stations at the national meteorological observation stations in the adjacent area. Since the CMPA analysis data are used in this paper, the number of observation stations is not taken as the standard, and the precipitation at any grid point in the study area exceeding 50 mm is used in the verification. However, for the classification of the rainstorm weather circulation situation, only the rainfall days with a cumulative area of more than 50 km² in the region are considered.

Considering that the rainstorm weather is a low-, middle-, and high-level coupled weather system, the rainstorm weather in Qinling Mountains and their surrounding areas is mostly related to the northward transport of warm and humid air flow around the Western Pacific subtropical high (subtropical high), especially from June to September, and the 700 hPa dynamic uplift and water vapor transportation directly affect the intensity and location of the rainstorm. Studies have shown that the generation of rainstorm weather in the Qinling Mountains and their surrounding areas is greatly related to the middle and high latitude circulation, subtropical high position, and the convergence system of the middle and low layers [5]. In fact, the analysis of climate data shows that the dominant mode of the rainstorm weather circulation background in this area is the interaction of the Western Pacific subtropical and westerly trough weather system [5]. Therefore, on the basis of the existing rainstorm classification, the rainstorm weather circulation situation is further refined by integrating 500 hPa pressure field, 700 hPa vertical velocity, and wind field.

In order to clearly show the direct effect of circulation based on the weather circulation classification of 74 rainstorm days from 2020 to 2021, typical weather systems such as westerly trough, subtropical high position, and southwest vortex are mainly considered.

From the weather system over Qinling Mountains and their surrounding areas, there are great differences in the circulation background in different seasons. The rainstorm weather from April to May is mainly affected by the westerly trough. From June to September, the rainstorm is related to the peripheral circulation system. With the northward movement of the subtropical high from June to September, most rainstorms are related to the warm and humid air flow around the subtropical high. Considering the influence degree of the subtropical high, the northern boundary of the subtropical high is one latitude distance away from or within the study area, and the circulation type is defined by the subtropical high and the direct influence system. When the distance exceeds one degree latitude, only the direct influence system is considered when defining the circulation system.

According to the CMPA precipitation data, 74 cases are selected during the study period. These cases are analyzed one by one, and four main circulation patterns of rainstorm weather in Qinling Mountains and their surrounding areas are obtained: subtropical high westerly trough (FGXFC, Figure 2a), westerly trough (XFC, Figure 2b), southwest vortex (XNW, Figure 2c), and low vortex shear (DWQB, Figure 2d). In addition, there are some rainstorm cases with no significant direct system impact, which are defined as the weak weather system type (RST). From the circulation pattern, the four weather types are obviously related to the eastward movement of the 500 hPa west wind trough. At the same time, 700 hPa has a strong upward movement and 850 hPa has a strong water vapor convergence. The rainstorm usually occurs near the high-value area of 700 hPa velocity rise. During the eastward movement of the trough, it develops and deepens into a low vortex, which is shown as a closed isobaric or cyclonic circulation in the wind field at 700 hPa or 850 hPa. The low vortex is northward in Sichuan and Chongqing and affects the area around Qinling Mountains which is a Southwest Vortex Rainstorm Weather System (Figure 2c), while the low vortex in other areas is the low vortex shear type (Figure 2d).

Detailed high- and low-altitude configurations for the selected cases are given in

Table 1. Circulation classification of rainstorm weather in Qinling and their surrounding areas.

Classification	Number of Cases	Subtropical High Location	Influence System			Rainstorm Location
			500 hPa	700 hPa	850 hPa
GXFC	25	Jianghan and southern Huanghuai	A low vortex in northern Inner Mongolia, Xinjiang	Shear line, southwest warm and humid air flow	Convergence line	South of 700 hPa shear line, rainstorm in warm area
XFC	16	Jiangnan and southern China	Westerly trough	Shear line, 700 hPa easterly flow	Convergence line	Near 700 hPa shear line
XNW	13	Jianghan Region	Trough of Qinghai Tibet developing and moving eastward	Cyclonic circulation	Southwest vortex	Central and Southern Shaanxi
DWQB	7	Eastern coastal area of Jiangnan	Westerly trough moving eastward	Cyclonic circulation	Vortex	Near 700 hPa vortex
RST	13	-	-	Shear line	Convergence line and high-energy region	Locally dispersed rainstorm

. In terms of the proportion of circulation typing, FGXFC are 25 cases, which is the most, accounting for 34%, followed by 16 cases of XFC, accounting for 22%, 13 cases of XNW and RST, accounting for 18%, and 7 cases of DWQB, accounting for 9%.

3.2. Verification of Results

3.2.1. Classical Verification

Precipitation score index obtained by the 2 × 2 contingency table: The POD, SCR, TS score and Bias can be displayed in a comprehensive precipitation score diagram to fully show the forecast ability of precipitation products. The overall forecast scores of 74 rainstorm weather cases are shown in Appendix A. Figure 2 presents a comprehensive diagram of forecast scores for different precipitation levels under different circulation classification conditions. The overall verification of all cases is shown in Figure 3a. Forecast performance of different precipitation magnitudes of the three products shows obvious differences. In light rain magnitude, the ECMWF model has high POD, low SCR, and a high TS score. Bias is significantly larger than SCMOC and CMA-Meso, and there is a high risk of a false alarm for light rain. However, for SCMOC, SCR is high, POD is low, Bias is less than 1, the TS score is low and there is a high underreporting risk. CMA-Meso Bias is the closest to the ideal value. With the increase of precipitation level, ECMWF and SCMOC Bias show an opposite trend. With increasing precipitation magnitude, ECMWF Bias decreases, and SCMOC Bias increases. In the magnitude of rainstorm and heavy rainstorm, the bias of ECMWF is slightly higher than 1, while the bias of SCMOC rainstorm and heavy rainstorm is 1.25 and 1.66, respectively. In the TS score, SCMOC performs best, which is associated with higher SCMOC Bias, and higher Bias increases the risk of vacancy, but has a positive effect on the TS score.

The forecast performance of the three FGXFC products (Figure 3b) and the whole (Figure 3a) are basically the same, but in the magnitude of rainstorm, SCMOC Bias is more significant than that of the other two products, which is similar in XFC (Figure 3c) and XNW (Figure 3d). SCMOC rainstorm POD and TS scores are the best, but the SCR is lower than that of ECMWF and the vacancy risk is higher than that of ECMWF. For the XFC, XNW, and DWQB rainstorm, the SCMOC POD and SCR forecasts are better than those of the other two products, and the TS score also has good performance, but the SCMOC Bias of XFC and XNW is slightly insufficient. Excluding the above four rainstorms with obvious influence on the system, the RST (Figure 3f) is not an obvious high-trough, medium-shear, and low-layer convergence system, and the performance of the RST rainstorm forecast is significantly different. For the rainstorm forecast frequency, SCMOC is the lowest, ECMWF is the second, CMA-Meso is the highest, and the SCMOC rainstorm TS score, SCR, and POD are lower than those of the other two products. Overall, CMA-Meso for the RST rainstorm forecast has advantages.

3.2.2. MODE Spatial Verification

Before giving the MODE spatial verification statistics, the MODE spatial verification is detailed based on one case. Figure 4 shows the actual precipitation from 00 UTC on 14 August to 00 UTC on 15 August 2020 and the precipitation forecast corresponding to the three products. According to the CMA CMPA actual precipitation display, the rainstorm area is mainly located in southern Shanxi, central Shaanxi Guanzhong, southwestern Shaanxi, and northern Sichuan. ECMWF, CMA-Meso, and SCMOC all forecast the rainstorm weather process. Intuitively, the ECMWF has the best forecast of rainstorm shape, but its location is north and west and the real distance deviation is large. CMA-Meso has a good forecast for the location of heavy precipitation in northern Sichuan and southwestern Shaanxi. The disadvantage is that the precipitation location forecast in Shanxi is in the south and the range of the heavy precipitation forecast is large. SCMOC has a good grasp of the precipitation in southwestern Shaanxi, hitting the entire rainstorm in southwestern Shaanxi, but the forecast for the rainstorm area is large, and the forecast of scattered heavy precipitation is not sufficient. The point-to-point rainstorm TS score of ECMWF is 0.05, the SCMOC score is 0.124, the CMA-Meso score is 0.048, and SCMOC performs best.

A comparison between the forecast and observation matching objects is also presented in Figure 5. In MODE verification, CMPA identifies five objects, and ECMWF, CMA-Meso, and SCMOC identify three, five, and two precipitation objects, respectively. In terms of the number of precipitation targets, CMA-Meso performs best, followed by ECMWF, and SCMOC is the worst.

Table 2. Corresponding property table of the clustered matching objects of precipitation cases. In the matching object number, “F” indicates the forecast and “O” indicates the observation.

Match Object Number		Centroid Distance (km)	Forecast/Observation Area (km2)	Forecast/Observation Intensity in the 90% Quartile (mm)	Axial Angle Deviation (°)	T(a)
ECMWF	F1-O1	21.99	6/8	51.71/58.71	21.99	0.458
	F2-O4	19.75	301/26	58.32/74.54	19.75	0.806
	F3-O5	18.38	580/295	94.26/84.27	18.38	0.707
CMA-Meso	F1-O1	5.91	126/36	60.95/72.96	3.42	0.5528
	F2-O2	3.83	44/1	64.34/50.14	72.40	0.3484
	F3-O3	19.25	702/17	107.12/78.85	31.27	0.3427
	F4-O4	11.81	22/268	56.01/74.54	12.78	0.4728
	F5-O5	9.11	781/295	106.98/84.27	2.21	0.680
SCMOC	F1-O5	14.49	1414/237	76.02/86.10	5.78	0.660
	F2-O4	20.29	217/277	52.75/76.31	13.69	0.764

shows the matching relationship and the comparison of attributes between the three product identification objects and the observation identification objects. It can be seen that ECMWF forecast object 2 and observation object 4, and forecast object 3 and observation object 5 have a good match. The precipitation intensity and area are basically consistent with the observation, and the object similarity is more than 0.7. However, the deviation from the ECMWF precipitation forecast location is large, with an average of more than 20 km, and there is also a large axial angle deviation. The forecast area and intensity of the CMA-Meso rainstorm objects are generally large, with the advantages of small precipitation position deviation and good forecast of dispersed precipitation, and the disadvantage of low similarity of observation and forecast objects. The MMI (Median Maximum Interest) performance of ECMWF is the best, and the maximum similarity of five matching objects is 0.68. The overall similarity of the SCMOC forecast object and the observation object is high. The disadvantage is that the forecast object area is too large, and the scattered small area object is omitted, which can avoid the double punishment caused by spatial position error in the point-to-point classical verification, but needs special attention in business applications.

The statistical results of independent object recognition of forecast and observation during the study period are shown in

Table 3. Comparison of rainstorm independent object attributes identified by MODE methods of observation and the three precipitation products.

Precipitation Products	Number of Objects	Object Area (Number of Grid Points)			Intensity (mm)
		Minimum	Maximum	Average	50% Quantile	75% Quantile	Maximum
Observed	1377	1	6044	68.39	57.30	62.46	242.87
ECMWF	427	1	5739	229.37	58.2	64.52	245.96
CMA-Meso	2214	1	6232	51.87	57.0	60.82	238.43
SCMOC	103	3	5129	1181.67	50.52	65.68	305.92

. It can be seen that 1377 rainstorm objects for the observation are identified, the number of ECMWF and SCMOC rainstorm objects is significantly lower, and CMA-Meso is significantly higher. In terms of the object area, the minimum rainstorm area of ECMWF and CMA-Meso is 1 grid point, which is consistent with the observations, while the minimum rainstorm area of SCMOC is 3 grid points; that is, the rainstorm object SCMOC with less than 3 grid points is not forecasted. The maximum area of the observed object is 6044 grid points, and the CMA-Meso is 6232 grid points, which is the most consistent. The maximum area of ECMWF and SCMOC is 5739 and 5129 grid points, respectively, which is smaller than the observation area. However, in terms of the average area, ECMWF and SCMOC are significantly larger than the observation area; in particular, the average area of SCMOC rainstorm objects is 17 times that of the observation area. The reason is that the number of scattered small area objects predicted by SCMOC is relatively small, while the range of regional large-scale rainstorm weather forecasts is relatively large, which is also in good agreement with the case analysis in Figure 2. In terms of rainstorm intensity, the CMA-Meso rainstorm object intensity matches the observations best, followed by ECMWF. The main characteristics of SCMOC are that the 50% quantile intensity is low, the maximum intensity is high, and the dispersion of rainstorm intensity is larger than that of observation.

The comparison between the number of independent object identifications and object areas of the three precipitation products and observations are shown in Figure 5. In order to make it clear, logarithmic coordinates are used. For the observations, the number of 102 km²–10³ km² objects is the most, and then as the area increases, the number of identified objects drops rapidly. The CMA-Meso trend is consistent with the observations, but the number of objects in the small area is much higher than those of the observations. Around 1000 km², the number of the identified objects is basically the same.

The change trend of the ECMWF and SCMOC object recognition quantity is relatively consistent. The number of precipitation objects in 10³ km²–104 km² is the largest. Compared with the observation, both of them show fewer small area objects and more large area objects. In particular, SCMOC has no forecast ability for precipitation objects below 100 km². From the perspective of object area and intensity quantile, the 50% intensity quantile dispersion of observed objects is large, and ECMWF and SCMOC objects have large area and small intensity. The dispersion of CMA-Meso objects is slightly larger than that of the other two products, but is still significantly smaller than the observation. In addition, CMA-Meso excessively predicts precipitation objects with an area less than three grid points, with a large number of vacancies. The return function of the matching object is greater than 0.6, indicating that the forecast and observation object have good similarity. ECMWF matches 103 objects with observation, and the return function of 71% matching objects exceeds 0.6. SCMOC has 91 matching objects and the return function over 0.6 reaches 80%. From the perspective of MMI, the MMI of ECMWF, SCMOC, and CMA-Meso is 0.662, 0.74, and 0.53, respectively. The rainstorm objects predicted by SCMOC and observations have good consistency, which can be accepted. At the same time, SCMOC and ECMWF have small centroid distances for forecast and observations and perform well in spatial locations. In contrast, only 80 of the 251 CMA-Meso matching objects have a return function of more than 0.6, with a ratio of only 32%, and the reliability is relatively low. The total area and overlapping area of matching objects also have similar performance. The forecast overlapping area of ECMWF and SCMOC is large; the overlapping degree of CMA-Meso is small. The overlapping area and total area of some large area precipitation objects of ECMWF and SCMOC are basically the same, indicating that the regional precipitation forecast ability of the two products is good.

Under different circulation conditions, the number of independent observation and prediction objects is consistent with the statistical results of all cases (Figure 6a). SCMOC forecast objects are significantly less, and CMA-Meso forecast objects are significantly more, which also leads to the largest total number of CMA-Meso matching objects. From the perspective of intensity (Figure 6b), the average forecast intensity of a single object of SCMOC is significantly higher, which may be related to the small number of SCMOC small area objects. ECMWF is slightly stronger than the observation under XFC and XNW circulation, and is basically consistent with the observation under other situations. The intensity of the matching combination object is only strong for SCMOC in XNW circulation situation, and the same situation is true in the ECMWF forecast, which is weaker than observation in other circulation situations. Whether independent or clustered objects, CMA-Meso forecast intensity and observation match best, but the problem with CMA-Meso is that the intensity of the objects matched with the observation is generally low. For combined objects with an average intensity of more than 65 mm, CMA-Meso does not match. The analysis shows that CMA-Meso has large position deviation under different circulation situations, especially for some precipitation objects with large intensity and area, which leads to the inability of prediction and observation to match or the low similarity of matching objects within the object centroid distance threshold. From the area point of comparison (Figure 6c), ECMWF has the best matching between forecast and observation under FGXFC circulation. However, on the whole, the area of the forecast object is larger than that of the observation, especially SCMOC.

Matching combination object is only strong for SCMOC in the XNW circulation situation, and the same situation is true in the ECMWF forecast, which is weaker than observation in other circulation situations. Whether independent or combined objects, CMA-Meso forecast intensity and observation match best, but the problem with CMA-Meso is that the intensity of the objects matched with the observation is generally low. For clustered objects with an average intensity of more than 65 mm, CMA-Meso does not match. The analysis shows that CMA-Meso has large position deviation under different circulation situations, especially for some precipitation objects with large intensity and area, which leads to the inability of prediction and observation to match or low similarity of matching objects within the object centroid distance threshold. From the area point of comparison, ECMWF has the best matching between forecast and observation under FGXFC circulation, and the circulation situation with the best matching of SCMOC is XFC. However, on the whole, the area of the forecast object is larger than that of the observation, especially SCMOC.

Figure 7 shows the position forecast errors of the three precipitation products in different circulation situations. To facilitate the analysis, only the spatial position of the clustered matching object is considered, while the centroid distance is decomposed into meridional errors and zonal errors. Overall, the SCMOC spatial position error is the smallest, followed by ECMWF. Under the three circulation conditions of FGXFC, XFC, and DWQB, the spatial position error regularity of ECMWF and SCMOC is obvious, the zonal error is less than 1°, the maximum meridional error is 1.5°, and the model zonal forecast error is less than the meridional error. SCMOC has a very good spatial position forecast of FGXFC, XFC, and DWQB rainstorm. The meridional and zonal errors of most objects are less than 0.5°. The disadvantage is that the SCMOC forecast objects of FGXFC rainstorm are significantly less, and there may be obvious underreporting. ECMWF forecast position for FGXFC and XFC rainstorm is obviously west and north, and the position error for DWQB rainstorm is mainly west. Relatively speaking, the regularity of the position error distribution of XNW rainstorm is lower than that of the other three circulation situations, which also exists in SCMOC. From the meridional and zonal error distribution of CMA-Meso, the errors under the four circulation situations are higher than those of SCMOC and ECMWF. Based on the comprehensive strength, area, and spatial location, SCMOC has good spatial location forecast of FGXFC, XFC, and DWQB rainstorm, but there is a risk of missing rainstorm objects. The general location of ECMWF rainstorm forecast is west and north, with strong regularity, and has good space for improving business applications.

4. Conclusions and Discussion

Based on the 74 rainstorm cases in the Qinling Mountains and their surrounding areas in 2020 and 2021, using ERA5 reanalysis data, on the basis of subjective rainstorm circulation classification, the dichotomy classical verification comprehensive diagram and object-oriented MODE methods are used to analyze the ECMWF, SCMOC, and CMA-Meso rainstorm forecast performance. The main conclusions are as follows:

(1)

The overall rainstorm circulation situation in Qinling and their surrounding areas can be divided into FGXFC, XFC, XNW, DWQB, and RST. FGXFC rainstorm is the most important circulation situation in Qinling and their surrounding areas, accounting for 34%, and the other four account for 22%, 18%, 9%, and 18%, respectively.

(2)

Comprehensive graph verification shows that SCMOC rainstorm has a high TS score and POD, which has obvious advantages over ECMWF and CMA-Meso, but SCMOC rainstorm Bias is slightly larger. In different rainstorm circulation situations, POD of SCMOC for FGXFC rainstorm is the highest, SCR is slightly lower than that of ECMWF, and the missed risk is greater than that of ECMWF. POD, SCR, and TS scores of SCMOC for XFC, XNW, and DWQB rainstorm are consistently better than the other two products. However, for RST rainstorm, POD and SCR of SCMOC are low and its forecast ability is weak.

(3)

The MODE verification shows that the number of ECMWF and SCMOC independent objects is significantly less than that of observation, and the CMA-Meso is significantly more. The forecast area of SCMOC regional rainstorm objects is significantly larger, scattered rainstorm objects are not reported, and CMA-Meso independent object forecasts are more. The median maximum interest of the cluster objects ECMWF, SCMOC, and CMA-Meso is 0.662, 0.74, and 0.53, respectively, and SCMOC performs the best. However, the area of ECMWF and SCMOC forecast objects is larger than that of observation, especially SCMOC. CMA-Meso rainstorm object has the best intensity match with the observation, followed by ECMWF. The main characteristics of SCMOC are that the 50% quantile intensity is low, the maximum intensity is high, and the dispersion of rainstorm intensity is larger than that of observation.

(4)

Overall, the SCMOC spatial position error is the smallest, followed by ECMWF. The rainstorm forecast position of FGXFC and XFC of ECMWF is obviously west and north, and the rainstorm position error of DWQB is mainly west. Relatively speaking, the regularity of the XNW rainstorm position error distribution is lower than that of the other three circulation situations. The CMA-Meso longitude and latitude position errors are higher than those of SCMOC and ECMWF in all four circulation situations. SCMOC has good spatial location forecast of FGXFC, XFC, and DWQB rainstorm, but it has the risk of missing small rainstorm objects. The position of the ECMWF rainstorm forecast is relatively west and north of the observation.

Two methods are used to analyze the performance of rainstorm forecasting over Qinling and its surrounding areas. Generally speaking, both methods have their own advantages and disadvantages. The traditional verification method evaluates the forecasting performance of the model precipitation through point-to-point comparative analysis. There are two main shortcomings: one is that dealing with the mismatched events of forecasting and observation in the same way cannot distinguish wrong categories; the other is that the verification results are easily affected by some small phase errors in the high-resolution model and cannot reflect the scale changes of the spatial structure and real forecast ability of the model. However, the physical meaning of the skill score index provided by the traditional verification method is clear and easy for researchers to understand and apply. The reasonable application of the traditional verification method still has good significance for understanding the forecastability of the high-resolution model. MODE can not only evaluate the traditional skill score of the model forecasting, but can also analyze the spatial matching of observation and prediction objects and calculate the overall performance of the model forecasting performance. In addition, the unique advantage of the MODE method is that it can analyze the spatial performance of precipitation forecasting and observation objects in attributes such as area, intensity, centroid distance, displacement distance, and so on; it provides users with various means to understand the forecasting performance of the model, but it is still difficult to improve the actual forecasting ability by using the verification results of the MODE method. In the application, it is suggested to comprehensively apply the two methods according to the needs of users, which may be able to better solve practical problems.