Regional Climate Models Validation for Agroclimatology in Romania

Abstract

Regional climate projections are widely used in impact studies such as adaptations in agronomy. The big challenge of the climate modeling community is to serve valuable instructions regarding the reliability of these simulations to encourage agronomists to use this kind of information properly. The study validates 15 high-resolution ensembles from the Coordinated Regional Climate Downscaling Experiment-European Domain (EURO-CORDEX) for maximum temperature, minimum temperature, and precipitation to fulfill this task. Three evaluation metrics are calculated (mean absolute error, root mean square error, and correlation) for the means and the 5th and 95th percentiles. The analyses are elaborated for annual and monthly means and the vegetation periods of maize and winter wheat. Only arable lands are considered to exclude the effects of the topography. Furthermore, an ensemble selection is applied based on the evaluation metrics to reduce the data use. The five models with the best performance in the case of winter wheat are CNRM-CM5-CLMcom-CCLM4-8-17_v1, MOHC-HadGEM2-ES-IPSL-WRF381P_v1, MOHC-HadGEM2-ES-KNMI-RACMO22E_v2, MOHC-HadGEM2-ES-CLMcom-CCLM4-8-17_v1, and MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1. In the case of the vegetation period of maize, the models with the best skills are MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1, CNRM-CM5-IPSL-WRF381P_v2, MPI-M-MPI-ESM-LR-SMHI-RCA4_v1a, MOHC-HadGEM2-ES-IPSL-WRF381P_v1, and MOHC-HadGEM2-ES-KNMI-RACMO22E_v2. Quantifying the errors in climate simulations against observations and elaborating a selection procedure, we developed a consistent ensemble of high time and space resolution climate projections for agricultural use in Romania.

1. Introduction

The IPCC has reported that the observed climate change is already affecting agriculture and the food security sector through increasing temperatures, changing precipitation patterns, and greater frequency of extreme weather events [1]. Furthermore, at the European level, in the Post-2020 Common Agriculture Policy, three of the policy’s nine “specific objectives” will focus on the environment and climate, and one of them aims to contribute to climate change mitigation and adaptation [2,3,4]. Studies on the assessment of the climate change impact on land suitability require high-quality climate information.

In current climate research, climate modeling represents the primary tool for analyzing the climate system’s response to natural and anthropogenic forcing, and thus for projecting the future climate. Global Climate Models (GCMs) provide reliable information on global and large regional scales, covering a vast and very different landscape with potential for various climate-correlated extreme events. However, the horizontal resolution of GCMs, which typically ranges from 250 to 600 km, limits the possibility of addressing smaller-scale processes. Regional Climate Models (RCMs), with a higher spatial resolution (up to 11 km) over a limited area driven by GCMs, provide more appropriate climate information on such smaller scales. Therefore, the impact and adaptation measures can be more reliable. In time, the use of RCMs has grown, their resolution has increased, documentation has developed, new components have been added, and various ensemble experiments have become more widespread [5]. The EURO-CORDEX project is a coordinated initiative to produce a more robust assessment of climate change projections over the European region [6].

Global and regional climate model projections present significant uncertainties, primarily due to issues of scale resolution, leading to incoherencies among the models [7]. For this reason, many studies use the outputs of multiple future climate projections to increase confidence [7,8,9,10]. This approach leads to large data sets to be processed and analyzed. Based on the data availability, resolution, and confidence in the model’s performance, some scholars choose to consider only one model or a small number of models in their analysis [11,12,13,14]. Under these circumstances, our paper aims to select a reasonable number of ensembles, but still keep the most important climate characteristics to estimate further agro-climatological conditions for impact studies. However, numerous studies have focused on validating EURO-CORDEX against observational time series [15,16,17,18,19,20]. Beyond these works, in this study, the skills of the regional climate models are quantified not only for the widely used periods (e.g., annual, seasonal) but for vegetation periods of two cereal crops.

This paper is part of a larger project, AGROCLIMRO (www.agroclim.ro, accessed on 1 July 2021), aiming at re-zoning the agroclimatic suitability areas in Romania for the two most-extended crops in the country, maize and winter wheat. It is in line with the Common Agricultural Policy and European Union Adaptation Strategy that suggests that each member state delineates the suitability for crop growth. At the same time, this topic fits very well to flagship question 1 of the Pannonian Basin Experiment Project (PannEx) [21].

This paper aims to present the methodology and results for choosing among many RCMs running under different driving models which best fit the specified crops in the focus area. However, the results are valid for Romania and agricultural land in the entire Pannonian Basin.

2. Materials and Methods

2.1. Data Used

Temperature and precipitation are the essential variables for different crops, especially for maize and winter wheat, the dominant ones in Romania. Thus, for validation we considered extreme daily temperatures (maximum and minimum) and daily amount of precipitation in this study.

The analysis was performed based on gridded data at the best spatial resolution available at this moment, 0.1° latitude × 0.1° longitude (~11 × 11 km).

As a reference, we used data from the national ROCADA gridded database developed by the Romanian National Meteorological Administration network [22]. It was chosen as it seems to return the smallest errors for average values against observations compared to other databases at the same spatial resolution covering the focus region, such E-OBS [23,24] or CarpatClim [25,26]. The ROCADA database was derived by interpolating 155 observation points (weather stations) for air temperature and 188 points for precipitation, compared to no more than 31 used for the other databases. Its original datasets are freely available online on PANGAEA [27]. Romania’s territory is covered by 3046 pixels (Figure 1) and has a temporal extension over a 53-year period (1961–2013). For this study, we considered the common period with historical data available of the RCMs: 1971–2000.

In terms of modeled data, we used the high-resolution EURO-CORDEX [28] simulations, similar to the observation-derived gridded data. First, we identified all RCMs having daily minimum and maximum air temperature outputs and daily precipitation for each historical experiment and RCP 4.5 and RCP 8.5 (the latter ones are not included in the present study). Since we identified 15 combinations of driving experiments and RCMs (

Table 1. Modeled data features.

Domain	EUR-11
EURO-CORDEX Simulation	Driving Model	Regional Climate Model	No.
	CNRM-CM5	CLMcom-CCLM4-8-17_v1 IPSL-WRF381P_v2 KNMI-RACMO22E_v2 SMHI-RCA4_v1	M1 M2 M3 M4
	ICHEC-EC-EARTH	KNMI-RACMO22E_v1 SMHI-RCA4_v1	M5 M6
	IPSL-CM5A-MR	IPSL-WRF381P_v1	M7
	MOHC-HadGEM2-ES	CLMcom-CCLM4-8-17_v1 IPSL-WRF381P_v1 KNMI-RACMO22E_v2 SMHI-RCA4_v1	M8 M9 M10 M11
	MPI-M-MPI-ESM-LR	KNMI-RACMO22E_v1 CLMcom-CCLM4-8-17_v MPI-CSC-REMO2009_v1 SMHI-RCA4_v1a	M12 M13 M14 M15
Experiment	historical
Ensemble *	r1i1p1
Time Frequency	Daily
Variable **	tasmax tasmin pr

* for each model, the first ensemble member (initial condition realization) was used (r1i1p1). ** tasmax—maximum temperature; tasmin—minimum temperature; pr—precipitation.

) and working with them could be time-consuming, we needed to make a hierarchy for all these models. To choose the most appropriate ones for further analysis, they were selected based on their best performance fit for the two crops’ vegetation seasons.

2.2. Methods Used

2.2.1. Evaluation Metrics for Validation

Three metrics were calculated between modeled and observation-derived datasets to evaluate regional climate simulations. The procedure was possible since both modeled and observation-derived historical data had the same horizontal resolution. The evaluation was elaborated based on multiannual means between each grid point. The evaluation metrics are the followings:

i. 

Mean absolute error (MAE) was used to calculate the absolute value of the difference between modeled and observed data (Equation (1)):

$$MAE=\frac{{{\displaystyle \sum }}_{i=1}^n\left|y_i-x_i\right|}{n},$$

(1)

where $y_i$ is the modeled value; $x_i$ is the observation-derived value; n is the number of grids.

ii. 

Root mean squared error (RMSE) detects the effect of the outliers in the difference between modeled and observation-derived values (Equation (2))

$$RMSE=\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-x_i\right)}^2}{n}}$$

(2)

where $y_i$ is the modeled value; $x_i$ is the observation-derived value; n is the number of grids.

iii. 

Pearson’s Correlation Coefficient (Corr) helped to measure the linear correlation between modeled and observation-derived values.

The metrics were calculated for multiannual means and the 5th and 95th percentile values of the chosen variables to analyze the ability of the models to represent the extreme values.

2.2.2. Algorithm for RCMs Ranking

The ranking of the 15 RCMs considered was developed in 4 steps, as follows:

Step 1. Based on the 3 validation metrics (MAE, RMSE, Corr), a d performance score was assigned to simulations in the case of the variables tasmax, tasmax 95th percentile, tasmin, tasmin 5th percentile, pr, and pr 95th percentile (Table S2). The scoring procedure is adapted after Bartok et al., 2019 [29]. The scores are calculated as described in Equation (3) for MAE and RMSE and Equation (4) for Corr.

$$d=\frac{\max \left(X\right)-X}{\max \left(X\right)-\min \left(X\right)}$$

(3)

$$d=\frac{X-\min \left(X\right)}{\max \left(X\right)-\min \left(X\right)}$$

(4)

where d is the performance score, the closer d is to 1, the better the simulation performs compared to the others; max(X) and min(X) are the maximum and minimum values of the given performance metrics among the 15 models.

Step 2. The d performance scores for each variable and evaluation metric were turned into an initial ranking between 1 and 15 (position_n) since we have 15 models. The highest d score (the model that best fit the observation-derived data) was assigned to 1 (position₁ in the ranking), and the lowest d score (the model with the lowest performance against the observation-derived data) was assigned to 15 (position₁₅ in the hierarchy). In this way, a uniform initial ranking was obtained (instead of the relative d performance scores, which vary among variables and metrics).

Step 3. The initial ranking (1 to 15) was converted into a weighted scale to assign values with different orders of magnitude for various positions. In this way, the final step summing up of weights is relevant, where models performing mostly well will have a score much higher than the models performing relatively weakly. The weighted ranking is elaborated based on a non-linear series which starts at 3 (1st initial position receives the value of 3) and is growing as presented in Equation (5):

$$x_n=x_{n-1}+positio{n}_n\times 3$$

(5)

where x_n—the weighted rank of the model on a given position; x_n−1 is the weighted rank obtained by the model on the previous position; position_n is the number of the position in the initial ranking (e.g., an x_n score of 9 is assigned to the model ranked in position 2 in the initial order: 3 + 2 × 3, where 3 is the rank for the model in position 1 in the initial ranking, 2 is the position of the model for which the weighted rank is calculated; similarly, a score of 18 is assigned to the model ranked in position 3 in the initial ranking (9 + 3 × 3), etc.). The series was introduced in reverse order to set the highest value (360) to the highest initial position (rank1).

The score was calculated for each model and each metric (see Table S3 for annual and vegetation periods in Supplementary material).

Step 4. We developed the final ranking of the models based on a final score (FS) obtained by summing up the weighted ranks calculated in Step 3 for each variable and evaluation metric. The higher the final score, the better the simulation performs compared to the others. Since temperature and precipitation act differently on cereal crops, for the final score, we applied weight for the three climatic parameters as follows: tasmin—40%; tasmax—30%, and pr—30% (Equation (6)).

$$\begin{array}{c}F{S}_{Mn}=0.4\times (x_{n}tasmi{n}_{MAE}+x_{n}tasmi{n}_{RMSE}+x_{n}tasmi{n}_{Corr}+x_{n}tasmin5p_{MAE}+x_{n}tasmin5p_{RMSE}+\\ x_{n}tasmin5p_{Corr})+0.3\times (x_{n}tasma{x}_{MAE}+x_{n}tasma{x}_{RMSE}+x_{n}tasma{x}_{Corr}+x_{n}tasmax95p_{MAE}+\\ x_{n}tasmax95p_{RMSE}+x_{n}tasmax95p_{Corr})+0.3\times (x_{n}p{r}_{MAE}+x_{n}p{r}_{RMSE}+x_{n}p{r}_{Corr}+x_{n}pr95p_{MAE}+\\ x_{n}pr95p_{RMSE}+x_{n}pr95p_{Corr})\end{array}$$

(6)

• FS_Mn—final score of the n model, n is the number given in Table 1;
• x_n tasmin_MAE—weighted rank obtained by the model n for tasmin for the MAE metric;
• x_n tasmin_RMSE—weighted rank obtained by the model n for tasmin for the RMSR metric;
• x_n tasmin_Corr—weighted rank obtained by the model n for tasmin for the Pearson Correlation metric;
• x_n tasmin5p_MAE—weighted rank obtained by the model n for 5th percentile of tasmin for the MAE metric;
• x_n tasmin5p_RMSE—weighted rank obtained by the model n for 5th percentile of tasmin for the RMSR metric;
• x_n tasmin5p_Corr—weighted rank obtained by the model n for 5th percentile of tasmin for the Pearson Correlation metric;
• x_n tasmax_MAE—weighted rank obtained by the model n for tasmax for the MAE metric;
• x_n tasmax_RMSE—weighted rank obtained by the model n for tasmax for the RMSR metric;
• x_n tasmax_Corr—weighted rank obtained by the model n for tasmax for the Pearson Correlation metric;
• x_n tasmax95p_MAE—weighted rank obtained by the model n for 95th percentile of tasmax for the MAE metric;
• x_n tasmax95p_RMSE—weighted rank obtained by the model n for 95th percentile of tasmax for the RMSR metric;
• x_n tasmax95p_Corr—weighted rank obtained by the model n for 95th percentile of tasmax for the Pearson Correlation metric;
• x_n pr_MAE—weighted rank obtained by the model n for pr for the MAE metric;
• x_n pr_RMSE—weighted rank obtained by the model n for pr for the RMSR metric;
• x_n pr_Corr—weighted rank obtained by the model n for pr for the Pearson Correlation metric;
• x_n pr95p_MAE—weighted rank obtained by the model n for 95th percentile of pr for the MAE metric;
• x_n pr95p_RMSE—weighted rank obtained by the model n for 95th percentile of pr for the RMSR metric;
• x_n pr95p_Corr—weighted rank obtained by the model n for 95th percentile of pr for the Pearson Correlation metric;

We granted the highest weights to tasmin as it is crucial for plant development and large area crops (such as cereals). It can severely impact the yields by shortening the growing season length and diminishing the growing degree days. Sometimes, it may lead to the incomplete maturity of the plant. Maximum temperature and precipitation received the same weight, 30%, as in the temperate regions; maximum temperature is usually not so high as to endanger the crops as in the Mediterranean or tropical climates. They do not last for many consecutive days when they occur, whereas precipitation can be successfully supplied by irrigation (ground water availability is not a problem in Romania). As the performance of the models differs with the season, the ranking was developed separately for monthly and annual means as well as for the vegetation periods of maize (April–October) and winter wheat (October–June) (Table 3).

2.2.3. Area Analysis

To increase the accuracy of the analysis, we considered only the arable lands for validation. Land use data were updated for 2020, and they were provided by the Agency for Payments and Interventions in Agriculture in Romania (http://www.apia.org.ro/, accessed on 1 July 2021). From the total number of 3046 pixels covering the entire country, after excluding the non-arable land, we retained only 2726 grids, representing 89.5% of the country territory. The selection of grids was made even where a very small arable land is included; for this reason, the number of remaining grids is still relatively high. Even though this decrease does not seem so important, it should be mentioned that the area excluded is from the complex topography regions (high hills and mountains), which usually impose the most important biases. We consider that accuracy for this validation is the best possible, based on the available data at present.

3. Results

3.1. Model Performance on Temperature

First, we compared the multiannual monthly averages of daily tasmax and tasmin derived from the 15 EURO-CORDEX ensembles with the values of the same grid point from the ROCADA database.

The validation metrics for individual models over the 12 months are presented in Figure 2, and the detailed results are included in Supplementary Material as Table S1. The multimodel mean of MAE is 1.3 °C for tasmax and 1.5 °C for tasmin (Table 1). However, there is a wide spread among the models (upper left and bottom left graphs in Figure 2), mainly during the spring and summer. The most significant mean errors are detected for M5, M3, and M7. The difference between RMSE and MAE highlights outliers in errors (in RMSE, the errors are squared before averaging). Larger outliers are detected in January, February, and March, particularly M4, M6, and M11 simulations (upper right and bottom right graphs in Figure 2).

Focusing on the vegetation periods of maize (April–October) and winter wheat (October–June), in the case of tasmax, the multimodel mean of MAE is 1.40 °C and 1.44 °C, and 1.47 °C and 1.62 °C in case of tasmin, respectively (

Table 2. Validation metrics for tasmax, tasmin, and pr: multimodel mean of 15 EURO-CORDEX simulations against ROCADA observation-derived datasets, over the period 1971–2000.

Variable/Metric			Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Annual	MVP *	WWVP **
tasmax		MAE	1.46	1.85	2.42	2.22	2.34	1.77	2.17	2.24	1.57	1.27	1.25	1.28	1.29	1.40	1.44
		RMSE	1.79	2.27	2.86	2.57	2.60	2.06	2.44	2.50	1.83	1.55	1.51	1.58	1.60	1.70	1.78
		Corr	0.78	0.78	0.81	0.89	0.90	0.91	0.92	0.91	0.92	0.91	0.89	0.83	0.90	0.91	0.89
tasmin		MAE	1.84	2.21	2.35	2.19	1.96	1.63	1.80	1.74	1.51	1.36	1.42	1.59	1.52	1.47	1.62
		RMSE	2.07	2.48	2.61	2.39	2.13	1.84	2.06	2.02	1.77	1.57	1.60	1.80	1.73	1.70	1.83
		Corr	0.87	0.89	0.90	0.91	0.91	0.90	0.89	0.87	0.89	0.89	0.89	0.87	0.92	0.91	0.91
pr		MAE	25.25	22.64	24.26	15.89	21.38	23.64	24.67	22.74	11.28	15.48	21.70	22.23	144.1	155.9	88.68
		RMSE	32.28	29.56	29.52	21.48	28.33	29.92	29.16	25.51	14.00	19.69	27.75	29.26	208.3	207.0	118.8
		Corr	0.73	0.58	0.61	0.70	0.72	0.73	0.75	0.74	0.71	0.72	0.59	0.71	0.80	0.77	0.79
tasmax	5th percentile	MAE	2.37	2.79	2.49	2.35	1.65	1.62	1.83	1.98	1.34	1.21	1.61	1.99	1.31	1.22	1.63
		RMSE	2.78	3.25	2.87	2.70	1.99	1.89	2.11	2.26	1.63	1.54	1.89	2.32	1.62	1.50	1.95
		Corr	0.77	0.75	0.80	0.88	0.89	0.92	0.93	0.93	0.92	0.87	0.83	0.78	0.91	0.92	0.89
	95th percentile	MAE	1.83	2.36	3.19	2.78	2.48	1.96	2.51	2.39	1.85	1.53	1.45	1.57	1.46	1.51	1.76
		RMSE	2.30	2.94	3.69	3.11	2.74	2.27	2.79	2.65	2.12	1.81	1.76	1.92	1.81	1.83	2.14
		Corr	0.78	0.83	0.83	0.85	0.89	0.90	0.90	0.89	0.89	0.88	0.85	0.83	0.89	0.90	0.88
tasmin	5th percentile	MAE	2.66	2.99	3.93	2.98	2.20	1.64	1.76	1.65	1.43	1.52	2.80	2.68	1.89	1.47	2.19
		RMSE	3.09	3.43	4.28	3.39	2.38	1.86	2.01	1.92	1.67	1.77	3.04	3.06	2.14	1.70	2.45
		Corr	0.76	0.82	0.88	0.87	0.90	0.91	0.91	0.89	0.89	0.85	0.86	0.78	0.91	0.92	0.90
	95th percentile	MAE	1.40	1.58	1.87	1.88	1.74	1.69	2.04	2.19	1.78	1.53	1.36	1.42	1.42	1.55	1.40
		RMSE	1.64	1.79	2.11	2.10	1.93	1.90	2.30	2.48	2.06	1.77	1.59	1.67	1.63	1.78	1.59
		Corr	0.79	0.86	0.85	0.89	0.88	0.87	0.85	0.82	0.83	0.86	0.86	0.83	0.90	0.88	0.89
pr	95th percentile	MAE	3.21	3.20	3.43	1.95	2.20	3.11	3.88	4.10	2.52	2.26	3.10	2.89	1.44	1.85	2.04
		RMSE	4.11	4.11	4.15	2.68	2.93	3.78	4.43	4.48	2.99	2.90	4.12	3.85	2.09	2.30	2.80
		Corr	0.65	0.50	0.48	0.60	0.61	0.59	0.63	0.70	0.58	0.65	0.45	0.60	0.73	0.74	0.69

* MVP—maize vegetation period; ** WWPV—winter wheat vegetation period.

).

The overall performance of the models for the two vegetation periods is also analyzed by using the Taylor diagrams. The best fit characterized the maize vegetation period with lower RMSE and standard deviation and stronger spatial correlation (Figure 3).

MAE and RMSE of the 5th and 95th percentiles were calculated to analyze the performance of the models in representing extreme values. Figure 4 depicts the spread of the models in terms of MAE by months. The larger errors were detected for the 5th percentile for tasmin. M3, M7, and M5 simulations indicated the most significant errors in representing low temperature (upper right graph in Figure 4). The spread among the models in the case of high extremes is smaller. Some outliers are also present, namely, M8 and M9 (lower graphs in Figure 4).

The spatial correlation was determined between modeled and observation-derived gridded values. Figure 5 presents the correlation coefficients over the 12 months for tasmin (left) and tasmax (right). A strong correlation was detected for annual mean values for both parameters (Table 2), but during the summer, three models revealed a weak correlation in tasmin (M9, M2, and M7).

In general, all the 15 RCMs can represent these variables within a reasonable range of uncertainty (Table S1). For tasmax, the MAE is between 0.7–5.3 °C having a median of 1.6 °C. More significant errors (higher than 4 °C) are present in March–April and July–August.

Regarding tasmin, the range of MAE is 0.6–5.4 °C with a median value of 1.5 °C. Larger errors (more than 4 °C) are present in February–April and July–August. The most extensive spread of errors was found for precipitation, where the MAE range of monthly precipitation is 6.2–59.4 mm with a median of 19.6 mm. Higher errors (more than 40 mm) were detected for May–August. The spatial representation of the variables was checked by a spatial correlation between model outputs and observations. The maximum temperature is represented very well. The median of the correlation coefficient throughout the year is 0.89, and the weakest correlation is 0.68 in February. In minimum temperature, the median value is 0.90, while the lowest correlation is 0.63 in August.

Even though the multimodel MAE, RMSE, and Corr are similar over the year, a significant spread among the models was found. The multimodel MAE for tasmax and tasmin is between 1.25 and 2.42 °C. More significant errors were found in tasmin in wintertime (in particular in the case of RCA4). This fact could be linked to a bias over snow-covered areas, as presented in Vautard et al., 2020 [20], assessing data from the northern part of Europe. The incoherencies are much more evident in the cold season for temperature and in the warm period for precipitation.

A similar weak representation of minimum temperature during summertime (here Figure 5) was also found in a previous study [20]. In summer, model biases are substantially dependent on stable boundary layer parameterizations due to lighter winds inducing less mixing than in winter.

3.2. Model Performance on Precipitation

Precipitation represents a critical variable in agroclimatology. The representation of precipitation in climate models shows significant uncertainties.

The validation metrics for the 15 RCMs are given in Figure 6, and the detailed results are presented in the Supplementary Material, Table S1. The multimodel mean of MAE is 144.1 mm (Table 2). There is a wide spread among the models over the year (upper left graph in Figure 6). The largest monthly MAEs are detected in the case of M7 and M8. The difference between RMSE and MAE highlights outliers in errors, especially in the summertime, particularly for the M4, M15, and M6 simulations (upper left graph in Figure 6). The monthly 95th percentiles were compared between modeled values and observation-derived data (lower left chart in Figure 6). Among the models, M8 seems to be an outlier.

The weaker spatial correlations between models and observation-derived data (lower right graph in Figure 6) are also explained by regional climate models’ weakness in representing precipitation, mainly during the cold season.

In Figure 6, the differences between models and observations are calculated for monthly sums (larger values). In general, RCMs tend to overestimate precipitation, but more in winter than in summer (here Figure 6) [20]. In summer, rain is essentially absent in observations (especially in gridded ones, such as ROCADA), while most simulations produce rain through predominantly (parameterized) convective events. Regarding the importance of boundary conditions, both large-scale drivers and local physical parameterizations (e.g., convection, microphysics) are important in determining the biases [20].

A weaker spatial representation was found for precipitation, with a median value of 0.71, and the lowest correlation detected was 0.34 in March. For precipitation, Vautard et al., 2020 [20] found smaller biases in summer than in winter in Europe. We found a similar result for the multimodel MAE for the same seasons. Still, the presence of outliers in summer (in particular for the RCA4 regional model) is canceled out by averaging.

Regarding vegetation periods of maize (April–October) and winter wheat (October–June), the performance of the models is presented in Figure 7. The Taylor diagrams indicated a better fit for the winter wheat vegetation period with lower RMSE and stronger spatial correlation (see Table 2).

3.3. Model Selection Procedure

As shown in Section 3.1 and Section 3.2, there is a large spread among different RCMs in representing tasmax, tasmin, and pr. In this study, we applied a selection procedure for individual models to rank them, taking into account their performance in the case of each variable (Section 2.2.2).

At an annual scale, the top 5 models in terms of their performance are M9 (MOHC-HadGEM2-ES-IPSL-WRF381P_v1), M1 (CNRM-CM5-CLMcom-CCLM4-8-17_v1), M2 (CNRM-CM5-IPSL-WRF381P_v2), M10 (MOHC-HadGEM2-ES-KNMI-RACMO22E_v2), and M12 (MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1). Analysis by month revealed that M12 is the model that best fits in September, October, November, and December and is in the second position in July and August. For the other models, we have a much more diverse ranking over the months. M1 is placed in the top three positions (among all models) in the winter and spring months and is relatively low-ranked (below ninth place) in July, August, September, October. M10 indicated a reverse situation, ranking in the top three positions within the first three places during summer and autumn (June, July, August, and September). Still, it was assigned ranks below ninth position for the winter period (December, January, February, and March). For M2, the highest ranking was detected for February, June, October, and November, and the lowest in April, May, and December. The most significant difference in ranking during the year was found in M9, which performed best in February and March, but in July and December had a very weak performance.

Consequently, one can state that there are models with high performance in a given period of the year but with lower skills in the other months. These results are extremely important for the agriculture sector to select the models with the best performance during the vegetation periods for specific crops. They should be considered in agroclimatology studies and make the model selection accordingly to reduce the inconsistency among the model data. Similar to the monthly and annual mean values, a model ranking was calculated for the vegetation periods of maize (April–October) and of winter wheat (October–June) (

Table 3. Regional climate model ranking for months, for annual, and for vegetation periods of maize (April–October) and winter wheat (October–June) (full names of individual models are given in Table 1).

Final Rank	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Annual	MVP *	WWVP **
1	M5	M1	M8	M8	M1	M13	M3	M3	M12	M12	M12	M12	M9	M12	M1
2	M1	M9	M9	M1	M13	M2	M12	M12	M10	M2	M6	M5	M1	M2	M9
3	M12	M8	M1	M10	M8	M10	M10	M10	M3	M5	M3	M1	M2	M15	M10
4	M9	M2	M5	M13	M12	M1	M2	M15	M4	M7	M2	M3	M10	M9	M8
5	M8	M12	M2	M15	M14	M14	M4	M13	M2	M11	M10	M4	M12	M10	M12
6	M11	M6	M11	M14	M10	M12	M5	M14	M11	M15	M1	M7	M11	M11	M11
7	M6	M5	M3	M12	M3	M9	M15	M4	M5	M3	M9	M6	M5	M7	M5
8	M2	M3	M4	M9	M9	M5	M7	M2	M13	M10	M4	M11	M8	M5	M2
9	M10	M4	M12	M11	M5	M3	M13	M5	M1	M4	M11	M8	M15	M4	M3
10	M4	M11	M13	M2	M15	M7	M14	M9	M15	M9	M15	M13	M3	M13	M13
11	M3	M7	M6	M5	M2	M11	M6	M7	M9	M6	M5	M14	M13	M3	M6
12	M7	M15	M7	M4	M11	M15	M1	M6	M7	M13	M8	M2	M4	M1	M15
13	M14	M13	M10	M3	M6	M6	M9	M11	M8	M1	M7	M10	M6	M8	M4
14	M13	M10	M14	M6	M4	M4	M11	M1	M6	M14	M13	M9	M14	M14	M14
15	M15	M14	M15	M7	M7	M8	M8	M8	M14	M8	M14	M15	M7	M6	M7

* MVP—maize vegetation period; ** WWPV—winter wheat vegetation period.

). In the case of the vegetation period for maize crops, models with the highest rank are M12 (MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1), M2 (CNRM-CM5-IPSL-WRF381P_v2), M15 (MPI-M-MPI-ESM-LR-SMHI-RCA4_v1a), M9 (MOHC-HadGEM2-ES-IPSL-WRF381P_v1), and M10 (MOHC-HadGEM2-ES-KNMI-RACMO22E_v2). For the winter wheat crop, the best-fitting models are M1 (CNRM-CM5-CLMcom-CCLM4-8-17_v1), M9 (MOHC-HadGEM2-ES-IPSL-WRF381P_v1), M10 (MOHC-HadGEM2-ES-KNMI-RACMO22E_v2), M8 (MOHC-HadGEM2-ES-CLMcom-CCLM4-8-17_v1), and M12 (MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1) (Table 3).

4. Conclusions and Discussion

In the study, 15 EURO-CORDEX simulations of tasmax, tasmin, and pr were validated against ROCADA observation-derived datasets for the Romanian arable land. The performance of the models was quantified by three validation metrics: MAE, RMSE, and spatial correlation. Consequently, we obtained different representations of maximum and minimum temperature as well as precipitation. For the vegetation periods of the two crops, a better fit for the maize vegetation period was revealed for temperature and for winter wheat crop when precipitation is considered.

Besides the significant differences in model skills over different months, several outliers were detected, inducing considerable errors in the multimodel means. Outliers are very model-dependent; they are the result of the model physics and setup, or they could be inherited from the driving GCMs. Thus, each case should be analyzed separately, but this is not the aim of this paper. For example, we found MOHC-HadGEM2-ES CLMcom-CCLM4-8-17_v1 as an outlier for tasmax and tasmin in high extremes (95th percentiles) and precipitation. References [15,20] also reported the unusual behavior of CCLM. In the case of CCLM, the spatial setup of the model (domain size and vertical resolution) has a more significant impact on performance than changes in the physical parameterization [30].

The larger ensemble spread for the lower quantiles of winter temperature (here Figure 4) can be due to the considerable sensitivity of winter temperatures on differences in the regional temperature field. Additionally, it may be due to the solid west–east air temperature gradient in winter in Europe and the snow-albedo feedback [31].

The presence of outliers could have a significant influence on random model selection. This is one of the main reasons why using a single model output for impact studies is not recommended. Furthermore, to capture the climate system’s main feature (dominant trend), the multimodel means are still not the perfect method because outliers also determine multimodel means. More realistic would be to use the entire ensemble climate sensitivity range to account for uncertainties in future climate. This type of interpretation, however, is quite difficult in impact modeling.

For this reason, the study recommends reducing the number of ensembles keeping the main climate features. Based on the validation metrics, a model ranking procedure was applied to quantify the overall skill of the individual models, including temperature and precipitation. The ranking procedure was elaborated for monthly and yearly means, as well as for vegetation periods.

In many cases, the model’s performance varies in the different periods of the year (e.g., seasons), and for this reason, the annual ranking is not recommended for the vegetation periods. Each vegetation period should be analyzed separately because the seasonal errors could lead to different rankings. In our case, the MOHC-HadGEM2-ES-IPSL-WRF381P_v1 simulation has the best performance at annual means. For the maize vegetation period, MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1 is the best model; for the wheat vegetation period CNRM-CM5-CLMcom-CCLM4-8-17_v1 is the best one. The best five models for the winter wheat crop are M1 (CNRM-CM5-CLMcom-CCLM4-8-17_v1), M9 (MOHC-HadGEM2-ES-IPSL-WRF381P_v1), M10 (MOHC-HadGEM2-ES-KNMI-RACMO22E_v2), M8 (MOHC-HadGEM2-ES-CLMcom-CCLM4-8-17_v1), and M12 (MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1). For the maize crop, the models with the highest five in the final ranking are M12 (MPI-M-MPI-ESM-LR-KNMI-RACMO22E_v1), M2 (CNRM-CM5-IPSL-WRF381P_v2), M15 (MPI-M-MPI-ESM-LR-SMHI-RCA4_v1a), M9 (MOHC-HadGEM2-ES-IPSL-WRF381P_v1), and M10 (MOHC-HadGEM2-ES-KNMI-RACMO22E_v2).

Regarding the structure of RCM and its driving model, there is no clear relationship between model type and performance. For example, the annual best rank was found for the MOHC-HadGEM2-ES-IPSL-WRF381P_v1 simulation, whereas the lowest position is also assigned to the IPSL-WRF381P_v1 RCM simulation but driven by the IPSL-CM5A-MR GCM. On the other hand, CLMcom-CCLM4-8-17_v1 driven by the same GCMs as the first one (MOHC-HadGEM2-ES) is situated in only eighth place.

Each RCM has its structure, and the performance depends on how its physics can capture large-scale and small-scale processes. Consequently, we obtained different representations of tasmax, tasmin, and pr. The models’ performance differs among variables and periods of the year. For this reason, we propose a ranking method taking into account the model performance for 18 metrics, also assessing the extremes. An example was given for five ensembles (having the lowest errors and highest correlations with observations) separately for maize and winter wheat vegetation seasons. The results are also presented on a monthly scale to combine the models for their particular use.

The results of this paper can be helpful for stakeholders for two main reasons: on the one hand, the number of the models can be significantly reduced, keeping only the models with the lowest errors, and on the other hand, the incoherencies among the models decrease by eliminating the outliers. These two aspects are very convenient for agronomical applications, where the climate is a crucial driving factor in a specific period of the year. It should be mentioned that bias-correction of raw model data is required in some applications, but this is beyond the scope of this article.