Monthly Scale Validation of Climate Models’ Outputs Against Gridded Data over South Africa

Abstract

The validation of climate models is important for ensuring accurate climate variability over a given region. This study evaluates the performance of multiple global climate model simulations from the Coupled Model Intercomparison Project Phases 5 and 6 and the downscaled regional climate model simulations from the Coordinated Regional Climate Downscaling Experiment against gridded observational data from the Climatic Research Unit gridded data during the historic period 1981–2000. Spatial analysis using monthly bias maps and statistical metrics (i.e., correlation coefficient, standard deviation, and centred root-mean-squared error) were employed to assess the model outputs’ ability to reproduce monthly temperature and precipitation patterns over South Africa. The results indicate an improvement in CMIP6 and CORDEX model simulation outputs compared to their CMIP5 predecessors, with reduced biases and enhanced correlation. The study underscores the importance of model selection for regional climate analysis and highlights a need for further model development to capture complex physical processes.

1. Introduction

The use of global climate models (GCMs) significantly contributes to the prediction of future climatic conditions [1], with many results focusing on regional analysis within the Coordinated Regional Climate Downscaling Experiment (CORDEX). However, the new CMIP6 results are a new addition to the Coupled Model Intercomparison Project Phase 5 (CMIP5), which served for driving inputs for the completed CORDEX simulations. Hence, it is of interest for climate change research, especially over the African domain, as it is considered one of the most vulnerable regions to global climate change. Therefore, it is important to ensure the validity of these models for accurate projections and adaptation plans [2].

Climate model validation has always been used to determine the ability of models to reproduce historic station or observational data sets [3]. This is important for ensuring that climate model projections are reliable for the use of regional climate studies. Studies across Africa have mostly focused on the validation of precipitation (e.g., [4,5,6]), as it is the most complex parameter, and the lack of station data has raised the question of whether or not climate models can represent the African domain. The global data sets of the Climatic Research Unit (CRU) offer reliable gridded data, which can be used for validation [7]. These studies concluded that climate models often struggle to represent the African regions, due to complex topographies and diverse climates. However, previous studies, e.g., [8], focused on a larger Southern African region and showed that the model ensembles of CORDEX, CMIP5 and CMIP6 considerably underestimated precipitation when validated against satellite and gauge data during the October to March seasons.

On the contrary, ref. [4] observed overestimation of precipitation over the western part of Southern Africa during December to March when using large ensemble means. Reference [9] highlighted the fact that model performance varies with region, and showed an overestimation of precipitation over Southern Africa. Other examples (i.e., from [10]) used observations and indicated that CMIP6 data had a better performance compared to the CMIP5 model simulations at reproducing precipitation over West Africa. However, ref. [11] noted variability in CMIP6 performance, with some exhibiting overestimations during dry seasons and underestimations during the wet season, over NorthAfrica, respectively. Whilst African studies indicate that regional climate models (RCMs) tend to perform better on a region-specific level, for example, ref. [12] indicated that the CORDEX-AFRICA model ensemble had high correlation with precipitation station data in Ethiopia.

This study evaluates models from CORDEX, CMIP5, and CMIP6 to reproduce monthly historical temperature and precipitation climatology over South Africa, facilitating model selection for projection studies. The main objective is to evaluate the model simulations by using gridded observational data sets from the CRU database. The results of this study will contribute to the scientific community by providing an analysis which explores the performance of the models on a regional country level. Overall, this study aims to determine (i) how the downscaling improves the GCM simulation results by the embedded RCM simulations; (ii) how the GCMs have improved from CMIP5 to CMIP6; and (iii) which of the simulations is the best performing in terms of precipitation and temperature reproduction among the GCM simulations and the GCM-driven RCM simulations, respectively.

For these purposes, the monthly precipitation and temperature data are validated against the gridded CRU data using correlation coefficients (r), standard deviation (SD) and centred root-mean-squared error (cRMSE). This study uses spatial validation, wherein grid-by-grid values from the climate model simulations are calculated against CRU data for the 20-year reference period between 1981 and 2000.

Study Area

The target area of this study is Domain 5 under the CORDEX program. More specifically, the validation focuses on South Africa (Figure 1), which lies at the southernmost part of the African continent. The interior of the country is a plateau surrounded by a 3408 m high boundary of the Great Escarpment. The South African climate is influenced by several factors such as oceanic influence, geographical location and distinct topographical features. Thus, the country has varied climatic conditions.

The country is predominantly classified as semi-arid, and receives an annual mean precipitation of 450 mm, experiencing wet summers and dry winters, with the exception of the Western Cape, which has a Mediterranean climate with an opposite annual course of precipitation. Though the country has faced adverse effects of climate change, it still heavily relies on the use of fossil fuels [13], which makes it even more vulnerable to the projected effects of climate change under a business-as-usual scenario.

2. Materials and Methods

2.1. Observational Data

The CRU high-resolution global data set developed by the University of East Anglia is used for this study. The data is freely available at https://crudata.uea.ac.uk/cru/data/hrg/ (9 April 2024). Both monthly precipitation and temperature are available from 1960 to 2010 at a high resolution of 0.5°. The data is a reliable gridded data set, as the university maintains credible data quality control, significantly contributing to filling the scientific gap where the lack of observational datasets has affected climate studies. CRU uses angular-distance weighting to interpolate averaged anomalies from monthly station data [7].

2.2. Simulation Data

This study uses monthly precipitation and temperature data derived from daily precipitation and temperature, respectively. The simulated data (

Table 1. Summary of CMIP5 and CMIP6 GCMs and the CORDEX RCMs driven by CMIP5 GCMS.

Model	Institution	Atmospheric Model	Ocean Model	Land-Surface Model	Carbon Cycle	Reference
CMIP6: HadGEM3-GC31-MM	Met Office Hadley Centre (MOHC)	Global Coupled Model 3 (GC3)	Nucleus for European Modelling of the Ocean (NEMO)	Joint UK Land Environment Simulator (JULES)	advanced carbon cycle and biogeochemical processes	[14]
CMIP5: HadGEM2-ES		Hadley Centre Global Environment Model 2	HadGEM2-Ocean	Met Office Surface Exchange Scheme2 (MOSES2)	carbon and chemistry	[15]
CMIP6: MPI-ESM1-2-HR	Max Planck Institute for Meteorology (MPI-M)	ECHAM6.3	MPIOM	JSBACH3.2	interactive carbon cycle	[16]
CMIP5: MPI-ESM-LR		ECHAM6		JSBACH		[17]
CMIP6: NorESM2-MM	Norwegian Climate Centre (NCC)	Community Atmosphere Model version 6 (CAM6)	Bergen Layered Ocean Model (BLOM)	CLM5 (Community Land Model version 5)	isopycnic coordinate Hamburg Ocean Carbon Cycle (iHAMOCC)	[18]
CMIP5: NorESM1-M		Community Atmosphere Model version 4 (CAM4)	Miami Isopycnic Coordinate Ocean Model (MICOM)	CLM4 (Community Land Model version 4)	interactive carbon cycle	[19]
CORDEX: CCLM5-0-15 Driven by all three 3 CMIP5 model outputs	Climate Limited-area Modelling Community (CLM-Community)	COSMO-CLM	N/A	TERRA-ML	N/A	[20]
CORDEX: REMO201 5 Driven by all three 3 CMIP5 model outputs	Max Planck Institute for Meteorology (MPI-M)	REMO		Basic surface module (HydroPy)		[21]
CORDEX: RegCM4-7 Driven by all three 3 CMIP5 model outputs	International Centre for Theoretical Physics (ICTP)	RegCM4		BATS or CLM		[22]

) is freely available from the Earth System Grid Federation (ESGF) data nodes (https://esgf-metagrid.cloud.dkrz.de/search (15 March 2024)). CMIP5 experiments ensure that the GCMs’ capacity in representing the internal variability and external forcing are taken into account through the historical simulations, which are available from 1850. For this study, we use simulation data, which are available at a nominal resolution of 100 km. The latest addition of the CMIP strengthens the climate model data archives for the future climate, which is important for downscaling studies. It provides insight into changes in the response to climate change under Shared Social–Economic Pathways (SSPs), coupled with the available Representative Concentration Pathways (RCPs). However, for this validation study, we downloaded the available data from historical simulations with 1° resolution. The CORDEX data provides experimental simulations with parametrisations specifically developed to address the African domain, thus providing an added value for regional-scale climatic information. We use 3 RCMs with a resolution of 0.22°; each has been driven by the three different GCMs selected from the CMIP5.

3. Methodology

The most common validation method includes the ensemble mean method, where the mean of two or more climate model simulations is calculated and validated against an observational data set. This method is reported to improve the climate representation of a region (e.g., [23]). In this study, we validate each climate model simulation’s output against the CRU data without an ensemble mean, to determine the performance of individual model simulations. For this purpose, statistical and spatial validation methods are used.

3.1. Spatial Validation Using the Monthly Mean Bias Maps

The multi-year monthly means were calculated for both temperature and precipitation for the reference period 1981–2000. This period was selected in accordance with the IPCC reports, which uses 20 years, as it is long enough for precipitation statistical analysis. To create a uniform grid, we selected the gridded CRU observational data set as our base resolution, since it serves as the reference, and the climate models from the CMIP and CORDEX simulations used different spatial resolutions. Therefore, the CRU grid was extracted to remap the CORDEX, CMIP5 and CMIP6 datasets using bilinear interpolation via the CDO (Climate Data Operator, https://code.mpimet.mpg.de/projects/cdo (4 April 2024)) script. A topography mask was then applied to the simulated data to ensure that data over the ocean was omitted. The monthly mean bias between each climate model simulation against CRU and against each other was calculated for all 12 months via a CDO script. Then, these bias fields (i.e., maps) were produced with R-codes [24].

3.2. Statistical Validation of Monthly Mean Fields

The average spatial fields for both temperature and precipitation were calculated with R-codes, where the monthly average for each grid cell was used to calculate the different statistical measures, namely, cRMSE, r and SD. The comparison of the spatial structure of simulated data to CRU is visualised in Taylor diagrams [25], which were produced using the Plotrix package in R. The analysis of the spatial comparison of monthly mean fields serves as the main tool to select the best performing model simulations per month among all three simulations sets (i.e., CMIP6, CMIP5, and CORDEX).

4. Results and Discussion

Most studies conducted in Africa used the validation of a large climate model ensemble against gridded data GCPP over the African region. However, there are not many studies that specifically focus on providing a broad overview in South Africa, even though there are existing studies focusing on the Southern African region. This study focuses specifically on South Africa, the country. This study will therefore improve and update previous validation results on temperature and precipitation in South Africa and compare the recent modelling results from different models, such as the high-resolution data versus coarser resolutions. The study focused on validating monthly temperature and precipitation simulated by the selected GCMs (from CMIP5 and CMIP6) and RCMs (from CORDEX) against CRU gridded data, during the reference period from 1981 to 2000.

The results demonstrate wide variations between the data sets. However, it was observed that the CORDEX and CMIP6 models used in this study had improvements on the CMIP5.

4.1. Mean Temperature Bias Patterns and Model Output Patterns

CMIP5 HadGEM2 generally underestimates temperature across the interior regions throughout the year, with largest negative biases in summer, DJF (3–5 °C) (Figure 2), and moderate biases in winter (JJA) (Figures S1–S10). This is consistent with patterns observed over Africa; for example, ref. [26] reported that CMIP5 models underestimate the Indian Ocean Equatorial Undercurrent (EUC) (thus explaining the observed underestimations, as South Africa’s climate is greatly influenced by the Indian ocean [27]). And their spatial distribution of these biases indicates that the topography influenced the accuracy of models, especially over areas of higher elevation, such as the plateau.

The CORDEX models can capture the overall temperature of the country; however, the models exhibit seasonal variability, wherein differences between the CORDEX and the CRU datasets observed in this study highlight the importance of model selection for seasonal analysis due to variable performance, especially over complex topographic regions (Figure 3). For instance, HadGEM2-CCLM5 accurately reproduces the observed temperatures across the Western Cape during the winter half of the year (March to August), while the other parts of the country are moderately underestimated (1–2 °C). The largest underestimations occur in SON, by 3–5 °C. The HadGEM2-REMO2015 exhibits an overestimation (1–3 °C) over Limpopo, Gauteng, Mpumalanga, North West and parts of KwaZulu-Natal (KZN), but underestimates temperature by 2–3 °C over the southern provinces in November. HadGEM2-RegCM4-7 generally overestimates temperature (3–4 °C) from May to October over the western parts of Northern Cape and Western Cape, Free State and major parts of North West, with decreasing biases (2–3 °C) during colder months in Western Cape and Eastern Cape. The model exhibits better performance in Free State and North West during the summer months, but tends to overestimate the northern parts of the country, indicating a variable performance in CORDEX models, depending on their GCM-RCM pairs.

Our results indicate that MPI-ESM-driven CORDEX models overestimate temperature by 2–3 °C over Limpopo, North West and Free State during warm months of October, November and December. Parts of the Western Cape are overestimated (2–4 °C) from January to April (Figure S5). The MPI-ESM-CCLM5 and MPI-ESM-driven RegCM4-7 accurately reproduces temperature over the Western Cape, with underestimations observed during hot months (November to February). However, the MPI-ESM REMO2015 model exhibits a good performance during the winter, with slight overestimation of 1–2 °C over some parts of the interior in June and July. The NorESM1 models exhibit the highest biases, underestimating temperature in JJA and SON months at different magnitudes. However, an overestimation of the western coast by 1–3 °C is observed from November to October.

On the contrary, overestimations of 5–6 °C occur over the western Free State and eastern Northern Cape, while the Northern Limpopo and the North West exhibit a substantial temperature underestimation (4–5 °C) during December. However, the intensity of the underestimation decreases from May to November by 2–4 °C. This is a systematic error observed in the models, as CORDEX models tend to overestimate the coastline. Other African studies have reported that the CMIP5 has a better ability to reproduce temperature than precipitation [28].

CMIP6 HadGEM3 generally exhibits a good ability to reproduce temperature, with improved correlations, with high accuracy in the interior from February to June, with high accuracy over the Northern Cape, Free State, North West, Gauteng, Limpopo and Mpumalanga in February to June. The models had accurate distribution and exhibited fewer errors. However, regional biases persist in some models; for example, the MPI-ESM1 and NorESM2 tend to overestimate temperature over the interior, indicating that different models’ accuracy may be influenced by variable topography. The modification of the CMIP6 includes an increase in the model’s forcing [29], which may result in CMIP6 having higher projected temperature values. Therefore, the results must be analysed with caution for impact studies

The CMIP6 MPI-ESM1 tend to underestimate temperature by 1–2 °C along the escarpment by 1–2 °C, but accurately capture the interior during cooler months (May to August) (Figure S5). This marks an improvement on the CMIP5, which generally underestimates the interior by 1–4 °C but accurately captures the climatology of the escarpment.

However, the NorESM2 model simulation in CMIP6 tends to overestimate temperature in parts of the Northern Cape and Western Cape by 1–2 °C in MAM. The greatest overestimations are observed in October, over different parts of the interior, wherein NorESM2 simulation within CMIP6 also exhibits temperature overestimation (2–4 °C) over Free State, Gauteng and northern Limpopo, from November to March.

4.2. Statistical Analysis of the Metrics for the Monthly Mean Temperature Fields

Statistical analysis indicates that the CORDEX and CMIP6 models show substantial improvements in reproducing temperature over South Africa compared to the CMIP5, exhibiting lower cRMSEs and stronger correlations with CRU observational data.

A comparison of CORDEX and CMIP5 indicates that CCLM5 models exhibit a generally lower temperature (4–6 °C) over the coastal provinces, with no substantial difference in performance over the Free State, Gauteng and Limpopo in JJA. The RCMs result in an overestimation (1–2 °C) of CMIP5 in January–May. Meanwhile, the REMO2015 model pairs exhibit variable overestimation (1–2 °C) over Free State, North West, Gauteng and Limpopo. The RegCM4-7 pairs exhibit great overestimation (3–4 °C) of the CMIP5, especially over Free State and Northern Cape, with some agreements over the Free State and the northwest in MAM. Meanwhile, the REMO2015 model pairs exhibit higher variability, with overestimation of CMIP6 over the Western Cape and Northern Cape. The RegCM4-7 pairs strongly overestimate CMIP6 by 4–5 °C over the Western Cape from December to September, and overestimate temperature over the eastern part of the interior from May to September.

The evaluation of the results showed that the CORDEX and CMIP6 datasets have substantially improved on the performance of their predecessor, CMIP5. It is observed that the CMIP6 has better spatial representation of temperature trends compared to the CMIP5, Iidicating that biases have been corrected in the recent temperature simulation. However, it should be noted that we have observed different spatial distributions in the case of CMIP6, resulting in variations in model performances, highlighting the fact that CMIP5 still outperforms CMIP6 in terms of SD. However, the CMIP6 exhibit lower cRMSE and correlation.

The average historic temperature statistics, cRMSE, r and SD across the models, indicate substantial improvement in the CMIP6’s ability to reproduce overall temperature climatology over South Africa, with the models exhibiting lower cRMSEs and high correlation with the CRU data. Figure 4 illustrates a statistical summary of model performance against CRU data through Taylor diagrams, which depict the models’ ability to better capture temperature variability over the country. CORDEX models exhibit a good ability to reproduce overall temperature over the country, but the models show higher SD compared to CMIP6, indicating persistent biases from CMIP5.

Model simulations from CORDEX-Africa generally exhibit a decrease in cRMSE compared to the driving CMIP5 global simulations (

Table 2. Summary monthly statistics (cRMSE, r and SD difference) of CMIP5 (left) and CMIP6 (right) model performance against CRU observational data for temperature during 1981–2000. The yellow background highlights the best-performing model simulation, and the green background highlights the second-best-performing model.

		CMIP5			CMIP6
Month	Model	cRMSE	Correlation	Std_Dev Difference	RMSE	Correlation	Obs_Std_Dev Difference
December	HadGEM3	1.40	0.87	−0.21	0.97	0.94	0.09
December	MPI_ESM1	1.72	0.81	−0.20	1.31	0.93	0.54
December	NorESM2	2.40	0.64	−0.02	1.53	0.88	0.41
January	HadGEM3	1.41	0.86	−0.24	0.92	0.94	−0.05
January	MPI_ESM1	1.96	0.72	−0.23	1.21	0.91	0.22
January	NorESM2	2.50	0.56	−0.13	1.53	0.86	0.22
February	HadGEM3	1.49	0.83	0.00	0.96	0.93	−0.02
February	MPI_ESM1	2.03	0.68	−0.11	1.42	0.85	0.03
February	NorESM2	2.35	0.55	−0.18	1.40	0.87	0.26
March	HadGEM3	1.39	0.85	−0.12	0.95	0.93	−0.02
March	MPI_ESM1	1.98	0.70	−0.18	1.43	0.86	0.16
March	NorESM2	2.27	0.60	−0.19	1.37	0.88	0.28
April	HadGEM3	1.39	0.86	−0.15	0.92	0.94	−0.03
April	MPI_ESM1	1.74	0.79	−0.13	1.31	0.90	0.24
April	NorESM2	2.21	0.66	−0.10	1.28	0.90	0.24
May	HadGEM3	1.31	0.89	0.06	0.95	0.95	0.09
May	MPI_ESM1	1.58	0.86	0.30	1.16	0.95	0.60
May	NorESM2	1.89	0.78	0.04	1.32	0.90	0.16
June	HadGEM3	1.22	0.92	−0.11	1.01	0.95	0.05
June	MPI_ESM1	1.46	0.89	0.20	1.04	0.96	0.41
June	NorESM2	1.69	0.85	0.05	0.98	0.95	0.13
July	HadGEM3	1.18	0.92	−0.07	1.01	0.94	0.00
July	MPI_ESM1	1.47	0.89	0.17	1.11	0.95	0.45
July	NorESM2	2.00	0.78	0.03	1.15	0.93	0.08
August	HadGEM3	1.27	0.90	−0.15	0.95	0.95	0.20
August	MPI_ESM1	1.47	0.88	0.19	1.09	0.95	0.49
August	NorESM2	1.94	0.77	0.02	1.11	0.93	0.12
September	HadGEM3	1.18	0.91	−0.14	0.94	0.95	0.17
September	MPI_ESM1	1.56	0.86	0.07	1.15	0.95	0.54
September	NorESM2	2.13	0.73	0.04	1.11	0.94	0.28
October	HadGEM3	1.30	0.91	−0.03	0.98	0.95	0.22
October	MPI_ESM1	1.55	0.88	0.20	1.27	0.95	0.71
October	NorESM2	2.07	0.77	0.02	1.10	0.94	0.19
November	HadGEM3	1.38	0.89	−0.13	1.03	0.95	0.22
November	MPI_ESM1	1.60	0.85	−0.16	1.20	0.94	0.48
November	NorESM2	2.19	0.74	−0.02	1.43	0.91	0.34

). More specifically, the HadGEM2-CCLM5 outperforms the CMIP5 HadGEM2 simulations; this is supported by lower cRMSE throughout the year (12 months) and higher correlation with CRU data in CORDEX models compared to CMIP5. However, the HadGEM2-CCLM5 within CORDEX tends to exhibit higher correlation. Meanwhile, HadGEM2-RegCM4-7 exhibits a decrease in cRMSE in 9 of 12 months compared to the driving global model, and only slight changes from February to November. This model pair also resulted in an increase in correlation and SD in all the months, compared to the driving global model. Similarly, NorESM1 and MPI-ESM global models show smaller cRMSE throughout the year, coupled with higher values of linear correlation coefficients and SD.

The evaluation of the results showed that the CORDEX and CMIP6 datasets have substantially improved on the performance of their predecessor, CMIP5. It is observed that the CMIP6 has better spatial representation of temperature trends compared to the CMIP5, indicating that biases have been corrected in the recent temperature simulation. However, it should be noted that we have observed different spatial distributions in the case of CMIP6, resulting in variations in model performances, highlighting the fact that CMIP5 still outperforms CMIP6 in terms of SD. However, the CMIP6 exhibits lower cRMSE and correlation.

CORDEX models generally exhibit higher cRMSE with substantial differences during summer months (DJF) compared to CMIP6 models, with NorESM1-RegCM4 exhibiting the highest cRMSEs during the summer months. Higher SD is observed throughout the year in all models paired with RegCM4 CORDEX simulations, coupled with a lower correlation compared to CMIP6, except for the CCLM5 and REMO2015 model pairs outperforming the CMIP6 in terms of linear correlation in May, June and July. CMIP6 models exhibit variable cRMSE, with insubstantially different correlation compared to the CORDEX models observed (Table 2).

The NorESM1-CCLM5 exhibits lower cRMSE compared to NorESM1-RegCM4-7 and NorESM1-REMO2015, with higher correlation and SDs, whereas in the Northern Cape, NorESM1-RegCM4-7 and NorESM1-REMO2015 perform poorly in terms of cRMSE, correlation and SD, with the RegCM4-7 exhibiting a lower correlation between October and March, June and July, while the REMO2015 has higher correlations than CMIP6 in 10 of 12 months (excluding January and February). The CMIP6 has greatly improved on the CMIP5, exhibiting lower cRMSE and higher correlation. However, CMIP5 still outperforms CMIP6 in terms of SD in all GCMS, with HadGEM3 within CMIP6 indicating lower SD in February as a result of a possible underestimation. The MPI-ESM1 and NorESM2 models exhibit improvements during the summer months. NorESM2 exhibits lower errors.

The models also exhibit a good ability to capture North West in January and February and capture Limpopo, Gauteng, North West and Free State in March and April. HadGEM2 REMO2015 exhibits better performance, indicating model improvement on CMIP5. However, underestimations are exhibited in the eastern parts of the country from November to May. The magnitude of underestimation decreases during cooler/colder months, thus performing better in cooler months over Limpopo, Gauteng, Mpumalanga, North West and parts of KZN. Meanwhile, the RegCM4-7 does not improve the model performance. The downscaling has reduced the intensity/magnitude of underestimation in NorESM1 models. However, it increased the spatial variability/distribution of underestimation in the country, wherein the entire country is underestimated by NorESM1-REMO2015 and MPI-ESM-REMO2015. The NorESM1-RegCM4-7 can better reproduce CRU data during cooler months. The overall mean and variability of CORDEX simulations show more accuracy compared to the CMIP5; this is consistent with previous studies, which have highlighted the improvement of CORDEX models [30]. These results indicate that model performance can vary within a region [12]; therefore, it is important to select the best-performing model in a specific region [31] for climate studies directed toward stakeholders and adaptation strategies.

The results highlight the effect of downscaling, exhibiting improvements in temperature, and therefore indicating substantial progress in model representation of climatology over the African domain. The statistical improvements highlight an enhanced presentation of the regional climatology, including a better ability of the models to capture complex topography and land-surface processes, thus providing accurate data for application in agriculture and water management.

The persistent biases inherent in the CMIP5 over complex topographies highlight a need for continuous model development and downscaling, especially for the African region, to ensure high-resolution models [32]. Indicated orographic effects, especially over the Drakensberg, demonstrate influences on regional climate simulation.

4.3. Analysis of the Monthly Mean Precipitation Bias Maps

Climate models exhibit differences in simulating precipitation across South Africa. With a general increase in precipitation gradient from west to east, CMIP5 models generally overestimate precipitation, with an observed, underestimation over some regions, particularly the southwestern parts of the county (Figures S13–S15). Our results are consistent with other studies that have highlighted that MPI-ESM models overestimate precipitation during the rainy season, the October to November months, in East Africa, highlighting Reference [33]. Meanwhile, some parts of southern Africa show an underestimation of precipitation patterns. This study mainly focuses on the performance of individual models, instead of the seasonal performance of the ensemble mean.

The performance of CMIP5, CORDEX and CMIP6 models varies by season and month. CORDEX simulations can best capture precipitation patterns over the central and eastern parts of the country during summer months, with DJF exhibiting an enhanced ability to capture regional variability and higher resolutions [34]. However, some models show overestimations during the wet season (Figures S1–S15). This aligns with previous studies that highlighted a regional wet bias in CORDEX models over the Southern African region [35]. Other studies have reported that the CCLM5 RCMs tend to underestimate the precipitation [36] over East Africa. The differences in the performance of the CORDEX models data sets mainly stem from the GCMs, highlighting systematic differences and parameterisations [35].

The HadGEM2-REMO2015 from CORDEX-Africa can reproduce the precipitation in JJA better than the other models, while HadGEM2-CCLM5 substantially underestimates precipitation over Limpopo, Mpumalanga and Gauteng, by −50% to −100% in January (Figure 5), February, and March. In contrast, the CMIP models substantially overestimate (200–300%) the western parts of the country, whereas substantial overestimation of precipitation is noted over the interior in Limpopo, Gauteng and Mpumalanga.

HadGEM2-CCLM5 generally outperforms CMIP6 HadGEM3, with agreement noted over the central plateau in April, where precipitation biases are similar. An overestimation of precipitation (by 80–100%) is noted over the southern part of the country. Similar monthly biases are observed in the case of HadGEM2-REMO2015 and CMIP6 HadGEM3 from November to February; however, variability in model performance is observed, with overestimation in the winter half of the year, and an underestimation over the western parts of the country, i.e., the Western Cape, Northern Cape, and parts of the Free State, during the summer half year. The HadGEM2-REMO2015 and HadGEM2-RegCM4-7 exhibit general overestimation (by 50–100%) of precipitation over the southern parts of the Western Cape from October to December. However, the HadGEM2-RegCM4-7 follows a similar series of patterns as the HadGEM3 within the CMIP6 during October to February, exhibiting overestimations in the winter half of the year (as shown for July in Figure 6), whereas the other months can be found in the Supplementary documents (Figures S1–S15).

Both the MPI-ESM from CMIP5 and MPI-ESM1 from CMIP6 underestimate (80–100%) the southwestern coastline in October and November. The CMIP global model simulations generally underestimate precipitation during the winter half of the year (March to August), exhibiting a large overestimation over the interior of the country.

MPI-ESM-driven CCLM5 simulations generally exhibit lower biases than the corresponding global models, both within CMIP5 and CMIP6, with overestimations observed over the Northern Cape, Western Cape, Free State and North West in November and December. However, MPI-ESM-REMO2015 also exhibits an overestimation over the stretch of the western coast and Western Cape compared to the MPI-ESM simulation within CMIP5. MPI-ESM-RegCM4-7 agrees with MPI-ESM of CMIP5 over the interior during November–August; however, the model pair tends to overestimate precipitation patterns in September and October.

The NorESM2 shows a great improvement, with better capacity to reproduce distribution; however, traces of the dry biases observed in CMIP5 over the Western Cape are still observed in all months. For example, the NorESM1-driven RCM simulations within CORDEX-Africa tend to underestimate CRU data; the models exhibit overestimations during the dry season in Limpopo, Gauteng, Mpumalanga and North West, with the NorESM1-REMO2015 showing lower biases over the Northern Cape. However, the model simulations overestimate precipitation over the Western Cape in all months.

In contrast, the NorESM2 (Figure S10) exhibits an underestimation over areas that were previously overestimated by the NorESM1 (Figure S13), indicating a substantial improvement in CMIP6 models (NorESM2) in terms of correcting the overestimation. This GCM is not able to accurately capture the precipitation climatology over southern Africa. A shift from overestimation to underestimation may result in new regional biases, which require careful evaluation of model output. CMIP6 models are valuable in precipitation analysis, due to better accuracy in capturing the annual cycle compared to CMIP5; they can be used for extreme-precipitation event projection [37].

4.4. Statistical Analysis of the Metrics for the Monthly Mean Precipitation Fields

Statistical metrics indicate that the downscaling resulted in a substantial model improvement in model performance, resulting in lower errors. The model performance of CORDEX simulations varies, depending on the RCM and GCM pairs, with the CCLM5 showing higher correlation indication, with approximately 66% overall improvement across all models, indicating improvements in the SD, wherein less variation is observed compared to CMIP5, especially in December to February and September to November. For example, HadGEM2-CCLM5 exhibit higher correlation and a substantial reduction in cRMSE, with the differences ranging from −6.2 to −6.9 in summer, and reduced variability (SD) in most months. Similarly, the MPI-ESM-CCLM5 exhibits a substantial increase in correlation in 9 of 12 months, coupled with a decrease in standard deviation, also noted. The model exhibits a good ability to reproduce historical precipitation patterns, particularly at the peak of the wet season (July), with a low cRMSE of 8.5. Meanwhile, NorESM1-CCLM5 also shows improvements on NorESM1, with a reduced cRMSE in 11 of 12 months and a decrease in standard deviation from August to March. Even with the improved performance in reproducing precipitation, the CCLM5 does exhibit biases in winter (JJA), overestimating SD in model pairs such as MPI-ESM-CCLM5 and NorESM1-CCLM5. The results show that the RCM has the strongest agreement with observational data, with a correlation ranging from 0.5 to 0.8.

On the other hand, REMO2015 models exhibit variable results among model pairs. HadGEM2-REMO2015 shows an increase in correlation during December to March and August, September, and November, suggesting improvements in reproducing deputation events; however, the model exhibits higher cRMSE and SD compared to CMIP5, with overestimations in December and January. Similarly, MPI-ESM-REMO2015 exhibit increases in cRMSE, especially in December, January and April; these are predominantly in the wet season; over the majority of South Africa, despite high variability, the REMO2015 consistently exhibit higher correlations relative to the driving GCM. This indicates that the RCM can capture precipitation patterns more linearly, despite inherent biases from the GCMs [38]. The NorESM1-REMO2015 follows the same trend; however, it exhibits higher cRMSE and SD, compared to the other model pairs.

On the contrary, the RegCM4-7 RCMs exhibit lower performance compared to the other CORDEX model pairs and the driving CMIP5 model, with notable disagreement between the models, where a decrease in correlation is observed, accompanied by increases in cRMSE and SD (Figure 7). A general model disagreement and precipitation biases across different months indicate that that the model pair is not able to accurately capture precipitation patterns over a complex domain. For example, MPI-ESM-RegCM4-7 exhibits a higher cRMSE and a general decrease in correlation in 8 of 12 months, underestimating SD during colder months (May to August), while the NorESM-RegCM4 exhibits the lowest ability to reproduce precipitation, with high cRMSE, especially in December (93.2), coupled with high SD, highlighting the fact that downscaling could not improve the representation of South African precipitation climatology in this model pair.

CMIP6 simulations exhibit statistical improvements on the CMIP5, with substantial reductions in cRMSEs and increased correlations across most months, thus exhibiting reduced overestimation biases. This aligns with other studies, e.g., [39], who have reported that the CMIP6 models had a better ability to reproduce precipitation climatology. However, the models still exhibit some inherent seasonal biases. For example, HadGEM3 exhibited larger errors and low correlation in March and December, respectively. However, the model exhibited a notable decrease in cRMSE and an increase in correlation in 8 of 12 months (e.g., a reduced cRMSE from 29.6 to 25 in December) and overestimations of SD in January to April and June.

The MPI-ESM1 and NorESM2 models exhibit improvements during the summer months, with MPI-ESM1 exhibiting a better ability to reproduce the rainy seasons, exhibitingthe least change in DJF, showing improvement in the case of correlation and cRMSE (e.g., 12.5 in July). The model overestimates precipitation in February and October, while March and November are underestimated. The most notable improvements are observed in NorESM2. The model exhibits lower errors and positive correlation, especially during the summer half of the year; for instance, a significant (48%) reduction, from 65 in CMIP5 to 34 in CMIP6, is observed (

Table 3. Summary monthly statistics (cRMSE, r and SD difference) of CMIP5 (left) and CMIP6 (right) model performance against CRU observational data for precipitation during 1981–2000. Yellow highlighting indicates the best-performing model simulation, and green highlighting indicates the second-best-performing model simulation.

		CMIP5			CMIP6
Month	Model	cRMSE	Correlation	Std_Dev Difference	RMSE	Correlation	Obs_Std_Dev Difference
December	HadGEM3	29.6	0.9	21.5	25.0	0.9	11.5
December	MPI_ESM1	26.4	0.9	14.7	21.3	0.9	8.8
December	NorESM2	65.0	0.9	55.3	34.1	0.9	28.4
January	HadGEM3	30.8	0.8	7.7	28.9	0.9	21.5
January	MPI_ESM1	32.4	0.8	12.3	29.3	0.8	9.6
January	NorESM2	63.0	0.8	51.1	32.2	0.9	22.1
February	HadGEM3	23.2	0.8	−1.4	24.2	0.8	1.9
February	MPI_ESM1	33.4	0.7	2.9	39.1	0.6	6.5
February	NorESM2	42.3	0.7	17.5	27.2	0.8	7.9
March	HadGEM3	15.8	0.8	−0.3	20.3	0.8	6.6
March	MPI_ESM1	27.3	0.8	14.8	25.7	0.7	10.9
March	NorESM2	41.1	0.7	27.7	21.8	0.9	14.5
April	HadGEM3	15.6	0.7	9.0	16.2	0.8	10.2
April	MPI_ESM1	26.7	0.5	16.5	20.1	0.7	13.7
April	NorESM2	25.2	0.5	15.2	21.8	0.7	15.9
May	HadGEM3	14.1	0.6	5.6	11.8	0.7	3.5
May	MPI_ESM1	17.9	0.5	8.4	13.0	0.7	5.9
May	NorESM2	20.2	0.3	6.7	13.0	0.6	2.4
June	HadGEM3	12.0	0.6	−1.8	12.4	0.7	1.3
June	MPI_ESM1	14.3	0.5	0.8	10.7	0.8	1.5
June	NorESM2	13.3	0.6	−0.4	10.3	0.7	-3.9
July	HadGEM3	11.4	0.7	−2.9	9.2	0.8	-2.5
July	MPI_ESM1	13.9	0.5	−2.4	12.5	0.7	2.2
July	NorESM2	14.5	0.5	−1.9	10.4	0.7	-5.4
August	HadGEM3	12.3	0.7	3.7	12.4	0.8	5.6
August	MPI_ESM1	18.4	0.6	9.2	13.2	0.7	5.9
August	NorESM2	19.3	0.5	9.1	8.1	0.8	-2.0
September	HadGEM3	16.7	0.8	10.9	14.5	0.9	10.2
September	MPI_ESM1	13.2	0.8	7.0	16.4	0.9	12.1
September	NorESM2	24.5	0.7	16.9	11.6	0.8	5.2
October	HadGEM3	18.1	0.9	10.6	18.3	0.9	13.0
October	MPI_ESM1	13.2	0.9	3.5	18.0	0.9	6.4
October	NorESM2	33.1	0.9	26.6	15.6	0.9	4.2
November	HadGEM3	27.5	0.9	20.0	28.8	0.9	23.1
November	MPI_ESM1	20.9	0.9	15.5	21.4	0.9	9.5
November	NorESM2	48.6	0.9	42.9	26.9	0.9	20.5

) in December. Despite these improvements, the model persistently shows interannual variability over South Africa. The results indicate that HadGEM is consistently the best-performing model in both model runs. However, some models tend to underestimate precipitation patterns during summer months, while substantial overestimations are noted in different models, between December and August. This is consistent with [40], who reported that CMIP6 models can reproduce the general precipitation climatology over West Africa; however, underestimations were observed during the rainy season. The CMIP6 shows substantial improvement in HadGEM3’s ability to reproduce precipitation climatology in the Western Cape, compared to CMIP5. For example, it was noted that HadGEM3 outperforms HadGEM2 in September, December, January, May, and July, in terms of cRMSE and correlation.

A comparison between CORDEX and CMIP6 indicates that the models have substantially improved from CMIP6. The CMI P6 model exhibits the lowest overall cRMSE and higher correlation with CRU data, especially in summer months. But the CORDEX model passes, especially the CCL M5 RCM pass, often outperforms CMI P6 every month; for example, HadGEM2 CCLM5 has lower cRMSE and SD than CMIP6 in August to December and March. However, this is coupled with a lower correlation, indicating that CMIP6 has a better linear relationship with CRU data. The MPI-CCLM5 outperforms CMIP6 MPI-ESM1, with lower cRMSE and SD in 9 of 12 months, exhibiting higher errors in winter (JJA). The model also has a higher correlation in 11 of 12 months. However, the model exhibits a great underestimation of the SD data in September to November and April. This highlights an improvement in the ability of the CMIP GCMS to reproduce precipitation over South Africa, especially during the JJA (rainy season) in the Western Cape, which can capture the Mediterranean climate. Although the CMIP6 model has advanced physical processes which contribute to its improved performance [41], their ability to reproduce precipitation climatology is still hindered by some parameterizations. Therefore, they may not be perfect [9] for some regional studies, as they tend to underperform against CORDEX models during the wet season, highlighting their importance in regional precipitation analysis. Therefore, CORDEX models are recommended for studies over complex topographic regions, as they may provide accurate insights into local climate adaptation and mitigation studies.

4.5. Limitations

This study had limitations that should be considered when interpreting the results; the model evaluation employed one observational dataset (CRU), which may result in uncertainties, especially over the regions without sufficient station data. Additionally, the study evaluated the individual models against statistical matrices, excluding more advanced methodologies such as trend analysis, extreme indices and ensemble means. Future studies should address these limitations through the use of multiple observational data sets and the evaluation of more climate model ensembles.

5. Conclusions

This study evaluated the performance of CMIP5, CMIP6 and CORDEX model outputs in reproducing temperature and precipitation climatology over South Africa during the historic period 1981–2000, using gridded observational data from CRU. Our findings indicate that CORDEX and CMIP6 model outputs indicate substantial improvements on their predecessors, with CMIP6 model outputs outperforming CMIP5. The models exhibited lower cRMSE, higher correlation and less bias in both temperature and precipitation. Despite observed improvements, the models showed biases in precipitation climatology. The results show that CORDEX model outputs exhibit better ability in capturing regional climate variability, especially over the complex topographies of South Africa. This is attributed to their finer resolution and model setup. Therefore, CORDEX model simulations are more preferable for regional studies, due to their ability to capture local climate. This study highlights the importance of model selection and the use of multiple model outputs in climate projection studies. Future studies should focus on the understanding of the underlying mechanisms that cause persistent biases, especially over complex topographies, and further employ multi-model ensemble approaches and bias correction for more robust results that will inform decision-making.