Assessing the Performance of CMIP5 GCMs for Projection of Future Temperature Change over the Lower Mekong Basin

Abstract

In this study, we assessed the performance of 34 Coupled Model Intercomparison Project Phase 5 (CMIP5) general climate models (GCMs) for simulating the observed temperature over the Lower Mekong Basin (LMB) in 1961–2004. An improved score-based method was used to rank the performance of the GCMs over the LMB. Two methods of multi-model ensemble (MME), sub-ensemble from the top 25% ranked GCMs and full ensemble from the entire GCMs, were calculated using arithmetic mean (AM) method and downscaled using the Delta method to project future temperature change during two future time periods, the near future (2006–2049) and the far future (2050–2093), under representative concentration pathways (RCP2.6, RCP4.5, and RCP8.5 scenarios) over the LMB. The improved score-based method combining multiple criteria showed a robust assessment of the GCMs performance over the LMB, which can provide good information for projecting future temperature change. The results showed a significant increase in temperature over the LMB under the two ensembles. However, there were differences in the magnitudes of the future temperature increase between the two ensemble methods, with a higher mean annual temperature increase from full ensemble and sub-ensemble at 1.26 °C (1.09 °C), 1.90 °C (1.70 °C), and 2.97 °C (2.78 °C) during 2050–2093 under the RCP2.6, RCP4.5, and RCP8.5 scenarios compared to the values at 0.93 °C (0.87 °C), 0.99 °C (0.95 °C), and 1.09 °C (1.06 °C) during 2006–2049, respectively, relative to the reference time period of 1961–2004. In the future (2006–2093), the temperature is likely to increase at 0.06 °C, 0.18 °C, and 0.39 °C decade⁻¹ under the RCP2.6, RCP4.5, and RCP8.5 scenarios by the sub-ensemble, while a higher temperature increase at 0.08 °C, 0.20 °C, and 0.42 °C was found by the full ensemble over the LMB, relative to the reference time period of 1961–2004. On the whole, the higher warming mainly occurred in the northern and central areas over the LMB, while the lower warming mainly occurred in the southeast and the southwest, especially under the RCP4.5 and RCP8.5 scenarios, with the warming increased with increasing RCP for both ensembles. Moreover, in order to reduce the uncertainty of temperature projection in further studies in the LMB, multiple methods of GCMs ensemble should be considered and compared.

1. Introduction

Climate change has become one of the most important topics to scientists, the public, and governments around the world. The past 30 years (1983–2012) were probably the hottest in the Northern Hemisphere in the last 1400 years, with the warmest being the first 10 years of the 21st century [1]. Regionally, mean temperature is expected to rise 0.79 °C by the year 2030 compared to 1951–2000 in the Mekong River Basin [2]. Observational evidence from mainland and oceans has shown that the regional climate change associated with the rise in temperature has significant impacts on the population and environment [3].

Recently, multiple criteria have been widely used for assessing the performance of the CMIP5 GCMs in climate variable simulation [4,5,6,7,8]. Rupp et al. [4] assessed 41 CMIP5 GCMs’ abilities in temperature and precipitation simulation of the Pacific Northwest, USA, based on observed metrics such as the correlation and variance of mean seasonal spatial patterns, amplitude of seasonal cycle, diurnal temperature range, annual-to decadal-scale variance, and other metrics. Miao et al. [5] assessed the performance of 24 CMIP5 GCMs in simulating intra-annual, annual, and decadal temperature over Northern Eurasia based on the criteria of correlation, the centered root-mean-square-error, and the amplitude of the standard deviations. Ahmadalipour et al. [6] used criteria such as mean, standard deviation, coefficient of variation, relative change (variability), Mann‒Kendall test, Kolmogorov‒Smirnov test (KS test), and other criteria to assess the performance of 20 CMIP5 GCMs for selecting suitable GCMs for climate change impact analysis over the Columbia River Basin in the Pacific Northwest, USA. Dong et al. [7] assessed the performance of 22 CMIP5 GCMs in simulating annual mean surface air temperature over the critical Belt and Road region using criteria of temporal, spatial, and trend analysis. Das et al. [8] assessed the performance of 34 CMIP5 GCMs in reproducing observed temperature statistics over the Western Himalayan Region of India using criteria including mean seasonal cycle, temporal trends, and spatial correlation. Furthermore, additional researches on the comparison of the ability to reproduce the observed climate variables between CMIP5 and CMIP3 have been done, and many of them showed that CMIP5 models showed improvements compared to CMIP3 models [9,10,11,12]. In addition, many studies have been conducted assessing the performance of the GCMs in reproducing patterns of large-scale climate variability, such as the North Atlantic Oscillation (NAO), Pacific/North America pattern (PNA), North Pacific Oscillation (NPO), and El Niño Modoki [13,14,15]. Lee and Black [13] found that the low-frequency mode structure from model biases has a significant effect on representing associated regional anomalies in surface air temperature and storm track behavior. Ning and Bradley [14] found that the winter temperature correlation changes over the eastern USA are usually highly affected by the NAO and PNA. Wang et al. [15] showed that future projections of El Niño Modoki of the multi-model simulations can be analyzed based on the changes in the warm wind‒evaporation‒SST (WES) feedback intensity and simulations of the NPO. Moreover, Hawkins and Sutton [16] suggested that model uncertainty was more important than internal variability for decadal time scales and regional spatial scales (~2000 km). Therefore, a GCM that can simulate observed temperature reasonably well should be selected before a climate change projection is made [5,17,18].

The fifth assessment report of IPCC put forward a new set of scenarios called representative concentration pathways (RCPs) for climate change projection. In detail, four typical pathways include the radiative forcing levels of 2.6 W/m² (RCP2.6), 4.5 W/m² (RCP4.5), 6.0 W/m² (RCP6.0) and 8.5 W/m² (RCP8.5) by the end of this century, which represent the low (RCP2.6), the low-medium (RCP4.5), the medium-high (RCP6.0), and the high (RCP8.5) greenhouse gas emissions [1]. However, because of the coarse spatial resolution of the GCMs, the outputs from the future projections of the GCMs are inadequate to project the regional climate scenarios in detail [19,20,21]. Thus, downscaling methods have been applied to transfer the coarse outputs of the GCMs to regional or local high-resolution climate change scenarios [21,22]. Currently there are three downscaling methods that are being applied: dynamic downscaling, statistical downscaling, and a combination of both. Statistical downscaling is advantageous in that it requires a small amount of calculation, simple to use and does not consider the effects of boundary conditions on the predicted results [23]. A statistical downscaling method called the Delta method, which is simple and easy to operate, has been widely used for projection of future climate change around the world [22,24,25,26,27,28,29].

Future climate change will affect water yield and rice cultivation in the tributary countries over the Lower Mekong Basin (LMB) [30]. Due to the population growth in the Lower Mekong Basin (LMB), the demands for agricultural products from the LMB is expected to increase by 20–50% in the next 30 years [31]. Additionally, it is likely that the demands for water for agriculture in 2010–2050 will increase as a result of the higher temperatures [32]. Therefore, assessing performance of the GCMs in temperature simulation is essential for projection of future temperature change and for policy-making over the LMB. In previous studies, performance of the GCMs and projection of future temperature change have been mainly focused on the watersheds in the Mekong River Basin [2,33,34,35]. However, fewer studies have focused on the LMB. Meanwhile, no research exits that assessing performance of the GCMs in simulating the observed temperature, based on an improved score-based method, using gauge-based data of the Asian Precipitation-Highly Resolved Observational Data Integration Towards Evaluation of Water Resources (APHRODITE) temperature data in the Lower Mekong Basin. Moreover, the multi-model ensemble projection can improve the temporal simulation relative to a single GCM in the Mekong River Basin [35]. Thus, in this study, we aim to assess the performance of the CMIP5 models in temperature simulation and use multi-model ensemble (MME) to project future temperature change during two periods of the near future (2006–2049) and the far future (2050–2093) under RCP2.6, RCP4.5, and RCP8.5 scenarios over the LMB.

2. Data and Methods

2.1. Data

2.1.1. GCM Data

Thirty-four GCM models from the CMIP5 [36] were used in this study.

Table 1. Basic information of the CMIP5 models used in this study.

Model Name	ID	Institution	Resolution (Lon × Lat)
ACCESS1.0	1	Commonwealth Scientific and Industrial Research Organization and Bureau of Meteorology, Australia	1.88° × 1.25°
ACCESS1.3	2		1.88° × 1.25°
BCC-CSM1.1	3	Beijing Climate Center, China Meteorological Administration, China	2.81° × 2.79°
BNU-ESM	4	College of Global Change and Earth System Science, Beijing Normal University, China	2.81° × 2.79°
CanESM2	5	Canadian Centre for Climate Modelling and Analysis, Canada	2.81° × 2.79°
CCSM4	6	National Center for Atmospheric Research, USA	1.25° × 0.94°
CESM1(CAM5)	7		1.25° × 0.94°
CESM1(WACCM)	8		2.50° × 1.88°
CMCC-CMS	9	Centro Euro-Mediterraneo sui Cambiamenti Climatici, Italy	1.88° × 1.88°
CNRM-CM5	10	Centre National de Recherches Météorologiques Centre Européen de Recherche et Formation Avancée en Calcul Scientifique, France	1.41° × 1.40°
CSIRO-Mk3.6.0	11	Commonwealth Scientific and Industrial Research Organization/Queensland Climate Change Centre of Excellence	1.88° × 1.88°
EC-EARTH	12	EC-EARTH consortium published at Irish Centre for High-End Computing, Netherlands/Ireland	1.13° × 1.13°
FGOALS-g2	13	Atmospheric Sciences and Geophysical Fluid Dynamics/Institute of Atmospheric Physics, Chinese Academy of Sciences, China	2.81° × 2.81°
FIO-ESM	14	The First Institute of Oceanography, SOA, China	2.80° × 2.80°
GFDL-CM3	15	NOAA Geophysical Fluid Dynamics Laboratory, USA	2.50° × 2.00°
GFDL-ESM2G	16		2.00° × 2.02°
GFDL-ESM2M	17		2.50° × 2.02°
GISS-E2-H	18	NASA/GISS (Goddard Institute for Space Studies), USA	2.50° × 2.00°
GISS-E2-R	19		2.50° × 2.00°
HadGEM2-AO	20	National Institute of Meteorological Research, Korea Meteorological Administration, Korea	1.88° × 1.25°
HadGEM2-CC	21	Met Office Hadley Center, UK	1.88° × 1.25°
HadGEM2-ES	22		1.88° × 1.25°
INMCM4.0	23	Russian Academy of Sciences, Institute for Numerical Mathematics, Russia	2.00° × 1.50°
IPSL-CM5A-LR	24	Institute Pierre-Simon Laplace, France	3.75° × 1.89°
IPSL-CM5A-MR	25		2.50° × 1.27°
IPSL-CM5B-LR	26		3.75° × 1.89°
MIROC4h	27	Atmosphere and Ocean Research Institute (The University of Tokyo), National Institute for Environmental Studies, and Japan Agency	0.56° × 0.56°
MIROC5	28	for Marine-Earth Science and Technology, Japan	1.41° × 1.40°
MIROC-ESM	29		2.81° × 2.79°
MIROC-ESM-CHEM	30		2.81° × 2.79°
MPI-ESM-LR	31	Max Planck Institute for Meteorology, Germany	1.88° × 1.87°
MPI-ESM-MR	32		1.88° × 1.87°
MRI-CGCM3	33	Meteorological Research Institute, Japan	1.13° × 1.12°
NorESM1-M	34	Bjerknes Centre for Climate Research, Norwegian Climate Center, Norway	2.50° × 1.89°

provides a detailed description of the origin and their spatial resolution. More details can be found on the CMIP5 website (http://cmip-pcmdi.llnl.gov/cmip5/index.html). Meanwhile, detailed information for the future projection of 2006–2093 on the RCP2.6, RCP4.5, and RCP8.5 scenarios were also acquired from the website.

2.1.2. Temperature Data

Gauge-based data from the Asian Precipitation-Highly Resolved Observational Data Integration Towards Evaluation of Water Resources (APHRODITE) temperature dataset were used as the observational data. APHRODITE temperature data have a high spatial resolution of 0.25° × 0.25° [37], which has been proven to be a better gridded temperature product for studies of climate changes in the Mekong River Basin [38]. APHRODITE temperature data are taken daily. Thus, we calculated them into monthly data before assessment.

Because of the different spatial resolutions of the GCMs and the observation, monthly temperature outputs for the 34 GCMs and APHRODITE temperature data were converted to 2.5° × 2.5° using bilinear interpolation, and 21 grids were selected for comparison (Figure 1). In this paper, we consider 1961–2004 as the reference time period. The LMB temperature data were calculated by taking the arithmetic mean of the 21 grids.

2.2. Methods

2.2.1. Assessment of the Performance for CMIP5 GCMs

A GCM that performs well for a specific statistic does not necessarily perform well for a different statistic; moreover, an individual assessment criterion can produce a biased assessment [39]. Thus, it is beneficial to use multiple criteria for comprehensively assessing the performance for CMIP5 GCMs. Thus, we used seven assessment criteria, including Mean value (M), Standard deviation (SD), Normalized Root mean square error (NRMSE), linear correlation coefficient (r) for spatial distribution, Mann‒Kendall test statistic Z, Sen’s slope, and Significance score (S_score). Then we used an improved RS (Rank Score) method [39,40] to rank the GCMs to comprehensively assess the performance of GCMs for temperature simulation over the LMB.

• Mean value
  The mean value (M) is defined as follows:

  $$\mathrm{M}=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{T}_i$$

  (1)
  Here T_i represents the monthly temperature for the LMB at ith time step, and n represents the total number of the time steps.
• Standard deviation
  The standard deviation (SD) is defined as follows:

  $$\mathrm{SD}=\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^n{(T_i-\overline{T})}^2}{n-1}}$$

  (2)
  Here T_i represents the monthly temperature for the LMB at ith time step, $\overline{T}$ represents the mean monthly temperature for the LMB, and n represents the total number of the time steps. A smaller value of SD indicates a better performance of a GCM.
• Normalized root mean square error
  The normalized root mean square error (NRMSE) is defined as follows:

  $$\mathrm{NRMSE}=\frac{\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{(T{m}_i-T{o}_i)}^2}}{\sqrt{\frac{1}{n-1}{{\displaystyle \sum }}_{i=1}^n{(T{o}_i-T\overline{o})}^2}}$$

  (3)
  Here Tm_i and To_i represent the monthly temperature for the GCM and the observed value for the LMB at ith time step, respectively. $T\overline{o}$ represents the mean value of the observed value for the LMB. n represents the total number of the time steps. A smaller value of NRMSE indicates a better performance of a GCM.
• Linear correlation coefficient (r) for spatial distribution
  The correlation coefficient (r) was used to as a measure to compare spatial distribution of temperature between the observation and the GCMs. The sample size is 21, and r was calculated between the observation and the GCMs for long-term mean values of each grid. The formula is defined as follows:

  $$\mathrm{r}=\frac{{{\displaystyle \sum }}_{i=1}^n(T{m}_i-T\overline{m})(T{o}_i-T\overline{o})}{\sqrt{{{\displaystyle \sum }}_{i=1}^n{(T{m}_i-T\overline{m})}^2·{{\displaystyle \sum }}_{i=1}^n{(T{O}_i-T\overline{O})}^2}}$$

  (4)
  Tm_i and To_i represent temperature for the GCM and the observation of the mean annual values at ith grid, respectively, and the $T\overline{m}$ and $T\overline{o}$ represent the corresponding mean values for temperature of the GCM and the observation of all grids, respectively. A larger value of the r indicates a better performance of a GCM.
• Mann‒Kendall test statistic Z and Sen’s slope
  The Mann‒Kendall test statistic Z and Sen’s slope were used to obtain the trends and their magnitudes for GCMs and observations. Thus, the ability of how well the GCMs represent the variation trend of the observations can be obtained. The statistics of the annual time series were used for analysis.
  The Mann‒Kendall test statistic Z is defined as follows [41,42,43]:

  $$\mathrm{Z}=\{\begin{array}{cc}\frac{\mathrm{S}-1}{\sqrt{\mathrm{var}\left(\mathrm{S}\right)}},& \mathrm{S}>0\hfill \\ 0,& \mathrm{S}=0\hfill \\ \frac{\mathrm{S}+1}{\sqrt{\mathrm{var}\left(\mathrm{S}\right)}},& \mathrm{S}<0\hfill \end{array}$$

  (5)

  and

  $$\mathrm{S}={{\displaystyle \sum }}_{i=1}^{n-1}{{\displaystyle \sum }}_{k=i+1}^n\mathrm{sgn}(x_k-x_i)$$

  (6)
  Here x_k, x_i are the sequential temperature values, n is the length (44) of the dataset, and

  $$\mathrm{sgn}\left(\theta \right)=\{\begin{array}{l}1,\theta >0\\ 0,\theta =0\\ -1,\theta <0\end{array}$$

  (7)

  and

  $$\mathrm{var}\left[\mathrm{S}\right]=\left[n\left(n-1\right)\left(2n+5\right)-{\displaystyle \sum }_{t}t(t-1)(2t+5)\right]/18$$

  (8)
  Here t is the extent of any given tie and ∑ denotes the summation over all ties.
  The Sen’s slope is defined as follows [44,45]:

  $$\beta =\mathrm{Median}\left(\frac{x_i-x_j}{i-j}\right),\forall j<i$$

  (9)
  Here 1 < j < i < n, and the slope estimator β represents the median of the entire dataset.
• Probability density functions (PDF)
  The Significance score (S_score) was used to assess the GCM’s probability density functions (PDF) for monthly temperature. The formula is defined as follows [46]:

  $${\mathrm{S}}_{\mathrm{score}}={{\displaystyle \sum }}_{i=1}^n\mathrm{Minimum}\left(B_{Mi},B_{oi}\right)$$

  (10)
  Here B_mi and B_oi represent the probability of GCM and observed temperature values at the ith of bin, respectively.
  The n is the number of bins, and it was set to 100 according to the data range. The S_score is a measurement of the degree of overlap between the simulated probability distribution and the observed value [46]. Thus, a larger S_score value indicates a better performance of a GCM [39].
• Improved RS (Rank Score)
  The improved RS distinguishes between the relative error index and non-relative error index in comparison to the RS method, which could avoid inconsistent results [40]. For example, a smaller value of NRMSE of the relative error index indicates a better performance of a GCM, while a larger value of r of the non-error index indicates a better performance of a GCM. Thus, the improved RS can be used for different assessment criteria and climatic variables to comprehensively assess the performance of GCMs in the regions. The Rank Score of each assessment criterion can be calculated by its statistic [40]:

  $${\mathrm{RS}}_i=\{\begin{array}{c}1-\frac{T_i{-T}_{\min }}{T_{max}{-T}_{min}},Trepresentstherelativeerrorindex\\ \frac{T_i{-T}_{\min }}{T_{\max }{-T}_{\min }},Trepresentsthenon-relativeerrorindex\end{array}$$

  (11)
  Here RS_i represents the score for GCM calculated by an assessment criterion i. For the relative error indexes of M, SD, Z and Sen’s slope, T_i represents absolute error that was calculated between a GCM and the observation (Equation (12)), and T_min and T_max represent the corresponding minimum and maximum among all GCMs. Moreover, for the relative error indexes of NRMSE, T_i represents the absolute value of statistic for a GCM, and T_min and T_max represent the corresponding minimum and maximum among all GCMs. For the non-relative error index of r and S_score, T_i represents the absolute value of the statistic for a GCM, and T_min and T_max represent the corresponding minimum and maximum among all GCMs.
  $$T_i=\left|T_{sm}{-T}_{so}\right|$$

  (12)
  Here T_sm and T_so represent the statistics of the GCM and the observation, respectively.
  Therefore, the overall RS for temperature can be calculated as follows:

  $$\mathrm{R}{\mathrm{S}}_T={{\displaystyle \sum }}_{i=1}^n\frac{\mathrm{R}{\mathrm{S}}_i·W_i}{W_s}$$

  (13)
  Here RS_T represents the overall RS of temperature for the GCM. Here n = 7 and i represents an assessment criterion, such as M, SD, NRMSE, Z, Sen’s slope, spatial distribution r, and S_score. W_i represents the weight for an assessment criterion i, W_s represents the sum weight of all the assessment criteria. Since Z and Sen’s slope are part of trend analysis, we set 0.5 weight for Z, Sen’s slope, respectively, while 1.0 weight for M, SD, NRMSE, r, and S_score, respectively.

2.2.2. Projection of Future Temperature Change

Based on the results of the performance for GCMs, we used two multi-model ensemble (MME) methods (sub-ensemble and full ensemble) to make comparison of future temperature projection under RCP2.6, RCP4.5, and RCP8.5 scenarios, which was calculated using arithmetic mean (AM) method. The formula can be defined as follows:

$$\mathrm{MME}=\frac{1}{n}{{\displaystyle \sum }}_{j=1}^n{T}_{ji}$$

(14)

For the sub-ensemble method, j represents the GCM that ranked within the top 25% of all the GCMs, $T_{ji}$ represents the future monthly temperature of GCM j at ith time step, and n represents the total number of the GCMs ranked within the top 25% of all the GCMs. For the full ensemble method, j represents the entire GCMs, which have outputs of three RCP scenarios, $T_{ji}$ represents the future monthly temperature of GCM j at ith time step, and n represents the total number of the GCMs that has data of three RCP scenarios.

Subsequently, we applied the Delta method to calculate temperature changes between the historical MME outputs of temperature for the reference period 1961–2004 and projections of temperature under the same time duration of 44 years in the future, which can be defined as follows:

$$\Delta T={\overline{T}}_{s,i}-{\overline{T}}_{r,i}$$

(15)

Here ${\overline{T}}_{s,i}$ and ${\overline{T}}_{r,i}$ represent mean temperature for month i of the projections of 44 years in the future and the historical MME outputs of the reference period of 1961–2004, respectively. We focused on two 44 years of 2006–2049 and 2050–2093 under RCP2.6, RCP4.5, and RCP8.5 scenarios for the projection of future temperature changes.

3. Results

3.1. Annual Cycle of Temperature

Temperature variation of the mean annual cycle for the observation and the 34 GCMs for the period 1961–2004 over the LMB is shown in Figure 2. Most of the GCMs had effectively reproduced the single-peak pattern of temperature in the mean annual cycle. The errors between the 34 GCMs and the observed temperature ranged from −3.02 °C to 1.06 °C, −2.68 °C to 3.08 °C, −3.21 °C to 1.30 °C, −3.19 °C to 0.09 °C, and −4.41 °C to 0.33 °C, with mean errors of −0.97°C, −0.02 °C, −0.58°C, −1.35 °C, and −1.94 °C for the mean temperature of annual, spring (March–April–May), summer (June–July–August), autumn (September–October–November), and winter (December–January–February), respectively (Figure 2). Moreover, the number of the negative errors showed by the 34 GCMs were 29, 18, 26, 32, and 32 for the mean temperature of annual, spring, summer, autumn, and winter, respectively. This indicated that most of the GCMs tended to underestimate temperature relative to the observation, especially during the winter.

3.2. Characteristics of Statistics in Criteria

Table 2. Statistics of assessment criteria of the GCMs over the LMB. “Obs” represents the observation, the ID numbers represent the GCMs, which the same as the ID in Table 1.

ID	Means (°C)	SD	NRMSE	r	Trend		Sscore
					Z	Sen’s slope (°C/yr)
Obs	24.62	2.13			2.80	0.01
1	25.49	2.94	0.81	0.952	3.15	0.013	0.62
2	25.44	3.77	1.04	0.972	1.18	0.0072	0.55
3	23.63	2.88	0.77	0.896	3.04	0.0123	0.61
4	23.68	3.21	0.84	0.886	2.42	0.0125	0.65
5	24.48	2.14	0.52	0.874	4.00	0.0215	0.53
6	22.73	3.18	1.12	0.897	2.54	0.0138	0.66
7	22.57	2.71	1.12	0.904	1.57	0.0096	0.65
8	23.51	2.58	0.74	0.876	2.17	0.0101	0.62
9	25.00	3.06	0.84	0.930	3.04	0.0233	0.60
10	23.08	2.73	0.93	0.910	3.35	0.0132	0.66
11	23.82	3.54	0.92	0.929	2.30	0.0098	0.59
12	22.13	1.57	1.30	0.930	3.35	0.0149	0.62
13	22.86	2.52	1.03	0.883	4.00	0.0223	0.65
14	23.65	3.36	0.88	0.901	2.78	0.0157	0.60
15	22.94	2.91	1.03	0.972	1.81	0.0088	0.51
16	22.41	3.30	1.32	0.947	3.55	0.0213	0.53
17	22.88	3.34	1.18	0.959	2.78	0.0213	0.51
18	23.90	2.57	0.63	0.927	2.58	0.0096	0.70
19	24.35	2.46	0.54	0.922	2.50	0.0111	0.64
20	25.68	3.09	0.88	0.939	2.76	0.0143	0.61
21	24.42	3.04	0.74	0.919	3.57	0.0155	0.60
22	24.79	3.04	0.74	0.933	2.68	0.011	0.59
23	21.60	3.13	1.57	0.829	2.68	0.0133	0.51
24	22.89	2.94	1.01	0.915	4.30	0.0186	0.64
25	23.49	3.08	0.84	0.916	5.23	0.0219	0.62
26	22.92	2.86	0.97	0.903	4.72	0.021	0.66
27	24.42	3.03	0.62	0.936	5.27	0.0228	0.67
28	24.25	2.82	0.60	0.947	0.31	0.0026	0.69
29	23.82	3.15	0.79	0.933	3.73	0.0189	0.57
30	23.87	3.14	0.79	0.931	2.52	0.0141	0.57
31	24.18	3.13	0.75	0.932	3.49	0.0226	0.61
32	24.28	3.17	0.75	0.925	2.48	0.0156	0.63
33	22.88	3.21	1.09	0.934	3.53	0.014	0.59
34	22.00	3.14	1.41	0.884	2.84	0.0135	0.62

shows the M, SD, NRMSE, r, Mann‒Kendall test statistic Z, Sen’s slope, and S_score of the temperature for the 34 GCMs. The M for the temperature of the GCMs ranged from 21.60 °C to 25.68 °C, with mean and median values at 23.65 °C and 23.67 °C, respectively. A smaller absolute error indicates a better correspondence with the observed temperature for the GCM. CanESM2, HadGEM2-ES, and HadGEM2-CC had the smallest absolute errors (0.14 °C, 0.17 °C, and 0.20 °C, respectively), while INMCM4.0, NorESM1-M, and EC-EARTH had the largest absolute errors (3.02 °C, 2.62 °C, and 2.49 °C, respectively).

The SD values of the GCMs range from 1.57 to 3.77, with mean and median values of 2.96 and 3.05, respectively. CanESM2, GISS-E2-R, and FGOALS-g2 had the lowest absolute errors at 0.01, 0.33, and 0.40, respectively, indicating good simulation of the observed temperature. However, ACCESS1.3, CSIRO-Mk3.6.0, FIO-ESM had the highest absolute errors at 1.64, 1.42, and 1.23, respectively, indicating poor simulation of the observed temperature.

The NRMSE values of the GCMs ranged from 0.52 to 1.57, with mean and median values at 0.86 and 0.92, respectively. CanESM2, GISS-E2-R, and MIROC5 had the smallest NRMSE values (0.52, 0.54, and 0.60, respectively), while INMCM4.0, NorESM1-M, and GFDL-ESM2G showed the largest values (1.57, 1.41, and 1.32, respectively).

Most of the GCMs showed relatively high r, with minimum and mean r values of 0.83 and 0.92, respectively, and 26 of the 34 GCMs had higher r (r > 0.9), indicating good reproducibility of characteristics of time series and spatial distributions of the observed temperature. ACCESS1.3, GFDL-CM3, and GFDL-ESM2M had the highest spatial r at 0.972, 0.972, and 0.959, respectively, but the lowest values for INMCM4.0, CanESM2, and FGOALS-g2 were at 0.829, 0.874, and 0.883, respectively.

The observed temperature showed a significantly positive trend, with a Z statistic of 2.80, at the 0.01 significance level and a Sen’s slope of 0.01 °C/year. The Z statistics and Sen’s slope of the 34 GCMs ranged from 0.31 to 5.27, 0.0026 to 0.0233, respectively. Out of the 34 GCMs, all were able to reproduce the positive trend of the observed temperature, and 30 (28) GCMs showed a significantly positive trend of observed temperature at the significance level of 0.05 (0.01) compared to that at 0.01 significance level for the observed temperature. Moreover, only six GCMs showed statistics of Sen’s slope smaller than the observed temperature, indicating that most of the GCMs simulate higher temperature increase during 1961–2004 compared to the observed temperature. The results indicated that more than 80 percent of the GCMs showed relatively good reproducibility of the positive trend of the observed temperature. FIO-ESM, GFDL-ESM2M, and NorESM1-M had Z statistic values of 2.78, 2.78, and 2.84 closest to the observed temperature. CESM1(WACCM), CSIRO-Mk3.6.0, and GISS-E2-H had Sen’s slope values of 0.0101, 0.0098, and 0.0096, which are in near agreement with the observed temperature.

The S_score of the GCMs ranged from 0.51 to 0.70, with mean and median values at 0.61 and 0.62, respectively. Out of the 34 GCMs, 21 showed the S_score values of more than 0.6. This indicated that more than half of the GCMs models had relatively good ability in reproducing the characteristics of the probability distribution function of the observed temperature.

3.3. Comparison of the Performance of the GCMs

Ranking scores of the seven criteria of the GCMs are shown in Figure 3, which were calculated based on the criteria statistics. Overall ranking scores of the GCMs are shown in Figure 4, which were calculated based on the ranking scores of the eight criteria. For a set of different criteria, a GCM may perform well for one criterion but badly for another. For example, CanESM2 had the highest ranking score value of 1 based on the M but the low ranking score value at 0.14 for the Sen’s slope. Although GFDL-ESM2M had the lowest ranking score value at 0 by S_score, it had the highest ranking score value at 1 by Z. For the overall ranking scores of the GCMs, a GCM may perform well for one or two criteria but badly for the overall result. For example, ACCESS1.3 had the highest ranking score value at 1 by r but low overall ranking scores at 0.51. Although INMCM4.0 had high ranking score value at 0.92 by Z, it had the lowest ranking score value at 0.21 for the overall result. This indicated that the results of GCMs performance relied mainly on the assessment of the criterion. Thus, using multiple criteria to comprehensively assess the performance of GCMs is essential.

3.4. Sensitivity Analysis of the GCMs Performance

The sensitivity analysis of the GCMs performance were made in two ways. First, the overall ranking scores were compared to the ranking scores after removing one criterion (Figure 5). The results showed overall ranking score had not been changed as a whole after removing each criterion individually, indicating that improved score-based method combining multiple criteria showed a robust assessment of the GCMs performance. Second, the temporal changes of the overall ranking scores were evaluated by comparing the results of two different reference time periods. In this paper, we selected the longest time series of 1961–2004 as the reference time period. Thus we selected a relatively short time series of 1975–2004 as another reference time period for comparison. As shown in Figure 6a, as a whole, the ranking scores of the GCMs had not changed much as the reference time periods changed. Moreover, the top 25% ranked GCMs were listed separately for making comparison under the two reference time periods (Figure 6b,c). Although the ranking scores of the top 25% ranked GCMs were different under the two reference time periods, ACCESS1.0, GISS-E2-H, GISS-E2-R, HadGEM2-ES, HadGEM2-CC, MIROC5, and MIROC4h were all ranked within the top 25% under the two reference time periods, except CanESM2 and CNRM-CM5 (red bar in Figure 6b) ranked within the top 25% for the reference time period of 1975–2004 and CESM1(WACCM) and MPI-ESM-MR (red bar in Figure 6c) ranked within the top 25% for the reference time period of 1961–2004. These indicated that the results of performance of the GCMs based on the improved score-based method are reliable and can be used as a reference for projecting temperature change in the future over the LMB.

3.5. Future Temperature Projection

The top 25% ranked GCMs (except the ACCESS1.0, HadGEM2-CC, and MIROC4h because of not having the outputs under the RCP 2.6 scenario) and the entire GCMs (except the ACCESS1.0, ACCESS1.3, CMCC-CMS, HadGEM2-CC, MIROC4h, INMCM4.0, and IPSL-CM5B-LR because of not having the outputs under the RCP 2.6 scenario) were used for calculating projections from the sub-ensemble and the full ensemble by arithmetic mean (AM) method, respectively. Then, we assessed the future temperature change under two periods of the near future (2006–2049) and the far future (2050–2093) under RCP 2.6, RCP 4.5, and RCP 8.5 scenarios based on the Delta method.

Table 3. Changes in mean annual and seasonal temperature (°C) over the LMB under RCP2.6, RCP4.5, and RCP8.5 relative to the period of 1961–2004 under two ensemble methods. Values in parentheses represent the results of the full ensemble.

Period	RCP2.6					RCP4.5					RCP8.5
	Annual	MAM	JJA	SON	DJF	Annual	MAM	JJA	SON	DJF	Annual	MAM	JJA	SON	DJF
2006–2049	0.87 (0.93)	0.86 (0.97)	0.89 (0.95)	0.85 (0.95)	0.87 (0.84)	0.95 (0.99)	1.01 (1.14)	0.97 (0.98)	0.92 (0.95)	0.88 (0.91)	1.06 (1.09)	1.12 (1.18)	1.05 (1.07)	1.02 (1.07)	1.04 (1.02)
2050–2093	1.09 (1.26)	1.16 (1.37)	1.04 (1.22)	1.03 (1.22)	1.13 (1.23)	1.70 (1.90)	1.70 (2.07)	1.61 (1.80)	1.65 (1.84)	1.86 (1.90)	2.78 (2.97)	2.76 (3.17)	2.69 (2.92)	2.76 (2.93)	2.91 (2.86)

shows that the LMB had higher mean annual temperature increase during 2050–2093 under RCP2.6, RCP4.5, and RCP8.5 scenarios compared to the period of 2006–2049, respectively, relative to the reference period of 1961–2004. Moreover, the projection by the full ensemble showed higher mean annual temperature increase compared to the sub-ensemble, with mean temperature increase at 1.26 °C (1.09 °C), 1.90 °C (1.70 °C), and 2.97 °C (2.78 °C) during 2050–2093 under RCP2.6, RCP4.5, and RCP8.5 scenarios, and at 0.93 °C (0.87 °C), 0.99 °C (0.95 °C), and 1.09 °C (1.06 °C) for the period 2006–2049. Additionally, the projection by the sub-ensemble showed that MAM (March–April–May) had the highest mean temperature increase during 2006–2049 under RCP4.5, and RCP8.5 scenarios and during 2050–2093 under RCP2.6 scenario, while DJF (December–January–February) showed the highest mean temperature increase during 2050–2093 under RCP4.5 and RCP8.5 scenarios. However, the projection by the full ensemble showed the highest mean temperature increase for the MAM for the RCP2.6, RCP4.5, and RCP8.5 scenarios during 2006–2049 and 2050–2093. Overall, the projected annual temperature change in future time period of 2006–2093 relative to the reference period of 1961–2004 showed highest warming for RCP8.5 scenario, followed by RCP4.5 scenario, with the RCP2.6 scenario showing the lowest warming, which is likely to increase at 0.06 °C, 0.18 °C, and 0.39 °C decade⁻¹ under RCP2.6, RCP4.5, and RCP8.5 scenarios by the sub-ensemble, while higher temperature increase at 0.08 °C, 0.20 °C, and 0.42 °C by the full ensemble, with Z statistics of annual temperature increase series at 5.6(8.6), 10.7(12.1), and 12.5(13.2) for the sub-ensemble and full ensemble, respectively (Figure 7), indicating significant temperature increase over the LMB in the future.

Future temperature change under RCP2.6, RCP4.5, and RCP8.5 scenarios of all 21 grids over the LMB are shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. It is projected that the temperature will increase over the LMB. The projected mean annual temperature increase from the far future (2050–2093) showed significantly higher than the near future (2006–2049), and the projection by the full ensemble showed higher temperature increase than the sub-ensemble, with the values range from 1.06 °C to 1.39 °C, 1.55 °C to 2.14 °C, and 2.46 °C to 3.33 °C for the far future (2050–2093) and 0.81 °C to 1.00 °C, 0.83 °C to 1.10 °C, 0.92 °C to 1.20 °C for the near future (2006–2049) by the full ensemble, while range from 0.92 °C to 1.21 °C, 1.45 °C to 1.86 °C, and 2.36 °C to 3.03 °C for the far future (2050–2093) and 0.77 °C to 0.99 °C, 0.82 °C to 1.06 °C, and 0.93 °C to 1.15 °C for the near future (2006–2049) by the sub-ensemble, relative to the reference period of 1961–2004 under RCP2.6, RCP4.5, and RCP8.5 scenarios, respectively. On the whole, the distributions of the mean annual temperature change showed relatively similar spatial distributions between the near future (2006–2049) and the far future (2050–2093) under projections from the sub-ensemble and the full ensemble, showing that the higher warming mainly occurred in the northern and central areas of the LMB, while the lower warming mainly occurred in the southeast and the southwest, especially under the RCP4.5 and RCP8.5 scenarios. Moreover, the warming for individual grid increased as the RCP increased for the sub-ensemble and the full ensemble. For the seasonal change, the similar spatial patterns of temperature changes were also found in the MAM, JJA (June–July–August), SON (September–October–November), and DJF, but the MAM warming showed relatively higher in comparison with other seasonal warming.

4. Discussion

In this paper, most of the CMIP5 GCMs leaned towards underestimating the temperature relative to the observation, with mean annual bias at −3.02 °C for winter over the LMB. Similar results also been shown by Huang et al. [35] that the CMIP5 GCMs underestimated the annual mean surface air temperature relative to the Climate Research Unit temperature data (CRU TS 3.21) in Mekong River Basin. Other study in monsoon region of Sichuan Basin conducted by Bannister et al. [17] showed that mean temperature was underestimated by CMIP5 GCMs, especially during the winter, with bias exceeding −3 °C. Furthermore, another study in the Qinghai–Tibetan Plateau also showed that CMIP5 GCMs underestimated annual and seasonal temperatures, with bias at −2.3 °C for the annual mean, and larger cold biases for autumn and winter [47]. However, a study by Miao et al. [5] showed that most of the CMIP5 GCMs overestimated the annual mean surface air temperature in Northern Eurasia, especially during the winter. In addition, Zazulie et al. [48] also found that most of the GCMs overestimated the monthly mean temperature compared to ERA-Interim over the complex orographic region of the subtropical central Andes. Thus, the uncertainty of CMIP5 GCMs should be considered when applying them to regional scale. Additionally, there remains a need to understand why most of the GCMs underestimated the observed temperature over the LMB.

We used an improved score-based method to assess the performance of the GCMs in reproducing the observed temperature from 1961 to 2004. The improved score-based method combining not only the relative error index but also the non-relative error index, which have been proven to be effectively for the assessment [39,40]. Based on the method, multiple criteria were selected in this study, including Mean value (M), Standard deviation (SD), Normalized Root mean square error (NRMSE), linear correlation coefficient (r) for spatial distribution, Mann‒Kendall test statistic Z, Sen’s slope, and Significance score (S_score). The results of sensitivity analysis showed that the improved score-based method combining multiple criteria showed a robust assessment of the GCMs performance. Thus, the methods can also be used for other study regions for assessing the GCMs performance for reproducing the climatic variables. After that, reliable information about the GCMs performance can be obtained for exploring future climate change.

Based on the results of the performance of the CMIP5 GCMs, the top 25% ranked GCMs were GISS-E2-H, GISS-E2-R, MIROC5, HadGEM2-ES, ACCESS1.0, MPI-ESM-MR, HadGEM2-CC, CESM1(WACCM), and MIROC4h. Study by Huang et al. 2014 [35] found that GISS-E2-H and MRI-CGCM3 was the best GCMs to simulate the annual surface temperature in the Mekong River Basin. A study by Das et al. [8] showed that CMCC-CM, GISS-E2-H, MIROC5, MPI-ESM-LR, MIROC4th, and CCSM4 were top-ranked models in reproducing the observed temperature over the Western Himalayan Region of India. Although different GCMs were likely to rank differently depending on the region [49], we highlighted the similar good performance of GISS-E2-H, MPI-ESM-LR, MIROC5, and MIROC4h over the LMB, which is in close proximity to the Western Himalayan Region of India, and is a part of the Mekong River Basin, both of which are affected by the southwest monsoon.

We used the top 25% ranked GCMs and the entire GCMs to build the multi-model ensemble (MME) for temperature projection over the LMB, respectively. A previous study by Huang et al. [35] also showed that it can improve the temporal simulation compared to a single GCM when assessing the performance of CMIP5 GCMs by MME of Bayesian moving average (BMA) in the Mekong River Basin. Miao et al. [5] also found that the performances of the MMEs by simple model averaging (SMA), reliability ensemble averaging (REA) and Bayesian model averaging (BMA) methods are superior to any single GCM output in simulating temperature over Northern Eurasia. Venkataraman et al. [50] showed that the ensemble mean based on the arithmetic mean (AM) method had advantages in performance of temperature and precipitation simulation compared to a single model. Similar results were also found by Ahmadalipour et al. [51] that the Bayesian model averaging (BMA) ensemble have advantages in simulating precipitation, maximum temperature, and minimum temperature than individual GCMs over the Columbia River Basin. These indicate that MME is superior to that of the single GCM, which can improve the skill scores for projecting future temperature. In this paper, our results showed that the sub-ensemble by the top 25% ranked GCMs and full ensemble by the entire GCMs have similar spatial distributions of temperature projection over the LMB. However, there existed differences about the magnitudes of the future temperature change between the two ensembles. In addition, previous studies also found that projection uncertainty varied as the model ensemble methods changed [5,52]. Moreover, Ahmadalipour et al. [51] found that model uncertainty is the major source of uncertainty in precipitation projection and near-future temperature simulations over the Columbia River Basin. Therefore, in order to reduce the uncertainty of temperature projection in further studies in the LMB, multiple methods of GCMs ensemble should be considered and compared.

5. Conclusions

This study focused mainly on the assessment of the performance of 34 CMIP5 GCMs in simulating the observed temperature using an improved score-based method and the projection of future temperature change over the LMB. The performance was assessed using seven criteria including the M, SD, NRMSE, r, Z, Sen’s slope, and S_score over the LMB. Then, based on the results of the performance for these GCMs, two ways of the multi-model ensemble (MME), sub-ensemble from the top 25% ranked GCMs, full ensemble from the entire GCMs were calculated using arithmetic mean (AM) method and downscaled using the Delta method to project future temperature during two future time periods, the near future (2006–2049) and the far future (2050–2093) under RCP2.6, RCP4.5, and RCP8.5 scenarios, respectively. The major findings of this study are summarized below:

The results of the GCMs assessment varied as the criterion changed. It is essential to comprehensively assess the GCMs by multiple criteria rather than using single criterion. Based on the overall ranking scores, the top 25% ranked GCMs were GISS-E2-H, GISS-E2-R, MIROC5, HadGEM2-ES, ACCESS1.0, MPI-ESM-MR, HadGEM2-CC, CESM1(WACCM), and MIROC4h, which can be used to build the multi-model ensemble (MME) for future temperature projection over the LMB. Meanwhile, the improved score-based method combining multiple criteria showed a robust assessment of the GCMs performance, which can provide good information related to the studies of future temperature change.

The results showed significant temperature increase in the future over the LMB, with higher mean annual temperature increase from full ensemble and sub-ensemble at 1.26 °C (1.09 °C), 1.90 °C (1.70 °C), and 2.97 °C (2.78 °C) during 2050–2093 under RCP2.6, RCP4.5, and RCP8.5 scenarios compared to the values at 0.93 °C (0.87 °C), 0.99 °C (0.95 °C), and 1.09 °C (1.06 °C) during 2006–2049, respectively, relative to the reference time period of 1961–2004. Moreover, the projected seasonal temperature increase from the far future (2050–2093) also showed significantly higher than the near future (2006–2049). In addition, the projection by the full ensemble showed higher temperature increase than the sub-ensemble. The two ensembles showed similar spatial distributions of temperature change, with the higher warming mainly occurred in the northern and central areas of the LMB, while the lower warming mainly occurred in the southeast and the southwest, especially under the RCP4.5 and RCP8.5 scenarios. Moreover, the warming for individual grid increased as the RCP increased for the two ensembles.

The results provided an application of an improved score-based method in the GCMs assessment and the multi-model ensemble (MME) to project future temperature change over the LMB under three scenarios, which can be useful for GCMs assessment and policy making in this region. However, there remains a need to understand why most of the GCMs underestimated the observed temperature over the LMB, and a need to establish more meteorological stations for long-term observation over the LMB as well as a need to extend temperature projection to a larger region such as the entire Mekong Basin or the Southeast Asia for better understanding regional climate change impacts. Moreover, there existed differences about the magnitudes of the future temperature change between the two ways of ensemble. Therefore, in order to reduce the uncertainty of temperature projection in further studies in the LMB, multiple methods of GCMs ensemble should be taken into considered and compared.