Evaluation of the 3DVAR Operational Implementation of the Colombian Air Force for Aircraft Operations: A Case Study

Abstract

This manuscript introduces an exploratory case study of the SIMFAC’s (Sistema de Información Meteorológica de la Fuerza Aérea Colombiana) operational implementation of the Weather Research and Forecasting (WRF) model with a 3DVAR (three-dimensional variational) data assimilation scheme that provides meteorological information for military, public, and private aviation. In particular, it investigates whether the assimilation scheme in SIMFAC’s implementation improves the prediction of the variables of interest compared to the implementation without data assimilation (CTRL). Consequently, this study compares SIMFAC’S 3DVAR-WRF operational implementation in Colombia with a CTRL with the same parameterization (without 3DVAR assimilation) against the ground and satellite observations in two operational forecast windows. The simulations are as long as an operational run, and the evaluation is performed using the root mean square error, the mean fractional bias, the percent bias, the correlation factor, and metrics based on contingency tables. It also evaluates the model’s results according to the regions of Colombia, accounting for the country’s topographical differences. The findings reveal that, in general, the operational forecast (3DVAR) is similar to the CTRL without data assimilation, indicating the need for further improvement of the 3DVAR-WRF implementation.

1. Introduction

Colombia is located in the northwestern part of South America. This geographical location results in a topography influenced by the Andes mountain range, creating unique meteorological conditions that favor the occurrence of precipitation events [1,2,3]. The Colombian Air Force (FAC, Spanish acronym) understands the importance of forecasting meteorological conditions for the strategic planning and development of aircraft operations and has created the Sistema de Información Meteorológica de la Fuerza Aérea Colombiana (SIMFAC), a system to provide applied meteorological products, including aeronautical meteorology, synoptic meteorology, mesoscale processes, aeronautical climatology, cloud physics, numerical weather prediction, remote sensing, and meteorological surveillance, to the public forces (army, navy, FAC, and police), as well as the civil aviation sector.

One of the tools used by SIMFAC to provide meteorological products is the Weather Research and Forecasting (WRF) model, which is a mesoscale model widely used in numerical weather prediction (NWP) [4]. Various studies have been conducted in Colombia to explore the capabilities of this model. For instance, WRF downscaling was employed to analyze the complex region of the Cauca River [5]. Another study focused on the ability of the WRF model to replicate meteorological conditions in several cities throughout Colombia [6]. Furthermore, investigations were carried out using subkilometer horizontal resolution in the Aburrá Valley [7], which demonstrated an improved performance in terms of the surface temperature and wind direction but also revealed the challenge of overestimating the wind speed at such resolutions. Furthermore, sensitivity analyses were conducted on the model’s planetary boundary layer (PBL) [8] and initial conditions [9], highlighting their impact on model outcomes. Notably, identifying the low-level jet, an atmospheric phenomenon associated with hazardous wind and consequently with aircraft, has also been documented [1].

Although the WRF is used to predict meteorological events, it is fundamental to recognize the inherent constraints and uncertainties associated with its output. Factors that contribute to these constraints, as discussed in [10,11], include the complex terrain characteristics, the horizontal resolution [12], the specification of the initial and boundary conditions (IC/BC) [13], the physical parameterizations [13,14,15], the representation of the topographic characteristics in the input data [14], or even that the mathematics are a simplification of reality, as highlighted in [16,17,18].

The preceding discussion emphasizes the weakness and potential of the WRF model and the need for its evaluation. One promising approach to enhance the performance of the model in regional settings, particularly in the complex tropical Andes region (TAR), by using data assimilation (DA) [19]. DA is a mathematical/statistical technique integrating the available information from various sources with a mathematical model [20,21]. Therefore, DA can potentially improve forecasts by reducing the accumulated errors [22]. Various DA techniques have been used within the framework of the WRF model. These include three-dimensional variational data assimilation (3DVAR) [23,24], four-dimensional variational data assimilation (4DVAR) [25,26,27], ensemble methods [28,29], and the ensemble transform Kalman filter–3DVAR (ETKF–3DVAR) system [30]. The operational implementation of the WRF in SIMFAC utilizes the 3DVAR method and assimilates METAR (Aviation Routine Weather Report), radio soundings, SYNOP, RADAR, and GOES-16.

The 3DVAR aims to improve the model’s initial conditions through observation [31] and reduce the errors propagating through time. Several studies have shown that the 3DVAR in the WRF improves the associated errors in the output. So, for example, ref. [32] used it in Polar WRF to predict the wind speed, temperature, and radiation, and ref. [33] showed that assimilating dropsonde and satellite data improved the forecast. However, if the uncertainty in the model output holds, it is necessary to assess the NWP implementations to determine whether the errors are controlled.

The study of the performance of the model is so important that the World Meteorological Organization (WMO) has designed comparison and verification mechanisms for the different products generated by Centers to provide information to their users [34]. In the same way, the European Center for Medium-Range Weather Forecasts (ECMWF) conducts daily evaluations of forecast performance to provide feedback to users and model developers [35]. Moreover, some research centers explore the potential for comparing different methods. For example, ref. [36] assessed the 4DVAR and 3DVAR DA techniques in a cyclone case study. The process of evaluation of the operational implementation usually includes comparing the model’s output with observations [37,38], the sensitivity analysis of the implementations [39], or comparing the ability of different models to represent a variable [40].

Despite the indicated importance of evaluating implementations of NWP models, there is as yet no evidence of research work to account for the operational implementation of the WRF model used by SIMFAC in its forecasts and the impact of the implemented DA scheme. This article discusses the outcomes of the operational forecast implementation of the 3DVAR-WRF (3DVAR from here on) model employed in SIMFAC, compared with a control run without 3DVAR (CTRL). This discussion covers two operational windows and focuses on the surface temperature, pressure, and precipitation variables. For the analysis of the outcome, we use the root mean square error (RMSE), the mean fractional bias (MFB), the correlation factor (Corr), the mean bias (MB), the normalized mean bias (NMB), the percent bias (PBIAS), the equitable threat score (ETS), and associated metrics of the contingency tables (the false alarm ratio and the probability of detection).

This paper is organized as follows: Section 2 presents a comprehensive overview of the WRF model, including the specific parameterization used in the SIMFAC WRF operational implementation and CTRL run. The data used in the assimilation process and the methods for evaluating the results obtained from the simulations are also described. Section 3 presents the results obtained from the ground-based and satellite data evaluations. Finally, in Section 4, we discuss the results and show the conclusions.

2. Model, Methods, and Data

This study focuses on SIMFAC’S operational implementation of the WRF that employed 3DVAR. The 3DVAR data assimilation process seeks the maximum of a posterior probability density function, which is achieved by minimizing a cost function as in [41]:

$$\mathbf{J}\left(\mathbf{x}\right)=\frac{1}{2}{\left(\mathbf{x}-{\mathbf{x}}^b\right)}^T{\mathbf{B}}^{-1}\left(\mathbf{x}-{\mathbf{x}}^b\right)+\frac{1}{2}{\left(\mathbf{y}-{\mathbf{y}}^0\right)}^T{\mathbf{R}}^{-1}\left(\mathbf{y}-{\mathbf{y}}^0\right),$$

(1)

where ${\mathbf{x}}^a=argmin\left(J\left(\mathbf{x}\right)\right)$ is the analysis to be found that minimizes the cost function given the data sources, ${\mathbf{x}}^b$ is the forecast for the problem of estimation or the first guess of the NWP model, and ${\mathbf{y}}^0$ is the assimilated observation. $\mathbf{B}$ and $\mathbf{R}$ are the background error covariance matrix and the observation error covariance matrix, respectively. Further, $\mathbf{y}=H\left(x\right)$ is the model-derived observation transformed from the analysis by the observation operator H.

The primary assumption in (1) is that the observation and background error covariances are described using Gaussian probability density functions (PDFs) with $\mathbf{0}$ mean error. Non-Gaussian PDFs, due, for example, to nonlinear observation operators, are permitted using an appropriate nonquadratic version of the cost function. If a model state $\mathbf{x}$ has n degrees of freedom (the number of grid points multiplied by the number of independent variables), calculating the whole background term ${\mathbf{J}}_b$ of the cost function requires $O\left(n^2\right)$ calculations. For a typical NWP model with $n^2$$({10}^{12}-{10}^{14})$, a direct solution is not feasible in the time interval allotted for DA in operational applications. One practical solution to reduce the computational cost is to calculate ${\mathbf{J}}_b$ with respect to the control variables defined through the relationship $\mathbf{x}=\mathbf{U}\mathbf{v}$. The $\mathbf{U}$ transform is designed to non-dimensionalize the variational problem and permit efficient filtering techniques that approximate the full background error covariance matrix [42].

The quality of a DA method depends on its analysis state, which relies on several components, including the impact of observations [43]. In the case of the 3DVAR method, the effect of observations depends on the background error statistics matrix $\mathbf{B}$, which is crucial for the optimal initialization of the method and, therefore, for its overall success [43,44].

The WRF model estimates the background error statistics matrix using the GEN_BE (Generalized Background Error Covariance Matrix Model), which assumes that the errors are Gaussian and calculated using control variables [45]. However, there are limitations to calculating this vital factor of the DA process, which can introduce uncertainty in the results. The SIMFAC 3DVAR-WRF, reviewed here, used the control variables (CV) for u, v (horizontal wind components in the x and y direction, respectively), T (temperature), $P_s$ (surface pressure), and $R{H}_s$ (pseudo relative humidity), denominated CV7 [46,47]. The matrix $\mathbf{B}$ is defined as:

$$\mathbf{B}=\overline{ϵ{ϵ}^T}\approx \overline{x^{\prime }{x}^{\prime T}},$$

(2)

with $ϵ=x-x^b$, and the background error and ${\mathbf{x}}^{\prime }$ are the perturbed states of the model used to calculate this matrix because $ϵ$ is unknown. The background error statistics matrix $\mathbf{B}$ in the SIMFAC’s implementation was estimated with six prognostics from one month of the WRF model (00, 06, 12, and 18 UTC) of 12 h of duration with six shared and $x^{\prime }$ being the difference between two consecutive outputs [47].

The methodology employed in this case study involved using SIMFAC’s WRF model as implemented without any modifications in its physical parameterizations, data source, and, in general, architecture. The implementation incorporated the 3DVAR DA techniques to assimilate the observational data.

2.1. WRF Model Setup

The WRF model operational implementation performs four daily runs at 00 UTC, 06 UTC, 12 UTC, and 18 UTC, and its parameterization, described here, is based on the study by [48]. SIMFAC’s 3DVAR implementation involves two forecast horizons in cold-start mode. The first horizon, spanning 73 h, is utilized for aircraft operations planning, while the second horizon, lasting 6 h, is specifically for executing these operations.

The SIMFAC application assumes the nested domains shown in Figure 1. This domain configuration is designed to simulate and predict weather patterns in Colombia and the surrounding continental area [48].

The first domain (D01) covers the entirety of Colombia. The second domain (D02) extends beyond the country’s borders and excludes the San Andrés, Providencia, and Santa Catalina islands in the north of Colombia. The topography includes the Andean mountain range in the northwest region of South America.

The San Andrés, Providencia, and Santa Catalina islands belong to Colombia but are not included in the domain D02. Consequently, information regarding these locations is obtained from the output generated by domain D01. The distribution of the domains, in terms of grid number and horizontal resolution, is presented in

Table 1. Number of grid points per domain in the WRF model configuration.

Domain	Latitude	Longitude	Grid
D01 (9 km)	$(-8.42264,17.5811)$	$(-85.1025,-60.8975)$	$300\times 326$
D02 (3 km)	$(-4.90034,12.8275)$	$(-80.0936,-65.7439)$	$532\times 661$

.

The SIMFAC’s WRF implementation provides IC/BC data every 3 h to the model and is obtained from the Global Forecast System (GFS) [49] to a resolution of $0.{25}^{\circ }\times 0.{25}^{\circ }$, consequently, these are the data used in the experiment.

Table 2. The WRF model setup in the SIMFAC operational implementation.

Parameter		Selection in WRF
Domain settings	Coordinate system	Mercator
	Vertical levels	42
	Nesting	One way, run using adaptive time steps
Physic Settings	PBL Scheme	MYJ
	Cumulus option	KF
	Longwave scheme	CAM
	Shortwave scheme	Goddard
	Microphysics	WSM 6-class graupel scheme
	Land surface	Noah Land Surface Model

presents the parameterization used in the SIMFAC operational implementation, which is based on the results of [47,48]. It includes the WSM six-class graupel scheme for microphysics, which is appropriate for high-resolution simulations when precipitation is of interest [50]. The Goddard scheme is used for short-wave radiation, considering both diffuse and direct solar radiation and cloud effects [51]. The Community Atmospheric Model (CAM) scheme is used for longwave radiation [52].

Moreover, the implementation uses the Mellor–Yamada–Janjic (Eta) TKE scheme for the PBL [53] and the Kain–Fritsch (new Eta) scheme for the cumulus option [54] for domain D01. The land surface model uses four soil layers, corresponding to the Noah land surface model. The urban canopy model is configured to activate shadow effects from neighboring points and slope effects. Refer to Table 2 for additional simulation setups.

2.2. The 3DVAR Implementation of SIMFAC

Various data sources are integrated into the SIMFAC’s 3DVAR operational implementation. These include the Geostationary Operational Environmental Satellite (GOES) 16, METAR, radio soundings, and radar [55].

METAR, SYNOP, and radio soundings used for the SIMFAC’s 3DVAR are provided by IDEAM (Instituto de Hidrología, Meteorología y Estudios Ambientales), a public institution in Colombia that offers scientific and technical support for studying nature and environmental resources. IDEAM operates stations to measure meteorological variables at 28 airports in Colombia, some of which provide frequent METAR information.

IDEAM also generates radio soundings and SYNOP. The first is retrieved from the Wyoming radio soundings database [56] and the other from the IDEAM database.

It is important to note that the SIMFAC’s 3DVAR implementation assimilates the radiance of GOES-16 and the radial velocity and reflectivity of radar. All the steps followed in the implementation are depicted in Figure 2. The procedures follow the WRF user guide [4]. It starts with processing static data (boundary and initial conditions) by WPS and continues with the real module. When DA is used, the METAR, SYNOP, and radio soundings observations are organized and transformed to a LITTLE_R file corresponding individually. After that, the data go to OBSPROC, where they are organized and filtered according to the domain and simulation time. In addition, information about altitude and pressure is retrieved based on the data. Finally, a text file is produced to be employed by the WRFDA module. PREPBUFR or BUFR checks the GOES-16 and radar information following a similar process to OBSPROC for the WRFDA module. Having the observation, the WRFDA module updates the boundary and initial conditions, and this information and that generated by the WPS go to the ARW, where the simulation is conducted.

In examining the SIMFAC’s implementation, it was possible to see that there was no bias correction when GOES-16 was assimilated; consequently, the implementation is exposed to systematic errors for this kind of data. Furthermore, it was observed that there was no defined warning when a low amount of information was assimilated or when one kind of data was unavailable. Additionally, it was observed that the bias accepted for the radar data was high. Consequently, the implementation admitted significant errors in this type of information, which could introduce further uncertainty into the forecast.

2.3. Evaluation of the Operational Implementation

The simulations in the experiment spanned the same duration as the 3DVAR operational implementation. While acknowledging that a comprehensive evaluation might be limited within a 73 h window, the obtained results were expected to contribute to the application utilized by SIMFA’s procedure to provide information for aircraft operations in Colombia.

The experiment for the evaluation of the case study consisted of two operational forecast windows (73 h long), where the model implementation was run with DA (3DVAR) and without DA (CTRL), and the results were compared with the actual observations. Both simulations used the same parameterization of the model given in Table 2.

The first simulation window corresponded to 18 August 2020, at 00:00:00, until 21 August 2020, at 00:00:00 UTC (W1), and the second simulation window was 1 September 2020, at 00:00:00, until 4 September 2020, at 00:00:00 UTC (W2).

The selection of the two windows, W1 and W2, offered a valuable opportunity to observe the model’s performance during a critical period of the year when Colombia experiences some of its highest precipitation accumulations. Additionally, these windows coincided with the transition from a drier period to a rainier one in the Andean region.

Each simulation followed the scheme shown in Figure 2 and comprised two stages. In the first stage, the SIMFAC architecture incorporated multiple observation sources (Section 2.2) through WRF-3DVAR directly before running the model. In the second stage, a control run (CTRL) was conducted without DA, where the BC/IC remained unchanged. However, the parameterization, static data, domain setup, and other essential components of the model remained identical to those of the 3DVAR run.

2.4. Ground-Based Data

The simulations obtained in Section 2.3 were compared with the actual surface temperature, surface pressure, and precipitation retrieved from IDEAM and IMERG (Integrated Multisatellite Retrievals for the Global Precipitation Measurement (GPM) mission). Particularly for the validation of precipitation, studies such as [47,57] have classified this variable into rain events for days where the accumulated precipitation is more significant than $0.1$ mm or no rain when the accumulated precipitation is less than or equal to $0.1$ mm. The categorization of precipitation is important for SIMFAC because the warning for aircraft is given in this term as the likelihood of rain or not. This operational need means we did not use the quantitative precipitation forecast (QPF), a method of verification more commonly used in the literature.

Furthermore, we used data from the integrated multi-satellite recovery for the Global Precipitation Measurement (GPM) mission (IMERG) [58]. The IMERG database provides precipitation information every half hour with a spatial resolution of $0.1^{\circ }\times 0.1^{\circ }$.

IMERG data for spatiotemporal validation is valuable because it provides a high level of detail regarding the precipitation patterns over time and space. Some evaluations have shown that it is one of the gridded precipitation products with the best performance in the country ([59]). This product can help identify areas where the model may not accurately predict precipitation. Furthermore, using satellite data for validation is beneficial because it provides an independent source of information not affected by ground-based measurement limitations.

Using actual observations for comparison is crucial for this case study. By comparing the model’s output with the actual observations, the experiment was able to determine how closely the model’s predictions matched real-world conditions.

2.5. Statistical Performance Metrics

The evaluation metrics included in the model study included the root mean square error (RMSE), the mean absolute fractional bias (MFB), the correlation factor (CF) [60], and a contingency table that includes the false alarm ratio (FAR) and the probability of detection (POD) [23]. Additionally, it used percent bias (PBIAS) and the success percentage, which is calculated as the ratio between the number of times the model successfully detects an event and the total number of occurrences of that event. The comparison was conducted by comparing the observed data with the corresponding nearest neighbor values obtained from the output of the WRF model.

The PBIAS is defined in Equation (3) as a percentage that indicates overestimation or underestimation.

$$\mathrm{PBIAS}[\%]=\frac{{\left(y^f\right)}_i-y_i^o}{y_i^o}\times 100,$$

(3)

where $y_i^f$ is the model simulation output, and $y_i^o$ is the observation.

The MFB, defined in Equation (4), is a statistical measure commonly used in assessing weather and climate models and represents the average of the normalized bias for each model–observation pair.

$$\mathrm{MFB}=\frac{2}{M}\sum _{i=1}^M\frac{{\left(y^f\right)}_i-y_i^o}{{\left(y^f\right)}_i+y_i^o}.$$

(4)

M is the number of observations from all valid monitoring station data for the comparison period of interest. A zero value indicates perfect agreement between the model and observations, while positive or negative values indicate overestimation or underestimation. The MFB has a numerical range from $-2$ to 2.

The RMSE represents the differences between the predicted values and the observed values (Equation (5)), defined as

$$\mathrm{RMSE}=\sqrt{\frac{1}{M}\sum _{i=1}^M{\left({\left(y^f\right)}_i-y_i^o\right)}^2}.$$

(5)

The RMSE tries to penalize a high variance by giving errors with larger absolute values more weight than errors with smaller ones. Finally, the Pearson correlation coefficient measures the linear relationship between the simulated values and the observations, written as:

$$CF=\frac{{\sum }_{i=1}^M\left({\left(y^f\right)}_i-\overline{\left(y^f\right)}(y_i^o-\overline{y^o})\right)}{\sqrt{{\sum }_{i=1}^M{\left({\left(y^f\right)}_i-\overline{\left(y^f\right)}\right)}^2}\sqrt{{\sum }_{i=1}^M{(y_i^o-\overline{y^o})}^2}}.$$

(6)

A CF value close to $1.0$ indicates a strong positive relationship, while values around $0.5$ suggest a moderate relationship. Values below $0.3$ indicate a weak relationship, and a low negative value approaching $-1$ represents a strong inverse relationship. A value near 0 indicates a small or negligible relationship between the variables.

The precipitation is considered a categorical event, and its performance is evaluated using contingency tables, which provide information about hits, false alarms, and errors. Hits occur when the model predicts rain, and it does rain, whereas false alarms occur when the model predicts rain, but it does not rain. Errors occur when the model does not predict rain, but it does rain. Correct rejections are identified for observations where the model predicts no rain, and it does not rain [61]. A perfect model would obtain all the values between the correct hits and rejections, leaving the other values at zero.

Table 3. Contingency table used to evaluate precipitation as a categorical event.

Event Forecast	Event Observed
	Yes	No	Marginal Total
Yes	Hit	False Alarm	Fc Yes
No	Miss	Correct Rejection	Fc No
Marginal Total	Obs Yes	Obs No	Sum Total

provides an example of the contingency tables used in this case study.

Contingency tables are an effective way to evaluate the quality of a model’s forecast. These tables can be used to calculate various verification measures for the categorical forecast performance’s probability of detection (POD) or sensitivity, the false alarm ratio (FAR), and other measures. These measures provide valuable insight into the model’s performance and can help identify areas for improvement.

The PC is a measure that indicates the percentage of forecasted events that the model correctly predicted. It is calculated by dividing the total number of correct event forecasts (hits) by the total number of events observed for the model. The POD, also known as the sensitivity, is the percentage of correct forecasts. It is computed by dividing the total number of correct forecasts (hits + correct rejections) by the total number of forecasts.

On the other hand, the FAR is the ratio of false alarms to the total number of event forecasts. The best possible FAR is zero, which means no false alarms, while the worst is one, indicating that all forecasts were false alarms.

3. Results

The results were focused on the temperature, surface pressure, and precipitation due to the importance of these variables for aircraft. Pressure influences the lift of the aircraft through density and is the basis for various flight instruments such as the altimeter, the anemometer, and the variometer.

Figure 3 illustrates the assimilated data within the forecast window W1. Each point in Figure 3a–d represents a location where METAR, radio soundings, SYNOP, and radar data were retrieved, respectively. The number on the right side indicates the quantity of data utilized in the 3DVAR process. Additionally, Figure 3e,f depicts the radiances and brightness temperature retrieved from GOES-16, respectively. It can be observed from Figure 3a–d that the availability of data was relatively low in the lower part (Amazonian and Orinoquia regions). A similar pattern was observed for the window W2.

3.1. Evaluation of Surface Temperature and Pressure

This section discusses the results of the 3DVAR and CTRL model implementation for the surface temperature and pressure in the locations corresponding to the Cali and Rionegro international airports. The data used for analysis corresponded to the available METAR data for the operational forecast windows of the experiment, W1 and W2. It is important to note that the selected aerodromes were located in areas with unique weather conditions because of their proximity to large bodies of water or mountain ranges.

Figure 4 and Figure 5 present the temporal series of the surface temperature and pressure for the aerodromes in Barranqilla, Bogotá, Cali, and Rionegro (Colombia) in the operational forecast window W1. The intention was to see the adjustment of the actual data with the model output (3DVAR and CTRL). The observed pattern can be attributed to the accuracy of the measurement (data report is only integers).

As noted in Figure 4 and Figure 5, there was a substantial similarity between the temporal behavior of the surface temperature and the pressure for the 3DVAR and CTRL run during the forecast window. Sometimes, it was impossible to discern differences between the two implementations. Similar results were observed for the forecast window W2 (not shown).

The previous observation suggests that the 3DVAR did not significantly improve the model’s forecast capabilities for these variables, at least for the studied periods.

Furthermore, Figure 4 and Figure 5 highlight the differences between the observations and the simulations for both variables. This indicates that there may be significant errors in the model predictions, particularly for localized weather conditions such as those found at aerodromes. Accordingly, the observations may have an impact locally and close to the ground, which may fade immediately when the model forecast starts.

In general, while Figure 4 and Figure 5 suggest that there may be few differences between the 3DVAR and CTRL model implementations, they also highlight the need for the further analysis and evaluation of the model’s performance. For this reason, the statistics such as the MFB, RMSE, and CF, as defined in Section 2.5 were introduced too. These statistics can provide valuable insights into the accuracy and reliability of the model’s predictions and can be used to compare the model implementations.

Continuing with the previous discussion,

Table 4. The RMSE, MFB, and CF metrics for the temperature and pressure in the forecast window W1.

Variable	City	3DVAR			CTRL
		MFB	RMSE	CF	MFB	RMSE	CF
Temperature (°C)	Barranquilla	0.047	2.835	0.617	0.0489	2.817	0.619
	Bogotá	0.140	2.343	0.942	0.154	2.473	0.937
	Cali	−0.004	1.687	0.836	−0.009	1.622	0.851
	Rionegro	−0.006	3.119	0.548	−0.004	3.183	0.542
Pressure (hPa)	Barranquilla	0.445	113	0.714	0.419	108	0.726
	Bogotá	−0.475	98.95	0.787	−0.549	100	0.781
	Cali	0.294	125	0.798	0.313	122	0.817
	Rionegro	0.866	192	0.066	0.434	192	0.066

and

Table 5. The RMSE, MFB and CF metrics for the temperature and pressure in the forecast window W2.

Variable	City	3DVAR			CTRL
		MFB	RMSE	CF	MFB	RMSE	CF
Temperature (°C)	Barranquilla	−0.026	1.535	0.810	−0.028	1.589	0.800
	Bogotá	0.032	2.088	0.870	0.028	2.040	0.876
	Cali	−0.029	3.577	0.733	−0.033	3.596	0.730
	Rionegro	−0.039	3.669	0.584	−0.039	3.591	0.604
Pressure (hPa)	Barranquilla	0.518	91.672	0.878	0.4732	89.316	0.877
	Bogotá	−1.145	84.516	0.882	−0.621	83.177	0.884
	Cali	0.313	106.192	−0.156	0.342	106.202	−0.158
	Rionegro	−1.219	183.950	0.111	−0.907	1.839	0.1178

present the statistical performance metrics for the surface temperature and pressure variables in the aerodromes of Barranquilla, Bogotá, Cali, and Rionegro. The tables compare the performance of the 3DVAR implementation with the CTRL simulation during the selected operational forecast windows, W1 and W2.

In accordance with Table 4 and Table 5, it was impossible to generalize the behavior of the model for the locations analyzed. However, the results suggest that, on average, there was either an underestimation or overestimation of the temperature variables depending on where they were observed. However, for the temperature, the MFB showed a better estimation. The results for both variables indicated that the model representation was similar for the two forecast windows studied.

Additionally, Table 4 and Table 5 indicate that except for the pressure and the Cali location in the window W2, the model results had a high linear correlation for both the evaluated variables (temperature and pressure) and the simulation windows studied. However, the RMSE for the temperature suggested, on average, a lower model bias than for the pressure concerning the observations used.

The results in Table 4 and Table 5 suggest that both scenarios exhibited comparable statistical measures for the variables examined and the operational forecasts, W1 and W2, indicating that the performance of the 3DVAR implementation is similar to the CTRL simulation in terms of precision and precision for the operational forecast and the variables evaluated. However, it is essential to note that this observation was based on the specific experimental setup (and station spot measurements) and may not necessarily hold for other scenarios or variables. Therefore, further analysis and evaluation are required to generalize the model’s performance for other applications and scenarios.

3.2. Evaluation of Precipitation

After analyzing the statistical metrics presented in Table 4 and Table 5, it is challenging to determine which implementation, 3DVAR or CTRL, performed better in terms of matching the observations or metrics due to the similarity or the minor differences in both the scenarios and the variables.

This section focuses on a composite variable, precipitation, which is influenced by various factors, including the temperature, pressure, and humidity.

Figure 6 illustrates the average accumulated precipitation over time and space for each domain throughout the 73-hour simulation. Additionally, it showcases the difference between the 3DVAR and CTRL simulations for the selected operational windows, W1 and W2. This comparison helps assess the possible impact of DA on the model’s forecast outcomes in the operational forecast selected.

Interestingly, on average, Figure 6 shows that the differences between the periods were similar temporally for domain D01. However, more spatial differences existed between the CTRL and 3DVAR implementation for the forecast window W2. These spatial differences may be related to incorporating additional data through DA and may impact the temporal differences in the forecast window W2 in the domain D02.

Figure 7 displays the differences between the original ICs and those updated by the 3DVAR for precipitation near the first time step for W2 (see Section 2.2). In particular, the impact was more significant in areas with more ground-based stations and other data sources.

Figure 7a shows that the most significant differences occurred in the Andean and Amazonian regions. Furthermore, Figure 7c shows that the differences between the 3DVAR and CTRL occurred throughout most of the forecast simulation window but became smaller at the end of the window, as we hoped, because the boundary conditions were the same for both simulations, and the forecast horizon was so long. Consequently, the effect of the updated ICs through the DA process vanished.

An evaluation of the IMERG database against the CTRL and 3DVAR implementations revealed that the 3DVAR implementation, on average, produced an IC closer to the database (see Figure 8d). However, Figure 8b,c demonstrates that the CTRL and 3DVAR results overestimated the precipitation across most of the domain, with the largest overestimations observed in the Amazonian and Andean regions.

Figure 9 shows the differences between the IMERG database and the 3DVAR (Figure 9a) and the CTRL (Figure 9b) run. It also presents the MFB results (Figure 9c for the entire domain at each time step to evaluate the total accumulated precipitation of the whole domain.

Figure 9a,b shows the similarity between the 3DVAR and CTRL runs; however, more bias was observed close to the Orinoquia and the Amazonian regions. Furthermore, Figure 9c shows that the 3DVAR had, on average, a more significant underestimate than the CTRL at the beginning of the simulation, similar to Figure 8, where the cumulative total precipitation domain led to a decrease in the error.

When considering the overestimation of precipitation, it is necessary to acknowledge that this discrepancy could stem from various factors. These include the inherent complexity of the model, the representation of the atmospheric processes, the influence of the topography, and the persistent uncertainties in the input data and the IC/BC. Figure 10 presents a division of Colombia into five distinct regions: the Caribbean, Andean, Pacific, Orinoquia, and the Amazonian. These regions are delineated based on heterogeneous characteristics, considering factors such as topographic variations, climate patterns, and vegetation throughout the country. Previous research has indicated that the model’s performance can vary across these different regions.

Additionally, Figure 10 displays the success percentage of the operational implementation and CTRL models for the precipitation during the two operational windows selected, W1 and W2. Figure 10a,b presents the information for W1 and W2, respectively.

Figure 10 displays that the success percentages for the operational forecast windows were similar in the two simulation scenarios (3DVAR and CTRL). However, a slight advantage of the operational implementation was evident for W1 concerning the CTRL run in the domain D01. In contrast, for the domain D02, the CTRL outperformed the operational implementation. In W2, as illustrated in Figure 10b, the CTRL model performed better in both domains.

Figure 10 shows that the model successfully identified the precipitation with percentages exceeding 50% in all regions, except the Andean and Amazonian regions during August and the Pacific region for September. The Andean, Amazonian, and Pacific regions have unique weather patterns and topography, which could have influenced the accuracy of the model in identifying the precipitation. These results suggest the need to account for regional variations in climatic conditions when developing precipitation models, as these can significantly impact the model’s accuracy and performance.

Figure 11 compares the IMERG observations with the implementations of the 3DVAR and CTRL for W2 in the Caribbean, Pacific, Andean, Orinoquia, and Amazonian regions, as defined in Figure 10. The comparison is made in terms of the MFB.

Figure 11 shows significant differences between the model output and the observations for all regions and that the overestimation of the 3DVAR under the initial conditions was more critical in the Amazonian region. In contrast, in other areas, there were less significant differences. This indicates that both model runs tended to overestimate the precipitation in both studied operational forecast windows, W1 and W2. This observation was consistent with the temporal series in Figure 4 and Figure 5 and the statistical measures presented in Table 4 and Table 5, which showed that the CTRL and 3DVAR had similar performances in terms of the MFB and RMSE, but the correlation factor was relatively low.

The results in Figure 11 suggest that the CTRL and 3DVAR implementations were limited in accurately representing the precipitation in the studied regions for the operational forecast windows used, particularly in the Orinoquia and Amazonian regions. The limited availability of the observation data, shown in Figure 3, may have contributed to the model’s inaccuracies. The benchmark used in processing GOES-16 data and radar in the data assimilation process can also contribute to the observed errors.

Further research may be necessary to improve the model’s representation of precipitation in these regions.

Finally, in order to obtain more evidence of the quality of the model forecast to represent the precipitation for the 3DVAR and CRTL implementation, contingency tables for the accumulated precipitation were obtained for W1 and W2.

Table 6. Results of the contingency tables for the 3DVAR and CTRL simulations in the two windows of study, W1 and W2.

Window	3DVAR		CTRL
	POD	FAR	POD	FAR
W1	$0.154$	$0.852$	$0.154$	$0.846$
W2	$0.556$	$0.737$	$0.500$	$0.750$

shows the POD and FA.

The results shown in Table 6 were similar between the CTRL and 3DVAR for each window W1 and W2, suggesting an important opportunity to improve the operational implementation and control run because they indicate a high probability of false alarm and a low rate of detection.

According to the previous results, the differences between the operational implementation and the CTRL were slight. The 3DVAR implementation offered an improvement in the second period, reducing the FAR and increasing the POD. However, these differences were lowest in the calculated statistics.

4. Discussion and Conclusions

This paper presented an exploratory case study of the 3DVAR-WRF used operationally in SIMFAC.

The results of the 3DVAR are used for aircraft operations and are a significant undertaking to provide weather information and improve air safety in the country. The study used two operational forecasts in W1 and W2 and two CTRL simulations conducted for this study in the same forecast windows, with identical parameterizations of the model and input data but with the 3DVAR module turned off. The operational output (3DVAR) and non-operational (CTRL) simulations were compared against observations from the IDEAM and the IMERG databases.

The results indicated that the 3DVAR and CTRL implementations analyzed tended to overestimate the METAR observations and precipitation, regardless of the locations or operational forecast windows selected. The analysis also indicated that both implementations performed poorly regarding the POD and FAR for precipitation. Therefore, there are suggestions that the implementation needs to be inspected.

In particular, the results for the operational forecast windows selected did not show any significant improvement in the model with the 3DVAR implementation. At some locations, the CTRL was slightly closer in terms of the precipitation to the observations.

In general, the 3DVAR implementation did not show significant differences compared to the CTRL, possibly due to several aspects. One may be because, in practice, only some of the information available in the model is being assimilated. It is also possible that there are problems with the observational operators in variables such as the radar reflectivity and the satellite brightness temperature, which generally involve nonlinear relationships with the variables available in the model.

The evaluation represents a step forward in advancing our comprehension of the 3DVAR operational implementation’s capability to forecast weather patterns in Colombia. Nevertheless, further research and in-depth evaluation are necessary to overcome the remaining challenges and to comprehensively understand the model’s performance. This will enable us to make any necessary adjustments to the implementation to enhance the model’s accuracy. Such improvements are vital for supporting aviation decision makers, ensuring reliable weather information for aircraft operations.

The limitation of the operational implementation in the forecast windows studied may stem from several factors, such as the complexity of the model, which affect the representation of atmospheric processes. There could also be uncertainties in the input data and BC/IC. The overestimation of certain variables, including the surface temperature, pressure, and precipitation, could be partly attributed to the model’s spatial resolution, which may not capture small-scale atmospheric features that affect these variables. Further studies could focus on refining the model’s parameterizations and improving the input data quality, especially in remote areas like the Amazonian and Andean regions.

Another aspect that requires further analysis is the construction of the background error covariance matrix, which is a crucial step in the assimilation process. It is necessary to review how this matrix is estimated and whether it is the most appropriate method to obtain it. It is possible that the selection of control variables could be more optimal or that the assumption of the Gaussianity of the error needs to be met. Previous evaluations using ensembles of operational implementation have shown that covariances vary depending on the location, highlighting the need for further research. Concerning this, the authors of [44] emphasized the importance of the background error covariance matrix $\mathbf{B}$ in the cost function (1) and the minimization process. This matrix $\mathbf{B}$ needs to be determined by the user, but it is challenging to obtain or estimate due to the problem’s dimensionality.