Reconstruction of Missing Data in Weather Radar Image Sequences Using Deep Neuron Networks

Abstract

Missing data in weather radar image sequences may cause bias in quantitative precipitation estimation (QPE) and quantitative precipitation forecast (QPF) studies, and also the obtainment of corresponding high-quality QPE and QPF products. The traditional approaches that are used to reconstruct missing weather radar images replace missing frames with the nearest image or with interpolated images. However, the performance of these approaches is defective, and their accuracy is quite limited due to neglecting the intensification and disappearance of radar echoes. In this study, we propose a deep neuron network (DNN), which combines convolutional neural networks (CNNs) and bi-directional convolutional long short-term memory networks (CNN-BiConvLSTMs), to address this problem and establish a deep-learning benchmark. The model is trained to be capable of dealing with arbitrary missing patterns by using the proposed training schedule. Then the performances of the model are evaluated and compared with baseline models for different missing patterns. These baseline models include the nearest neighbor approach, linear interpolation, optical flow methods, and two DNN models three-dimensional CNN (3DCNN) and CNN-ConvLSTM. Experimental results show that the CNN-BiConvLSTM model outperforms all other baseline models. The influence of data quality on interpolation methods is further investigated, and the CNN-BiConvLSTM model is found to be basically uninfluenced by less qualified input weather radar images, which reflects the robustness of the model. Our results suggest good prospects for applying the CNN-BiConvLSTM model to improve the quality of weather radar datasets.

1. Introduction

Weather radar can receive high temporal and spatial resolution information about reflectivity factors. The subsequently generated radar images (where pixel values represent radar reflectivity factors) reflect locations and intensity of precipitation, which have become an indispensable tool in quantitative precipitation estimation (QPE) [1,2] and quantitative precipitation forecast (QPF) [3,4] studies. However, various reasons, such as malfunctions, may cause missing observations and discontinuity of radar datasets. Therefore, reconstructing missing data in weather radar datasets becomes an inevitable procedure in utilizing these data. The most conventional way is to replace missing frames with the nearest image (i.e., persistence) [5]. However, this will definitely reduce the accuracy of QPE and QPF products since the movements and intensities of radar echoes are non-negligible, especially in the circumstance in which multiple successive frames are missing. Efforts have also been made by using interpolation methods to reconstruct missing frames. For instance, interpolation techniques are widely used in meteorological data processing [6,7,8,9,10]. Methods based on optical flow (i.e., motion) [11,12], complex empirical orthogonal functions [13], and fractals [14] have been developed and have been found to be more accurate than persistence or linear interpolation.

Optical flow is one of the most popular techniques that is used for weather radar data interpolation as well as for weather radar echo extrapolation [15,16]. The motion of radar echo features is first estimated from a sequence of radar images [17,18,19]. Then, the intermediate or future radar echoes are obtained according to their motion [20]. There are several schemes to calculate optical flow, such as gradient-based algorithms [21,22], correlation-based algorithms [23], and spectral-based algorithms [9,24], and all of them are used in meteorological studies. For instance, a local Lucas–Kanade optical flow method (gradient-based) is used in precipitation nowcasting [18]; both gradient-based and correlation-based optical flow algorithms are used for operational nowcasting in Hong Kong [15]; and studies show that correlation-based interpolation can improve the accuracy of hourly and daily accumulated rainfall [25,26]. Fast Fourier transform is also used to estimate optical flow for nowcasting and interpolation [9,24].

Despite the successful application of the optical flow method in the interpolation of radar and precipitation data, it cannot overcome the basic assumption that the intensity of the features remains constant [12]. This assumption is not appropriate because of the intensification and disappearance of radar echoes. Similar empirical assumptions have also been used in the nearest neighbor approach and the linear interpolation method. The former is based on persistence assumption, and the latter only takes account of the temporal evolution of radar echoes at each grid point but without considering spatial movements. At this point, a deep-learning method could be a better choice [27,28]. The main advantage is the deep neuron network (DNN) learns motions from a radar image sequence, rather than taking account of only two frames as optical flow methods usually does [12]. A DNN tries to learn regularities in all radar sequences or, in other words, it tries to learn the dynamical regularities underlying the motions of all the radar echoes involved. Therefore, a DNN may, to some extent, capture developing and dissipating processes of radar echoes. On the basis of this idea, this study will take advantage of DNNs to address the issue of missing radar data in QPE and QPF products.

The key of this study is how to fully reconstruct radar image sequences from those filled with missing frames. The typical encoder–decoder architecture is well suited to our study. The encoder transforms the inputs into encoded features, and then the decoder transforms the encoded features into the outputs. For this study, the encoder learns regularities or high-level features first from the spatial–temporal radar image sequences with missing frames by convolutional neural networks (CNNs) and/or recurrent neural networks (RNNs) [29,30]. After that, the decoder generates reconstructed radar image sequences without missing frames based on either CNN or RNN. The encoder–decoder architecture is widely used in many fields, including computer vision, natural language processing, and remote sensing, to solve various sequence-to-sequence (Seq2Seq) problems [31,32,33,34]. DNNs with such an architecture has already been used in operational precipitation nowcasting based on radar images [27,28], representing one of the most successful applications of the deep-learning method in geosciences. In addition to basic encode–decode networks, such as CNNs and long short-term memory networks (LSTMs) [35,36,37], more complicated and powerful networks are being developed to learn high-level representations of spatial–temporal sequences. Shi et al. [28] propose a convolutional long short term memory (ConvLSTM) network to improve precipitation nowcasting. Another RNN-based model, namely the trajectory gated recurrent unit (TrajGRU), was introduced in a subsequent study and it shows higher accuracy [38]. Some other sophisticated encoder–decoder networks, such as a predictive recurrent neural network (PredRNN) [39,40], memory in memory (MIM) networks [41], and eidetic 3D-LSTM [42], show much greater skills in radar echo extrapolation. Apart from RNN-based models, CNN-based models, such as U-Net [43,44], have also been used in precipitation nowcasting [27]. In addition, a combination of CNN and ConvLSTM also demonstrates application potential in radar echo nowcasting [45].

In sharing Seq2Seq problems, there are similarities between predicting future radar images and reconstructing missing radar images from several radar images. However, the latter would benefit more from the radar sequences due to higher correlations between input radar images and target images. In this study, we propose an encoder–decoder DNN model, which combines CNN and bidirectional ConvLSTM (CNN-BiConvLSTM), to establish a deep-learning baseline for reconstructing missing radar data [45,46]. The bidirectional structure is used mainly because reverse directional information can also provide substantial skills, unlike the case of prediction. Furthermore, the CNN enables the model to deal with large radar images without the problem of memory shortage. The CNN-BiConvLSTM model is further compared with several baseline models, including linear interpolation, optical flow methods, and two DNNs: CNN-ConvLSTM and three-dimensional CNN (3DCNN). A training scheme that enables the DNN models to deal with arbitrary missing patterns in a radar sequence is also proposed.

The article is organized as follows: The related models are described in Section 2. In Section 3, the data, experimental design, and evaluation metrics are introduced. In Section 4, the experimental evaluations and comparative analysis are presented. Section 5 summarizes the main results of the study and provides some discussion.

2. Model Description

2.1. CNN-BiConvLSTM Model

In terms of environment configuration, Tensorflow1.14 was employed to construct and train the DNN models. The CNN-BiConvLSTM model used in this study consisted of 3 components: A CNN encoder, a bi-directional recurrent neural networks (BiRNNs) encoder, and a CNN decoder. As shown in Figure 1a, a CNN encoder was used to extract high-level features of input radar images. It consisted of 3 CNN layers with 8, 16, and 32 filters and a kernel size of 5 × 5, where each CNN layer was followed by an average pooling layer with a stride of 2. The spatial sizes of the feature were simultaneously reduced, which served the CNN-BiConvLSTM model well since it required less memory usage. Furthermore, a larger receptive field was obtained for the BiRNN encoder compared to the original inputs. Thus, it would be helpful in dealing with the rapid spatial movements of radar echoes. The BiRNN encoder (Figure 1a,b) was used to capture spatial–temporal variations from high-level features. It was constructed by stacking 4 BiConvLSTM layers with 32 filters and a kernel size of 5 × 5. The CNN decoder (Figure 1a) was finally used to transform the high-level features to radar images. It consisted of 3 CNN transpose (Deconv) layers with 32, 16, and 8 filters and a kernel size of 5 × 5, corresponding to the CNN encoder. It is worth noting that the CNN encoder and decoder share parameters for all input and output radar images to ensure unique relationships between high-level features and radar images. The leaky rectified linear unit (ReLU) activity layer was used for all CNNs and CNN transpose layers to avoid the vanishing gradient problem. The formula is as follows:

$$f\left(x\right)=\{\begin{array}{c}\alpha x \mathrm{for} x<0\\ x \mathrm{for} x\ge 0\end{array} \alpha =0.1$$

(1)

2.2. Baseline Models

2.2.1. Traditional Baseline Models

The linear model is a basic and simple model since it takes only the temporal evolution of radar echoes into account. It calculates the linear trend between 2 radar images for each pixel and then interpolates or extrapolates these pixels using the corresponding trends. The optical flow model was constructed based on the Dense Inverse Search (DIS) algorithm. The DIS algorithm is a global optical flow algorithm proposed by Kroeger et al. [47], which allows the explicit estimation of the velocity of each image pixel based on an analysis of 2 adjacent radar images. The DIS algorithm was selected as a benchmark because of its high accuracy in the extrapolation and interpolation of radar images [12]. The implementation of the optical flow model can be summarized as follows:

• Calculate velocity field using the global DIS optical flow algorithm [47] based on the radar images at times $t_a$ and $t_b$.
• Use a backward constant-vector [48] to interpolate or extrapolate each pixel according to the velocity field.
• Obtain an irregular point cloud that consists of the original radar pixels. Then interpolate the intensity of these displaced pixels in the original radar grid. The inverse distance weighted interpolation technique is used in the interpolation.

Figure 2 demonstrates an example of interpolating 3 radar frames from 2 adjacent radar images. The features of Radar (t) were propagated according to the calculated velocity field to generate intermediate frames Radar (t + 1), Radar (t + 2), and Radar (t + 3). Apparently, the observed radar image Radar (t + 4) was only used in velocity field calculation, and the shape of the echoes was fully dependent on the first frame.

2.2.2. Basic DNNs

The CNN-ConvLSTM model was constructed by replacing the BiconvLSTM module in the BiRNN encoder (Figure 1b) with the ConvLSTM module. The 3DCNN model was also constructed as an encoder–decoder structure, as shown in Figure 3. The encoder consisted of 3 3DCNNs (Conv3D), each followed by an average pooling (AvgPooling3D) layer. The decoder consisted of 3 3DCNN transpose (DeConv3D) layers, as shown in Figure 3. Unlike RNN-based models, the 3DCNN model extracted high-level features by using all radar images together. Meanwhile, all the output radar images were decoded from the high-level features. The leaky ReLU activity layer was used for all 3DCNN and 3DCNN transpose layers.

3. Data and Experimental Design

3.1. Dataset

The radar dataset was constructed based on the radar reflectivity data of 11 S-band Doppler weather radar (Beijing Metstar Radar Co., Ltd., China) stations covering the Guangdong province from 2010 to 2019. The products of Constant Altitude Plan Position Indicator (CAPPI) [49] images with a resolution of 0.1 km were generated and used in the study. The dates of the radar data were chosen according to the rule that short-term heavy precipitation (greater than 20 mm/h) occurred at more than 3 automatic weather stations over Guangdong province. The size of the radar echo images was 256 × 256, and the limits of the values of radar reflectivity factors were 0 to 80 dBZ. To facilitate the study, the length of the sample sequences was extracted and fixed at 10. The training set and validation set were split by samples in 2010–2018 with sizes of 18,000 and 2000, respectively. The test set of size 2000 was retrieved from samples in 2019 to ensure independence.

3.2. Experimental Configuration

The CNN-BiConvLSTM model and basic DNNs were all optimized with Adam [50] by setting ${\mathsf{\beta }}_1=0.9$ and ${\mathsf{\beta }}_2=0.999$, and the initial learning rate was set to 0.0005 with an exponential decay according to the formula:

$$lr=0.0005\times {0.98}^{\frac{i}{2085}},$$

(2)

where $i$ denotes the number of iterations. The loss function is a combination of $L1$ losses according to the formula:

$$\mathcal{L}=L{1}_{\mathrm{all}}+w_1\times L{1}_{\mathrm{all}\_\mathrm{thrd}}+w_2\times L{1}_{\mathrm{missing}},$$

(3)

where $L{1}_{\mathrm{all}}$ denotes $L1$ losses for all output radar images and $L{1}_{\mathrm{missing}}$ denotes only missing positions of output radar images; $L{1}_{\mathrm{all}\_\mathrm{thrd}}$ denotes $L1$ losses for values of reflectivity factors larger than 30 in all input radar images; $w_1=3$ and $w_2=8$ are chosen as the weights of $L1$ losses. $L{1}_{\mathrm{missing}}$ and $L{1}_{\mathrm{all}\_\mathrm{thrd}}$ are given more weight in order to emphasize the importance of missing frames and strong radar echoes. All the inputs were scaled by 80, and the outputs were limited to the range of 0 to 80. To simulate missing frames in real radar sequences, the input radar images were randomly masked before they were fed to the model during the training, following the rules below:

(1) Generate a random mask for a radar sequence of length 10. The number of masked radar images ranges from 0 to 10 with equal probability. All the masked images are set to zero.

(2) Randomly replace two positions of the mask by ones to ensure at least two real radar images in a sequence.

During inference, the input radar images were masked according to the real missing patterns. The scheme chart for the mask procedure and DNN model is shown in Figure 4. This procedure enabled the model to deal with any possible missing patterns. If no masks were given to the input radar images, the model would abstract features from every input radar image in the training procedure. However, in the inference model, the model will extract features from zero inputs as real radar images and subsequently obtain poor results. In the study, the missing pattern of a radar sequence of length 10 was denoted as 10 binary numbers, where element ‘0′ indicated missing frames and element ‘1′ indicated normal frames. The batch size was set to 8, and the training was stopped after ~160,000 iterations (i.e., 80 epochs). The mask ratio was gradually increased up to 100% at 10,000 iterations. The models were evaluated for every epoch, and the optimal model was chosen according to the highest validation loss for each model.

3.3. Evaluation Metrics

We evaluated the performances of all models with the mean absolute error (MAE), critical success index (CSI), and Heidke skill score (HSS) [51]. Mathematically, the CSI score was computed as

$$\mathrm{CSI}=\frac{\mathrm{hits}}{\mathrm{hits}+\mathrm{misses}+\mathrm{false} \mathrm{alarms}},$$

(4)

where hits correspond to true positives, misses correspond to false positives, and false alarms correspond to false negatives in the confusion matrix. The critical threshold was chosen to be 30 dBZ (CSI-30) or 40 dBZ (CSI-40). A higher CSI score denotes a better prediction with respect to a reflectivity factor larger than the threshold. The HSS score was computed as

$$\mathrm{HSS}=\frac{\frac{1}{N}{{\displaystyle \sum }}_{j=1}^{K}n\left(F_j,O_j\right)-\frac{1}{N^2}{{\displaystyle \sum }}_{j=1}^{K}N\left(F_j\right)N\left(O_j\right)}{1-\frac{1}{N^2}{{\displaystyle \sum }}_{j=1}^{K}N\left(F_j\right)N\left(O_j\right)},$$

(5)

where $N$ is the number of predictions (i.e., pixels in a predicted image), $K$ denotes forecasts categories, $n\left(F_j,O_j\right)$ denotes the number of forecasts and observations both in category $j$ and $N\left(F_j\right)$ and $N\left(O_j\right)$ denote the total number of forecasts and observations in category $j$, respectively. The predictions are classified into 5 classes according to the intervals (0, 20), (20, 30), (30, 40), (40, 50), and (50, 80). The HSS score reflects the accuracy of a prediction relative to that obtained by random chance. A higher HSS score denotes a better model.

4. Results

In this section, the CNN-BiConvLSTM model and baseline models were evaluated for the test datasets with comparative analysis. Section 4.1 evaluates the results for the case of interpolation, corresponding to several successive or discrete intermediate missing frames in a radar sequence. Because the extrapolation can also be used to reconstructing missing frames, Section 4.2 further evaluates and compares the results for the case of extrapolation to verify whether the effectiveness of a model was better for interpolated cases. The influences of the quality of input radar sequences and model sizes on the performance of the models are investigated in Section 4.3.

4.1. Evaluation Results for the Case of Interpolation

Six different missing patterns were involved in the evaluations in order to investigate the influence of the numbers of missing frames on the models. The first 5 patterns miss 1, 2, 4, 6, 8 successive intermediate frames, respectively. The last missing pattern misses 4 discrete intermediate frames. The chosen missing patterns were representative because the number of successive missing frames was usually not too high in real radar data sets.

Table 1. Evaluation results of mean absolute error (MAE), Critical success index (CSI)-40 scores, and Heidke skill score (HSS) scores for the bi-directional convolutional long short-term memory networks (CNN-BiConvLSTM) model and baseline models and for 6 missing patterns with respect to interpolation.

Missing Patterns	Metrics	Nearest Frame	Linear	Optical Flow	CNN- ConvLSTM	3DCNN	CNN- BiConvLSTM
	MAE	2.5205	2.0783	1.8766	1.7332	1.5993	1.4650
1111011111	CSI	0.3431	0.4044	0.4223	0.4194	0.4763	0.5085
	HSS	0.6088	0.6790	0.6947	0.7023	0.7264	0.7465
	MAE	3.0002	2.3783	2.1617	1.9517	1.7244	1.5675
1111001111	CSI	0.2835	0.3412	0.376	0.3640	0.4424	0.4828
	HSS	0.5532	0.6404	0.6546	0.6697	0.7071	0.7305
	MAE	3.7057	2.8644	2.6966	2.3433	1.9619	1.7777
1110000111	CSI	0.2121	0.2549	0.3029	0.2744	0.3781	0.4296
	HSS	0.4773	0.5804	0.5871	0.6129	0.6721	0.6976
	MAE	4.2260	3.2542	3.2146	2.7135	2.2522	1.9966
1100000011	CSI	0.1708	0.2039	0.2446	0.1996	0.3049	0.3784
	HSS	0.4250	0.5338	0.5287	0.5606	0.6298	0.6651
	MAE	4.6448	3.5841	3.6964	3.5725	2.8397	2.3731
1000000001	CSI	0.1431	0.1666	0.2007	0.0999	0.1883	0.3067
	HSS	0.3849	0.4967	0.4781	0.4460	0.5442	0.6131
	MAE	2.5156	2.0735	1.8717	1.8304	1.6085	1.4670
1010101011	CSI	0.3446	0.4052	0.4236	0.4026	0.4454	0.5067
	HSS	0.6092	0.6796	0.6950	0.6892	0.7244	0.7457

presents the MAE, CSI-40 scores and HSS scores for the CNN-BiConvLSTM model and baseline models with respect to several specific missing patterns. It was found that the performances of the CNN-BiConvLSTM model were the best among all the models and for all evaluation metrics and missing patterns involved. Apart from the CNN-BiConvLSTM model, the 3DCNN model generally performed better than the other models. Whereas for the CNN-ConvLSTM, the performances were unstable and are highly dependent on evaluation metrics and missing patterns. Specifically, the MAE and HSS scores for the CNN-ConvLSTM model were generally better than those of the optical flow model, but the CSI-40 scores were poorer than those of the optical flow model, especially under conditions of multiple successive missing frames (e.g., ‘1000000001’). Considering identical network structures between CNN-BiConvLSTM and CNN-ConvLSTM except for the bidirectional structure, it can be inferred that the spatial–temporal information in both directions is quite important. This was also the reason why the 3DCNN model achieved a better performance. The output radar images in the 3DCNN model used encoded features from all input radar images (Figure 3), while the output radar images in the CNN-ConvLSTM model used only encoded features from previous radar images. When features from subsequent radar images were taken into account in the CNN-ConvLSTM model (i.e., CNN-BiConvLSTM), the performances were superior to those of the 3DCNN model. This result also implied the higher effectiveness of RNNs in extracting spatial–temporal features compared to CNNs. In addition to higher average scores, the scores of each sample for the CNN-BiConvLSTM model were found mostly higher than that of the baseline models. Figure 5 shows histograms for the missing pattern of ‘1110000111’ as a representative case. The differences of scores between the CNN-BiConvLSTM model and baseline models for test samples were found mostly less than zero for the MAE and larger than zero for the CSI scores and HSS scores. The results further highlighted the effectiveness of the CNN-BiConvLSTM model.

As the number of successive missing frames increased, the performance of all the models got worse. This was reasonable since the models received less information but generated many more outputs. Nonetheless, the CNN-BiConvLSTM model performed well enough when missing 4 successive frames, and even better than traditional interpolation models when only 1 frame was missing. For the case of 4 discrete intermediate frames missing (i.e., 1010101011), the performances of the linear model, the optical flow model, and CNN-BiConvLSTM were found to be nearly equivalent to those of 1 intermediate frame missing. This implied that the CNN-BiConvLSTM model did well in reconstructing discrete intermediate missing frames without losing accuracy, as the linear model and the optical flow model did. While the latter models used only 2 adjacent frames before and after missing frames to generate an interpolation, the CNN-BiConvLSTM model learned regularities from the whole radar sequence and may obtain higher accuracy. By contrast, the performances of the CNN-ConvLSTM model and the 3DCNN model in the case of 4 discrete intermediate frames missing were slightly unstable.

To provide more detail, Figure 6 shows an example of reconstructing missing radar images with the CNN-BiConvLSTM model for all 6 missing patterns. It was found that the reconstructed radar images were quite similar to the ground truth with respect to the shapes and values of the radar echoes. The most encouraging success was the good reproduction of newly emerging and moving radar echoes in the west of the radar images (red boxes in Figure 6). This can clearly be observed for the results of missing patterns ‘1100000011’ (the 5th line in Figure 6) and ‘1000000001’ (the 6th line in Figure 6). The details of the radar echoes were indeed less distinct as the number of missing frames increased, which agreed with the evaluation results.

The reconstructed radar images for the selected example with respect to different models are shown in Figure 7, but only for the missing pattern ‘1110000111’. The performances of the CNN-BiConvLSTM model and the 3DCNN model were better on the whole, but with slightly more detailed echoes for the CNN-BiConvLSTM model. The reconstructed radar images from the CNN-ConvLSTM model lost much more detail compared to the ground truth. For the linear model, a large area of weak radar echo appeared around ground truth, and the strong radar echoes were diluted to some extent. For the optical flow model, the intensity of the radar echoes behaved quite well but exhibited an incorrect and fixed shape. More seriously, the reflectivity factors in some of the marginal areas in the reconstructed images were filled with zeros since the radar echoes moved out to other regions following the velocity field. Two subareas were selected to further demonstrate the effectiveness of different models for the marginal area (red and black boxes in Figure 7). The corresponding scores are shown in Figure 8. It was obvious that the traditional methods were quite unsuccessful in the two subareas, especially for subarea-2. The locations of the clear sky were quite distinct and correct for the reconstructed radar images in subarea-2 for the 3DCNN model and the CNN-BiConvLSTM model, but were not quite correct for the other models. In addition, the movements of radar echoes in subarea-1 can also be clearly observed for the 3DCNN model and the CNN-BiConvLSTM model but did not occur for the other models. In addition, the scores, which exclude the marginal area, were also tested, and they indeed showed slightly higher scores, but the order of the scores did not change. In summary, the CNN-BiConvLSTM model outperformed all the baseline models involved.

4.2. Evaluation and Comparison for the Case of Extrapolation

Relative scores for models with respect to 5 missing patterns in the case of extrapolation were evaluated and presented in

Table 2. Evaluation results of MAE, CSI-40 scores, and HSS scores for the CNN-BiConvLSTM model and baseline models, and for 5 missing patterns with respect to extrapolation.

Missing Patterns	Metrics	Nearest Frame	Linear	Optical Flow	CNN- ConvLSTM	3DCNN	CNN- BiConvLSTM
	MAE	2.5520	3.5274	1.9849	1.7530	1.8870	1.7283
1111111110	CSI	0.3518	0.2467	0.4232	0.4304	0.3613	0.4558
	HSS	0.6147	0.5067	0.6883	0.7075	0.6893	0.7112
	MAE	3.0322	4.5577	2.3037	1.9694	2.1186	1.9465
1111111100	CSI	0.2903	0.1954	0.3718	0.3723	0.3238	0.4043
	HSS	0.5585	0.4367	0.6443	0.6746	0.6544	0.6784
	MAE	3.4213	5.4307	2.6010	2.1713	2.3390	2.1469
1111111000	CSI	0.2475	0.1651	0.3285	0.3210	0.2594	0.3543
	HSS	0.5157	0.3874	0.6062	0.6448	0.6221	0.6495
	MAE	3.7422	6.1667	2.8694	2.3512	2.5471	2.3260
1111110000	CSI	0.2160	0.1440	0.2938	0.2778	0.2216	0.3111
	HSS	0.4813	0.3505	0.5736	0.6181	0.5919	0.6236
	MAE	3.3575	5.2202	2.5439	7.7030	2.3127	2.1017
0001111111	CSI	0.2411	0.1643	0.3264	0.	0.3203	0.3490
	HSS	0.5093	0.3935	0.6020	0.0367	0.6140	0.6431

. To investigate the influence of the positions and number of missing frames on the models, the 5 specific missing patterns were chosen as follows: The first 4 patterns missed 1, 2, 3, and 4 successive radar images at the end of the sequence, respectively; The 5th pattern missed 3 successive radar images at the beginning of the sequence. Overall, the performances of the CNN-BiConvLSTM model were found to be the best in the case of interpolation, but were very close to those of the CNN-ConvLSTM model apart from missing pattern ‘0001111111’. This result suggested that the bidirectional architecture provided only a little improvement for the extrapolation compared to one direction. The good performance of the CNN-ConvLSTM model was consistent with previous studies in which the effectiveness of the ConvLSTM model was better than for optical flow methods [28]. The performances of 3DCNN were comparable to the optical flow model for the metrics of the MAE and HSS scores but were poorer for the CSI-40 scores. The linear model failed for all the missing patterns since the differences in pixel-wise reflectivity factors between two adjacent radar images were extremely unstable. The direction of extrapolation was found to have little effect on the performance of the models except for CNN-ConvLSTM when considering the scores for the missing pattern ‘0001111111’. In addition, the performances of all models become poorer as the number of missing frames increases in the case of interpolation.

Figure 9 shows an example of extrapolation with missing pattern ‘1111111000’ for all the models tested in this study. The failure of the linear model was quite obvious. The clear sky area in the western margin area (the fourth line in Figure 9) revealed the poor ability of the optical flow method in dealing with newly emerging radar echoes. The extrapolated radar images of the CNN-ConvLSTM model, the 3DCNN model, and the CNN-BiConvLSTM model showed features close to the ground truth, but more detailed radar echoes can be obtained by the CNN-ConvLSTM model and the CNN-BiConvLSTM model. All these features can be clearly observed in the red and black boxes in Figure 9. The corresponding scores are shown in Figure 10. For this example, apart from the nearest neighbor approach and linear interpolation, which showed poor results, the other methods showed comparable results for the whole area and for subarea-2. The optical method was quite unsuccessful for subarea-1 because of the movements of the radar echoes.

On the whole, the effectiveness of the extrapolation was poorer than that in the interpolation experiments. To reconstruct 1 or 2 missing frames, extrapolation of 1 frame can be substituted. The evaluation results showed lower scores of the extrapolation models than the interpolation models. For instance, HSS scores of 0.7465 and 0.7305 for the case of interpolation by the CNN-BiConvLSTM model with missing patterns ‘1111011111’ and ‘1111001111’, and 0.7112 for the case of extrapolation with missing pattern ‘1111111110’, respectively. Similarly, to reconstruct 3–4, 5–6, and 7–8 missing frames, extrapolation of 2, 3, and 4 frames, respectively, can be substituted. For the case of reconstructing 5–6 missing frames, the performances of interpolation models were still better. While for the case of reconstructing 8 missing frames, the performances of interpolation and extrapolation models were comparable. This may imply the availability of both kinds of models when plenty of successive radar images are missing. However, this was not a solid conclusion since there were only two valid radar images in the input sequences. A better performance may be obtained when longer radar image sequences are used, and the effectiveness of interpolation models is probably still superior to that of extrapolation models for the case of multiple frames being missing.

4.3. Influence of Data Quality and Model Size

In addition to missing frames, the quality of radar images is another tough problem. The poor quality of radar images is mainly exhibited by the disappearance of part of the radar echo. This may originate from a quality control process or missing data at some radar station, which causes incomplete radar images when radar images are concatenated. Whether the quality of the radar echo sequences may influence the effectiveness of interpolation models is another important issue to be considered. We tested 1000 low-quality radar sequences for all interpolation models. Since the ground truth here is not a robust representation of the reasonable truth, the statistical evaluations were not quite suitable. Indeed, the performances of models on different low-quality radar sequences were quite consistent. Thus, several examples are presented to illustrate the influence of data quality as a representative case for the models involved.

Figure 11 shows an example of the influence of data quality for all interpolation models with missing pattern ‘1010101011’. Apparently, the 2nd, 4th, and 7th frames were less qualified images. For the linear model with missing pattern ‘1010101011’, the reconstructed frames 6 and 8 showed poor performances due to the large and unreasonable differences between input frames 5 and 7, and frames 7 and 9, respectively. For the optical model, reconstructed frame 8 showed poor performance. The poor result was caused mainly by the incorrect optical flow calculation of input frame 7 and frame 8. It is worth noting that reconstructed frame 6 was quite reasonable, although the adjacent frame 7 was a less qualified image and was used in the optical flow calculation. This is because the calculation of optical flow relies on the order of the image sequences. For three DNN interpolation models, all the reconstructed frames were surprisingly reasonable, although more details were shown for the CNN-BiConvLSTM model. For interpolation of frame 2 and frame 4 for all the models, it was found that the less qualified images can be well replaced by the reconstructed images, provided that the adjacent frames were well qualified. This was an encouraging result since it implies the potential of using interpolation methods to improve the quality of radar data by treating the less qualified frame as missing, especially with an effective interpolation model.

Whether a DNN model, specifically the CNN-BiConvLSTM model, can deal with all the conditions with less qualified images was further investigated. By exploring plenty of samples, it was found that quite well-reconstructed frames can be obtained for the DNN models even with two less qualified adjacent frames. The only exception occurred when the less qualified images located at the ends of the sequence and the missing frames, were numerous enough. Figure 12 shows an example of the first frame being missing and interpolation by the CNN-BiConvLSTM model for missing patterns ‘1010101011’ and ‘1000000001’. The failure of the interpolation model was quite evident. Despite this failure, the problem can simply be avoided by extending the radar sequences by a step or two. Meanwhile, it was also found that the model performed well enough when the third frame was well qualified (third line in Figure 12).

In this study, the CNN-BiConvLSTM model was constructed as quite a small model with 2.3 million parameters in total. Larger models denoted CNN-BiConvLSTM-M with 3.3 million parameters in total, and CNN-BiConvLSTM-L with 10.3 million parameters in total were also constructed and evaluated.

Table 3. Evaluation results of MAE, CSI-40 scores, and HSS scores for the CNN-BiConvLSTM model with different model sizes and for 6 missing patterns with respect to interpolation.

Missing Patterns	Metrics	CNN-BiConvLSTM	CNN-BiConvLSTM-M	CNN-BiConvLSTM-L
	MAE	1.4650	1.4357	1.4159
1111011111	CSI	0.5085	0.5137	0.5287
	HSS	0.7465	0.7519	0.7548
	MAE	1.5675	1.5431	1.5206
1111001111	CSI	0.4828	0.4811	0.5010
	HSS	0.7305	0.7351	0.7382
	MAE	1.7777	1.7562	1.7294
1110000111	CSI	0.4296	0.4217	0.4477
	HSS	0.6976	0.7018	0.7054
	MAE	1.9966	1.9723	1.9420
1100000011	CSI	0.3784	0.3684	0.3965
	HSS	0.6651	0.6696	0.6732
	MAE	2.3731	2.3486	2.3010
1000000001	CSI	0.3067	0.2880	0.3214
	HSS	0.6131	0.6169	0.6234
	MAE	1.4670	1.4371	1.4182
1010101011	CSI	0.5067	0.5132	0.5273
	HSS	0.7457	0.7515	0.7540

presents the results. The performances of the CNN-BiConvLSTM model were found to have improved monotonously for all missing patterns as the size of the models increased, indicating that the spatial–temporal variations can be better simulated by models with more parameters and probably by more sophisticated RNN modules.

5. Conclusions and Discussion

In this paper, we propose a DNN model, the CNN-BiConvLSTM model, to reconstruct missing data in weather radar image sequences. The model is able to deal with arbitrary missing patterns by using a proposed training scheme. To verify the effectiveness of the model, the nearest neighbor approach, linear interpolation, the optical flow method, and two DNN models, CNN-ConvLSTM and 3DCNN, were included in the study and were evaluated to perform a comparative analysis. The performances of the CNN-BiConvLSTM model were better than for all baseline models for the missing patterns and evaluation metrics involved. Two baseline DNN models, the CNN-ConvLSTM model, and the 3DCNN model generate reasonable radar images but lose much more detail. The weaknesses of traditional approaches are quite obvious: The nearest neighbor approach generates fixed radar echoes and obtains the lowest scores; the linear model generates lots of unreal weak radar echoes; the optical flow model generates radar echoes of quite fixed shape and zero values in the marginal area.

The DNN models, especially for the CNN-BiConvLSTM model, are quite robust to the quality of the input data sequences. The linear model and the optical flow model largely failed in less qualified input radar sequences. Furthermore, the performances of the CNN-BiConvLSTM mode were found to improve with the number of model parameters, indicating potential improvements to the model. In summary, the CNN-BiConvLSTM model shows a powerful ability in reconstructing missing observations in radar data sequences. The main reason is that the DNN models are able to learn intrinsic spatial–temporal features based on numerous radar image sequences, especially for the CNN-BiConvLSTM model, while the traditional methods are based on empirical assumptions, which is not quite appropriate, such as persistence (nearest neighbor approach), linear temporal evolution at each grid (linear interpolation), and linear motion of radar echoes (optical flow method).

To the best of our knowledge, no previous research has experimented with DNNs in reconstructing missing radar images. Most studies focus on predicting future radar images rather than improving radar data themselves [52]. Experimental results (Section 4.2) show that the effectiveness of the models for the interpolation cases is superior to that of extrapolation (i.e., prediction) cases in reconstructing radar images, indicating the significance of our study. The results also suggest that the input radar sequences should include valid first and last frames in the reconstruction process. Other relative studies focus mainly on interpolating satellite images using DNNs to improve temporal resolution [53,54]. However, the quantitative evaluations for these temporal super-resolution experiments are quite limited because not all interpolated frames have ground truths. Therefore, the accuracy of reconstructed radar images is expected to be higher. For these reasons, we implemented an RNN-based DNN model with encoder–decoder architecture as a benchmark in order to improve the quality of weather radar datasets. Based on the formulation of our work, more sophisticated RNN modules, such as PredRNN and MIM [40,41], can easily be applied to further improve the model performance. Slim architecture can also be used in the CNN encoder and CNN decoder to reduce computational complexity but without reducing accuracy [55].

In addition to missing radar images, we noticed that the less qualified radar images could also be reconstructed by the CNN-BiConvLSTM model as long as they are fed to the model as missing data (Figure 11). This result suggests the potential use of DNN models in weather radar data quality control. Although no relevant research has been reported, applications of image restoration in the field of computer vision may provide references [56,57]. It should be noted that the models and their evaluations are based on radar data sequences of length 10. Nonetheless, the results are largely representative since there are not usually longer lengths of successive missing frames in reality. Models that deal with much longer lengths of radar data sequences can be easily constructed by following this study. In addition, based on the improved weather radar datasets through reconstructing missing radar data, improvements in QPE and QPF products are to be expected and are worth investigating in the future.