End-to-End Prediction of Lightning Events from Geostationary Satellite Images

Abstract

While thunderstorms can pose severe risks to property and life, forecasting remains challenging, even at short lead times, as these often arise in meta-stable atmospheric conditions. In this paper, we examine the question of how well we could perform short-term (up to 180 min) forecasts using exclusively multi-spectral satellite images and past lighting events as data. We employ representation learning based on deep convolutional neural networks in an “end-to-end” fashion. Here, a crucial problem is handling the imbalance of the positive and negative classes appropriately in order to be able to obtain predictive results (which is not addressed by many previous machine-learning-based approaches). The resulting network outperforms previous methods based on physically based features and optical flow methods (similar to operational prediction models) and generalizes across different years. A closer examination of the classifier performance over time and under masking of input data indicates that the learned model actually draws most information from structures in the visible spectrum, with infrared imaging sustaining some classification performance during the night.

1. Introduction

Thunder and lightning are violent atmospheric events that must have impressed humans since prehistoric times. Lightning itself, as well as strong winds and precipitation, hail, or even down bursts or tornadoes that might accompany thunderstorms, are not only impressive appearances in satellite images but can pose significant risk to life and property. Even in modern times, fatalities and severe damages from thunderstorms are still occurring at unfortunate rates, and many commercial operations, such as airports or public outdoor events (sports, music, gatherings), rely on risks assessments and prediction of lightning in order to operate safely. In this context, accurate forecasts for the next few hours are particularly important. Usually, thunderstorms are associated with lightning, which easily can be detected by triangulation using multiple radio wave antennas.

Unfortunately, the prediction of thunderstorms and lightning (Cb) is a very difficult problem: The development of atmosphere’s electric field with locally strong charges of different signs is crucially dependent on cloud processes. Charge generation is based on the collision of ice particles (ice crystals, graupel or hail particle) in the presence of super-cooled water droplets [1]. Furthermore, other processes of this kind are proposed for charge generation; see, e.g., [2]. Nevertheless, the collisions are enhanced in strong, localized atmospheric updrafts that are typically formed by frontal movements or spontaneous convective events induced by heat in situations of unstable atmospheric layering. In particular, the latter effects are highly sensitive to small-scale perturbations and thus are hard to predict.

Although significant improvements in Cb forecasting have been achieved with numerical weather prediction (NWP) [3,4], accurate forecasts of Cb location and strength are still a major challenge. Hence, nowcasting methods are typically used to issue warnings with short lead times (in the range of up to a few hours) [5]. An early overview of nowcasting of (severe) weather phenomena is given in [6]. Nowcasting can in general be viewed as a special type of video prediction problem, that is, classification or regression based on a temporal sequence of images. Thus, methods proposed for, e.g., video frame prediction in natural videos are applicable to nowcasting and are related; see [7].

Radar-based methods are often used for regional nowcasting—in particular, the (Doppler-) radar-based remote sensing of (heavy) precipitation events and their movement, depending on the method in combination with lightning data [8]. The forecast is typically based on two subsequent images and extrapolation of the movement of the observed storm cells, e.g., by optical flow methods [9] or tracking methods [10]. This limits the ability to predict newly forming or decaying cells and limits long-term precision. For example, operational systems such as NowCastMIX [10] struggle with predictions for lead times beyond two hours. That is why for longer lead times, NWP models are used.

An alternative source of information are images recorded by geostationary satellites. They cover larger regions than methods based on weather radar. For example, the SEVIRI instrument onboard Meteosat’s second generation (MSG) platform provides data in twelve spectral bands with a temporal resolution of 15 min and a spatial resolution with $3 \mathrm{k}\mathrm{m}\times 3 \mathrm{k}\mathrm{m}$ sub satellite point, which corresponds to $0.05°\times 0.03°$ in Central Europe. These data can be used like radar data to determine atmospheric motion vectors (AMVs) using optical flow and to provide information about the brightness temperature of clouds (BT), which can be used to indicate thunderstorms. For example, the physically motivated usage of the “sandwich” method, which is based on the BT difference between the $6.3$ water vapor (WV) channel and the IR window channel [11], or alternatively the $7.3$ WV channel [12]. Yet, Cbs are defined by lightning. Hence, in order to improve the accuracy of the detection of Cbs, which is the basis for the nowcasting, it is recommended to use lightning data in addition. For example, [13] use the Advanced Baseline Imager (ABI) and flash-extent density (FED) from the Geostationary Lightning Mapper (GLM) on board GOES-16, as well as satellite and solar zenith angles and geo-coordinates to predict “intense convection”. In [14], satellite-based information is combined with lightning data from the Vaisala Global Lightning Detection Network (GLD360) and information from NWP. Moreover, a combination of satellite and radar information is applied; for example, [15] make use of five temporal images from two ABI channels to predict convective regions, derived from Multi-Radar Multi-Sensor (MRMS) precipitation types.

However, it remains unclear whether physical nowcasting methods are capable of making the best possible use of the available information given in the satellite image and lightning data. As discussed before, current approaches, e.g., [14,16,17], do not take into account the decay or development of (new) cells. Corresponding information may be hidden in the input data and unused. The visual analysis of thunderstorms based on lightning data and satellite imagery suggests that there may be more information in the data that can be extracted. Thus, an obvious alternative is to resort to machine learning [18,19] and in particular deep representation learning techniques [13,20], which have, in recent years, become able to automatically build highly predictive statistical models even from data with extremely complex statistical dependency structure, such as differentiating different breeds of dogs in general photography [21,22,23].

Our study follows this approach: We predict lighting events from image sequences containing satellite imagery and a map of recent lightning events through an image-to-image translation performed by (a variant of) a convolutional U-Net [24], which could be considered a “canonical” network approach to image-segmentation and translation tasks. Our study is designed in particular to address the question of what insight can be gained from satellite images alone for short-term Cb-forecasting, and which parts of the data are most important to this end. The use of machine learning methods, and specifically deep neural networks, has been studied previously in the literature [25,26,27,28,29]. Specifically, U-Nets have been used in related studies for precipitation forecasting [30,31]. Regarding convection, [32] published a study that uses radar images (2 min, 1 km grid) to predict precipitation with a lead time up to 6 h using a (residual) U-Net. Similarly, Shi et al. [33] train a Trajectory GRU model to predict precipitation based on radar images (6 min, ca. 1 km grid). A random forest classifier to detect convective initiation (CI) from geostationary satellite data training on hand-crafted features, where CI is derived from radar data, was proposed in [34]. A similar model based on four commonly available surface weather variables (air pressure at station level (QFE), air temperature, relative humidity, and wind speed) to predict lightning events for lead times with up to 30 $\min$ training a decision tree was developed in [35].

However, lighting prediction comes with the problem of heavy class imbalance which is a general challenge for classification with statistical machine learning: In many relevant scenarios, lighting events are rather rare; thus, a classifier that just predicts “no (nearby) lightning” for all outputs can easily reach accuracies close to $100\%$. Previous work has mitigated this issue by resampling in evenly balanced examples [18] or considering conditions with high prevalence [20], which then also assumes balanced base-rates when making predictions. In our paper, we develop a method to automatically balance class weights to optimize a deep learning classifier for high predictive power (such as high critical success index, CSI). This permits a simple “end-to-end” training with operationally meaningful predictions (i.e., using recent lightning and satellite observations, the classifier predicts future events with above-chance accuracy). To obtain an indicator of the quality achieved, our results are compared with those of an optical flow-based nowcasting method of the German Weather Service (DWD).

A second and possibly even more interesting question is understanding which information actually contributes to making better predictions. Having a strong classifier that outperforms hand-engineered operational models by simple, automatic statistical learning opens up the opportunity to study this question: By withholding data at the learning stage and tracking the reduction in performance, we can attribute how much information the classifier was able to draw from these sources (formally as lower bound of how much statistical information about the event is contained in portions of the data). We conduct an experiment where we measure the variability of the prediction performance over different times of the day, and retrain classifiers for visible (including a 1.6 $\mathsf{\mu }$m near-infrared channel), infrared, and two water vapor bands.

Overall, our study makes two key contributions: (i) a simple method for end-to-end training of lightning events from image data that is both practical (applicable to data with real-world, skewed prior class frequencies) and accurate (outperforming state-of-the-art optical flow-based systems), and (ii) we obtain some novel insights of which sources of information are useful for making predictions by examining the performance characteristics of the learned classifier.

2. Materials and Methods

Our method performs statistical learning to learn a mapping that takes satellite images and measurements of recent lightning activity as input and predicts future lighting activity as output, with a specific lead time.

Satellite images: Formally, we define the geostationary satellite images as functions

$$S:\mathbb{R}\times [-{79}^{\circ },{79}^{\circ }]\times [-{81}^{\circ },{81}^{\circ }]\times \mathbb{R}\to \mathbb{R},$$

(1)

denoted as $S_t(\theta ,\varphi ,\lambda )$, where t is the time at which the image capturing has begun, $\theta$ and $\varphi$ are the longitude and latitude, respectively, and $\lambda$ refers to the wave-length band measured. The specific satellite imagery used is provided via the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) of the Meteosat Second Generation (MSG) system, and obtained from EUMETSAT [36]. Images are available in 12 discrete frequency bands with a finite temporal resolution of $t_s=15$ min. Eight bands represent the thermal infrared (IR) range, providing radiance resulting from the emission of the atmosphere and the Earth surface. They can by used to estimate the brightness temperature of the atmosphere and the surface. Three channels in the visible (VIS) spectrum measure the reflection of solar light at clouds or the Earth’s surface. This information can be used to retrieve the albedo of clouds and the surface. Lastly, the High-Resolution Visible (HRV) channel contains multiple broadband detection elements to scan the Earth with a lower sampling distance. The spatial resolution of the satellite images in Central Europe is roughly $0.05°$ latitude and $0.03°$ longitude (except the HRV band, which we do not consider). All satellite data are projected to an equirectangular projection with an equal spatial resolution of $0.05°$. We represent the satellite data as collections of 2D images $S_{\lambda ,t}$ as data with values quantized to a 16-bit integer. Figure 1b–d show examples of processed satellite imagery of the VIS006 ($\lambda =0.6 \mathsf{\mu }$m), WV062 ($\lambda =6.2 \mathsf{\mu }$m), and IR120 ($\lambda =12 \mathsf{\mu }$m) channels at 4 June 2016 at 12:00 h UTC, where a lot of lightning occurred across Central Europe. The images already indicate that the optical band might be more informative, as the infrared channels only see the cloud tops. However, the patterns in the visible spectrum are complex and not easy to capture in a hand-designed classifier.

Lightning images: Measurements of lightning activity are given as point sets $\{l_1,\dots ,l_{n_l}\}\in {\mathbb{R}}^4$, with two spatial geo-coordinates, a time coordinate, and the electrical current $I_i$. The lightning data used in this study were obtained from the LINET lightning detection network [8]. LINET is a low-frequency long-range lightning detection network (VLF/LF) using the time-of-arrival (TOA) method, consisting of several ground-based lightning sensors. For further processing, we convert the point set into images, with the same spatial and temporal resolution of the satellite images. To this end, we bin all lightning events in an aggregation time span of $t_a=15$ min, during which the satellite data have been measured, or, during experiments with varying lead-time $\Delta t$, offset by the corresponding $\Delta t$, and binarize them by setting the closest pixel to 1 (with background set to 0). To filter out potential noise, only lightning events with an electric current of at least 1 kA were considered. We denote the resulting lightning images by $L_t$. Figure 1a shows an example of a processed lightning image for the same example date (4 June 2016, 12 h UTC).

The region of interest (ROI) in our work is the mid-latitudes in Central Europe, between $2.0°\mathrm{W}$ and $21.5°\mathrm{E}$, and $44.5°\mathrm{N}$ and $57.5°\mathrm{N}$. The data are cropped to the defined ROI and split up in smaller patches, according to Appendix A.3. We focus our work on the months May to August in the years of 2016 and 2017. As additional test data set, we use data from August and September of 2021. Detailed information about the data and their use can be seen in

Table A1. Train, test, and validation splits for 2016/2017 data set. Events are shown in millions ($\times {10}^6$) in the area ($2.0^{\circ }$W–$21.5^{\circ }$E, $44.5^{\circ }$N–$57.5^{\circ }$N).

Year	Month	Week	Events	Data Set
2016	May	17	$0.03$	Train
2016	May	18	$0.10$	Test
2016	May	19	$0.24$	Train
2016	May	20	$0.13$	Validation
2016	May	21	$0.70$	Train
2016	May/June	22	$0.94$	Test
2016	June	23	$0.80$	Train
2016	June	24	$0.56$	Validation
2016	June	25	$1.81$	Train
2016	June/July	26	$0.58$	Test
2016	July	27	$0.36$	Train
2016	July	28	$1.08$	Validation
2016	July	29	$0.70$	Train
2016	July	30	$1.18$	Test
2016	August	31	$0.95$	Train
2016	August	32	$0.28$	Validation
2016	August	33	$0.67$	Train
2016	August	34	$0.50$	Test
2016	August	35	$0.56$	Train
2017	May	17	$0.01$	Validation
2017	May	18	$0.15$	Train
2017	May	19	$0.26$	Test
2017	May	20	$0.28$	Train
2017	May	21	$0.33$	Validation
2017	May/June	22	$1.10$	Train
2017	June	23	$0.37$	Test
2017	June	24	$0.61$	Train
2017	June	25	$1.88$	Validation
2017	June/July	26	$1.05$	Train
2017	July	27	$0.81$	Test
2017	July	28	$0.77$	Train
2017	July	29	$1.29$	Validation
2017	July	30	$1.13$	Train
2017	July/August	31	$0.89$	Test
2017	August	32	$1.25$	Train
2017	August	33	$0.45$	Validation
2017	August	34	$0.43$	Train
2017	August/September	35	$0.95$	Test

and

Table A2. Additional test split for 2021 data set. Events are shown in millions ($\times {10}^6$) in the area ($2.0^{\circ }$W–$21.5^{\circ }$E, $44.5^{\circ }$N–$57.5^{\circ }$N).

Year	Month	Week	Events	Data Set
2021	July	30	$0.23$	Test
2021	August	31	$1.09$	Test
2021	August	32	$1.06$	Test
2021	August	33	$0.89$	Test
2021	August	34	$1.33$	Test
2021	August/September	35	$0.82$	Test
2021	September	36	$0.97$	Test
2021	September	37	$0.56$	Test
2021	September	38	$0.88$	Test
2021	September/October	39	$0.29$	Test

for 2016/2017 and 2021, respectively. The processing of the data is done using multiple tools, such as pyPublicDecompWT [37], SatPy [38], Pyresample [39], and Cartopy [40].

The learning task can now be posed as a probabilistic prediction of a future lightning image: Let $t_0$ denote the current time, $\Delta t$ the lead time for the prediction, $t_s$ the temporal sample spacing, $t_a$ the aggregation time span (in our case: $t_s=t_a=15$ $\min )$, and k the number of past satellite images used for the prediction. With this information, we want to determine a probabilistic classifier $f_{\theta }$ that computes a probability map that specifies for each pixel the likelihood of a lightning event being marked in the future lightning image:

$$f_{\theta }\left(L_{t_0},S_{t_0},L_{t_0-1·t_s},S_{t_0-1·t_s},\dots ,L_{t_0-k·t_s},S_{t_0-k·t_s}\right)\approx L_{t_0+\Delta t}.$$

(2)

This problem is solved by representing the function $f_{\theta }$ by a deep neural network with parameters $\theta \in {\mathbb{R}}^d$. Learning is performed using maximum-likelihood: We assume that we are given a set of N training images with pixel-dimensions $(W,H)$ and determine a good-fitting $\widehat{\theta }$ as a (local) maximum of the likelihood function with L1 regularization:

$$\widehat{\theta }=\underset{\theta \in {\mathbb{R}}^d}{arg\; max}-\frac{{\lambda }_{\mathrm{R}}}{d}\sum _{j=1}^d\left|{\theta }_j\right|+\sum _{\begin{array}{c}i\in \{1,\dots ,N\}\\ x\in \{1,\dots ,W\}\\ y\in \{1,\dots ,H\}\end{array}}w\left(f_{\theta }^{\left(i\right)},L_{t_0+\Delta t}^{\left(i\right)},r_s\right)[x,y]·\ell \left(f_{\theta }^{\left(i\right)},L_{t_0+\Delta t}^{\left(i\right)}\right)[x,y],$$

(3)

with

$$\ell \left(f_{\theta }^{\left(i\right)},L_{t_0+\Delta t}^{\left(i\right)}\right)=\underset{\mathrm{positives}}{\underbrace{log{f}_{\theta }^{\left(i\right)}·L_{t_0+\Delta t}^{\left(i\right)}}}+\underset{\mathrm{negatives}}{\underbrace{log\left(1-f_{\theta }^{\left(i\right)}\right)·\left(1-L_{t_0+\Delta t}^{\left(i\right)}\right)}},$$

(4)

$$f_{\theta }^{\left(i\right)}=f_{\theta }\left(L_{t_0}^{\left(i\right)},S_{t_0}^{\left(i\right)},L_{t_0-1·t_s}^{\left(i\right)},S_{t_0-1·t_s}^{\left(i\right)},\dots ,L_{t_0-k·t_s}^{\left(i\right)},S_{t_0-k·t_s}^{\left(i\right)}\right).$$

(5)

Here, the operator $[x,y]$ refers to the pixel at position $x,y$ in the respective images; and w denotes a per-pixel class weight, based on the prediction skill of the sample. L1 regularization is scaled by the factor ${\lambda }_{\mathrm{R}}$.

The main issue with the cross entropy (CE)-based loss function shown in Equation (4) is the naive (unweighted) summation over all pixels, regardless of their classification. Even when including all pixels in a search radius $r_s$, the amount of pixels “with lightning” is extremely sparse. Therefore, we add the event-based, per-pixel weight w in Equation (3), which addresses the issue of heavy class imbalance in a sample. This differs from per-sample weighting strategies based on class balance, such as [41]. Per-sample weights work well with known class imbalances, e.g., when data from one class are underrepresented in the data set, by scaling the importance of single samples. This is only partially useful in our case: Thunderstorms are comparatively rare, but when they occur, lightning activity is reasonably high, spanning over multiple pixels in a sample.

In the following, we will construct a per-pixel weight map w, based on the samples class labels $L_{t_0+\Delta t}^{\left(i\right)}$, the predictions $f_{\theta }^{\left(i\right)}$, and a (fixed) “search radius” ($r_s$) as follows:

$$w\left(f_{\theta }^{\left(i\right)},L_{t_0+\Delta t}^{\left(i\right)},r_s\right)[x,y]=\left\{\begin{array}{cc}{\omega }_{\mathrm{lightning}},\hfill & \mathrm{if}{L}_{t_0+\Delta t}^{\left(i\right)}[x,y]=1\mathrm{or}\left((x,y)\in {\mathrm{FP}}_{r_s}\left(f_{\theta }^{\left(i\right)},L_{t_0+\Delta t}^{\left(i\right)}\right)\right)\hfill \\ {\omega }_{\mathrm{no}-\mathrm{lightning}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(6)

where ${\mathrm{FP}}_{r_s}(p,l)$ is the set of pixels classified as false positive given the search radius $r_s$, the predictions p, and ground truth labels l of the sample, and ${\omega }_{\mathrm{lightning}}$ is the modified class-weight of the “lightning” class in the data set, and ${\omega }_{\mathrm{no}-\mathrm{lightning}}$ its corresponding counterpart. The class-weight ${\omega }_{\mathrm{cw}}$ of a class “cw” is defined as

$${\omega }_{\mathrm{cw}}=\frac{1}{2}\times \frac{1}{\left|\mathrm{cw}\right|},$$

(7)

where $\left|\mathrm{cw}\right|$ is the amount of instances of the class. Scaling by $\frac{1}{2}$ keeps the overall loss at a similar magnitude, so that the sum of the weights of all examples roughly stays constant. The values of ${\omega }_{\mathrm{lightning}}$ and ${\omega }_{\mathrm{no}-\mathrm{lightning}}$ are pre-computed for the whole training split of the data set, and are shown in Table 1. Taking false positives into account, w will adapt to the performance of the classifier f. The training split of the data set is detailed in Table A1.

Table 1. Computed pixel-weights of the training split for the 2016/2017 data set, based on class-weights shown in

Table 2. Measured class-weight of the training split for the 2016/2017 data set.

Class	Class-Weight
lightning	$0.15$
no-lightning	$99.85$

and Equation (7).

Class	Pixel-Weight
${\omega }_{\mathrm{lightning}}$	$3.33$
${\omega }_{\mathrm{no}-\mathrm{lightning}}$	5 × 10−5

Table 1. Computed pixel-weights of the training split for the 2016/2017 data set, based on class-weights shown in Table 2 and Equation (7).

Class	Pixel-Weight
${\omega }_{\mathrm{lightning}}$	$3.33$
${\omega }_{\mathrm{no}-\mathrm{lightning}}$	5 × 10−5

The search radius $r_s$ is used to model the label uncertainty introduced by various factors, such as the dislocation between lightning and the fictitious center of the Cb, the geolocation error of the satellite, and the movement of the cloud [12]. During the computation of w, it is used to determine false positive predictions. The weight map w can efficiently be computed at every training step in parallel. During training, the loss values are averaged over all pixels and samples in a batch.

The network f used in our study is based on the U-Net [24] architecture, combined with ResNet-v2 [42] residual blocks, adapted to work with three-dimensional input.

The input to our model is of the form $(B,H,W,T,C)$, where B is the batch size, H and W the height and width of an image, T the amount of time-frames, and C the amount of channels. We fixed the height and width of our model to be 256 px × 256 px, which equals 12.8° × 12.8°, or roughly 1425 km × 1425 km. Larger input areas are split up according to the domain decomposition scheme described in Appendix A.3. Further, each input possesses a boundary region, overlapping with neighboring inputs, which allows for a more precise prediction of the non-overlapping region, but which is excluded from the optimization process and evaluation.

Figure 2 shows an overall view of the used network architecture. The U-Net structure, including skip-connections, can be seen. Varying from the original architecture, we replace the stacked convolution layers and the pooling layer at each down/up sampling step with residual blocks (with stride where necessary). Instead of cropping the feature map of the contracting path, we use the feature map after the down-sampling operator. We use convolutions for down-sampling, but deterministic trilinear up-sampling operations. Deviating from the original architecture, we replace batch normalization layers with instance normalization [43], which normalizes each element of the batch independently, i.e., only across the spatial and time dimension.

The residual blocks consist of full pre-activation units, optionally paired with pre-activated convolution shortcuts, as described in [23]. Figure 3 illustrates the used residual (Figure 3c), down-sampling (Figure 3a), and up-sampling blocks (Figure 3b). Convolution layers use $3\times 3\times 3$ filters for up- and down-sampling, and $1\times 1\times 1$ otherwise. Non-linear layers consist of ReLU [44] activation functions.

As usual, we increase the amount of channels per block (representing coarser spatial scales). Empirically, we found good validation results for a stronger increase (only) for larger lead-times—details are shown in

Table 3. Detailed number of channels in the network, depending on the lead time.

Lead Time	Block 1	Block 2	Block 3	Block 4	Block 5	Block 6	Bottleneck
0 $\min$	13	13	14	14	15	16	16
30 $\min$	14	16	20	28	46	81	81
60 $\min$	14	16	21	31	52	97	97
90 $\min$	14	16	21	32	56	106	106
120 $\min$	14	16	22	33	59	114	114
180 $\min$	14	16	22	35	63	124	124

. When growing, it follows an exponential curve countering the spatial down- and up-sampling per block in the U-Net, which restraints memory use for weights and activations. The resulting network can be trained on consumer graphics cards at a reasonable batch sizes. Together with the training settings, the expressivity of the neural network is adapted to “match” the complexity of the modeled problem.

Our model is trained using stochastic gradient descent (SGD) with momentum [45] using decoupled weight decay (SGDW) [46] to optimize the loss function, described in Equation (3). The base learning rate (LR) is set as $\lambda =0.5$, and the weight decay (WD) and L1 regularization are set based on the number of channels of the network, as shown in

Table 4. Detailed training settings, depending on lead time and network size.

Lead Time	Bottleneck Channels	Base LR	WD	Regularization Factor	Epochs
0 $\min$	16	0.5	5.0 × 10−6	0.1	14
30 $\min$	81	0.5	7.0 × 10−6	0.15	14
60 $\min$	97	0.5	7.0 × 10−6	0.2	14
90 $\min$	106	0.5	7.0 × 10−6	0.2	14
120 $\min$	114	0.5	1.0 × 10−5	0.3	14
180 $\min$	124	0.5	1.0 × 10−5	0.3	14

. We use a batch size of 64 and train the network for 14 epochs. In each epoch, the network trains on a random permutation of the complete training data set. At the end of each epoch, we validate the network’s performance on the validation data set and save its parameters to disk. Every 8th batch, we adapt the decision threshold based on the current best performance. During training, the LR is scheduled with a 1-cycle LR scheduler [47] (two-phase, cosine decay). The learning rate schedule throughout the training process is discussed in more detail in Appendix A.5 and is depicted in Figure A2. Additional to the use of our custom weighted loss function, samples with no lightning activity are discarded during training.

All parameters (of all experiments) were tuned based on data given in Table A1. Our method uses the following input features: SEVIRI channel VIS 0.6, VIS 0.8, nIR 1.6, IR 3.9, WV 6.2, WV 7.3, IR 8.7, IR 9.7, IR 10.8, IR 12.0, IR 13.4 and the most recent lightning events from LINET.

We use PyTorch [48] for all of our experiments and train on a single machine, equipped with an Intel Core i7-8700 Processor and a single NVIDIA TITAN RTX GPU. A reference implementation is provided under a free license.

3. Results

We evaluate our method on the testing data set aside from the original 2016/17 data set (Table A1 in the Appendix A.1), as well as testing data from the month of August and September of 2021, for which comparison data with a nowcasting method [14] at DWD were available. In line with good experimental practice, all hyperparameter tuning has been concluded using validation data before running any inference on any test data has been performed (with frozen, final hyperparameter settings). In particular in our case, where substantial hyperparameter tuning was required, this protocol minimizes the risk of reporting a coincidental success. Further, training and testing were performed 5 times with independent random initialization and training of the network, which can always lead to (a bit of) spread in the results, and mean and standard deviation are reported.

We report the critical success index (CSI) (see Appendix A.2 for details) of the deep network and a base-line “persistence” method, which just assumes that the most recent lightning events at the start of the prediction period persist at the same spot indefinitely. We also compute an “improvement factor” that measures the factor by which the deep network outperforms the base-line persistence model in terms of CSI. As discussed in more detail in Appendix A.4, this makes it easier to compare to other methods while avoiding misjudgement due to small deviations in the evaluation protocols for the corresponding CSIs.

The findings are shown in Figure 4 and

Table 5. CSI of our method for various data sets, SR 30.61 $\mathrm{k}$$\mathrm{m}$ and varying lead time (LT).

Data Split\LT	0 min	30 min	60 min	90 min	120 min	180 min
Validation 16/17	$99.9\pm 0.0$	$89.2\pm 0.8$	$78.5\pm 1.6$	$74.4\pm 1.6$	$66.8\pm 2.9$	$38.2\pm 2.3$
Testing 16/17	$99.7\pm 0.0$	$88.6\pm 1.4$	$77.8\pm 1.9$	$76.2\pm 1.6$	$70.0\pm 2.7$	$41.8\pm 3.1$
Testing August 21	$99.9\pm 0.0$	$87.9\pm 1.1$	$75.4\pm 2.0$	$69.7\pm 2.7$	$61.3\pm 3.7$	$31.9\pm 3.0$
Testing September 21	$99.1\pm 0.0$	$87.2\pm 1.3$	$72.9\pm 2.7$	$66.9\pm 3.5$	$56.9\pm 5.2$	$24.7\pm 4.2$

.

3.1. Classifier Performance and Base-Line Comparison

As a first sanity check, for zero lead time, the network reaches very close to $100\%$ success (i.e., matching the base-line when tasked with just predicting the last known events), which means that the network has been able to successfully learn to base its decisions solely on the lightning data in this setting.

With growing lead-time, the results diverge from base-line, with substantial advantages for the learning-based model. We obtain a CSI of about 57–80% at 120 min and 25–42% at 180 min lead time (depending on the data set), corresponding to a factor of improvement of ≥1.5 at 3 h and ≥2.0 at 2 h lead time.

This can be interpreted as a good outcome: Usually a CSI value of $50\%$ is used to decide if the forecast provides a useful prediction. The CSI value is well above $50\%$ for all testing periods up to a lead time of 120 min, demonstrating the quality and potential practical relevance of the classifier obtained from statistical machine learning.

Specifically, our model was able to transfer to unseen data from September 2021, as neither the year nor the month was in the training or validation data set. As expected, testing results for this period show a decline in performance, but are still above $50\%$ CSI for a 120 min lead time. Data from 2021 in general perform a bit worse than testing data from the corresponding months of the 2016/17 data. We believe that the close temporal proximity of the 2016/17 testing data to training data is likely responsible for this. While we reduce direct data leakage of capturing the same convective cell by maintaining a 12 h gap in between adjacent weeks, the statistical similarity of the overall meteorological conditions appears as a plausible explanation. The increased drop for September 2021 is consistent with this hypothesis as the atmospheric conditions in September generally differ from May to August, which are used for training. As shown in Table 5, the performance on validation data is not systematically better than testing performance, showing that the hyperparameter choice has most likely not been overly specific to training and validation data used for method development.

3.2. Prediction Structure

To provide an impression of the outputs produced by the network, Figure 5 shows the model logits (inputs to the softmax-layer that yields normalized probabilities) and skill (True/False positives and negatives per pixels) for the previously used example of a severe weather event on 4 June 2016 at 12:00 h UTC [49]. It shows how the predicted lightning map significantly reduces false negatives and positives over base-line persistence. Visually, one can see that the predictions are automatically enlarged by the network to match the search radius, but our formulation of the adaptively weighted loss (Equation (6)) does not lead to “overblown” predictions, thereby reducing false positives (a naive dilation operation on all input and output data to model the search radius would lead to smudged, unsharp predictions at the border regions and could seriously harm false positive rates).

Further details of the results are presented and discussed in the following subsections.

3.3. Comparison against Physical Nowcasting Method

The DWD currently works on an improvement of the nowcasting applied within the 24/7 nowcasting method, discussed in detail in [14]. In order to obtain a first hint of the possible improvements of a machine-learning-based method, the factor of improvement between persistence and nowcasting was calculated for the 120 min prediction for both approaches. Due to a deviation in the lead time reference point, a 120 min lead time in our work transfers to a $120+10$ min lead time in the physical nowcasting method. The validation method has been adapted as far as possible for this comparison. Nevertheless, it is not a direct comparison and hence only provides an indicator for possible improvements using the method presented in this manuscript. The authors of [14] state a CSI for the persistence algorithm of $26\%$, which is consistent with the values reported in our work; see

Table A3. CSI of the persistence model for various data sets, SR 30.61 $\mathrm{k}$$\mathrm{m}$, and varying lead time (LT).

Data Split\LT	0 min	30 min	60 min	90 min	120 min	180 min
Validation 16/17	$100.00$	$81.38$	$56.11$	$38.08$	$27.73$	$17.10$
Testing 16/17	$100.00$	$80.89$	$58.42$	$40.37$	$28.64$	$16.11$
Testing 21 August	$100.00$	$78.66$	$48.79$	$31.01$	$21.36$	$12.25$
Testing 21 September	$100.00$	$80.08$	$57.03$	$39.50$	$28.42$	$16.79$

. For the physical nowcasting method, they state a CSI of $38\%$; thus, the improvement factor is of about $1.4$ and hence significantly below the values achieved by our learned classifier. This is not proof but a strong indicator that the developed method is able to gain hidden information and to improve the prediction of Cbs in the time frame of 0–3 h.

3.4. Feature Attribution

After observing that our classifier is able to make statistically strong predictions (in comparison to alternative methods), we ask the question which “features” (input channels) are most informative in making the decision.

Figure 6 shows the result of training the classifier again, but with some inputs omitted. The graph shows the CSI over the course of the day, with an overlay of the number of lightning events recorded by LINET (dotted black curve) at these times of day on average (clearly peaking in the afternoon UTC time, which is late afternoon local time in the observed region). The time on the x-axis denotes to the prediction time and corresponds to the number of lightning events shown (the actual data used for the prediction are taken 2 h prior).

The full classifier model (dark blue curve) shows a significantly lower CSI at night-time than during the day: With the sunrise starting, performance improves, and drops again later in the evening; the increase in performance during 4–6 am UTC coincides with the beginning of dawn in Central Europe (local daylight-saving time being offset by +2 h) in the relevant time frame of May to August. As more daylight becomes available closer to the start of summer, more visible information becomes available earlier in the day, which could explain the transition in accuracy. Similarly, accuracy declines towards the end of the day.

A second phenomenon, apparently overlayed with the hypothesized daylight effect, is a further increase in CSI between 9 and 16 h UTC, peaking around noon (notably, at about 80% CSI). This corresponds to the time where most LINET events have been recorded. Apparently, prediction becomes easier if events are more common (which is a statistically plausible finding and would be expected).

The hypothesis that the network bases its decision mostly on information in the visible spectrum of light is solidified by looking at the CSI-curves when using only images from the visible spectrum (VIS/light orange) versus various infrared channels only (light blue/green): During daylight, the model using only visible features drops only slightly in performance compared to the full model (often within the $1\sigma$-margin of error of the full model), while the infrared models perform significantly worse. Only at night, the prediction benefits from IR-information (with visible-only falling behind), although at an overall poorer level of performance. Surprisingly, the water-vapor bands alone yield an even slightly worse performance than the other IR-channels.

The comparison against base-line persistence shows an occasional drop below base-line for the restricted models, which might be explained by training noise (with variance increasing during low-event-count night times); this also suggests to not over-interpret smaller differences (WV vs. IR), but the gap to visible light appears very large and correspondingly very unlikely coincidental.

In summary, we observe that visible light has by far the largest contribution to classification performance, indicating the image recognition in the visible spectrum plays a major role. When visible light is available, the model performs very well at levels of CSI of 60–80%, which might have some operational relevance for applications to daytime activities. Infrared imagery appears to be much less informative to our classifier, which can only sustain a generally lower prediction performance consistently at day and night times. The water vapor channels alone, which are a target for physically motivated, “hand-crafted” models [12], lead to overall worse prediction results than visible or other infrared bands.

4. Conclusions

Our paper provides an end-to-end training technique for Cb nowcasting. As it works directly on unbalanced data, it can easily be used for predictions, as no pre-filtering of data is required. The experimental results provide clear indications that deep learning can be used to improve the Cb nowcasting even in this general scenario of naturally unbalanced classes. It should be noted, however, that the data used are not maximally unbalanced: All training and testing has so far been taken from the “warm” months of May to September, where lightning is much more prevalent in the observed region of Central Europe; including winter data and/or a more global excerpt of the planet might likely lead to a decrease in performance.

Comparison to operational nowcasting approaches: Most physical models are based on the extrapolation of the detected Cbs with atmospheric motion vectors (AMV; see [14] and the references therein). This approach works well as long as the cells do not decay during the prediction period. Unfortunately, cells usually do decay after a certain lifetime, such that this effect occurs regularly. Further, newly developed cells cannot be captured by extrapolation of detected cells. These are serious drawbacks of the AMV approach. The quite good results achieved with deep learning indicate that the training process might enable the network to gain information on the life cycles of cells (decay, newly developed cells). The network seems to be able to learn to a certain extent whether Cbs decay or newly develop within the prediction period or not, and we believe this is crucial for lead times around or larger than 180 min, as the lifetime of regular Cbs is usually shorter than this; see, e.g., [50].

Importance of the visible-light spectrum: The results diagrammed in Figure 6 clearly show that the learning process benefits tremendously from the visible channels. In the IR, only the cloud top contributes to the emission and hence to the signal measured at the satellite, with exception of semi-transparent clouds. However, in the VIS, the complete cloud contributes to the signal, and the reflections hold the information of the cloud optical thickness and effective cloud droplet radii [51]. Thus, the VIS provide much more information about the cloud textures, shapes and micro physics. It is likely that this information is used to learn information about the life cycle of Cbs. Otherwise, better results compared to physical methods are hard to explain. Although the prediction of Cbs with NWP has been significantly improved, nowcasting is still assumed to outperform NWP in the first 0–3 h. Thus, a lot of scientists aim to develop a seamless transition between nowcasting and NWP. Currently, mainly physical methods are used to achieve this goal, but deep learning might be a powerful alternative.