Opportunistic Rainfall Monitoring from Single Pictures Using Artificial Intelligence ^†

Abstract

Urban flood risk mitigation requires fine-scale near-real-time precipitation observations that are challenging to obtain from traditional monitoring networks. Novel data and computational techniques offer a valuable potential source of information. This study explores an unprecedented, device-independent, artificial intelligence-based system for opportunistic rainfall monitoring through deep learning models that detect rainfall presence and estimate quasi-instantaneous intensity from single pictures. Preliminary results demonstrate the models’ ability to detect a significant meteorological state corroborating the potential of non-dedicated sensors for hydrometeorological monitoring in urban areas and data-scarce regions. Future research will involve further experiments and crowdsourcing, to improve accuracy and promote public resilience.

1. Introduction

In the last 50 years, flooding has been the most common and widespread natural disaster [1]. Climate changes can exacerbate extreme precipitation events [2], heightening the flood risk, especially in urban areas where impervious surfaces alter natural hydrological response [3]. Fine-scaled and near real-time rain observations are essential to assess the rainfall-runoff response in urban areas [4] and therefore the risk scenarios and alerts for exposed population [5]. The dedicated monitoring networks—from in situ devices (rain gauges, disdrometers) and remote sensors (radars, satellites)—differ in spatio-temporal representativeness and availability both regionally and temporally, thus require additional information to fill gaps [4].

In this context, newly available data sources and computational techniques offer enormous potential in risk understanding, analysis, forecasting, and warning [5]. Repurposing data that are continuously being produced can yield useful information so that non-dedicated devices can serve as low-cost “opportunistic” hydrological sensors [4].

This study explores the potential and limitations of hydrometeorological monitoring using generic consumer cameras as smart, non-traditional, opportunistic sensors.

2. Materials and Methods

2.1. Rainfall and Computer Vision

Digital pictures and videos are affected by weather conditions, in particular by precipitation. In fact, the rainfall produces local variations in the pixel’s intensity value of raster images depending on the following: raindrops characteristics (shape, size, velocity, density), camera parameters (exposure time, F-number, depth of field, etc.), and environment settings (scene brightness, background, etc.) [6]. Rain-induced artifacts reduce the performance of computer vision (CV) systems, thus several rain detection and removal techniques were developed to improve algorithms robustness in adverse weather conditions. Publications that concentrate on video-frame sequences adopt both classical CV and deep learning (DL) techniques—whereas studies that use single images adopt mainly DL-based methods. Instead, a smaller body of literature is concerned with meteo-hydrological purposes. Machine earning (ML) and DL methods have become the standard for classifying weather conditions from single outdoor images (e.g., sunny, cloudy, snowy, rainy, and foggy [7]). Rain measurement methods are based on classical image processing algorithms and exploit temporal information derived from video-frame sequences thus presenting stringent acquisition requirements (time step, device settings, resolution, etc.). For an overview of existing techniques, the reader is directed to [8].

All studies support the notion that cameras can function as rainfall sensors. However, such methods remain narrow in focus using only one of the following specific image sources at one time: vehicle-mounted cameras; surveillance cameras; outdoor scenes; and adjustable cameras [8].

An effective monitoring network must maximize sensors redundancy to improve accuracy and reliability and, at the same time, must minimize processed data amount to reduce computational resources and time. Thus, our research approach hinges on the following: application simplicity and readiness; adherence to the perceptual reality; cost-effectiveness.

As opposed to existing works, our methodology exploits general-purpose cameras (e.g., smartphones, surveillance cameras, dashcams, etc.) without requiring parameter setting, timeline shots, or videos. Using DL models based on transfer learning (TL) with convolutional neural networks (CNNs), our method can be seen as a two-stage process:

• Detection task: the CNN descries rainfall presence through a binary classification;
• Estimation task: the CNN estimates the quasi-instantaneous rainfall intensity through a multi-class classification;

A CNN-based approach was chosen since it is the state-of-art among other image recognition methods [9]. Contrary to traditional CV techniques, CNNs avoid the problem of manually engineered feature descriptors since such algorithms recognize visual patterns directly in images, learning automatically the most salient features [9,10].

Rain-induced visual effects are complex and often inconspicuous [6]. Furthermore, public big datasets are lacking and data creation is challenging. The non-triviality of the task and the size of available data drove the choice to exploit the common visual semantics of images by TL strategies combining fine-tuning and freezing layers [9].

2.2. Dataset and Model Setup

Datasets for training, validation, and test (ratio of 60:20:20) mimic real-world scenarios. Different sources have been combined following the criteria of availability, weather conditions representativeness, location, and lightning conditions diversity. Each input image was paired with exactly one output label describing the rain class.

For the binary classification [8], the dataset included 4788 images—crowdsourced images from Image2Weather set [7]; images from dashcams moving around Tokyo metropolitan area from 19 August 2017 for 48 h (©NIED) coupled with rainfall rates retrieved from XRAIN multiparameter radar (©NIED) [11] by capture time and GPS location (threshold 4 mm/h); pictures taken during experiments in the NIED Large-scale Rainfall Simulator located in Tsukuba (Japan) with different devices (Canon XC10, Sony DSC-RX10M3, Olympus TG-2, XiaoYI YDXJ 2, XiaoMi MI8) coupled with the nominal values of produced intensity ranging from 20 mm/h to 150 mm/h.

For the multiclass classification, the dataset included 55,600 images from: dashcams moving around Tokyo metropolitan area from 19 August 2017 for 48 h and in several Japanese prefectures from 8 July 2017 to 19 June 2018 (©NIED), coupled with XRAIN rainfall rates (©NIED) [11] by time and location; experiments in the NIED Simulator coupled with the nominal produced intensity (from 20 mm/h to 150 mm/h). Rain rates were binned into 6 non-overlapping classes.

Using a TL approach, the VGG16 network [12] (Figure 1)—pre-trained on the large-scale dataset ImageNet—was employed as convolutional base because of its generality and the portability of learned features [9].

To mitigate overfitting, different regularization techniques were adopted during training and validation phases, such as holdout train-valid-test data split; training data augmentation; dropout; adaptive learning rate; small batch training; early stopping; etc.

The models were entirely implemented with open-source tools, namely R language with RStudio using Keras framework and Tensorflow engine backend [9].

3. Results

To evaluate unbiasedly the predictive skills, the test sets were exposed to the models trained as described. Each test set (holdout set) was locked away completely during the training and evaluation steps [9]. The binary prototype was deployed in a real-world environment in Matera (Italy) in [8], showing great robustness and portability, contrary to the other algorithms [8]. The rainfall estimator presented the best performances with no-rain and heavy rainfall (the most critical condition for urban flood risk).

Table 1. Performance metrics.

Metric 1		Binary Model 2	Multiclass Model	Reference Values
Overall accuracy	$\frac{\left(TP+TN\right)}{\left(TP+FP+FN+TN\right)}$	85.28%	77.71%	worst = 0%
		85.13%		best = 100%
F1 score	$\frac{TP}{TP+0.5\left(FP+FN\right)}$	0.8600	0.73 3	worst = 0
		0.8482		best = 1
Cohen’s kappa	$\frac{p_o-p_e}{1-p_e}$	0.7057	0.7095	worst = 0
		0.6669		best = 1

¹ True Positives (TP), False Positives (FP), True Negatives (TN) and False Negatives (FN); p_o observed agreement and p_e hypothetical probability of chance agreement. ² Test set and Deployment set. ³ macro-averaged.

shows the overall metrics; the obtained classifications are significantly better than random predictions in both models. The non-informative accuracy values correspond to 50% for a binary classification and 16, 67% for a balanced 6-way classification. F1 score and Cohen’s kappa shows consistency, such metrics considers the balance of recall and precision and compare observed accuracy with an expected accuracy (random chance), respectively.

4. Discussion—Conclusions

This study presented a prototyping method for smart opportunistic near real-time rainfall monitoring in urban areas. Starting from the authors’ previous work [8], it represents an update of an Artificial Intelligence-based approach to gather rainfall observations from single images shot in very heterogeneous conditions.

The results achieved by the models test and deployment [8] proved that our system can provide approximated but robust near-real-time precipitation observations. In comparison with algorithms based on frame sequences or video, our single-image classification algorithms process less data volume and are more robust in urban scenarios where the background contains non-static elements, such as moving pedestrians or vehicles, as they don’t depend on background subtraction techniques.

The major limitation concerns the approximations inherent in the following output types: rainfall presence and intensity ranges. Another limitation can stem from visual ambiguities in scenes that are also misleading for human perception. However, redundant observations from a dense network with many inaccurate nodes can enhance overall performances.

Our methodology offers an expeditious operative opportunistic sensors system intending not to substitute, but instead to support traditional measurement methods. The readiness level, cost-effectiveness, and limited operational requirements allow an easy implementation by exploiting pre-existent devices. Hence, observations retrieved from camera-based opportunistic sensors can effectively complement dedicated rainfall observational methods especially in areas where the traditional monitoring networks is sparse, such as urban areas but also low- or lower-middle-income countries.

Future research will involve further experiments and crowdsourcing, to improve accuracy and at the same time promote public resilience through a smart city perspective.