There has been a steady increase in the occurrence of natural disasters since 1980 globally. The number of people that are prone to disasters is also increasing [1
]. Amongst natural disasters, the most catastrophic disaster includes hurricanes that occur in areas with warm seawaters that are in tropical and subtropical areas. The sun heats seawater, leading to the formation of enormous clouds, which cause excessive rainfall, floods and very fast-moving winds [2
]. Damage of approximately USD 265 billion was estimated in the US in the year 2017 due to three major hurricanes (Harvey, Maria and Irma). These hurricanes affected thousands of people and caused many fatalities. During such difficult times, the affected people required assistance. Hence, it is very essential assess the destruction brought about due to hurricanes [1
Satellite images have been used to determine whether there has been damage inflicted by the hurricane or not. Satellite images have been gaining immense popularity for monitoring hurricanes. Ground surveys are time-consuming and also labor-intensive [3
In artificial intelligence, transfer learning is a technique that involves reusing an already trained model on a different but related problem. This technique is now being popularly used in deep learning when the dataset is not large. This technique helps in the reduction in resources and the labeled data required for training newer models. It helps reduce training times [4
A convolutional neural network consists of feature extraction as the first stage and classification as the final stage. In transfer learning, the classification stage is altered. The initialization of the network has been performed with weights from the ImageNet dataset [4
]. The convolutional layers and the max-pooling layers are frozen so that no modification of weights takes place. Only the dense or the fully connected layers are left free to be altered. After this, the retraining of the model is performed. The advantage of the feature extraction stage is taken, and only the final classifier is tuned, which works better with smaller datasets. This is the reason for why it is called transfer learning, as the advantage of the knowledge of one problem can solve the second problem [5
This paper involves the study of hurricane damage detection using satellite images. The estimation of the intensity of hurricanes was performed by using deep convolutional neural networks. Infrared images were used for estimation and were obtained from the satellite source. The adopted method is known as Deep Phurie, and it produced a very low root mean square (RMS) value in comparison with the method adopted earlier, which is known as Phurie. Deep Phurie is completely automatic, but this paper does not evaluate the damage post-disaster [2
]. Furthermore, deep convolutional neural networks were used to estimate of the intensity of the tropical cyclones or hurricanes that took place over a period from 1998 to 2012. Regularization techniques were used along with many convolutional and dense layers. This technique helped in extracting features from hurricane images effectively. A low RMS value and an improved accuracy were obtained [6
]. However, these data were noisy and not of good quality.
A multilayer perceptron was proposed for the determination of the connection between the appearance of hurricanes and the high-energy particles that flow out from the sun. A multilayer perceptron is an artificial neural network accompanied by backpropagation. It was found that hurricane appearances could take place a few days before the breakout of a solar wind [7
As a deep learning method, a single-shot multibox detector (SSD) was employed for the calculate of the destruction inflicted on buildings due to hurricane Sandy, which occurred in the year 2012. The Vgg16 model and the SSD model were used, and improvements of 72% and 20% in mAp and mF1, respectively, were observed [8
]. The CNN model was used to determine areas that were severely affected by Hurricane Harvey. Satellite images were used for the extraction of man-made features such as roads and buildings before and after the occurrence of the disaster. An F1 score of 81.2% was achieved [9
Damage assessment after a hurricane is of utmost importance. In this paper, the author created a benchmark dataset for the property that became damaged by Hurricane Harvey. The dataset consisted of both undamaged and impaired building images, and they were obtained from satellite imagery. FEMA and TOMNOD were the sources of this dataset [10
The destruction brought about because of hurricane Dorian has been determined using satellite imagery and artificial intelligence. The austerity of the destruction caused due to the hurricanes has been determined, and an accuracy of 61% was achieved [11
Earlier studies focused on finding the intensity of hurricanes and providing a benchmark dataset for damage detection. Fewer studies have focused on classifying hurricane images into damaged and undamaged classes. In this paper, a comparative analysis of the four transfer learning models that include DenseNet121 [12
], VGG16 [13
], MobileNetV2 [14
] and InceptionV3 [15
] has been performed with respect to confusion matrix parameters. These models have also been used for determining the destruction brought about on buildings because of Hurricane Harvey.
The objectives of this study include the following:
Addition of a newer set of layers to the pre-trained models for classification of the satellite images of hurricanes into damaged and undamaged categories;
To generalize the model by applying data augmentation techniques to images;
To perform a comparative study based on accuracy, precision, recall and F1-score for the four pre-trained models, which include VGG16, MobileNetV2, InceptionV3 and DenseNet121, at a learning rate of 0.0001 and 40 epochs.
To compare the best performing models for various optimizers, which include SGD, Adadelta, Adam and RMSprop.
The rest of the paper is organized as follows: proposed methodology in Section 2
; results and discussion in Section 3
; conclusion and future scopes in Section 4
2. Proposed Methodology
The model that has been presented for automatic damage detection due to hurricanes is shown in Figure 1
. The platform used to create and run the algorithm is Kaggle. The model classifies satellite images into damaged and undamaged categories. The methodology comprises two main steps: The first is preprocessing [16
], which is further divided into normalization and data augmentation, and the second is classification using the pre-trained CNN models. Each stage has been described below.
The satellite images of the Houston region used in this study were captured by optical sensors. The images could be covered with clouds either partially or fully. This implies that the images obtained from the satellites have been corrupted by noise. The nature of the noise is unknown, meaning that it could be a result of fluctuations in light, the sensor of the camera or artifacts. Improving the quality of the images so that good results can be obtained is imperative. For this purpose, a denoising operation needs to be performed, which could be based on wavelets [18
] or can be acquired from a compressive sensing method [19
For the suppression of unwanted distortions or enhancement of some of the features of the images, pre-processing steps such as resizing were used. The original size of the satellite images of hurricanes is 128 × 128. The resizing of the satellite images of the hurricane was performed. The resizing of the images was performed at 224 × 224 when Vgg16, MobileNetV2 and DenseNet121 transfer learning techniques were applied. The images were resized to 299 × 299 on the application of the InceptionV3 technique.
The two main steps of the preprocessing stage, which include normalization and data augmentation, have also been explained in this section.
Normalization is a very important step for maintaining numerical stability in a model. Normalization helps in learning faster and brings about stability in gradient descents. The input image pixels have, thus, been normalized in the values between 0 and 1. Normalization is brought about by multiplying pixel values by 1/255.
2.1.2. Data Augmentation
The augmentation of data is a technique utilized to generalize the model by applying random transformations with respect to input images [20
]. It increases the variability and robustness of the model as the model becomes new and modified versions of the input data. An image data generator is utilized for augmenting the data, which is an on-the-fly data augmentation method because augmentation is performed during training time. The image data generator returns only the randomly modified images and not the original images. Data augmentation has been applied only to training images and not to testing images.
The techniques adopted for data augmentation in this study are rotation, width shifting, height shifting, horizontal flipping and zoom operation.
2.2. Hurricane Damage Detection Using Pre-Trained CNN Models
In this paper, four pre-trained models, which include VGG16 [22
], MobileNetV2 [23
], InceptionV3 [24
] and DenseNet121 [25
], have been used for classifying satellite images into damaged and undamaged classes.
Transfer learning models are models trained on very large datasets that include millions of images. As the models have been trained on such a large dataset, a generalization of the model takes place. The features that have been learned from the larger datasets help in solving a different problem consisting of lesser data or a smaller dataset. This helps eliminate the need to train a model from scratch.
The description of the architectures of these models is shown in Table 1
The VGG16 model comprises 16 layers that have weights and has approximately 138 million parameters. There are 13 convolutional layers and 3 fully connected layers. The VGG16 model is widely used because of its ease of implementation [22
]. MobileNetV2 consists of 53 layers and 3.4 million parameters. It has been derived from the MobileNetV1 model, which utilizes depth-wise convolution as the building block of the model. However, the additional feature from the previous models is that it has an additional inverted residual layer [23
]. This model is used because it is smaller in size and also cost-effective. There are nineteen bottleneck layers that were residual. There are 42 layers in the InceptionV3 model and 24 million parameters. InceptionV3 is an advanced version of InceptionV2. It reduces the amount of computations as it uses factorization methods [24
]. For InceptionV3, the input image size is (299, 299, 3). Densenet121 consists of 121 layers with trainable weights. DenseNet121 has 8 million parameters. In this model, the network proceeds deeper as each layer is connected to all the other layers; for example, the first layer is connected to the second, third, fourth and so on layers. This leads to a improved maximum flow of information amongst the layers [25
2.3. Tuning the Hyper-Parameters
The training of the four models has been performed for 40 epochs and a batch size of 100. The total epochs refer to how often the learning algorithm will be working through the complete dataset. Batch size refers to the number of training examples that would be utilized in a single iteration [26
]. A batch size of 100 implies that 100 samples from the training dataset would be used for the estimation of the error gradient before the weights of the models are updated. The learning rate (LR) is another important hyperparameter that should not be either too big or too small [27
]. It is used for finding the learning speed of the proposed models. The model would take a lot more time to reach the minimum loss if the LR is too small and if the LR is too high due to the fact that overshooting the low loss areas can take place. A learning rate of 0.0001 has been chosen in this paper. The batch size, number of epochs [28
] and the learning rate have all been decided empirically.
Furthermore, the activation function [29
] used is a rectified linear unit (ReLU) [30
]. The fully connected head, used along with all the four pre-trained models, is shown in Figure 2
. The pre-trained block is followed by a flattening layer and two dense layers. The flattening layer size of the DenseNet121 model is 50,176; for VGG16, the size is 25,088. For the MobileNetV2 model, the flattening layer size is 62,720; for InceptionV3, the size is 131,072. After the flattening layer, a dense layer of size 256 is applied. Finally, a dense layer with two classes that is damaged and undamaged is used.
The block diagram of the four pre-trained models, which include DenseNet121, VGG16, MobileNetV2 and InceptionV3, is displayed in Figure 3
The block diagram of DenseNet121 is shown in Figure 3
a. The input is of 224 × 224 × 3 sizes. This is applied to the DenseNet121 model, and the output obtained is of 7 × 7 × 1024 size. This is then applied to the new fully connected head, which comprises the flattening and dense layers. The output of the flattening layer is of 50,176 in size. The output of the first dense layer is 256, and the last dense layer classifies the images into two classes, which include damaged and undamaged classes.
The block diagram of the VGG16 model is demonstrated in Figure 3
b. An input size of 224 × 224 × 3 was applied to the model, and an output of 7 × 7 × 512 was obtained. The output of the flattening layer is 25,088, and the outputs of the dense layers are 256 and 2 in size.
The block diagram of MobileNetV2 has been demonstrated in Figure 3
c, for which its input image size is 224 × 224 × 3. This is applied to the model and an output of 7 × 7 × 1280 is obtained. The output after the application of the flattening layer is 62,720, and the outputs of the two dense layers are 256 and 2, respectively.
d presents the InceptionV3 model, for which its input image size is 299 × 299 × 3. The output when this input is applied to the model is 8 × 8 × 2048. The output obtained after the flattening layer is of size 131,072.
4. Conclusions and Future Scope
In this paper, four pre-trained models, includingDenseNet121, VGG16, MobileNetV2 and InceptionV3, based on transfer learning have been put forward for the detection of destruction inflicted on buildings due to Hurricane Harvey, which took place in the Greater Houston region in the year 2017. The comparison of the four models has been performed based on training accuracy, training recall, training loss, validation accuracy, validation recall and validation loss. The highest training accuracy of 0.9727 and training recall of 0.9735 was obtained by the DenseNet121model at the 40th epoch and learning rate of 0.0001. The highest validation accuracy of 0.9670 and validation recall of 0.9658 was obtained by the InceptionV3 model at the 40th epoch. The lowest training loss of 0.0666 and validation loss of 0.0956 was obtained by the DenseNet121 model at the 40th epoch.
A comparison was also performed in terms of the classification report’s parameters, and it was found that VGG16 outperformed other models by obtaining an accuracy of 0.75, an F1 score of 0.83 and a recall of 0.95.
When the comparison was performed for the best-performing models for various optimizers in terms of the classification report parameters, it was found that VGG16 performed better by obtaining an accuracy of 0.78 for the RMSprop optimizer.
Furthermore, an improvement could be brought in with values of the confusion matrix parameters. Moreover, the model could be made more generalizable by including images of other hurricanes.