Earthquakes accounted for over 60% of all natural disaster-related deaths from 2001 to 2011—a danger that will likely increase due to rapid global urbanization [1
]. Immediately after an earthquake occurs, satellite imagery is a critical component of damage mapping. Hussain et al. noted that “information derived from remote sensing data greatly helps the authorities in rescue and relief efforts, damage assessment, and the planning of remedial measures to safeguard such events effectively” [2
]. For immediate rescue operations, rapid damage maps derived from satellite imagery must be developed quickly. A study of the 1995 Kobe earthquake in Japan showed a drastic reduction of the total rescued and the proportion of survivors after the third day of recovery efforts [3
]. However, because rapid mapping is required to balance immediacy with in-depth analysis, early mapping efforts often yield coarse damage assessments [5
Remote sensing has been used widely to map the effects of major disasters such as earthquakes. Numerous studies have utilized electro-optical (EO), synthetic aperture radar (SAR), light detection and ranging (LiDAR), ancillary data, or a combination thereof for post-earthquake damage detection [1
]. One technique for damage detection involves fusion of SAR and EO data in pixel-based damage detection. Stramondo et al. used a maximum likelihood (ML) classifier on SAR features derived from the European Remote Sensing mission in combination with EO data provided by the Indian Remote Sensing satellite in order to identify damaged structures following the 1999 Izmit, Turkey earthquake [7
]. A similar approach combined SAR from COSMO/SkyMed mission and very high resolution (VHR) EO data from the Quickbird satellite to improve damage detection at block level after combining the two datasets in a pixel-based classification following the 2009 L’Aquila earthquake [8
As early as 1998, object-based image analysis (OBIA) has been used to detect earthquake damage from remote sensing [9
]. More recently, OBIA has been a continual focus in earthquake detection damage with many studies focusing on the use of unmanned aerial systems (UAS), LiDAR, and the popular image segmentation and classification software eCognition. Hussain et al. [2
] fused GeoEye-1 VHR EO data and airborne LiDAR elevation models derived from the RIT-ImageCAT UAS for image segmentation using the Definiens (now eCognition) software suite. The data were classified using nearest neighbor and fuzzy membership sets to detect damaged buildings and rubble following the 2010 Haiti earthquake. Similarly, Pham et al. [6
] used aerial VHR RGB composite and LiDAR data (also from the RIT-ImageCAT UAS) along with eCognition for object segmentation and damage detection.
The application of machine learning algorithms (MLAs) to earthquake damage detection is a relatively new area of study. MLAs actively adapt and learn the problem at hand, often mimicking natural or biological systems, instead of relying on statistical assumptions about data distribution [10
]. In addition to overall improved accuracy [11
], MLAs have several advantages compared to traditional classification and change detection methods. MLAs work with nonlinear datasets [11
], learn from limited training data [12
], and successfully solve difficult-to-distinguish classification problems [15
Ito et al. [16
] used learning vector quantization (LVQ), a type of artificial neural network (ANN) to classify SAR features signifying damage after the 1995 Kobe earthquake. Li et al. [17
] used a two-class support vector machine (SVM) on pre- and post-earthquake Quickbird imagery along with spatial relations derived from the local indicator of spatial association (LISA) index to detect structures damaged by the Wenchuan earthquake of 2008. Haiyang et al. utilized a SVM approach in combination with eCognition image segmentation on the RIT-ImageCAT RGB and LiDAR data, as well as the textural features of contrast, dissimilarity, and variance derived from the gray level co-occurrence matrix (GLCM) to detect urban damage in Port-Au-Prince [18
]. Kaya et al. [19
] used OBIA in combination with support vector selection and adaptation (a type of SVM) on pansharpened Quickbird imagery to conduct damage detection for specific buildings within Port-au-Prince after the 2010 earthquake. While OBIA using SVMs have been researched extensively in the past, ANNs, particularly radial basis function neural networks (RBFNNs), and Random Forests (RF) have shown promise in pattern recognition and image classification [15
] and have yet to be examined in the application of earthquake damage detection. All three algorithms require parameter-tuning process to achieve optimal performance and a cross-validation approach can be applied to automate the parameter-tuning. SVM has an advantage in dealing with small sample size problems due to its sparse characteristics. However, for applications where a large number of training samples are available, SVM often yields a large number of support vectors, resulting in unnecessary complexity and a long training time [21
Evaluating structural dimensions such as the Laplacian of Gaussian (LoG) and object-based metrics in addition to spectral and textural information could greatly increase damage detection rates. LoG, a blob detection technique, has been used for medical applications in nuclei mapping [22
] and for the detection of buildings in bitemporal images [23
]. As discussed earlier, OBIA has shown strong results in urban scenes and earthquake damage detection. Huang and Zhang had success applying the popular mean-shift segmentation algorithm for urban classification in hyperspectral scenes while statistical region merging (SRM) is another segmentation approach which is robust to noise and occlusions [24
]. Additionally, various metrics such as rectangular fit, morphological shadow index, and morphological building index can describe the structure of objects in the scene before or after segmentation [26
]. Applying structural descriptors such as a LoG filter and segmentation derived metrics to high resolution satellite imagery as an additional input to an MLA could evince damage in difficult to detect scenarios such as a pancake collapse [22
]. The robustness and generalizability of RF and ANN along with the additional dimensions of texture and structure may provide higher accuracies in the face of imperfect input data.
In past disasters, by the time an automated change detection scheme is ready for implementation, a crowdsourced team of visual interpreters is already mapping damaged buildings [29
]. Dong and Shan mention that while manual digitization of damaged structures requires trained image analysts and is unsuitable for large areas, “visual interpretation remains to be the most reliable and independent evaluation for automated methods” [1
]. Additionally, many previous studies suggest detection schema which require ancillary data such as UAS products, LiDAR, or GIS databases. Many of these products are unavailable in developing regions where the death toll is highest [30
]. Using MLAs (RF and ANN), a rapid damage map derived from readily available multispectral imagery could allow for a minimal compromise between time and accuracy and allow first responders to more rapidly allocate their resources in a crisis. RF and ANNs along with derived textural and structural features may provide improved balance between rapid and accurate damage detection. The main purposes of this study include:
Assess the performance of neural networks (to include radial basis function neural networks), and Random Forests on very high resolution satellite imagery in earthquake damage detection
Investigate the usefulness of structural feature identifiers to include the Laplacian of Gaussian and rectangular fit in identifying damaged regions
(supplemented by the confusion matrices in Appendix A
) compares overall performance of three MLAs using building-by-building and kernel density accuracy assessment. Our findings showed that the multilayer feedforward ANN outperformed both the RBFNN and random forests with an omission error rate of 37.7%. Figure 4
matches the spatially explicit locations of damage detected by the ANN algorithm with the digitized buildings marked as damaged or undamaged (please refer to Appendix B
for the other algorithms’ damage maps). Both RBFNN and RF had higher overall accuracies, yet drastically underestimated damage. RBFNN and RF created a high user’s accuracy for the damaged building class, but a lower producer’s accuracy. The feedforward ANN also had the shortest runtime, an advantage that was primarily gained through the extremely fast implementation of the ANN for testing. The kappa value for each of the algorithms indicated that the distribution of damaged and undamaged buildings could not be accounted for by random chance, however both of the ANNs clearly outperformed Random Forests in this measure as well.
For the kernel density accuracy assessment, a good result was measured as a value in the comparison map that ranged between −1 and 1 in accordance with the Tiede et al. [36
] approach. RBFNN reached a 90% kernel density map match (shown in Figure 5
), outperforming the standard ANN and RF, which were not far behind. This indicates that the radial basis function ANN may be able to generalize to a larger area with greater success than either a feedforward network or Random Forests. Even so, each of the algorithms performed at higher accuracies when generalizing the distribution of damage over a wide area instead of detecting individual, building level damage.
As well as examining an algorithm’s wide area generalizability, kernel density accuracy assessment also allowed for investigation into areas of common error of both omission and commission. One of the more interesting results was that these areas were common in all three algorithms. A common error of commission occurred in the north-center of the test area and coincided with the development of an internally displaced persons (IDP) camp (see Figure 6
). According to the algorithms, this camp broke up the “structure” of the underlying field and increased the randomness of the texture, which led it to misclassifying the area as a damaged building. Figure 7
shows a common area of omission error contained within the map in the central region of the testing area. The underlying cause of error in this region is the scene complexity and high density of small structures before the earthquake occurred. This preexisting randomness and highly variable structure and texture were difficult for the algorithm to interpret, leading to an error of omission. Even so, with kernel density matching occurring in nearly 90% of cells for each algorithm, a wide area damage density classification was successful for both of the algorithms tested.
Finally, variable importance shows similar trends for ANN and RF algorithms. There were a few variables that showed rather different utilization between the algorithms. Figure 8
shows the change in error between each variable and was developed by averaging the assigned variable importance between the pre- and post-earthquake datasets. Overall, the multispectral variables had lower changes in out-of-bag error and cross entropy in comparison to the textural and structural values, however the panchromatic images and the near infrared band was useful to each of the algorithms. Of the two texture measures, entropy was utilized more than dissimilarity in all three algorithms. Rectangular fit was marked as important for both the ANN and RF however the Laplacian of Gaussian filter was more impactful on the RBFNN assessment and also had a moderately high impact on RF performance.
It is difficult to determine outright which algorithm is better. While the multilayer ANN outperformed RF in building by building assessments and required slightly less overall training time, it required a large number of training samples in order to perform well. This is not necessarily the case for the RF algorithm. Also, it is rather easy to overfit an ANN to the training data, which can be avoided using RF due to its nature as an ensemble classifier [41
]. Finally, ANNs can also become stuck at local minima in the performance surface without reaching the global solution, yielding an insufficient result [41
]. However, in our study, the multilayer ANN had the lowest rates of error in detecting damaged buildings, without sacrificing much performance in wide area generalization or overall accuracy. SVMs, while not examined here, have shown promise in earthquake detection in previous studies [17
]. While our study focused on ANNs and RF, as little research has been done on their applicability to earthquake detection problems, future studies may investigate the performance of these algorithms (to include SVM) with respect to training sample size.
Beyond damage detection performance, practical considerations require an investigation into time complexity, particularly when considering any kind of operationalization of an algorithm in automatic damage detection. RF took much longer than either of the ANNs to train and test the datasets. The time complexity of a single classification and regression tree is O(mn∙logn) where m is the total number of variables and n
is the number of samples [48
]. Because RF is an ensemble classifier, the overall time complexity of Random Forests can be summarized as O(M(mn∙logn)) where M is the number of trees grown. For a large number of samples with moderate dimensionality, this can be quite time consuming. In contrast, neural network complexity is highly dependent on network architecture. Time complexity for the scaled conjugate descent algorithm is often polynomial, overall complexity is determined by the problem and the number of free parameters (weights, biases, or basis functions (in the case of RBFNN)) required to describe that problem [49
]. As such, testing showed that the ANNs trained faster and tested faster, which is important to consider given the requirement to process a potentially large amount of imagery in an operational context.
As previously discussed, a number of preprocessing steps are required to develop each of the textural and structural dimensions. Also, a k-means unsupervised clustering was used as part of the RBFNN algorithm to intelligently center the basis functions before training. Each of these steps adds time and complexity to the final product. For future studies, the parallelization of many of these processes is one way to greatly reduce computational time. Our data were gathered using serial processing (primarily because a parallel implementation of k-means was not immediately available) in order to establish a baseline and fairly assess each algorithm, however parallel implementations (which include graphics processing units [GPUs]) of both ANN and RF training are readily available for use and will greatly speed up training and implementation of these machine learning algorithms.
The actual results from this study mirrored what was expected quite well; the areas of imagery where texture and structure were broken up were often identified as damage, as one would expect. As mentioned in the results section, one interesting finding was that each of the algorithms erroneously identified IDP camp areas as building damage. These IDP camps are ad hoc structures (tents, tarps, and shanties) built primarily on open spaces. As these IDP camps were erected, they broke up the coherent texture and structure of the underlying terrain, causing the algorithm to mark them as damage. While this is technically an error of commission, it is nevertheless a useful result in showing the power of MLAs in seeking out patterns as well as their ability to simultaneously detect damage and displaced persons. In an operational context, the MLA results in combination with a priori knowledge of building distribution via a GIS database would allow first responders and emergency planners to easily distinguish damaged structures from these IDP camps.
As the experiment on variable importance showed, the textural and structural features were some of the most important factors which allowed both ANNs and RF to detect damage and IDP camps. Stramondo et al. [7
] also used important textural features for earthquake damage detection although in their study a maximum likelihood classifier was used. This line of thinking, paramount to computer vision applications, is expanded here by using more intelligent algorithms and readily available data. The importance of the panchromatic features along with the texture and structure products derived exclusively from that panchromatic imagery presupposes that future implementations of machine learning may be able to perform earthquake damage detection from panchromatic imagery alone. One reason that the multispectral imagery was not a driving variable is simply due to resolution; the native 2.4 m is too coarse to detect many of the features associated with earthquake damage. Interestingly, the only multispectral product which was found to be an important variable for each of the algorithms was the near infrared band, which may have resulted from a correlation between exposed rubble and a higher near infrared reflectance. These findings may guide future research in determining which variables to focus on in earthquake damage studies.
Our focus on simple panchromatic imagery is a departure from many previous earthquake studies. The state-of-the-art focuses on LiDAR, SAR, unmanned aerial vehicles, and the software suite eCognition [1
]. However, the access and availability of these additional data requirements may be limited in the aftermath of a destructive earthquake in a less developed region such as Haiti. A return to easily accessible data products such as multispectral or even panchromatic imagery alone could allow a MLA (potentially even one that is pre-trained) to detect imagery without the requirement of ancillary data. One potential disadvantage of the reliance on bitemporal VHR imagery is the requirement for precise coregistration. Different look angles can cause problems in classification and change detection. While image registration is still important to our study, a small look angle difference may not be critical due to our use of textural and structural features rather than the VHR imagery alone. Additionally, registration errors can be seen as a source of noise in the system; each of the MLAs used has been shown to be robust to noise [41
]. The difference in our look angles (~7°) did not appear to cause any damage detection errors in a visual survey of our results. Future research may investigate the limits of acceptable look angle differences or use a complex coregistration approach to eliminate the issue altogether [50
The future of earthquake damage detection may lie in deep convolutional neural networks (DCNN) coupled with high performance computing and GPUs. DCNNs have already pushed the boundaries of artificial intelligence and image recognition; rather than being told which textural, structural, or spectral inputs should be used, these networks automatically learn and identify the defining features (convolutions) of the problem at hand in order to classify future samples [51
]. Initial results are promising. Using MatConvNet (a deep learning library for Matlab), we experimented with training a DCNN with the VGG-F architecture following the approach and using the hyperparameters described by Chatfield et al. [52
]. We segmented the post-earthquake pansharpened image using SRM and trained the DCNN on each labeled, extracted object. The DCNN was not only able to detect buildings at a comparable rate (>55% detection rate), it was able to distinguish between damage and IDP camps (>65% detection rate) and did so using an after-only pansharpened image, reducing data requirements and eliminating the need for coregistration. A pre-trained DCNN optimized specifically for earthquake detection may offer a robust and operationally implementable solution to the much studied topic of earthquake damage detection.
This study analyzed the use of machine learning algorithms to include feedforward neural networks, radial basis function neural networks, and Random Forests in detecting earthquake damage caused by the 12 January 2010 event near Port-au-Prince, Haiti. The algorithms’ efficacy was improved by providing coregistered 0.6 m multitemporal imagery, texture features (dissimilarity and entropy), and structure features (Laplacian of Gaussian and rectangular fit) as inputs. Detection results were assessed on a structure specific basis by digitizing more than 900 buildings and comparing each MLA’s response to the UNITAR/UNOSAT validation set. For a wide area generalization, a kernel density map comparison was performed between each of the algorithms’ classification results and the UN damage validation points.
The feedforward ANN consisting of two hidden layers had the lowest error rate (<40%) without sacrificing much performance in overall accuracy or generalization to wider area damage estimates. Additionally, textural and structural features derived from panchromatic imagery were shown to be more important than spectral variables in the algorithms’ classification process. Each algorithm had common errors of commission occurring around ad hoc IDP camps that were spontaneously formed in open spaces following the earthquake; this technically incorrect result can be easily integrated with a GIS layer containing building footprints.
The results of this study show that not only do MLAs have potential for use in earthquake damage detection, but that panchromatic or pansharpened imagery can be the exclusive data source for training and testing. Measures of variable importance found that the panchromatic derived texture and structure products are the main drivers behind the success of these “shallow” machine learning algorithms. Future research into an operationally implementable machine learning method is warranted. An attractive next step is to transition into deep learning where convolutional neural networks move beyond pixel-based or object-based paradigms and begin to detect remotely sensed features in ways akin to natural image recognition.