A Review of Emerging Technologies for an Assessment of Safety and Seismic Vulnerability and Damage Detection of Existing Masonry Structures

: The construction sector has proven to be one of the slowest sectors to embrace technology—a problem that must be addressed. This problem can be quickly and e ﬃ ciently addressed in certain aspects of seismic engineering: from seismic risk assessment to damage detection, as well as condition assessments existing structures before or after an earthquake. In this paper, the literature review of assessment methods and damage detection technologies for existing (mainly) masonry structures is presented. Traditional methods are brieﬂy explained, and modern are critically discussed. Special focus is given to unmanned aerial vehicles, as well as, photogrammetry and close-range remote sensing as a technology that can complement traditional ways of assessment and give us data about a structure that is often di ﬀ erent to obtain. Graphical interpretation of one post-earthquake case study is provided. Open challenges and opportunities of emerging technologies for faster and easier assessment of seismic safety and vulnerability are presented.


Introduction
Earthquake is an unpredictable and unexpected natural disaster that results in the devastation of both natural and human-made environments. Studying the complexity of an earthquake, including its prediction and its consequences, requires knowledge from various areas of scientific study, ranging from seismology, geology, geodesy, mathematics, and applied statistics all the way to psychology, humanities and social science and of course, structural engineering.
The seismic risk of a structure is a measure of the expected future damage caused by the earthquake, which is expected to occur in the site of construction. It depends on three factors: hazard, vulnerability, and exposure [1,2]. Most casualties from earthquakes are associated with collapsing buildings. In this regard, the continuous assessment and monitoring of the seismic safety and vulnerability of buildings is a challenging task, especially when large-area evaluations are required [3]. The construction sector plays a significant role in the alleviation of the effects and consequences caused by earthquakes and new technologies, products, systems, design approaches are regularly introduced to a broader community [4][5][6][7][8][9][10][11][12].
It is generally assumed that an earthquake cannot be predicted (although it can be expected) or simulated (although we can approximate it), and that its destructive effects cannot be prevented (although they can be minimized or, at worst, optimized). However, building our infrastructure with the basic state-of-the-art (STAR) principles in mind will provide us with the ability to keep it safe

The Assessment Methods for Condition Inspection and Identification of Seismic Vulnerability of Existing Masonry Buildings
Regarding economy and sustainability, it is of great significance for a society to maintain existing building stock instead demolish and rebuild them. The contribution of the assessment phase to the decision-making process is crucial for the correct assessment, verification, and maintenance of existing buildings [15][16][17].
The assessment of an existing structure can be done in various stages with increased accuracy, the level of accuracy, thereby depending on the quantity and the quality of available data as well as on the significance of the structure. The developments in the areas of inspection, nondestructive techniques (NDT), structural health monitoring (SHM), and structural analysis of existing structures, together with recent guidelines for reuse and conservation, allow for safer, economical and more adequate remedial measures [18].
The assessment techniques focus mainly on damage identification, damage localization, and damage evaluation, as well as the determination of certain material properties of existing structures. There are several new NDT and semi-NDT methods on the market, different in sophistication and purpose. As the focus of this research is mainly on masonry structures or combined masonry and concrete/timber structures, the most important NDT methods for existing masonry structures can be summarized: visual inspection [19], rebound hammer [20,21], measurement of reinforcement location [22], stress wave transmission [23], ultrasonic velocity testing [24], sonic velocity testing [25,26], surface penetrating radar [25,27], infrared thermography [28], flatjack tests [29,30], damage identification using impact vibrations [31], acoustic emission [32], etc. The basic advantages and disadvantages of the mentioned methods are described in Table 1. More detailed information can be found in [33].
The issue of seismic vulnerability assessment and restoration of underperforming existing buildings is a very significant and difficult problem [34]. To develop efficient assessment methods, it is necessary to draw knowledge and information from various other scientific fields. The seismic vulnerability requires specialized technical skills [35].
The preliminary work on seismic assessment was based only on visual inspection of the buildings [19,36], NDT [15,24,32,37,38], and the experience of an engineer. The last decade has seen a growth in the technological advancement of various instruments that can considerably help in the estimation of structural safety and seismic behavior of existing structures. The applicability of the thermal camera for building diagnostics and detection of energy-related building defects has been investigated by numerous authors [39][40][41][42][43][44]. The importance close-range remote sensing and using multispectral cameras for damage assessment and crack detection on buildings was emphasized by numerous researchers [45][46][47]. In the last few years, a growing number of studies have shown the great potential of hyperspectral cameras for detection and mapping cracks on buildings [48,49]. Unmanned aerial vehicles (UAV) can be used for i.e., disaster management [50,51], crack identification using image processing [47,52], seismic vulnerability [53], architectural assessment of heritage sites and structures [54], and structural assessment using image processing techniques [55]. UAV-based photogrammetry [56][57][58] and LiDAR (Light Detection and Ranging) [59][60][61] devices are indeed used after earthquakes to scan affected areas, but this method is mostly used for larger areas only. The advantages and disadvantages of the new technologies which should complement traditional methods are given in Table 2. Mentioned techniques are especially proven to be good at a post-earthquake evaluation of the complete urban/rural areas and buildings individually.

UAV Platform for Multi-Sensor Photogrammetry Aerial Mapping
Nowadays, technology and sensor minimization enable the development of the multi-sensor custom-made UAVs (unmanned aerial vehicles) for multi-sensor aerial mapping. As already mentioned, UAVs can be applied to acquiring spatial data about the condition of buildings [59], cultural heritage [56], but also for acquiring spatial data of larger areas for risk assessment of damage detection and mapping, e.g., after earthquakes [57,58]. These platforms are equipped with the newest sensors for in-flight position determination. These sensors include GNSS (Global Navigation Satellite System), RTK (real-time kinematic), and IMU (inertial measurement unit) systems that allow positioning in centimeter accuracy in real time. Today, significant importance in the development of the UAV platform was devoted to developing new technologies for stabilization and orientation of the multi-sensor system [62][63][64]. The UAV platforms for multi-sensor aerial mapping can be made based on the open-source or commercial flight controllers and technologies and are able to carry out mapping missions autonomously [65][66][67]. For high-accuracy photogrammetric measurement of the safety and seismic vulnerability of existing structures, the systems are equipped with sensors that measure the obstacle distance and therefore enable flights at a close distance to buildings (e.g., microradar and/or LiDAR). Furthermore, these systems are the foundation for fast and accurate collection of large amounts of spatial data for building damage detection and mapping. In Figure 1, one scenario of the post-earthquake assessment with a UAV is shown. An engineer cannot see what happened to a building at higher floors or roof of a structure when doing on-site assessment from the street. Very often, the main problems (especially for moderate intensity earthquakes) are chimneys, damaged roof structures, damaged gable walls, etc. With a UAV device, all of the mentioned information can be easily assessed.
sensors for in-flight position determination. These sensors include GNSS (Global Navigation Satellite System), RTK (real-time kinematic), and IMU (inertial measurement unit) systems that allow positioning in centimeter accuracy in real time. Today, significant importance in the development of the UAV platform was devoted to developing new technologies for stabilization and orientation of the multi-sensor system [62][63][64]. The UAV platforms for multi-sensor aerial mapping can be made based on the open-source or commercial flight controllers and technologies and are able to carry out mapping missions autonomously [65][66][67]. For high-accuracy photogrammetric measurement of the safety and seismic vulnerability of existing structures, the systems are equipped with sensors that measure the obstacle distance and therefore enable flights at a close distance to buildings (e.g., microradar and/or LiDAR). Furthermore, these systems are the foundation for fast and accurate collection of large amounts of spatial data for building damage detection and mapping. In Figure 1, one scenario of the post-earthquake assessment with a UAV is shown. An engineer cannot see what happened to a building at higher floors or roof of a structure when doing on-site assessment from the street. Very often, the main problems (especially for moderate intensity earthquakes) are chimneys, damaged roof structures, damaged gable walls, etc. With a UAV device, all of the mentioned information can be easily assessed.

Multi-Sensors System for Photogrammetry Data Acquisition
The aforementioned advances in technology and sensor minimization give us the opportunity to the development of unique, highly specialized multi-sensors system for photogrammetry data

Multi-Sensors System for Photogrammetry Data Acquisition
The aforementioned advances in technology and sensor minimization give us the opportunity to the development of unique, highly specialized multi-sensors system for photogrammetry data acquisition. Today, these systems can contain four independent types of cameras: RGB camera, thermal, multispectral, and hyperspectral camera. All these cameras complement each other in spectral, radiometric, and geometric characteristics. The new multi-sensor system is capable for the detection of damage and anomalies on a building that is not visible to the human eye. The importance of using various kinds of different cameras, from thermal to multispectral cameras for damage assessment and crack detection on buildings, was emphasized by numerous researchers [48,52]. Although all these cameras were already used for building diagnostics assessment, so far, a multi-sensor system has not been developed for this purpose. It should be noted that the multi-sensor system should be modular, and it should be able to use different camera lenses, depending on building size and a priori accuracy. The newly built multi-sensor system should enable accurate data acquisition for building damage assessment, and the system harness the potential of all implemented cameras.
After the design and development of the multi-sensor system, it is necessary to develop new methods for calibration, synchronization, and fusion of the collected data. For each camera, the development of the new calibration method based on photogrammetric methods must be obtained. The geometric calibration for internal camera parameters calculation must be done [68,69] and followed by the radiometric calibration for each camera [70][71][72][73]. Further, for the autonomous spatial data acquisition from all cameras in a multi-sensor system, the system synchronization is crucial [70,74,75]. Adequately synchronized multi-sensor system are able to autonomously and/or remotely capture images from all cameras at the same time. Once the data has been properly collected, it is necessary to research and develop new imagery fusion methods. The new fusion methods developed for the multi-sensor system with various independent cameras allow the collection of a fused image that contains the data of all the above sensors (visible part of the spectrum, thermal, multispectral and hyperspectral data). The fusion process increases the spatial resolution of final multi-band imagery. The usage of fused, high-resolution, multi-band imagery enables better and more accurate damage detection on buildings.

Remote Sensing Methods for Automatic Detection and Mapping of Weak Structural Parts and Components
After the multi-sensor system has collected the data, to speed up the process, the mapping of weak structural parts and components should be performed. Such methods allow for the rapid analysis of large amounts of spatial data, which cannot be processed manually. The current development of new methods is based primarily on the analysis of all available data of the multi-sensor system. Taking advantage of individual bands outside the visible spectrum, such as images captured by a thermal, multispectral, and hyperspectral camera, enable detection, highlighting, and mapping of features that are not visible to the human eye [45,46]. The use of fused imagery further enhances the spatial resolution of the imagery and enables the detection of even the smallest cracks in the object [76,77]. The development and use of these methods enable rapid detection and accurate mapping of the smallest cracks that may not be visible on the objects themselves. The application of developed methods significantly accelerates the process of gathering information on the structure and condition of existing buildings.

Photogrammetric Three-Dimensional (3D) Building Modeling Methods
Optimal photogrammetric methods need to be developed for an accurate and measurable high-resolution photorealistic 3D building model based on a multi-sensor system and GeoSLAM system [78]. Photogrammetric methods are primarily based on an adequate phototriangulation procedure for all images. During the process of phototriangulation, the self-calibration of cameras control the stability of their internal elements [71]. The development of methods for the rapid creation of 3D models include the test of sensor accuracy for determining the elements of the external orientation of real-time images (GNSS + RTK, IMU) [62,69]. Markers can be used on the objects to control the phototriangulation process and the final 3D model. Nowadays, indoor spatial data for 3D models are usually acquired with the GeoSLAM system. The final 3D model can be made by the combination of data acquired with the multi-sensor system and GeoSLAM system data. Furthermore, spectralon and other markers to perform radiometric calibration can be placed on the objects [71]. This ensures the accuracy of the spectral signature of the recorded objects. For controlling thermal images, the temperature of the object can be measured by a measuring device and sensors at the time of data collection. This enables obtaining an accurate 3D model in the thermal spectrum, i.e., the accuracy of the temperature obtained by the multi-sensor system. The development and implementation of methods for photogrammetric creation of SfM (Structure-from-motion) point clouds enable the creation of high-resolution, dense 3D point clouds [79][80][81]. The dense point clouds are filtered by newly developed outlier detection algorithms for gross error points detection (e.g., glass area) [82][83][84]. For quality and accurate point cloud collection on objects with uniform textures, methods for radiometric enhancement of textural features were developed. Radiometric equalization of photogrammetric images based on new methods allow for the creation of seamless photorealistic 3D models [85,86]. Further, with new 3D modeling methods, the point cloud can be transformed into a high-resolution photorealistic 3D building model (Figure 2).
Appl. Sci. 2020, 10, x FOR PEER REVIEW 8 of 16 data acquired with the multi-sensor system and GeoSLAM system data. Furthermore, spectralon and other markers to perform radiometric calibration can be placed on the objects [71]. This ensures the accuracy of the spectral signature of the recorded objects. For controlling thermal images, the temperature of the object can be measured by a measuring device and sensors at the time of data collection. This enables obtaining an accurate 3D model in the thermal spectrum, i.e., the accuracy of the temperature obtained by the multi-sensor system. The development and implementation of methods for photogrammetric creation of SfM (Structure-from-motion) point clouds enable the creation of high-resolution, dense 3D point clouds [79][80][81]. The dense point clouds are filtered by newly developed outlier detection algorithms for gross error points detection (e.g., glass area) [82][83][84]. For quality and accurate point cloud collection on objects with uniform textures, methods for radiometric enhancement of textural features were developed. Radiometric equalization of photogrammetric images based on new methods allow for the creation of seamless photorealistic 3D models [85,86]. Further, with new 3D modeling methods, the point cloud can be transformed into a high-resolution photorealistic 3D building model (Figure 2).

Automatic Methods for Building Damage Detection and Mapping
The photorealistic 3D models and products (e.g., from orthophoto) can be used for the detection and mapping of building damage. The developed methods are based on a combination of unsupervised and supervised image classification methods. Furthermore, the methods can be technologically developed on object-based machine learning algorithms [87][88][89][90]. Such an approach allows rapid damage detection on buildings, which is important because of the large amount of highresolution spatial data. Preliminary research related to the topic of developing automatic methods for the classification has been made [91]. Newly developed methods for the detection and mapping of building damage can use all bands of the multi-sensor system, but also spectral indices. Spectral indices allow further highlighting of certain phenomena on earth or objects [91][92][93]. The use of spectral indices has been shown to increase the accuracy of classifications. In some cases, spectral indices allow features to be highlighted on an object that cannot be detected on any single band in the visible or out of the visible part of the spectrum. In addition to the spectral indices, the newly developed methods also use the Haralick texture features (Gray Level Co-occurrence Matrix) to detect and emphasize the edge between damaged and undamaged parts on objects [94][95][96][97][98]. Using an object-based approach enables rapid and accurate mapping of damage. After the mapping, the system immediately assigns statistical indicators such as area, depth, etc. (Figure 3). The development

Automatic Methods for Building Damage Detection and Mapping
The photorealistic 3D models and products (e.g., from orthophoto) can be used for the detection and mapping of building damage. The developed methods are based on a combination of unsupervised and supervised image classification methods. Furthermore, the methods can be technologically developed on object-based machine learning algorithms [87][88][89][90]. Such an approach allows rapid damage detection on buildings, which is important because of the large amount of high-resolution spatial data. Preliminary research related to the topic of developing automatic methods for the classification has been made [91]. Newly developed methods for the detection and mapping of building damage can use all bands of the multi-sensor system, but also spectral indices. Spectral indices allow further highlighting of certain phenomena on earth or objects [91][92][93]. The use of spectral indices has been shown to increase the accuracy of classifications. In some cases, spectral indices allow features to be highlighted on an object that cannot be detected on any single band in the visible or out of the visible part of the spectrum. In addition to the spectral indices, the newly developed methods also use the Haralick texture features (Gray Level Co-occurrence Matrix) to detect and emphasize the edge between damaged and undamaged parts on objects [94][95][96][97][98]. Using an object-based approach enables rapid and accurate mapping of damage. After the mapping, the system immediately assigns statistical indicators such as area, depth, etc. (Figure 3). The development of a new automatic method for structural damage detection and mapping is a major scientific contribution in the field of photogrammetry and remote sensing applications in civil engineering.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 16 of a new automatic method for structural damage detection and mapping is a major scientific contribution in the field of photogrammetry and remote sensing applications in civil engineering.

Discussion and Conclusions
The knowledge of seismic behavior of structures has seen significant advancements in recent decades. Still, there is much left to be considered. We are still unfamiliar with a number of things that must be taken into account, such as the magnitude, direction, and scale of earthquakes themselves, as well as their stochastic (random) nature. Some fundamental decisions still need to be made, e.g., whether to opt for deterministic or probabilistic modeling, static or dynamic load modeling, linear or nonlinear seismic behavior of structures.
Are we able to minimize the effects of an earthquake, or is optimizing them still the only option? How much do we know about the seismic risk or the potential consequences of seismic activity on our infrastructure? How accurate are we in assessing the seismic vulnerability of existing buildings? Alas, most of these questions remain unanswered as more research needs to be done on seismic risk and seismic vulnerability assessment. Concentrating only on visible issues cannot resolve this questions.
Conventional approaches only allow us to assess the condition of an existing structure once its stability has already been compromised. They also only help us when we know from experience that a certain building is vulnerable to seismic activity. But what about any invisible parameters? Are we ready to appropriate new technologies in the engineering sector? Can the use of thermography and UAVs help us expand our abilities in seismic vulnerability assessment? Can any new methods point us to other parameters crucial to the development of this scientific field? How do we precisely characterize data gathered with the help of new technologies? Answering these questions may give way to new principles in the field and foster the use of new technologies of photogrammetry and close-range remote sensing much needed in such a traditional sector as civil engineering.
Insight in material and structural properties obtained by assessments and structural analyses and evaluation can help us is in modeling, analyzing, and predicting the performance of existing structures and is always a base for the preparation of the data for more precise analyses. However, it is not always possible to have a comprehensive on-site assessment for rapid evaluation of buildings.
The basic idea of this paper is to show the possibilities for the development of an easy and fast procedure for seismically generated damage detection of buildings with new technologies such as UAVs and optical inspections. The open questions and possible future trends in the assessment of the seismic vulnerability of the existing structures is also discussed. The seismic vulnerability of a structure is a quantity associated with its weakness in the case of earthquakes of a given intensity so

Discussion and Conclusions
The knowledge of seismic behavior of structures has seen significant advancements in recent decades. Still, there is much left to be considered. We are still unfamiliar with a number of things that must be taken into account, such as the magnitude, direction, and scale of earthquakes themselves, as well as their stochastic (random) nature. Some fundamental decisions still need to be made, e.g., whether to opt for deterministic or probabilistic modeling, static or dynamic load modeling, linear or nonlinear seismic behavior of structures.
Are we able to minimize the effects of an earthquake, or is optimizing them still the only option? How much do we know about the seismic risk or the potential consequences of seismic activity on our infrastructure? How accurate are we in assessing the seismic vulnerability of existing buildings? Alas, most of these questions remain unanswered as more research needs to be done on seismic risk and seismic vulnerability assessment. Concentrating only on visible issues cannot resolve this questions.
Conventional approaches only allow us to assess the condition of an existing structure once its stability has already been compromised. They also only help us when we know from experience that a certain building is vulnerable to seismic activity. But what about any invisible parameters? Are we ready to appropriate new technologies in the engineering sector? Can the use of thermography and UAVs help us expand our abilities in seismic vulnerability assessment? Can any new methods point us to other parameters crucial to the development of this scientific field? How do we precisely characterize data gathered with the help of new technologies? Answering these questions may give way to new principles in the field and foster the use of new technologies of photogrammetry and close-range remote sensing much needed in such a traditional sector as civil engineering.
Insight in material and structural properties obtained by assessments and structural analyses and evaluation can help us is in modeling, analyzing, and predicting the performance of existing structures and is always a base for the preparation of the data for more precise analyses. However, it is not always possible to have a comprehensive on-site assessment for rapid evaluation of buildings.
The basic idea of this paper is to show the possibilities for the development of an easy and fast procedure for seismically generated damage detection of buildings with new technologies such as UAVs and optical inspections. The open questions and possible future trends in the assessment of the seismic vulnerability of the existing structures is also discussed. The seismic vulnerability of a structure is a quantity associated with its weakness in the case of earthquakes of a given intensity so that the value of this quantity and the knowledge of seismic hazard allows us to evaluate the expected damage from future earthquakes [99]. In modern assessment methods, the seismic vulnerability is represented by vulnerability curves, considering, in individual cases only, some structural characteristics of the affected buildings [100]. Throughout history, a variety of methods have been used to assess the vulnerability that can be divided into empirical and analytical (both approaches can also be used in different hybrid methods) and an approach based on engineering judgment by experts. Methods of vulnerability assessments mainly model damage to a discrete scale of damage commonly using three to six categories [101], but still, there are no unified approaches on a European level. High-definition precision images of buildings using satellites, LiDARs, drones, Google Street View and the like can give us information on floor dimensions, height, floors, etc. It is important to emphasize that such information alone is not sufficient, and it needs to be processed by experts, but also supplemented by other data relevant for damage assessments (e.g., structural system, function, and usage).
A compilation of all the data can be obtained to get a unique view of the seismic vulnerability of existing structures. A combination of the data from "traditional" assessments with the results from modern techniques can help us in the identification of weak structural parts and components (faults, big cracks, and openings, short cantilevers, and walls, short columns, soft storeys). With the knowledge of the architectural background and styles of the city, a quick estimation of the period when the building was built can be obtained, i.e., in Croatia, tall windows can tell us that the building was built during the Austro-Hungarian times. This data can relate to construction methods, wooden ceilings, lack of glazing and confiners, greater wall thickness, different material quality, etc. All these important factors are influencing the seismic behavior of the existing structure. With measurement data of foundation settlings, soil stiffness can be assessed or may indicate some differential settling. In addition, the building stiffness can be estimated, which can lead to an estimation of the number of internal walls. Assessment of roofs with UAVs can identify the deformation of roof plates and can serve as a basis for the identification of internal walls, which can also lead to estimation of stiffness. With thermography, different building materials can be identified (masonry vs. concrete) and existing confining elements with their dimensions and location in the structures can be pointed out. With a combination of methods, and estimation of wall thicknesses (with basic geometry) quick interpretation of structural safety can be estimated. Deformation of outer walls obtained by photogrammetry combined with energy losses by thermography and architectural rules of the construction period, and with expert opinions can lead us to a rough estimation of the number of walls (considering the knowledge of load-bearing plate system). All of the mentioned activities can give us the level of seismic vulnerability and safety of the existing structures. The obtained data, new procedures, and techniques can help us in the estimation of seismic risks of bigger regions or building blocks (Figure 4).
A conceptual design of building blocks can be easily identified (weak parts, floors on different heights, and influence of one building to another), which gives the information of seismic risk.
The development of a modern procedure for structural damage detection and estimation of seismic risk and vulnerability must be improved with new and accessible technologies. A conceptual design of building blocks can be easily identified (weak parts, floors on different heights, and influence of one building to another), which gives the information of seismic risk.
The development of a modern procedure for structural damage detection and estimation of seismic risk and vulnerability must be improved with new and accessible technologies.