Case Study—Spiking Neural Network Hardware System for Structural Health Monitoring

This case study provides feasibility analysis of adapting Spiking Neural Networks (SNN) based Structural Health Monitoring (SHM) system to explore low-cost solution for inspection of structural health of damaged buildings which survived after natural disaster that is, earthquakes or similar activities. Various techniques are used to detect the structural health status of a building for performance benchmarking, including different feature extraction methods and classification techniques (e.g., SNN, K-means and artificial neural network etc.). The SNN is utilized to process the sensory data generated from full-scale seven-story reinforced concrete building to verify the classification performances. Results show that the proposed SNN hardware has high classification accuracy, reliability, longevity and low hardware area overhead.


Introduction
Earthquake is an oscillatory movement caused by the abrupt release of strain energy stored in the rocks within the crust of earth surface. Areas are always vulnerable to natural disasters, which can lead to extreme damages in nearby populations in terms of fatality, communication and infrastructure loss. Flood, earthquake, cyclones and so forth, are among the most common occurring natural disasters across the world. The impact of these disasters differs in different geological and geographic locations. These disasters come with no advance warning but an effective, well prepared and maintained infrastructure will decrease the potential impact of future disasters. The structural health of buildings and other infrastructure suffers degradation due to environmental catastrophes caused by ageing, hazards and natural disasters [1]. In any area, public infrastructures, like schools, hospitals, fire stations, administrative buildings, bridges and treatment plants, are more prone to being highly affected by these disasters. Therefore, regular structural health monitoring is required to ensure the heath and endurance of these mega structures. In the event of a disaster, it is particularly important (i) to detect and quantify the severity of damage caused by environmental disasters at an early stage; (ii) to assess the current structural health and reliability of buildings to ensure their safe use; and (iii) to estimate repair costs for damage to minimize economic losses [2]. Traditional monitoring methods rely on an inspection and assessment of the buildings and requires experienced inspectors. Many structures are not convenient for on-site monitoring due to the terrain obstacles, that is, the lack of access to such buildings, which sometimes make it too late due to the retrospective nature of

System Architecture
SHM is a multi-layered hardware system that is comprised of multiple sensors for data acquisition, communication and processing architecture to assess the health of structural integrity. Figure 1 shows the structure of the proposed SHM system. The system is equipped with wired or wireless sensors, such as accelerometers, to collect the data from under observation structure. Through the analysis of the raw data, appropriate features can be selected and extracted from the time domain or frequency domain. After feature extraction, the data is fed into the SNN hardware system for the structure damage level assessment. The SNN encodes the pre-processed data into input spiking signals. This work proposed two SNN models to explore an efficient and cost-effective solution for the SHM system. A fully connected SNN network based on Leaky Integrate and Fire (LIF) neurons with SpikeProp as a learning algorithm for feature classification. The second model is based on the Neucube framework [27] using the Spike Timing Dependent Plasticity (STDP) rule for the unsupervised training and deSNN [28] algorithm for supervised learning. Both models can classify the level of structural damage to identify structural health status.
SNNs use time as an input dimension and record valuable information in a spatial domain. The information received by the spiking neuron is a pulsed time series, so the analogue sensory data needs to be encoded into the spatial dimension for input to the spiking neural network. The spiking neuron

Feature Extraction
Considering different sensors used in the structure, the selection of damage-sensitive features is generally based on multiple tests, so as to determine which features can indicate the health state of the structure accurately and are robust to the influence of the structural conditions and environments. These features can be extracted from the time domain (e.g., mean, variance, peak to peak amplitude, Zero crossing rate, energy, maximum amplitude, etc.) and frequency domain such as Fourier transform. Mean, variance and zero crossing rate are defined as: where a is the input sensor data, N is the number of the samples. After feature extraction, supervised or unsupervised learning methods can be used for data analysis and structure health status classification.

Structure Damage Classification
Temporal coding schemes, such as Address Event Representation (AER), Bens Spike Algorithm (BSA) and Step Forward (SF), are used to represent information as an input to SNNs. Figure 2 shows different encoding results for the same temporal input data. The spike trains will carry the key information of the original signals. Different spike encoding algorithms have distinct characteristics

Feature Extraction
Considering different sensors used in the structure, the selection of damage-sensitive features is generally based on multiple tests, so as to determine which features can indicate the health state of the structure accurately and are robust to the influence of the structural conditions and environments. These features can be extracted from the time domain (e.g., mean, variance, peak to peak amplitude, Zero crossing rate, energy, maximum amplitude, etc.) and frequency domain such as Fourier transform. Mean, variance and zero crossing rate are defined as: where a is the input sensor data, N is the number of the samples. After feature extraction, supervised or unsupervised learning methods can be used for data analysis and structure health status classification.

Structure Damage Classification
Temporal coding schemes, such as Address Event Representation (AER), Bens Spike Algorithm (BSA) and Step Forward (SF), are used to represent information as an input to SNNs. Figure 2 shows different encoding results for the same temporal input data. The spike trains will carry the key information of the original signals. Different spike encoding algorithms have distinct characteristics Sensors 2020, 20, 5126 5 of 14 when representing input data. BSA, shown in Figure 2c, is suitable for high frequency signals, so there are few spikes encoded from the low frequency signals, while AER and SF are better to represent the signal intensity.
Different spiking neuron models can be used to model spike generations at different description levels of biology [9], such as leaky integrate-and-fire (LIF), Izhikevich and Hodgkin-Huxley. The LIF neuron is one of the simplified models, which can be modelled as: where V mem is the membrane potential of the neuron, I ext is the external driving current, τ m is the membrane time constant, R is the input resistance and V eq is the equilibrium potential of the leakage conductance.
Sensors 2020, 20, x FOR PEER REVIEW 5 of 14 when representing input data. BSA, shown in Figure 2c, is suitable for high frequency signals, so there are few spikes encoded from the low frequency signals, while AER and SF are better to represent the signal intensity. Different spiking neuron models can be used to model spike generations at different description levels of biology [9], such as leaky integrate-and-fire (LIF), Izhikevich and Hodgkin-Huxley. The LIF neuron is one of the simplified models, which can be modelled as: where is the membrane potential of the neuron, is the external driving current, is the membrane time constant, R is the input resistance and is the equilibrium potential of the leakage conductance.  Figure 3 shows the state of the neuron updated by the membrane potential under the synaptic stimuli. When the membrane potential of the neuron crosses the threshold, the neuron then generates an output spike, which acts as an input stimulus for subsequent layer neurons.  Figure 3 shows the state of the neuron updated by the membrane potential under the synaptic stimuli. When the membrane potential of the neuron crosses the threshold, the neuron then generates an output spike, which acts as an input stimulus for subsequent layer neurons.
SNN can be trained using unsupervised and supervised approaches. An unsupervised SNN, using the Spike Timing Dependent Plasticity (STDP) learning rule, was demonstrated with competitive accuracy [31]. The weight update in the STDP learning rule [32] can be described as: where ∆w is the weight change rate, τ + and τ − are STDP time constants, α + (> 0) and α − (< 0) are constant coefficients and ∆t is the time difference between a post-neuron and a pre-neuron spike. When ∆t ≥ 0, the synaptic plasticity is a long-term potentiation (LTP) process; otherwise it is a long-term depression process. Two different SNN structures are adopted in this study, where one is a fully connected SNN and the other one is a model based on NeuCube [27].
For performance comparisons, the commonly used classification algorithms of K-means and ANNs are also used in this work for benchmarking. A supervised learning algorithm of ANN is used in this work, where the network weights are adjusted in every iteration by comparing the difference between actual output and the targeted output. A multi-layer feedforward architecture with input layer for sensory input, hidden layer for learning and an output layer to generate spiking output. The number of input neurons equals the number of sensors, whereas the output layer neurons represent the number of structure level classifiers. For K-means, the unsupervised K-means algorithm for SHM can be described with the following steps, where k is the number of desired clusters-(a) Given the features' matrix as an input, find the k centroids (random or select); (b) Calculate the distances between features' vectors and centroids; (c) Group the features' vectors based on their intra-cluster distance; and (d) Iterate the algorithm and update the centroids for a better clustering result. SNN can be trained using unsupervised and supervised approaches. An unsupervised SNN, using the Spike Timing Dependent Plasticity (STDP) learning rule, was demonstrated with competitive accuracy [31]. The weight update in the STDP learning rule [32] can be described as: where ∆ is the weight change rate, and are STDP time constants, ( 0) and are constant coefficients and ∆ is the time difference between a post-neuron and a pre-neuron spike. When ∆ 0, the synaptic plasticity is a long-term potentiation (LTP) process; otherwise it is a longterm depression process. Two different SNN structures are adopted in this study, where one is a fully connected SNN and the other one is a model based on NeuCube [27]. For performance comparisons, the commonly used classification algorithms of K-means and ANNs are also used in this work for benchmarking. A supervised learning algorithm of ANN is used in this work, where the network weights are adjusted in every iteration by comparing the difference between actual output and the targeted output. A multi-layer feedforward architecture with input layer for sensory input, hidden layer for learning and an output layer to generate spiking output. The number of input neurons equals the number of sensors, whereas the output layer neurons represent the number of structure level classifiers. For K-means, the unsupervised K-means algorithm for SHM can be described with the following steps, where k is the number of desired clusters-(a) Given the features' matrix as an input, find the k centroids (random or select); (b) Calculate the distances between features' vectors and centroids; (c) Group the features' vectors based on their intra-cluster distance; and (d) Iterate the algorithm and update the centroids for a better clustering result.

Experiments
This section explains experimental setup to generate damage level report for SHM system. Furthermore, this case study analyses and compares the results of three classification methods, K-means, ANN and SNN to identify the best performing SHM system.

Dataset
This case study used a full-scale seven-story reinforced concrete building dataset for experimentation [1]. The building is installed with 45 accelerometers operating at a sampling rate of 240 Hz. A sequence of dynamic tests was applied to the building over several months, including ambient vibration tests, free vibration tests and forced vibration tests using the shake table of Network for Earthquake Engineering Simulation at University of California, San Diego (UCSD-NEES). A 0.03 g root-mean-square (RMS) acceleration white noise base excitation and ambient vibration tests were performed on the structure before and between earthquake shake-table tests. For 45 channels, the signal to noise ratios (SNR) are −36.97 db~22.81 db. The building was damaged progressively through several historical earthquake ground motions and damage states of the building can be described as shown in Table 1. In 1st to 3rd earthquakes, the roof drift ratio, defined as the ratio between the maximum lateral displacement at the roof level of the building and the height of the roof relative to the base of the building, was measured as 0.28%, 0.75% and 0.83%, respectively. The maximum tensile strain in the longitudinal reinforcing steel was measured close to the base of the wall as 0.61%, 1.73% and 1.78%, respectively [1].

Feature Extraction
The raw data collected from 45 channels in the building at different health states are shown in Figure 4. Raw accelerometer data of different structure states show different features, such as maximum amplitude and mean value and so forth. By considering the building's physical movements in different states [33], the deformation degree of buildings can result in large differences in the mean and fluctuation range of the accelerometer data. Based on this analysis, zero-crossing rate, mean and variance are used for feature extractions. NEES). A 0.03 g root-mean-square (RMS) acceleration white noise base excitation and ambient vibration tests were performed on the structure before and between earthquake shake-table tests. For 45 channels, the signal to noise ratios (SNR) are −36.97 db~22.81 db. The building was damaged progressively through several historical earthquake ground motions and damage states of the building can be described as shown in Table 1. In 1st to 3rd earthquakes, the roof drift ratio, defined as the ratio between the maximum lateral displacement at the roof level of the building and the height of the roof relative to the base of the building, was measured as 0.28%, 0.75% and 0.83%, respectively. The maximum tensile strain in the longitudinal reinforcing steel was measured close to the base of the wall as 0.61%, 1.73% and 1.78%, respectively [1].

Feature Extraction
The raw data collected from 45 channels in the building at different health states are shown in Figure 4. Raw accelerometer data of different structure states show different features, such as maximum amplitude and mean value and so forth. By considering the building's physical movements in different states [33], the deformation degree of buildings can result in large differences in the mean and fluctuation range of the accelerometer data. Based on this analysis, zero-crossing rate, mean and variance are used for feature extractions.  After the data have been pre-processed, three methods (including zero-crossing rate, variance and mean value) are used to extract data in order to select the damage-sensitive features. The features are presented in Figure 5. The zero-crossing rate, which is the rate of sign-changes along a signal, is too weak to separate the different damage states (indicated by colors). Among them, calculating the mean value of the sensor data has the potential to differentiate the four damage states.
After the data have been pre-processed, three methods (including zero-crossing rate, variance and mean value) are used to extract data in order to select the damage-sensitive features. The features are presented in Figure 5. The zero-crossing rate, which is the rate of sign-changes along a signal, is too weak to separate the different damage states (indicated by colors). Among them, calculating the mean value of the sensor data has the potential to differentiate the four damage states.

SHM Classification Results
For different classification methods, 70~80% samples (including mean samples and raw data) are used for training and the rest for validation and testing.

K-Means
A 50 step-length sliding window with 100 sample points is used to get more mean samples, which are used as an input for the k-means algorithm. K-means parameters are shown in Table 2. It can be seen from Figure 6a that using the mean value of the data as an input of the k-means algorithm can classify the health status of the building. The dots represent historical records and the circles represent new data inputs. The classification accuracy of the structural health status is 100%. In Figure 6b, the raw data are used directly as the input of the k-means algorithm. In the case of overlapped data, including State-0, State-1 and State-3, the k-means algorithm cannot separate these data. There are 45 channels in total and only two of them are used for the demonstration in Figure 6.

SHM Classification Results
For different classification methods, 70~80% samples (including mean samples and raw data) are used for training and the rest for validation and testing.

K-Means
A 50 step-length sliding window with 100 sample points is used to get more mean samples, which are used as an input for the k-means algorithm. K-means parameters are shown in Table 2. It can be seen from Figure 6a that using the mean value of the data as an input of the k-means algorithm can classify the health status of the building. The dots represent historical records and the circles represent new data inputs. The classification accuracy of the structural health status is 100%. In Figure 6b, the raw data are used directly as the input of the k-means algorithm. In the case of overlapped data, including State-0, State-1 and State-3, the k-means algorithm cannot separate these data. There are 45 channels in total and only two of them are used for the demonstration in Figure 6. By incorporating the hardware design process [34] to implement K-means, the input data dimension area will be about 3.46 mm 2 and 1.23 mm 2 for the parallel mode and multiplexed architecture, respectively.

ANN
The ANN with 45 input neurons, 20 hidden neurons and 4 output neurons can get similar accuracy with different input samples (mean samples and raw data). Table 3 shows that ANN slightly confuse between State-0 and State-1 when trained on raw data samples. The hardware area of the neuron is estimated as about 1.347 mm 2 based on a 45 nm CMOS technology [35]. It can also be calculated from Reference [36] that the total hardware area of the ANN is >0.798 mm 2 . In NeuCube, raw data samples are fed into a dynamic SNN. One channel of an input sample was shown in Figure 2a. Table 4 shows network parameters used by NeuCube. The model is established with 45 input neurons, 50 hidden neurons and output neurons (the number of samples). Due to the dynamic structure, the overall area overhead of NeuCube SNN is about 4.655 × 10 −3 mm 2 is calculated according to the neuronal and synaptic hardware area estimation proposed in References [37,38]. Results shows that overall classification accuracy of NeuCube SNN is 98.9% (as shown in Figure 7). By incorporating the hardware design process [34] to implement K-means, the input data dimension area will be about 3.46 mm 2 and 1.23 mm 2 for the parallel mode and multiplexed architecture, respectively.

ANN
The ANN with 45 input neurons, 20 hidden neurons and 4 output neurons can get similar accuracy with different input samples (mean samples and raw data). Table 3 shows that ANN slightly confuse between State-0 and State-1 when trained on raw data samples. The hardware area of the neuron is estimated as about 1.347 mm 2 based on a 45 nm CMOS technology [35]. It can also be calculated from Reference [36] that the total hardware area of the ANN is >0.798 mm 2 .

NeuCube
In NeuCube, raw data samples are fed into a dynamic SNN. One channel of an input sample was shown in Figure 2a. Table 4 shows network parameters used by NeuCube. The model is established with 45 input neurons, 50 hidden neurons and output neurons (the number of samples). Due to the dynamic structure, the overall area overhead of NeuCube SNN is about 4.655 × 10 −3 mm 2 is calculated according to the neuronal and synaptic hardware area estimation proposed in References [37,38]. Results shows that overall classification accuracy of NeuCube SNN is 98.9% (as shown in Figure 7).  The weight is calculated as a modulation factor (the variable mod) to the power of the order of the incoming spikes. 0.55-0.6 Drift Initial connection weights are further modified to reflect the following spikes, using a drift parameter. 0.015 Figure 7. Classification result by using NeuCube (raw data). Table 5 shows the breakdown of performance accuracy for the classification of damage states observed by NeuCube. Enough samples will contribute to a higher probability of making the correct decision about the damage states. As a comparison, mean samples are input into NeuCube with the same parameter settings above. The accuracy is not as stable as raw data input, as NeuCube is more sensitive to temporal raw data [39].  [40]. The three-layered fully connected SNN is designed and modelled in MATLAB. Table 6 shows network topology, size and hardware area of the LIF based SNN model. Mean sensory samples are fed through 45 spiking input neurons to propagate spike towards 10 hidden neurons in order to generate 4 state output at 1 output neuron. The estimated hardware area of the SNN chip shown in Table 6 is calculated using References [37,38].  Table 5 shows the breakdown of performance accuracy for the classification of damage states observed by NeuCube. Enough samples will contribute to a higher probability of making the correct decision about the damage states. As a comparison, mean samples are input into NeuCube with the same parameter settings above. The accuracy is not as stable as raw data input, as NeuCube is more sensitive to temporal raw data [39].  [40]. The three-layered fully connected SNN is designed and modelled in MATLAB. Table 6 shows network topology, size and hardware area of the LIF based SNN model. Mean sensory samples are fed through 45 spiking input neurons to propagate spike towards 10 hidden neurons in order to generate 4 state output at 1 output neuron. The estimated hardware area of the SNN chip shown in Table 6 is calculated using References [37,38]. Damage states are encoded with the time of spike of output neuron (SNN output). The experimentation results show the classification accuracy using the mean samples input. The results show in Table 7 that the proposed customized SNN classifies the structural damage with 99.18% accuracy for the mean dataset. Moreover, the overall accuracy can be higher, up to 99.46%, by increasing the number of iterations, as compared to the 98.9% NeuCube average accuracy for raw sensory input.

Discussions
A summary of results using K-means, ANN and SNN in SHM applications, is shown in Table 8. ANN used raw data and feature samples as an input and there is little difference in classification accuracy. The final decision making can be the same within a certain confidence interval. Thus, if ANN combines the feature extraction into the learning process, it improves the computing speed and also reduces the hardware consumption. The structural damage occurrence detection can be assessed as health (State-0) and damage (State-1, State-2 & State-3), then the sensitivity (true positive rate) and specificity (true negative rate) of three typical methods can be obtained with the input of raw data samples, as shown in Table 9. Compared with the other two algorithms, SNN can accurately determine whether the structure is healthy. Meanwhile, the hardware area consumption of SNN is much less than ANN, the classification accuracy has a little difference of 0.9% and the sensitivity and specificity are higher. In summary, the proposed method based on SNNs apparently achieves a good trade-off between classification, reliability and hardware resource consumption.

Conclusions
The structural health state detection in this study involves the feature extraction from periodic observation measurements of a structure, where these features are analyzed to determine the current health state of the structure. Based on the detected states, the appropriate repair and strengthening of structures can keep the structure operational and longeval. Through the analysis of ZCR, Mean and Variance of the raw sensor data, it is found by experiments that the mean value is more sensitive to the structure state. Therefore, mean values and raw data were used as inputs and several classification methods, including K-means, conventional ANN and SNN, were used to detect the health state of the structure. Analysis and comparison results show that the SNN algorithm proposed in this study has advantages including (a) High classification accuracy can be obtained by directly using the raw data as input without manual feature extraction; (b) The small part of misclassification (1.92%) only exists in State-3, where the output health states can be clearly distinguished; (c) The hardware area of SNN is lower compared to ANN or K-means. In summary, the proposed SNN hardware solution for SHM has a stronger survivability and reliability than conventional approaches. Further work will further optimize the SNN for SHM systems in two respects including (a) to develop multi-layer (deep) SNNs to improve the accuracy and (b) to further analyze the sensor data to enhance the system functionalities, such as reporting the location of damage or life forecast of the structure.