Real-Time Monitoring of Parameters and Diagnostics of the Technical Condition of Small Unmanned Aerial Vehicle’s (UAV) Units Based on Deep BiGRU-CNN Models

Abstract

The paper describes an original technique for the real-time monitoring of parameters and technical diagnostics of small unmanned aerial vehicle (UAV) units using neural network models with the proposed CompactNeuroUAV architecture. As input data, the operation parameter values for a certain period preceding the current and actual control actions on the UAV actuators are used. A reference parameter set model is trained based on historical data. CompactNeuroUAV is a combined neural network consisting of convolutional layers to compact data and recurrent layers with gated recurrent units to encode the time dependence of parameters. Processing provides the expected parameter value and estimates the deviation of the actual value of the parameter or a set of parameters from the reference model. Faults that have led to the deviation threshold crossing are then classified. A smart classifier is used here to detect the failed UAV unit and the fault or pre-failure condition cause and type. The paper also provides the results of experimental validation of the proposed approach to diagnosing faults and pre-failure conditions of fixed-wing type UAVs for the ALFA dataset. Models have been built to detect conditions such as engine thrust loss, full left or right rudder fault, elevator fault in a horizontal position, loss of control over left, right, or both ailerons in a horizontal position, loss of control over the rudder and ailerons stuck in a horizontal position. The results of estimating the developed model accuracy on a test dataset are also provided.

1. Introduction

The paper proposes an original technique for the control systems of small unmanned aerial vehicles to monitor parameters and diagnose the technical condition of their units. This technique can be used as part of both real-time control systems directly on board a small unmanned aerial vehicle (UAV) and remote ones generating a control action via telemetry channels.

A technique for the technical monitoring and diagnosis of UAV’s onboard systems with decision-making support and a set of on-flight state checking equipment with a smart decision-making support system that allows to implement it are currently known [1]. This technique involves classifying the values of critical monitoring parameters characterizing the UAV onboard the system’s critical components according to any feature. Thereat, qualitative estimates are assigned to each parameter to identify pre-failure conditions of the UAV’s onboard system (OBS) and perform proactive diagnostics. Optimal diagnostic sequences are also formed as ranked lists of three-level precedents. The technique uses a knowledge base engine, a precedent library, and a Bayesian belief network for troubleshooting; however, it supposes checking the UAV tool kit-telemetry system while generating recommendations for the decision maker. Herewith, this technique does not allow using the diagnostic results by the UAV’s onboard control system to generate control actions.

Other known techniques for troubleshooting UAVs, in particular, smart aircraft troubleshooting and maintenance systems described, respectively, in [2,3], have a similar disadvantage—providing possible aircraft fault diagnoses without the possibility of autonomously using the diagnostic result on board the UAV. Note that fairly many studies are currently underway on the application of machine learning technologies using compact neural processing units (NPUs) on board small UAVs to solve image recognition and object identification problems [4,5,6,7,8,9], as part of control systems to implement sliding mode control in the fixed-wing type UAV flights [10], compensate Gated Recurrent Unit (GRU) network-based navigation system errors [11], land on a moving target [12], plan UAV formation flights [13], etc.

Many studies are devoted to diagnosing fixed-wing type UAV sensors and actuators, e.g., in Ref. [14], to estimate the fixed-wing type UAV’s inertial measurement unit condition, a nonlinear proportional integral unknown input observer-based system is implemented, and in Ref. [15], a wavelet transform-based approach is proposed. In Ref. [16], an RBF (radial basis function) neural network-based smart system is implemented to detect faults of the UAV’s gyroscope and accelerometer sensors. Paper [17] describes a technique for diagnosing a small UAV’s angular velocity sensor based on the support vector machine combined with the principal component analysis to detect faults.

Some studies are devoted to monitoring the UAV’s rotor condition: in Ref. [18], a duplicating simulation-based decentralized system is implemented for a group of UAVs; in Ref. [19], a linear parameter varying proportional integral unknown input observer-based monitoring system is used.

A Bayesian classifier-based UAV aileron condition monitoring using the emergency landing system is described in Ref. [20].

In Ref. [21], a fault-tolerant UAV control system is implemented to maintain the planned trajectory based on the random forest technique.

The drawback of most studies devoted to the monitoring of the UAV unit conditions is diagnosing the UAV condition at a specific time instant without considering the factor of the cumulative fault effect, such as a delay in the reaction time of actuators or changes in parameters in the early stages of a malfunction, or a change in the smoothness of the reaction to a jump.

Currently, there are concepts of fault-tolerant control over not only a single UAV but also a UAV formation, e.g., Ref. [22] describes the concept of a fault-tolerant formation for the case of loss of any UAV in it. When any UAV is lost, the objective function is corrected, which allows for maintaining the entire group’s performance. The assumption of a working diagnostic module capable that can timely detect faults is among the significant drawbacks of this concept. Herewith, it also has problems associated with the possible collision with other UAVs in the group and cascade destruction of the formation when any UAV is lost for the group of UAVs located at different heights or close to each other. Such a problem is solved by implementing a predictive diagnostic module. Thus, developing a diagnostic module is required, which will timely predict the UAV fault to further adaptively reconfigure the entire group. A decentralized formation adaptive reconfiguration technique capable of successfully using the diagnostic module concept described therein is given in Ref. [23]. To implement such diagnostic models and estimate their diagnostic ability, an experimental dataset is required that contains data on the UAV performance in both normal flight mode and the fault tolerant-mode.

Thus, the novelty of the diagnostic and monitoring technique herein is implementing a neural network model of original architecture to monitor the UAV parameters and determine the probability of faults and pre-failure conditions, consisting of convolutional layers to create a compact aggregated representation of parameters and bidirectional GRU layers to consider the cumulative fault effect and the actuator response inertia.

2. Description of the Proposed Diagnostic and Monitoring Technique

The objective of the invention proposed herein is to expand the opportunities for monitoring technical condition parameters of small UAV units and diagnosing them during operation.

The technical result comprises identifying deviations of the small UAV system parameters from the expected values, the failed system, the fault cause and type, and the pre-failure conditions.

The technique proposed herein involves the following: the reference model of the original CompactNeuroUAV neural network architecture generates the expected parameter value based on the operation parameter values for a certain period preceding the current and actual commands to the UAV actuators and defines the deviation of the actual value of individual parameters or their combination from the reference model. Faults that have led to the deviation threshold crossing are then classified. A smart classifier is here used to identify the failed system and the fault cause and type. The relationship of parameters in the CompactNeuroUAV neural network architecture is based on a combined neural network consisting of convolutional layers (CNN, Convolutional Neural Network) [24,25,26] to compact data and bidirectional recurrent layers with controlled units (GRU, Gate Recurrent Unit) [26,27,28] to encode the time dependence of the parameters.

Figure 1 provides the graphical diagram of a single GRU unit within the proposed architecture, processing the parameter vector values according to the system of equations:

$$\begin{array}{cc}\hfill z_t& ={\sigma }_g(W_z{x}_t+U_z{h}_{t-1}+b_z)\hfill \\ \hfill r_t& ={\sigma }_g(W_r{x}_t+U_r{h}_{t-1}+b_r)\hfill \\ \hfill h_t& =z_t\circ h_{t-1}+(1-z_t)\circ {\sigma }_h(W_h{x}_t+U_h(r_t\circ h_{t-1})+b_h)\hfill \end{array}$$

(1)

Here, $x_t$ is an input vector, $h_t$ is an output vector, $z_t$ is an update gate vector, $r_t$ is a reset gate vector, W, U, and b are parametric matrices and a vector.

Figure 2 shows a bidirectional network consisting of GRUs designed to encode a vector of values obtained at discrete time instants.

Processing UAV parameter inputs by CNN allows for forming a generalized abstract feature for several parameters. Sets of bidimensional time matrices form three-dimensional matrix units, and then, convolution is applied to them. The convolution result is a highly abstract feature, and after the convolution, its results are processed by a separate layer with subsampling. Figure 3 shows a diagram of a CNN for data processing.

Figure 4 shows the general diagram of the proposed CompactNeuroUAV architecture.

The diagnostic model is built by processing a previously collected database containing parameters recorded by the onboard control system and faults that have occurred during UAV flights. The parameter relationship reference model is built by the maximum sampling frequency of the recorded parameters with the normalization of values within the range [0, 1] and further defining the missing time series values by linear interpolation. The parameter relationship reference model is then fixed by training neural networks using the obtained dataset. To optimize the number of computational operations required to detect deviations in parameters and identify pre-failure conditions, subsampling of the recorded CNN data is used.

To fix the time dependence of parameter values and estimate the cumulative fault nature, the GRU layer is used with the number of units defined in the course of training the reference model by the number of discrete values in the time interval. Thereat, the time range’s lower interval is limited by the value corresponding to the minimum sampling frequency of the UAV parameters chosen to build the model.

The reference model output values are:

• Estimated deviation of the expected single parameter value, specifying the deviation magnitude calculated by reducing the normalized parameter value to its measurement range;
• Estimated deviation of the set of parameters, specifying the deviation as a percentage of the normalized value of the expected set of parameters;
• The fault classifier output value, which is one of the possible types, specifying the estimated probability of this fault class.

Thus, the proposed technique for the real-time monitoring of parameters and diagnostics of the technical condition of the unmanned aerial vehicle units is implemented as a sequence of the following operations:

• Preparing the training data—the use of archived data on the UAV unit parameters and faults during flights with normalization and reduction to the maximum processing frequency.
• Training the neural network with the proposed CompactNeuroUAV architecture with the generation of the required target value:
  • Estimating the individual parameter deviation from the expected value (Algorithm 1 with $SensorCount=1$);
  • Estimating the deviation of the set of parameters (Algorithm 1 with $SensorCount>1$);
  • Estimating the availability and probability of a fault or pre-failure condition (Algorithm 2).
• Validating the developed models.
• Integrating the developed models into the UAV control system.

Algorithm 1 Algorithm for estimating the deviation of any UAV parameter or a set of parameters from the expected reference as a neural network model with the CompactNeuroUAV Architecture

Require:$N\ge 1$ Require:$SensorCount\ge 1$   fori = 1 to N do        ▹ For each deviation estimation model       P[i] ← GetRealtimeSensorValues(SensorCount, i)       Pnorm[i] ← Normalization(P[i])       e ← EstimateDeviationCompactNeuroUAV[i](Pnorm[i])       if $N>MaxDeviation\left[i\right]$ then           $Alarm\leftarrow true$       else           $Alarm\leftarrow false$       end if       WriteToLog(e, Pnormi)   end for

The developed models are integrated into the UAV control loop as follows:

• At each $t_i$ time instant, a set of input values of normalized parameter-time matrices is generated, where the timestamp is within the range $t_{i-N}$, …, $t_{i-1}$;
• This set of normalized parameters is processed by the CompactNeuroUAV neural network; as a result, for each time instant, an actual compact aggregate representation of the parameter sets is generated in real-time;
• Deviations of the compact aggregated representation from the expected reference are estimated; the used model is marked as EstimateDeviationCompactNeuroUAV[i] in Algorithm 1;
• In the presence of a pre-trained classifier, a compact aggregated representation is used to estimate the presence of a certain-class fault or a pre-failure condition as ClassifierCompactNeuroUAV[i] in Algorithm 2.

Algorithm 2 Algorithm for identifying the UAV faults or pre-failure conditions based on neural network models with the CompactNeuroUAV architecture

Require:$N\ge 1$ Require:$SensorCount\ge 1$   for i = 1 to N do        ▹ For each deviation estimation model       P[i] ← GetRealtimeSensorValues(SensorCount, i)       for j = 1 to M do                ▹ For each sensor           if $P\left[i\right]\left[j\right]>MaxSensorValue\left[i\right]$ or $P\left[i\right]\left[j\right]<MinSensorValue\left[i\right]$ then              return Identifying fault           end if       end for       Pnorm[i] ← Normalization(P[i])       classes ← ClassifierCompactNeuroUAV[i](Pnormi)       WriteToLog(classes, Pnormi)   end for

The integration of the developed models into the control system of a single UAV makes it possible to implement a program for notifying the operator or for automatic safe landing of the aircraft when pre-fault conditions are detected in real time, as well as a controlled exclusion from the formation of a group flight of the UAV upon detection of pre-failure conditions of the aircraft.

The main disadvantage of integrating such models into the control system is the need for the presence of specialized neural processing units (NPU) on board the UAV for the effective operation of the models. As a result, when designing a UAV, it is necessary to take into account the additional weight and energy costs required for the operation of such neural accelerators (NPUs).

3. An Example of Implementing the Technique for a Fixed-Wing Type UAV

The proposed technique has been estimated on a simulation model using an experimental dataset [29,30] obtained for the Carbon Z T-28 fixed-wing type UAV consisting of the following:

• Two-meter wingspan;
• Front engine;
• Ailerons;
• Flaperons (flap ailerons);
• Elevator;
• Rudder.

To simulate faults in various UAV units, a modified control system was used to collect experimental flight data, which allows for the implementation of the following types of faults for an aircraft flying in autopilot mode:

i

Complete engine fault (shutdown);

ii

Elevator fault in a horizontal position;

iii

Rudder fault in the full right, full left, or middle position;

iv

Fault of the ailerons with the simulation of right, left, or both ailerons stuck in a horizontal position.

The ALFA dataset, which was used as part of the research work, presents flight data lasting from 26 to 233.4 s. To simulate errors, the authors of this dataset introduced an additional UAV control system in such a way that, at the command of the operator, the engine was turned off or the passage of the control command to the actuators was disabled.

Experimental flight dataset files are tagged to contain only autopilot flight data and include valid fault data only if any simulated fault actually occurs. Each file contains data on flights with one fault only. The UAV position GPS data, actual and target roll, pitch, speed, and yaw values measured by sensors and set by autopilot, the local unit conditions, and the overall UAV state estimate are used as the parameters monitored to estimate the technical UAV unit conditions.

Table 1. Fixed-wing type UAV’s simulated fault types.

Fault Type	Number of Experimental Flights	Flight Duration Before Fault (s)	Flight Duration with Fault (s)
No fault	10	558	-
Engine thrust loss	23	2282	362
Full left rudder fault (rudder control loss)	1	60	9
Full right rudder fault (rudder control loss)	2	107	32
Elevator fault in a horizontal position (loss of elevator control)	2	181	23
Left aileron control loss in a horizontal position	3	228	183
Right aileron control loss in a horizontal position	4	442	231
Loss of control over both ailerons in a horizontal position	1	66	36
Loss of control over the rudder and ailerons stuck in a horizontal position	1	116	27

provides a general summary list of the simulated types of faults and the UAV flight duration before the fault and after its simulation.

To implement the proposed technique for the real-time monitoring of parameters and diagnostics of technical conditions of the unmanned aerial vehicle units, the following sequence of actions was performed:

i

Labeling the experimental time series data:

(a)

Experimental flight data are divided into time intervals of 1 second; for signals with a sampling frequency less than the maximum 25 Hz, the dataset is completed by linear interpolation;

(b)

A label characterizing the presence of a fault is assigned to each interval from paragraph 1.a: flight without a fault, flight with a fault, transition interval for the fault manifestation. Transition intervals are chosen with the fault simulation start moments falling on different interval regions;

(c)

The resulting set of intervals is split into training (40%), cross-validation (10%), and test (50%) sets, considering the interval type (Table 1) and fault label (paragraph 1.b).

ii

The time series obtained at stage 1 is normalized by the measured parameter range for the UAV position GPS data and the actual and target roll, pitch, speed, and yaw values measured by the sensors and set by the autopilot.

iii

Deep neural network models with the proposed CompactNeuroUAV architecture are trained using training and cross-validation sets:

(a)

To estimate the deviation of a single pitch, roll, or yaw parameter from the expected reference considering the actuator response inertia;

(b)

To estimate the deviation of the set of roll, pitch, speed, and yaw parameters from the expected reference;

(c)

Classifier model to estimate the presence of a fault or pre-failure condition, specifying the estimated probability of this fault class

iv

The trained models are tested on a test set.

For the implementation and testing of the algorithms, the environment was used corresponding to that used in the collection of experimental data for the ALFA dataset [30]: Linux Ubuntu 16.04 (Xenial) and Robot Operating System (ROS) Kinetic Kame and a Nvidia GPU corresponding in performance to the Nvidia Jetson TX2 NPU. The processing of time series of experimental data was carried out in the described environment with the simulation of all types of errors presented in the dataset as a sequence of incoming data with reference to a timestamp

Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 provide the estimation results for the deviation of an individual pitch, roll, or yaw parameter and the inertia of the monitored parameter generated by the BiGRU layers of the CompactNeuroUAV network for a single flight without fault. As mentioned above, the data of each flight was split into short intervals and used as training, validation, and test sets. Therefore, it is obvious that in the figures, a part of each curve generated by the NN model is the training data re-processing result. However, such an additional validation allows estimating the NN model efficiency over the entire flight range and not only on short data sets.

In Figure 5, Figure 7 and Figure 9, the X axis shows discrete time instants, and the Y axis shows the UAV onboard the system’s monitored parameter change command (commanded), the actually measured value (measured), and the NN model-generated expected value (expectedGRU).

In Figure 6, Figure 8, and Figure 10, the Y axis shows the actual deviation of the monitored parameter and the expected one generated by the CompactNeuroUAV model. The deviation curves allow visually estimating the accuracy of considering the inertia of the UAV unit’s response to the control commands by the NN model. The figures demonstrate that the parameter deviation estimation accuracy is quite high, and the shape of the deviation curve generated by the CompactNeuroUAV model virtually repeats that of the actual value curve.

Thereat, the accuracy of the models for estimating the deviation of the pitch, roll, and yaw values from the reference ones was, respectively, 91, 87.2, and 89.6%.

The accuracy of the model for estimating the deviation of the set of roll, pitch, speed, and yaw parameters from the expected reference was 96%.

Table 2. Results of testing the CompactNeuroUAV fault classification neural network models on a test dataset for the carbon Z T-28 fixed-wing type UAV.

Fault Type	Class Identification Accuracy	FP Proportion (of the Total Dataset)	FN Proportion
No fault	99%	0.1%	0.25%
Engine thrust loss—fault	99.6%	0.5%	0.05%
Engine thrust loss—pre-failure condition	91%	1.4%	1.56%
Full left rudder fault (rudder control loss)—fault	94.6%	0.14%	0.03%
Full left rudder fault (rudder control loss)—pre-failure condition	89.3%	5.1%	3.2%
Full right rudder fault (rudder control loss)—fault	96%	0.26%	0.15%
Full right rudder fault (rudder control loss)—pre-failure condition	88.1%	6.7%	3.6%
Elevator fault in a horizontal position (loss of elevator control)—fault	97.4%	0.12%	0.83%
Elevator fault in a horizontal position (loss of elevator control)—pre-failure condition	90.25%	4.85%	4.19%
Left aileron control loss in a horizontal position—fault	94.65%	0.51%	0.3%
Left aileron control loss in a horizontal position—pre-failure condition	86%	7%	3.3%
Right aileron control loss in a horizontal position—fault	95.1%	0.63%	0.66%
Right aileron control loss in a horizontal position—pre-failure condition	88.1%	6.09%	2.91%
Loss of control over both ailerons in a horizontal position—fault	97.9%	0.08%	0.03%

provides the pre-failure condition identification testing results, specifying the proportion of false positive (FP) and false negative (FN) detections of fault or pre-failure condition of any UAV unit.

The presence of any class of fault or pre-failure condition was estimated by the corresponding CompactNeuroUAV model output with a probability of at least 75%. Validation of the developed models was done in two ways: validation over short time intervals and validation when simulating a full flight—the data of each particular flight was the input information for the models, while the situation of sequential receipts of these data in real time was simulated.

Figure 11 shows the results of testing the trained NN model for detecting faults and pre-failure conditions as a confusion matrix, where F is the fault, PF is the pre-failure condition, (L) is left (aileron or full left rudder position), and (R) is right (aileron or full right rudder position).

As result, the realized common fault or pre-fault state detection CompactNeuroUAV model accuracy is 93.36%, which is 5% better than the original approach, based on using the Recursive Least Squares algorithm in Ref. [30]. Using other types of recurrent neural networks for encoding time series data such as LSTM and BiLSTM was also less effective, as these models showed an accuracy 83% and 91.1% and it took one and a half and two times longer to train the models.

4. Conclusions

This paper provides the results of developing an approach to monitor and diagnose the state of an UAV in real-time. An original NN architecture was developed, which allows monitoring the deviation of parameters from the expected values and detecting faults and pre-failure conditions.

The solution implementation and evaluation results are provided for a small fixed-wing type UAV. The proposed approach can further be used to generate compensating control actions when any parameter significantly deviates directly on board the UAV using compact onboard NPUs. The identified pre-failure conditions can also be used to build control programs for the UAV control system for decision-making, and the individual UAV condition identification results can be used to promptly reconfigure formations during group flights.

The proposed approach offers a complex model for identifying faulty and pre-faulty states of the UAV, and its limitations also expire from here. The implementation of identifying a certain class of faults from the list presented in this work requires the presence of all input data, and the absence of any indications or sensor failure leads to the fact that the proposed models can no longer be used to make decisions about the direct state of the UAV elements.

Another limitation is the need for archival information about the flight parameters preceding the failure states in order to implement such models for a specific type of unmanned aerial vehicle. The adaptation and implementation of diagnostic models for other types of UAVs requires additional research, taking into account the operational information available in UAVs about flight parameters. The aim of the authors was to show an efficient neural network architecture for data processing during the flight based on the division of time data into short intervals to form a diagnosis of the state of the UAV elements. The authors hope that the approach presented in the paper will help other researchers in the tasks of predictive fault detection based on time series data.