Integrating Convolutional Neural Networks with Finite-State Machines for Fault Detection in Mobile Robots

Abstract

This paper highlights a communal fault detection and isolation framework integrating a convolutional neural network (CNN) with a finite-state machine (FSM). The proposed concepts ensure state-based controlled discriminate pattern recognition and enable the diagnosis of different anomalies in the mobile robot in a multi-robot environment. The framework processes the time-series sensor data through the convolution layer upon experiencing different types of fault and governs different states based on fault diagnosis and recovery. The proposed concept has been validated using a Python 3.11 and Webot environment featuring the shrimp robot in a multi-robot arena. The model obtained an accuracy of 97% in identifying and classifying faults, enabling automated recovery of faulty robots in the multi-robot environment. Experiments conducted on different simulators demonstrate that effective fault management can be achieved with low training loss.

1. Introduction

In recent days, the deployment of autonomous mobile robots in dynamic and mission-critical environments ranging from industry automation to remote exploration has increased [1]. The system grows in complex environments and ensures reliability as well as robustness against faults. The fault may lie in the sensor, actuator, or any structural component, making it imperative to identify and address the issue promptly. To deal with fault detection, identification (FDI) techniques have become vital to enhance system resilience, ensuring safety and minimizing downtime [2]. Several fault detection techniques have been proposed in the past, but many of the approaches fail in meeting real-time requirements. Early research leveraged adaptive estimation and multiple model-based strategies such as Multiple Model Adaptive Estimation (MMAE) and Kalman filter banks to detect sensor and mechanical faults in mobile robots [3]. Such systems rely on computing residuals, which are determined as the discrepancy between the actual and the predicted sensor values. For fault isolation and classification across a range of operational conditions, such techniques have been further enhanced using neural networks like the Back Propagation Network (BPN) [4]; however, it suffers from high computational costs as well as not being able to handle nonlinearity and unseen fault conditions [5].

Recent studies such as those on convolutional neural networks (CNNs) and multi-scale convolutional capsule networks (MCCNs) using the deep learning concept have addressed the issues of fault management. Hybrid models such as CNN-LSTM have also been used for discriminating feature extraction from spatial and temporal data for fault diagnosis [6,7,8]. Promising results in learning the complex environment with different fault patterns have been achieved. Noise and dynamic uncertainty do not have much effect on performance; however, an extensively labeled dataset may be required. The methods also suffer from dealing with real-time implementations when a fault is exercised which is not a part of the training dataset. Model integration has been considered by researchers to be able to overcome these challenges [9]. FSM has been integrated with CNN to provide an interpretable rule-driven mechanism to monitor robot states and transitions combined with pattern recognition; thereby the fault detection ability is enhanced. The FSM framework allows the system to track robot behavior in real time, while the CNN serves as an intelligent observer that identifies deviations corresponding to potential faults [10,11].

The authors have focused on the detection and identification of failures over time in a mobile robot using adaptive estimation, i.e., to find the outcome of various faults and train them as a pattern of failure in a system. Parallel multiple Kalman filter estimators have been implemented to integrate the faults under a system [12]. A set of residuals such as the difference in actual sensor reading with predicted readings are taken as indicators. The Back Propagation Network (BPN) has been implemented to process those residuals in a certain pattern to find the fault that occurred. Multiple Model Adaptive Estimation (MMAE) has been used for fault detection and isolation with the help of the neural network in a mobile robot [13]. Fault has been diagnosed in an industrial robot to make it efficient and safe. The authors have implemented a data acquisition strategy based on a transmission mechanism by using an attitude sensor which helps in data collection for learning discriminative features of the robot. The multi-scale convolutional capsule network (MCCN) and capsule network (Caps Net) detect fault in industrial robots and implement transfer learning. The authors have argued the inability of the approach for timely fault finding as the altitude data must be collected in advance [14]. Fuzzy logic neural network, wavelets, and Kalman filters are used in FDI to analyze and detect errors [15]. FDI is designed with the help of multiple models; each model has one of the possible faults embedded and relative observers evaluate the state of the vector and a bank of observers. Such architecture is suitable for offline implementation, and this algorithm is also used in the cooperative FDI layer to isolate faults in GPS and other types of sensor [16].

Adaptive estimation has been implemented with multiple parallel estimators for effective fault detection and identification in a mobile robot. It addresses both the sensor and physical failure. But the proposed approach lacks in dealing with the modification of the control strategy and actuator failure [17]. Faults like a change in the size of tire (or general kinematic disturbances) that cause model skidding or slipping in the kinematic and dynamic model have been addressed using torque filtering. The system enables fault detection despite parametric uncertainty in this wheeled robot model. The proposed approach may not always able to identify the fault in a single attempt. Further, it is not able to address parametric uncertainty [18]. A deep learning observer which enables us to combine CNN and long-term short-term memory network (LSTM) is implemented to approximate the nonlinear driving control system. The hybrid method focuses on both spatial features and temporal correlations of the target system, ensuring accurate evaluation of the target system. But it only focuses on the prediction of rotation speed [19]. An interacting multiple model (IMM) technique has been developed to detect and identify sensor failure [20,21]. To find a better result in fault decision, mode probability averaging and heuristic decision-making rules are incorporated into an IMM-based algorithm. It only focuses on hard sensor failures (physical failures) but it can be used for soft sensor failures by implementing various fault models which may lead to a complex structure [22]. Fault detection on the joints of an industrial robot has been achieved using a back propagation multi classification neural network. It introduces the forward kinematics algorithm for the generation of trajectories by the robotic arm and a neural network algorithm for classification [23]. The major bottleneck of the approach lies in not correcting the faults autonomously.

The FDI problem has been addressed based on residual values [24]. The FDI technique is used to monitor multiple faults and the actuators and sensors of robot manipulators without adding any redundant sensors. Exponential convergence is implemented for error estimation simultaneously for multiple faults in actuators and sensors and the auxiliary variable. The stability of this technique overall in the presence of unknown actuator faults and sensor faults, as well as modeling uncertainties and noise, has become the major issue for such an approach. It focuses on fault identification in robot manipulators to develop a reliable and fault tolerance robot based on the generation of residuals through the filtered torque estimate discussed in this paper [25]. It does not rely on the acceleration of quantities and an adaptive and robust detection algorithm is implemented for uncertainty parameters in the robot. Beyond the basic supervised learning approaches, recent development has been emphasizing the robustness of diagnostic models across variable operating scenarios. To address the limitations due to equipment diversity and varying workloads, the unsupervised domain adaptation has been focused on [26]. For fault diagnosis of the elevator in the subdomain of a contrastive adaptation approach, a time–frequency feature mechanism has been implemented. The authors have employed dimension segmentation and the cross-channel multi-head self-attention approach to extract high-dimensional frequency domain features.

The paper proposes a hybrid approach that integrates finite-state machines (FSMs) with convolutional neural networks (CNNs) for real-time fault detection, identification and automated recovery in mobile robots in multi-environment systems. The approach leverages CNN due to its discriminative pattern recognition from the obtained time-series sender data and the FSM for its state-based control logic for managing the transition across different operational states. The primary objectives of the paper lie in (i) developing a CNN module to classify the faults, (ii) implementing the FSM with six operational states enabling the interpretability, real-time state transition with automated recovery, (iii) combining the CNN and FSM for the communal benefit of achieving a state-based control logic with discriminative pattern recognition, and (iv) validation of the framework in a Python 3.11 simulation environment and Webot simulator with shrimp robots for justifying its performance over Kalman filter, MMAE and CNN-LSTM through detection latency, fault handling and false positive values. Unlike the standard CNN-FSM combination, the novelty in the proposed framework has been grounded theoretically in the following subsection describing hysteresis-augmented diagnosis and state-stable control.

The paper has been furnished into five sections. Introduction of the problem statement and the literature survey have been incorporated into Section 1. Section 2 describes the methodology to solve the addressed problem. Results and simulation in different simulators are represented in Section 3. Section 4 includes the discussion and the performance comparison of the proposed method with existing approaches. The conclusion with future work is discussed in Section 5.

2. Materials and Methods

In real-time defect diagnosis and detection in mobile robots, this study suggests a hybrid framework that combines convolutional neural networks (CNNs) and finite-state machines (FSMs). Time-series sensor data is gathered under a variety of operational and fault settings, preprocessed into structured inputs, and then used to train a CNN to identify fault patterns. A preset FSM model, which controls the robot’s reaction to errors, uses the CNN outputs to initiate state changes. The FSM allows for quick, understandable, and automated fault management during robot navigation by controlling behavior through distinct states like normal operation, fault suspicion, and recovery [27]. An FSM can be formally represented as a 5-tuple, δ: S × Σ→S

$$\mathrm{F}=\left(\mathrm{S},\mathsf{\Sigma },\mathsf{\delta },\mathrm{S}0,\mathrm{F}\right)$$

(1)

where S is a finite set of states, Σ: input alphabet (events and sensor readings), and the state transition function is denoted in Equation (2)

δ: S × Σ → S

(2)

S0 ∈ S: initial state; F ⊆ S: set of final or accepting states. At time $t$, the FSM is in state $s_t$ ∈ S and receives input $x_t$ ∈ Σ. It transitions to: ${\mathit{s}}_{\mathit{t}+1}=\mathbf{\delta }({\mathit{s}}_{\mathit{t}},{\mathit{x}}_{\mathit{t}})$ and a time series can be defined as a sequence of states or state-derived characteristics as

$$X_T=[X_{t-k},X_{t-k+1}, ..........,x_t].$$

(3)

The framework of the convolutional neural network (CNN) serves as a binary classifier and feature extractor for mobile robot malfunction detection [28]. The CNN is defined formally as a mapping function,

$$f_{\mathrm{ө}}:P^{k\times n}\to P^2$$

(4)

where θ stands for the model’s learnable parameters, such as weights and biases. A time-series data window made up of readings fros $k$ sensor channels over n distinct time steps is called the input $X_T\in R^{k\times n}$ $\times$, after processing $X_T$. As the output vector, the CNN generates $y=f_{\theta }\left(X_T\right)=[y_0,y_1]$, where $y_0,y_1$ are represent the expected odds that the input is in a “normal” or “faulty” operational state. The softmax function is used to generate the output probabilities from the raw logits q0 and q1, which are the final dense layer’s pre-activation outputs:

$$y_i={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(z_i\right)}{\exp \left(z_0\right)+\mathrm{e}\mathrm{x}\mathrm{p}\left(z_1\right)}}$$

(5)

where $i=0, 1$ and $z$ = CNN features before softmax.

A rule-based thresholding technique is applied to the CNN output probabilities in order to improve the robustness of defect identification. Assume that $\tau \mathsf{ϵ} \left[0,1\right]$ is a decision threshold and that the system determines the binary fault status based on the output probability $y_1$ corresponding to the CNN’s “faulty” class. If $y_1\ge \tau$, a fault is deemed to be detected, and the binary output is set to 1; if not, it is set to 0, signifying normal behavior. This technique can be stated as

$$\widehat{Y}=\{\begin{array}{c}1 if{ y}_1\ge r Fault detected \\ 0 Otherwise Normal\end{array}$$

(6)

Optional smoothing can be used over a sliding window of size $W_s$ to lower noise and avoid false alarms caused by brief variations in the CNN output. The average of the most recent $W_s$ CNN outputs is used to obtain the smoothed probability:

$$y_p={\displaystyle \frac{1}{W_s}}\sum _{i=1}^{W_S}\widehat{y}$$

(7)

Only when the smoothed probability $y_p$ surpasses a higher-level threshold λ is a fault warning triggered. Here,$\mathsf{\lambda }\in \left[0,1\right]$. By reducing the impact of noise and erroneous predictions, this two-tier thresholding approach, which is based on both instant classification confidence and smoothed historical data, increases the reliability of fault identification. The output of a CNN is used by the composite fault detection function $ℱ$ to classify the input sensor data ${\chi }_T$. CNN calculates a vector of probabilities as ${\mathcal{F}}_{\theta }\left({\chi }_T\right)=\left[y_0,y_1\right]$, where the probability of a fault is represented by $y_1$. For this value, a judgment threshold $\mathsf{\tau }\in [0,1]$ is applied. The system indicates a fault if $y_1\ge \mathsf{\tau }$ else, it is regarded as normal. This rule lowers false positives and increases the overall dependability of autonomous fault management by guaranteeing fault detection only in situations where confidence is high. The CNN is trained in the supervised learning framework using the labeled dataset ${\left\{{\chi }^t,y^t\right\}}_{t=1 to N}$ where the input data (such as a window of sensor measurements) is indicated by ${\chi }^t$ and the matching one-hot encoded label for “normal” or “fault” is represented by $y^t\in \left\{\right[1,0],[0,1]$ }. Finding the model parameter θ that minimizes the cross-entropy loss between the true labels and the predicted probability is the goal. The definition of the loss function is:

$$L\left(\mathrm{ө}\right)=\sum _{t=1}^N\sum _{u=0}^1{y}_u^t\mathrm{l}\mathrm{o}\mathrm{g}(f_{\mathrm{ө}}({\chi }^i)u)$$

(8)

When the model is confident, this function penalizes inaccurate predictions more severely, which directs the network to give correct classifications larger probabilities. Reducing this loss guarantees that the CNN correctly discerns between typical and malfunctioning system activity. Interpretable and reliable fault classification judgments are guaranteed by integrating a rule-based thresholding method with a softmax-based probabilistic output [29].

Theoretical Novelty

Integrating CNN with FSM puts forward state-based control logic fused with a discrete pattern recognition mechanism. The traditional supervised classifiers are static. However, the proposed framework utilizes CNN as the dynamic feature extractor mapping the time-series data with events for real-time recovery. The novelty in the fault-tolerant design has been represented as a mathematical model in Equation (9) describing either the fault detection and autonomous recovery success or the safety shutdown. It is based on the probabilistic input and ensures that the autonomous robot maintains a healthy baseline even in unseen noise. To design the framework beyond the traditional hysteresis, a two-tier decision layers has been implemented. An instantaneous threshold has been utilized for rapid response. Further, a secondary smoothing threshold has been applied for enhancing the temporal consistency. Thereby, it reduces the sensitivity to model mismatch and momentary sensor spikes which is considered a common point of failure.

The suggested system offers a dependable and scalable solution for autonomous fault handling in mobile robots by integrating CNN-based classification, dual-threshold decision logic, and FSM-driven fault management. Timely detection of the faults, minimized false positive values through smoothing and automated fault recovery have been guaranteed by the above-mentioned methodologies. Thereby, the overall system robustness and autonomy are enhanced. The FSM is initially set up in a defined starting position at the start of the algorithm as mentioned in Algorithm 1. After that, it gathers data in vector format from a variety of sources, such as cameras, Lidar, motor current, wheel sensors, and temperature sensors. A transformation function is used to improve the data quality for the learning algorithm, producing input data that has been normalized or reshaped. A convolutional neural network is used to detect faults. If the expected output reaches or surpasses a preset threshold, as indicated by the condition, a fault is detected. Finally, the transition of the FSM to the subsequent state has been determined by the present state and the classifier output through the transition function which ensures a responsive as well as reliable fault detection mechanism based on real-time data.

Algorithm 1: FSM_Fault($\mathrm{₣}$)
$INPUT:\chi \left(t\right)Rn:$ inputs from and $\chi \left(t\right):$ vector of realdata sensor inputs at time $\mathit{t}$ with $n$ components.
1	Initialize the FSM $F=(Q,\mathsf{\Sigma },\mathsf{\delta },{\mathrm{q}}_0,\mathrm{F}$)
2	Initialize:    $FS{M}_{State}=0$
3	Collect data  ${\chi \left(t\right)=\left[x_1\left(t\right),x_2\left(t\right),\dots \dots \dots \dots x_n\left(t\right)\right]}^T$
4	Apply transformation function ϕ    $x^{\mathit{\prime }}\left(t\right)=\mathsf{\varphi }\left(\mathrm{x}\right(\mathrm{t}\left)\right)$.
5	Detect Fault by using CNN $\widehat{y}\left(t\right)=\left(f_{CNN}\left(x^{\prime }\left(t\right);\mathrm{ө}\right)\right)If\widehat{y}\left(t\right)\ge \tau \Rightarrow \mathrm{f}\mathrm{a}\mathrm{u}\mathrm{l}\mathrm{t} \mathrm{d}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}$
6	Determine the FSM’s transition to the next state $q_{t+1}=\delta (q_t,\widehat{y}(t\left)\right)$.
7	Return  $\widehat{y}\left(t\right)$

The FSM operates through six states, ${\mathrm{S}}_0,{\mathrm{S}}_1,{\mathrm{S}}_2,{\mathrm{S}}_3,{\mathrm{S}}_5,\mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{S}}_6,$ denoting normal, fault detected, fault classified, fixing, verification, and manual intervention, respectively. It begins in the normal state and transitions to fault detection upon identifying an issue. If classified, it moves to fixing and then verification. If verification fails, manual intervention may occur. The developed framework implements a single-stage multi class CNN architecture to map the input signals to the six states where one is the normal class and the other five belong to the fault category. The output layer has been configured with six neurons and a softmax activation to serve as a detector–classifier. Categorical cross-entropy governs the training process for optimizing the granular fault variation rather than a binary detection process. The approach treats normal as the baseline class within a broader multi-fault identification.

Figure 1 ensures systematic fault management, enhancing the reliability and responsiveness of the system.

Algorithm 2 describes the fault recovery process of a faulty robot and prompts a safe stop or notifies the operator based on the trigger. The finite-state machine (FSM) of the system is initialized at the first step. At the initialization time when $t=0,$ suppose the system is working normally, without any fault occurring in the normal state. This step guarantees that a known, healthy baseline is used to begin the fault detection and recovery process. In the next step for each time, the system gets real-time data from sensors and actuators and stores it in a vector $\chi \left(t\right)$ which is verified by a diagnosis classifier: a machine learning model or a rule-based system. The diagnosis classifier is a model which is used for decision making to find an indication of unusual conduct and also assigns a system status classification. This step is responsible for finding whether the system is in normal operation mode or any fault has occurred. The symbol $\widehat{O}=0$ indicates operation and $\widehat{O}=1$ indicates a fault detected. FSM transition logic defines the transition process of different states based on the current state $S\left(t\right)$, the diagnostic classifier output $\widehat{O}$ and the success or failure of attempts at recovery. The transition of states as $S_0\to S_0 and \widehat{O}=0 \mathrm{w}\mathrm{i}\mathrm{l}\mathrm{l}$ continue normal operation $S_0\to S_1; when \widehat{O}=1,$ a fault is automatically detected.

Algorithm 2: Fault-Recov (R)
Input: $\chi \left(k\right)\mathrm{є}{R}^n$: Sensor and actuator input at time t
1	Initialize the states of FSM as $S\left(k\right)\mathrm{є}\{S_0,S_1,S_2{,S}_3{S}_4,S_5\}$
2	Initialization start in Normal State as $S\left(0\right)\leftarrow S_0$
3	For every time t, do Modify sensor/actuator value as $\chi \left(t\right)=input\_system\_state()\mathrm{є}{R}^n$ Execute diagnostic classifier $\widehat{O}\left(t\right)=Diagnosis(\chi (t))$
4	Compute the transition logic based on FSM $\mathit{S}\left(\mathit{t}\right)$ using Equation (9)
5	Obtain the output state as: $S\left(t\right)=S_3;$ access recovery procedure $Attempt\_Recovery\to Return\{Success,Failure\}$
6	$If S\left(t\right)=S_5$ trigger safe stop and notify operator
7	Else Repeat with transition logic.

$S_1\to S_2;$ fault identified.

$S_2\to S_3$ fault recovery process started.

$S_3\to S_4$ fault recover completed.

${ S}_3\to S_5$ enter failure and shutdown mode.

$S_4\to S_0$ enter into normal operation, i.e., system remains in healthy baseline.

Finally $S_4\to$ new fault detected and reenter fault detection mode.

$S\left(t\right)=S_3;$ access recovery procedure and Attempt_Recovery → Return{Success,Failure}.

$$\mathit{S}\left(\mathit{t}\right)=\{\begin{array}{c}{S}_0 \leftarrow IF S\left(t\right)=S_0 AND \widehat{O}\left(t\right)=0 normal operation \\ S_1 \leftarrow IF S\left(t\right)=S_0 AND \widehat{O}\left(t\right)=1 automatically fault detected\\ fault identiified \\ fault recover started \\ fault recovery fails\\ system reamains healthy \\ new fault occurs \\ \end{array}$$

(9)

The above Equation (9) describes what the system does when a fault has been identified and recovery is attempted, and what happens if recovery fails. The system is now in state S₃: fault recovery. Here, the system executes a recovery procedure to fix the fault. This could involve resetting a component, rerouting control, or restarting subsystems. The outcome may be success → fixed error, or failure—where the system cannot safely recover or the fault persists. This model or approach uses a diagnostic classifier and a finite-state machine (FSM) to recover from faults in mobile systems. It manages and recovers from failures in a mobile robot by switching between states and continuously observing system inputs.

$If S\left(t\right)=S_5,$ trigger a safe stop and notify the operator. This describes a critical safety step. Recovery failures can be overcome by shutting off the system, requiring human intervention.

Figure 2 demonstrates the process of fault identification and recovery in mobile robot starts by initializing the system state. The finite-state machine (FSM) begins with its starting state, represented as $S\left(0\right)\to S_0$. The process of fault identification and recovery in a mobile robot starts with initializing the system state. The finite-state machine (FSM) begins with its starting state, represented as $S\left(0\right)\to S_{0.}$ This state depicts the system’s typical working condition, often known as the healthy baseline. The baseline serves as the point of reference for additional monitoring, and the system makes the assumption that all sensors and actuators are operating as expected. The system assumes that all sensors and actuators are operational, and it serves as a reference for future monitoring [30]. The system continuously checks the real-time data from sensors and actuators after it has been initialized. The mathematical form of these inputs is $O\left(t\right)ϵR^n$, where $O\left(t\right)$ is the vector of observations at time t. This stage ensures that a steady flow of data regarding the robot’s condition and functionality reaches the FSM. After availing of all inputs, the robot enters into the next stage for assessing the condition of both actuators and sensors. A diagnostic function $Diagnosis\left(\chi \right)$ has been used by the FSM for inputs $O\left(t\right)$ to determine the operating status of the components as designed. The input function includes the sensor signal, actuator signal, and feedback from the environment. The function determines the acceptability of the numbers in the operational range by analyzing the input data. Further, the current state $S(t$) has been computed based on the assessment. The $Diagnosis\left(\chi \right)$ function examines these inputs to confirm their uniformity and authenticity by checking their range validation (here all reading from a sensor and actuator is compared to predetermined thresholds), temporal consistency check (In order to identify abrupt spikes or declines that are either physically impossible or extremely impractical, the system compares current values with those from earlier time steps. This stops temporary noise from being mistakenly categorized as a flaw.), correlation analysis (Actuators and a few sensors are interconnected. For example, associated position or velocity sensors should exhibit equivalent changes if a motor actuator is in operation. A malfunction could be indicated by an error amongst associated signals.), and residual/error calculation (By contrasting real sensor outputs with values anticipated by an algorithm used for machine learning or mathematical model, the system can calculate residuals [31]. The residual is marked as abnormal behavior if it is over a threshold.). Now the robot enters into a decision state. In this state, the FSM determines if the system remains in an optimal condition or if there is an anomaly once these diagnostic tests are completed. The FSM sets $S\left(t\right)=0$ (normal operation) if every test is passed. However, the FSM sets $S\left(t\right)=1$, which indicates a fault status if any inconsistency, deviation, or failure is found. This choice serves as the catalyst for entering the phases of fault detection and rehabilitation efforts. The diagnosis is then used by the FSM to make a decision. The system stays in the regular operational state (S(t) = 0) if no anomaly is found, which means the robot keeps functioning without being interrupted. Further, the FSM enters into the fault-detected state (S(t) = 1) if a fault is found. The system recognizes it and executes fault-handling procedure. The FSM transits to the fault identification state ($S_1)$ indicating the occurrence of a fault. After that, it enters state S2, where the problem is more precisely located. It is important to identify the issue location and its types too. The system transits to the fault recovery state ($S_3$), where the recovery procedure is started. Corrective measures are implemented here in order to return the system to normal operation. Depending on the severity of the malfunction, this may entail applying backup mechanisms, rerouting control commands, or recalibrating sensors. Finally, the robot reaches the results analysis stage to verify the status of the following recovery attempt and the FSM assesses the results. If recovery is successful, then the system moves to state S4, when the robot continues regular operations after equilibrium has been confirmed, otherwise the FSM enters state $S_5$. The operator is notified to manually intervene when the robot enters a shutdown phase, which stops more harm or dangerous operation in the mobile robot [32].

Table 1. FSM states description.

State ID	State Name	Description
S0	Normal Operation	Robot operating normally
S1	Fault Detected	Fault detected by CNN
S2	Diagnosis Mode	Diagnose the fault type and severity
S3	Recovery Attempt	Execute recovery procedures
S4	Critical Shutdown	Fault not recoverable, shut down safely
S5	Manual Intervention	Require human intervention

describes the state name and its description. Furthermore, the fault types are discussed in

Table 2. Fault type description.

Fault ID	Fault Type	Description/	Affected Component	Measured Parameters	Normal Value Range	Recoverable	Faulty Value Range	Label
F1	Motor Fault	Caused by overload or friction in motor bearings	DC Motor/Servo Motor	Temperature, Current, RPM	Temp: 25–45 °C; Curr: 0.5–1.2 A; RPM: 1000–1500	No	Temp: 60–85 °C; Curr: 1.8–2.5 A; RPM: 400–800	1
F2	Sensor Fault	Gradual deviation in sensor readings due to aging	Ultrasonic/IMU Sensor	Distance (cm), Acceleration (m/s2)	Dist: 10–100; Acc: 0–9.8	Yes	Dist: 120–200; Acc: −3–12	2
F3	Wheel Misalignment	Mechanical distortion or uneven surface	Wheel Encoder/Drive Unit	Encoder Pulse Count, Velocity (m/s)	Count: 250–300; Vel: 0.4–0.8	Yes	Count: 100–180; Vel: 0.1–0.3	3
F4	Battery Voltage Drop	Battery degradation or excessive current draw	Power Unit/Battery	Voltage (V), Current (A)	Volt: 11.5–12.6 V; Curr: 0.8–1.5 A	Yes	Volt: 8.5–10.2 V; Curr: 2.5–3.0 A	4
F5	Communication Fault	Data loss due to interference or software glitch	Wi-Fi/Serial Bus	Packet Loss (%), Delay (ms)	Loss: <1%; Delay: <100 ms	No	Loss: 10–30%; Delay: 200–500 ms	5

.

3. Results

The proposed logic has been validated by computer simulation through Python 3.11 and Webot simulation. The simulation environment enables controlled fault injection with reproducible experiments and systematic evaluation. The section highlights a simulation-based proof of the concept for a feasibility and performance analysis approach. The following subsection describes the results obtained in both environments.

3.1. Training Configuration and Implementation Details

CNN architecture has been designed for high-resolution feature extraction [33,34]. The specific network configuration designed to handle the six-class classification task is described in

Table 3. Layer-by-layer CNN architecture.

Layer Type	Output Shape	Activation	Regularization	Kernel/Stride
Input	128 × 128 × 1
Conv2D	126 × 126 × 32	ReLU	Batch Normalization	3 × 3/1
MaxPooling	63 × 63 × 32			2 × 2/2
Conv2D	61 × 61 × 64	ReLU	Batch Normalization	3 × 3/1
MaxPooling	30 × 30 × 64			2 × 2/2
Flatten	57,600
Dense	128	ReLU	Dropout (0.5)
Output	6	Softmax

.

To ensure stable reproducibility and convergence, several pieces of training information have been considered and furnished here. An Adam optimizer has been utilized with an initial learning rate = 0.001. Categorical cross-entropy as the loss function has been implemented for optimizing the multiclass objective across the fault categories. A Learning Rate Schedule has been implemented as a reduce-on-plateau strategy that scales the learning rate by a factor of 0.2. This was executed if the validation loss could not be improved for five consecutive epochs. The training was conducted through a mini batch size of 32 and 100 maximum epochs. To prevent overfitting, early stopping was triggered for the failed validation accuracy in 10 consecutive epochs. As the hardware specification concerns, the whole experiment was conducted on a workstation with 24 GB VRAM and 64 GB RAM, ensuring consistent throughput.

3.2. Computer Simulation

Fault identification and recovery using the proposed logic has been exercised on the computer using Python programming language [33]. Fault rate, accuracy, fault identification and FSM transition have been recorded to validate the proposed logic. Figure 3 depicts the temporal progression of fault states as detected by a finite-state machine (FSM) over a series of 100 time steps (sample indices). The Y-axis represents the detected fault states, labeled F1 through F5, which likely correspond to different fault severity levels or types. The plot shows that the FSM frequently transitions between states, indicating dynamic changes in system behavior over time. Few states, such as F1 and F5, appear frequently, indicating the occurrence of the intermittent minimal and severe faults, respectively. Similarly, the intermediate states F2–F4 indicate varying persistence. The transitions seem to be irregular as well as non-periodic, reflecting the stochastic or event-driven nature of the underlying fault occurrences. Overall, this temporal progression highlights FSM’s sensitivity in tracking fluctuating fault conditions, capturing both rapid shifts and longer periods of stability in specific fault states.

Table 4. FSM transaction.

Time	Previous State	Event	New State
21 05:33:41	S3	Fix result	S5
22 05:33:41	S5	fault detected	S5
23 05:33:41	S5	Diagnosis	S5
24 05:33:41	S5	recovery attempt	S5
25 05:33:41	S5	fix result	S5

represents a transaction log of FSM, showing the sequence of state transitions over time. The same has also been reflected using Figure 4. Each row lists the timestamp, the previous state, the event that occurred, and the resulting new state. For example, at 05:33:41, the FSM moved from state $S_3$ to $S_5$ due to the event fix result. Subsequent events such as fault detected, diagnosis, recovery attempt, and another fix result all occurred while the FSM was already in state $S_5$, leaving it in the same state. This indicates that state $S_5$ is stable under multiple events, possibly representing a critical or recovery state where the system can handle repeated events without changing state. The table effectively captures the temporal progression and event-driven behavior of the FSM.

Table 5. Fault detection.

Iteration	Fault	Confidence	Final State	Recovered
15	F05	0.797612	S0	TRUE
16	None	0	S0	TRUE
17	None	0	S0	TRUE
18	F04	0.875081	S5	FALSE
19	F03	0.855506	S5	FALSE

summarizes the fault detection outcomes over several iterations. Each row shows the iteration number, the fault detected (if any), the confidence level of detection, the final FSM state after processing the fault, and whether the system was successfully recovered. For instance, in iteration 15, fault F05 was detected with a confidence of 0.798, the FSM transitioned to state S0, and the system recovered successfully. Iterations 16 and 17 had no faults detected, maintaining state S0 with successful recovery. In contrast, iterations 18 and 19 detected faults F04 and F03 with high confidence, transitioned the FSM to state S5, but the system did not recover, indicating these faults may represent critical or unresolved issues. Overall, the table reflects the effectiveness of fault detection and recovery mechanisms in the FSM-based system.

The bar chart illustrates the frequency distribution of state transitions within a finite-state machine (FSM). The x-axis represents the different FSM states ($S_0$, $S_4$, and $S_5$), while the y-axis indicates the number of times transitions into each state occurred. From the figure, it is evident that state S0 is the most frequently visited, with a frequency of 16, indicating that the system predominantly remains or returns to the initial or idle state. In contrast, states $S_4$ and $S_5$ are rarely reached, each with a frequency of 2, suggesting that transitions into these states are infrequent or correspond to specific fault or operational conditions. Overall, the distribution highlights highly skewed behavior in the FSM, where the majority of transitions concentrate in the primary state, and other states are visited sparsely.

Figure 5 illustrates the recovery success rate for different fault types (F01 to F05) in a finite-state machine (FSM)-based system. Among the five fault types, F02 and F05 exhibit the highest recovery success, achieving a perfect rate of 1.0, indicating that the system can fully recover from these faults. Fault F04 shows a moderate recovery rate of approximately 0.67, suggesting partial effectiveness in handling this fault. F01 demonstrates a lower recovery success rate of 0.5, highlighting a potential area where the FSM’s fault-handling mechanism may require improvement. Notably, F03 has a recovery rate of 0, indicating that the system fails to recover from this particular fault entirely, representing a critical vulnerability in the FSM’s design. Overall, the system shows variable recovery performance across fault types, with certain faults fully mitigated while others remain challenging.

Figure 6 illustrates an FSM that models a system’s operational and fault-handling lifecycle. The system begins in the $S_0$: normal operation state. Upon a fault detected event, it transitions to $S_1$: fault detected, followed by a move to $S_2$: diagnosis mode after a diagnose action. In the diagnosis state, if the issue is deemed critical, the system transitions to $S_4$: critical shutdown. However, if the issue is recoverable, the system enters S₃: recovery attempt. From $S_3$, a successful recovery leads back to $S_0$: normal operation via the recovered transition, while a fix failed event moves the system to $S_5$: manual intervention. Both the $S_4$: critical shutdown state, through a restart action, and the $S_5$: manual intervention state, through a manual override action, can return the system to $S_0$: normal operation, closing the loop and indicating the primary goal is always to restore normal function.

An FSM state transition graph visually represents the system’s states and the events that trigger transitions between them. In the graph, each node corresponds to a specific state, while directed arrows indicate how the system moves from one state to another in response to events or conditions. The arrows may be labeled with the triggering event, showing the cause of each transition. This type of graph helps us to understand the system’s dynamic behavior, identify stable or critical states, and analyze how the system responds to different inputs, making it a useful tool for designing, debugging, and monitoring event-driven processes.

Figure 7 displays a confusion matrix for a five-class classification problem (F1 through F5). The matrix shows a strong overall performance with high accuracy across most classes, indicated by the large values along the main diagonal (true label = predicted label). Specifically, classes F1, F3, and F4 are classified perfectly, with 750 instances each of correctly predicted classes and zero misclassifications. Class F2 is highly accurate with 703 correct predictions, but it shows some confusion, primarily with class F3 (36 instances) and minor confusion with F4 (6) and F5 (5). Class F5 is also very accurate, correctly classifying 738 instances, but it exhibits a noticeable confusion with class F2 (11 instances) and a single misclassification with F4 (1 instance). The low number of off-diagonal entries (false positives and false negatives) suggests the model is robust, with the minor confusions between F2, F3, and F5 being the main areas for potential improvement.

Figure 8 shows the per-class precision, recall, and F1 scores for a five-fault classification problem (F1 through F5). Overall, the classification model demonstrates excellent performance across all fault classes, as all scores are very close to the ideal maximum of 1.0.

• Classes F1 and F4 show perfect scores (1.0) for precision, recall, and F1, indicating the model correctly identifies all instances of these faults with no false positives or false negatives.
• Classes F3 and F5 also exhibit near-perfect performance, with precision and F1 scores close to 1.0, though recall for F3 is slightly higher, and recall for F5 is visibly lower than its corresponding precision.
• Class F2 is the lowest performing class, where both recall (approximately 0.93) and F1 (approximately 0.95) are noticeably lower than its precision (approximately 0.98).

  ◦

  The lower recall for F2 indicates that a small percentage of actual F2 faults were missed and classified as other faults (false negatives).

  ◦

  The lower recall for F5 (around 0.85) is the most significant drop across all classes, suggesting that a notable number of actual F5 faults were missed (false negatives), although when the model does predict F5, it is highly likely to be correct (high precision).

In summary, the model is highly effective, with the minor degradation in performance mainly stemming from missed instances (lower recall) in class F2 and more prominently in F5.

The provided bar chart shown as Figure 9 clearly demonstrates that the classification model exhibits exceptionally high and consistent performance across all standard aggregated metrics. The scores for precision, recall, F1 score, and ROC-AUC are all at or extremely close to the maximum ideal value of 1.0. This indicates that, on average across all fault classes, the model is highly effective: it makes very few incorrect positive predictions (high precision), successfully identifies nearly all actual positive instances (high recall), maintains an excellent balance between these two metrics (high F1 score), and is outstanding at distinguishing between the different classes (high ROC-AUC). This near-perfect overall performance metric is consistent with the high diagonal counts observed in the prior confusion matrix and suggests that, despite minor per-class variations (as previously noted for F2 and F5), the system is extremely reliable for the majority of classification tasks.

The provided images Figure 10 and Figure 11 illustrate the training progress of a convolutional neural network (CNN) over 15 epochs. Figure 10 shows that the model learns very quickly. Both the training accuracy (blue line) and validation accuracy (orange line) start low and experience a sharp increase within the first few epochs (0 to 4), reaching an accuracy of over 97%. After this rapid initial phase, the accuracy stabilizes at a high level, fluctuating slightly between approximately 97% and 98%. The validation accuracy generally tracks the training accuracy closely, often slightly above the training accuracy in the latter half of the training (epochs 7–14). The close convergence and consistently high levels of both lines indicate that the model is generalizing well to unseen data and is not significantly overfitting. Similarly, Figure 11 mirrors the accuracy plot in its convergence. Both the training loss (blue line) and validation loss (orange line) drop sharply from a high initial value to very low values within the first 3–4 epochs. After this initial drop, the loss curves flatten out, stabilizing at a very low value. The validation loss remains slightly below the training loss throughout most of the training process, which, along with the high accuracy, confirms that the model is not overfitting and is performing excellently on the validation dataset. The continued low and stable loss suggests that further training may yield minimal additional benefit.

3.3. Webot Simulation

The proposed concept has been validated in a Webot simulation environment deployed with three shrimp robots. The robot is smaller in size with open-source software on board [34]. The structure of the shrimp robot is as shown in Figure 12. It is a six-wheeled mobile platform as described in

Table 6. Shrimp robot description.

Characteristics	Description
No. of wheels	6
Dimension	639 × 428 × 278 mm
No. of motors	6 DC motor
Weight	5.4 kg
Sensor	6 Wheel Encoders	Odometry and motor speed in each wheel
	1 IMU	Orientation/acceleration
	1 Lidar	Obstacle detection
	1 Camera	Vision for CNN fault detection

. The robot is deployed with passive articulated suspension for rough terrain adaptation up to 2× wheel diameter. It has six DC motors for precise control. Webot simulation enables risk-free fault testing such as wheel slip or sensor occlusion at 300× speed. The robot validates the CNN-FSM algorithms cost-effectively with customizable sensors and a multi-robot arena.

A rectangular arena with floor (3, 3) and tile (0.8, 0.7) has been set up to model fault scenarios for the shrimp robot, enabling the integration of convolutional neural networks (CNNs) with finite-state machines (FSMs). It detects anomalies such as sensor failures or locomotion impairments in real time. Figure 13 represents the initial environment of the robots. Figure 14 and Figure 15 depict the intermediate and final transition, respectively. The visualizations prove the feasibility of the hybrid CNN-FSM approach prior to physical deployment, confirming robust fault detection accuracy across diverse environmental conditions.

4. Discussion

The proposed framework has been compared with other existing algorithms such as Kalman, MMAE and CNN-LSTM [36]. The performance superiority of the proposed framework has been validated collectively using different parameters: accuracy, precision, robustness and fault recovery [37,38].

Table 7. Quantitate performance comparison.

Method	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)	Real Time
Proposed	97	97	97	97	Yes
MMAE	88	85	87	86	Partial
Kalman Filter	92	90	89	92	Yes
CNN-LSTM	93	92	93	94	No

represents a quantitative discussion demonstrating the framework’s effectiveness in terms of accuracy, precision, F1 score and recall, outperforming existing approaches such as MMAE, Kalman filter and CNN-LSTM.

Table 8. Characteristic comparison.

Characteristic	Proposed	MMAE	Kalman Filter	CNN-LSTM
Core Mechanism	Hybrid CNN pattern recognition + FSM states	Multiple parallel model voting	Model-based residual estimation	Sequential deep temporal modelling
Fault Coverage	5 types (motor/sensor/wheel/etc.)	Sensor/physical discrete	Sensor/actuator parametric	Time-series vibration/motor
Real-time Speed	Excellent	Moderate	Good	Poor
Robustness	Noise-resistant via dual-thresholding	Good for known faults	Sensitive to model mismatch	Over fits without augmentation
Interpretability	High	Low	Medium	Low
Data Needs	Moderate labeled datasets	Multiple fault models	Model priors + measurements	Large temporal sequences

illustrates the state transitions and fault logs of the FSM, demonstrating its state stability in terms of recovery speed, fault coverage, etc.

Table 9. Output metric comparison.

Metric	CNN-FSM	MMAE	Kalman	CNN-LSTM
Detection Latency	<32 ms	100–200 ms	50–100 ms	200 + ms
False Positive Rate	1–2%	3–8%	5–10%	2–5%
Recovery Automation	Full	Partial	Manual	None
Multi-fault Handling	Yes	Yes	Limited	Limited
Computational Load	Low	Medium	Low	High

discuss the superiority of the proposed framework by comparing different characteristics such as detection latency, false positive, etc. It can be observed in all of the discussions that the proposed framework outperforms the existing methods in the literature as mentioned above.

5. Conclusions

The proposed framework achieves superiority in fault detection with 97% accuracy in precision and recall across five different types of faults. In the first phase it identifies the fault, and in the later phase recovers it. The entire fault management has been addressed by the CNN and FSM such that CNN works on discriminate pattern recognition and FSM on state-based control logic. The communal benefit has been validated through the Python 3.11 simulation environment and Webot simulator using a shrimp robot navigating in a multi-robot environment. The integration leverages pattern recognition for accurate fault classification and automated recovery through six-state transition. It outperforms the traditional Kalman filter and MMAE in a real-time environment.

The paper can be further extended to implement a complex system with mission-critical resilience for autonomous mobile robots. Further, the work can be extended to deal with the lightly addressed or unaddressed fault category.