Bearing Fault Diagnosis Grounded in the Multi-Modal Fusion and Attention Mechanism

Abstract

This paper proposes a novel method called Fusion Attention Network for Bearing Diagnosis (FAN-BD) to address the challenges in effectively extracting and fusing key information from current and vibration signals in traditional methods. The research is validated using the public dataset Vibration, Acoustic, Temperature, and Motor Current Dataset of Rotating Machines under Varying Operating Conditions for Fault Diagnosis. The method first converts current and vibration signals into two-dimensional grayscale images, extracts local features through multi-layer convolutional neural networks, and captures global information using the self-attention mechanism in the Vision Transformer (ViT). Furthermore, it innovatively introduces the Channel-Based Multi-Head Attention (CBMA) mechanism for the efficient fusion of features from different modalities, maximizing the complementarity between signals. The experimental results show that compared to mainstream algorithms such as Vision Transformer, Swin Transformer, and ConvNeXt, the Fusion Attention Network for Bearing Diagnosis (FAN-BD) achieves higher accuracy and robustness in fault diagnosis tasks, providing an efficient and reliable solution for bearing fault diagnosis.The proposed model outperforms ViT, Swin Transformer, ConvNeXt, and CBMA-ViT in terms of classification accuracy, achieving an accuracy of 97.5%. The comparative results clearly demonstrate that the proposed Fusion Attention Network for Bearing Diagnosis yields significant improvements in classification outcomes.

1. Introduction

In mining operations, mechanical equipment such as cranes, conveyors, and ventilation fans are widely used for processing ore and materials, with bearings playing a crucial role in ensuring the stability of the rotating components [1]. However, when bearings fail, they may cause vibrations, overheating, or even shutdown among the equipment, resulting in serious economic losses and safety risks [2,3]. Therefore, the timely diagnosis and of bearing faults and replacement of faulty equipment can effectively prevent these problems.

Traditional fault diagnosis methods grounded in vibration signal analysis typically include data feature extraction and fault classification [4]. Feature extraction relies on time domain, frequency domain, and time–frequency domain analysis, while fault classification uses shallow models such as support vector machines [5], neural networks [6], and K-nearest neighbors [7]. Sun Yawei et al. proposed an autoregressive data generation method based on wavelet packet transform and cascaded stochastic quantization. This method can generate high-quality pseudo-samples to address the unbalanced sample problem in bearing fault diagnosis, improving the diagnosis performance [8]. However, these methods often depend on complex signal processing techniques and experience, and manual feature extraction has drawbacks such as strong subjectivity and poor universality. Moreover, shallow models perform poorly in handling complex nonlinear problems, limiting the effectiveness and applicability of traditional diagnostic methods.

Therefore, it is necessary to overcome the limitations of traditional methods and shallow models while effectively integrating and fusing vibration and current signals for fault diagnosis.

In recent years, deep learning has played an important role in solving these problems, especially regarding the successful application of deep convolutional neural networks (CNN) in image processing, which has improved research progress in the field of fault diagnosis [9]. CNN can effectively extract key features from rolling bearing fault data, thereby improving the accuracy and efficiency of fault diagnosis. For example, Guo et al. [10] combined Attention CNN (ACNN) and BiLSTM for rolling bearing fault diagnosis. They processed raw vibration signals using a wide one-dimensional CNN to extract their spatial features, which are enhanced by the Convolutional Block Attention Module (CBAM), which prioritizes important features. These features are then passed to BiLSTM for temporal feature extraction. The model is robust against noise interference and does not require preprocessing. While it achieves high accuracy even in noisy environments, its drawbacks include longer training times and potential instability under extreme noise conditions. Additionally, it does not address data imbalances. Lin et al. [6] conducted detailed fault diagnosis experiments based on wavelet analysis and neural networks to extract feature vectors from rolling bearing signals. They compared fault diagnosis methods based on BP neural networks and radial basis function neural networks, and analyzed the diagnostic results based on the relaxation neural networks and closed neural networks. Tian et al. proposed a rolling bearing fault diagnosis method combining a power spectrum analysis and support vector machine (SVM). The experimental results showed that this method could be effectively applied to rolling bearing fault diagnosis.

Furthermore, Vision Transformer (ViT) [11,12], a new visual sensor technology based on attention mechanism, can effectively integrate the global information of vibration signals and has a strong parallel computing ability and storage mechanism. Although ViT is inferior to CNN in its local information extraction ability and parameter efficiency, it has obvious advantages in terms of global information integration and computational speed. Therefore, ViT provides a new method of rolling bearing fault diagnosis and has great application potential.

Guo Haike et al. [13] adopted data-level fusion, directly combining the raw data of vibration signals and current signals. Although this method can increase input information in some cases, due to differences in physical characteristics, sampling frequencies, and value ranges between vibration and current signals, this direct fusion may lead to information loss or noise interference, increasing model complexity and training difficulties. Hongfeng Tao et al. proposed a semi-supervised planetary gearbox fault diagnosis model based on GAT using fuzzy distance nearest neighbor clustering. This can better extract fault features with few labeled samples, providing a new means of gearbox fault diagnosis [14]. Hongfeng Tao et al. also developed an indirect iterative learning control scheme for batch processes with time-varying uncertainties, input delays, and disturbances. This scheme is based on the repetitive process stability theory and can effectively control batch processes [15]. Zhongmei Wang et al. [16] adopted decision-level fusion, performing feature extraction and classification separately for each modality before decision fusion. While this method is simple to implement, it cannot fully explore the deep associations between different modalities and tends to ignore complementary information between modalities. Therefore, following a comparison of different fusion strategies, this paper chose to perform fusion at the feature level [17], which can avoid heterogeneity problems in data-level fusion while fully extracting the deep features of each modality through convolutional networks and the Vision Transformer (ViT) [18] self-attention mechanism. It used the Channel-Based Multi-Head Attention (CBMA) mechanism [19] to efficiently fuse features from different modalities, maximizing the complementary advantages of multi-modal data to achieve more accurate fault diagnosis results. This strategy is relatively mature and has obvious advantages, effectively improving the accuracy and robustness of fault diagnosis.

This study introduces a novel approach that converts the current and vibration signals into 2D images, which facilitates enhanced feature extraction through advanced image-processing techniques. Multi-layer convolutional networks are used to extract local features, while the self-attention mechanism of the Vision Transformer (ViT) improves the global feature representation. The Channel-Based Multi-Head Attention (CBMA) mechanism effectively fuses the two signals, improving the overall signal analysis process. These contributions represent a significant advancement in signal processing for fault diagnosis.

The paper is structured as follows: experiments were conducted using a multi-sensor, multi-modal public dataset, and multi-modal fusion was employed to verify the accuracy of the results. The proposed model was compared with CBMA-VIT, ViT, Swin Transformer, and ConvNeXt, demonstrating its superior performance in terms of classification accuracy. The results highlight the effectiveness and practical potential of the proposed methodology in signal processing and classification tasks.

2. Methods

2.1. Data Processing

First, the vibration and current signals underwent data preprocessing, where they were divided into data groups of size $256\times 4$ and $256\times 3$, respectively, and normalized according to Formula (1):

$$X_{norm}={\displaystyle \frac{X-X_{min}}{X_{max}-X_{min}}}\times 255$$

(1)

where $X_{max}$ is the maximum value of the sample and $X_{min}$ is the minimum value of the sample. After normalization, the data values were confined to between 0 and 255. In grayscale images, 0 represents black, 255 represents white, and the values in between represent different shades of gray. Therefore, the data can be converted into grayscale image groups of $16\times 16\times 4$ and $16\times 16\times 3$.

Next, the feature information of the grayscale image groups was integrated through convolutional neural networks, converting them into single-channel 16 × 16 grayscale images. The neural network structure was designed as shown in Figure 1.

The model processed an input vibration signal grayscale image of the shape (16, 16, 4) through three convolutional layers and two activation layers, ultimately outputting a single-channel grayscale image of the shape (16, 16, 1). In convolutional layers 1 and 2, 3 × 3 convolution kernels and “same” padding were used to increase the number of channels from 4 to 16, then to increase this number to 32, effectively extracting features within the same time dimension. Subsequently, convolutional layer 3 reduced the number of channels to 1 using 1 × 1 convolution kernels, generating grayscale images containing key information. Throughout the process, ReLU activation functions were used in activation layers 1 and 2 to introduce non-linearity, while activation layer 3 used linear activation to preserve features. Normalization helped to reduce the impact of outliers, enabling these grayscale images to simultaneously reflect the characteristics of both vibration and current signals. After adding labels and classifying these grayscale images, they can be further used in training and prediction classification models.

2.2. FAN-BD

The overall structure of the proposed convolutional vision neural network method is shown in Figure 2. This method aims to achieve precise bearing information recognition through the effective processing of current and vibration signals.

Since the original VIT image feature extraction method focuses on the global characteristics of targets while neglecting local characteristics, we used small convolution kernels and small strides to extract more detailed local information, and used the CBMA attention mechanism for feature calibration and fusion to reach our conclusions. The specific steps are as follows.

This process first extracts features from the input vibration signal grayscale image through convolutional and activation layers, then adds position encoding information, as shown in Equation (2), for both vibration and current signal two-dimensional grayscale images. Next, the Vision Transformer (ViT) is used to extract features from current signals (3) and vibration signals (4). These two types of signal features are then fused through the CBMA module (5), generating queries, keys, and values (6), and calculating channel attention through the attention mechanism (7). The multi-head attention mechanism further processes these features, enhancing feature representation through concatenation and linear transformation (8) and (9). Subsequently, the fused features undergo nonlinear transformation through a multilayer perceptron (MLP) (10), and finally, classification results are obtained through layer normalization and linear layers (11). This model structure can effectively process and analyze multi-modal data, extract key information, and achieve accurate classification.

$$z_0=\left[\begin{array}{c}{x}_{\mathrm{dass}};x_p^{1}E;x_p^{2}E;\cdots ;x_p^{n}E\end{array}\right]+E_{pes}$$

(2)

$$X_{current}=\mathrm{ViT}\left(I_{current}\right)$$

(3)

$$X_{vibration}=\mathrm{ViT}\left(I_{vibration}\right)$$

(4)

$$X_{combined}=Concat(X_{current},X_{vibration})$$

(5)

$$Q=X_{combined}{W}_Q,K=X_{combined}{W}_K,V=X_{combined}{W}_V$$

(6)

$$Attention(Q,K,V)=softmax\left({\displaystyle \frac{Q{\kappa }^T}{\sqrt{d_k}}}\right)V$$

(7)

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}({\mathrm{head}}_1\dots {\mathrm{Head}}_h)W^0$$

(8)

$${\mathrm{Head}}_i=\mathrm{Attention}\left(Q_i^e,K_i^{\kappa },V_i^v\right)$$

(9)

$$zß=MLP\left(LN\right(MultiHead\left)\right)$$

(10)

$$y=LN\left(z\iota \right)$$

(11)

The detailed layer information is shown in

Table 1. FAN-BD model layer information.

Layer Name Detailed Architecture Description	Output Tensor Shape	Layer Description
Conv1	[2, 64, 120, 100]	Convolutional layer: input channels: 1; output channels: 64; kernel size: 3 × 3; padding = 1. Extracts low-level features.
ReLU1	[2, 64, 120, 100]	Activation function: applies ReLU activation to Conv1 output
Conv2	[2, 128, 120, 100]	Convolutional layer: input channels: 64; output channels: 128; kernel size: 3 × 3; padding = 1. Extracts higher-level features.
ReLU2	[2, 128, 120, 100]	Activation function: applies ReLU activation to Conv2 output.
Padding	[2, 128, 120, 100]	Padding operation: ensures image dimensions are divisible by patch_size = 8.
Unfold (Patch Division)	[2, 15, 12, 128, 8.8]	Divides image into patches of size 8 × 8 and flattens each patch, resulting in (batch_size, num_patches_h, num_patches_w, channels, patch_size, patch_size).
Patch Embedding	[2, 180, 256]	Linear projection layer, flattens each patch and maps to embed_dim = 256 dimension. Results in [batch_size, num_patches, embed_dim].
Position Embedding	[1, 180, 256]	Dynamically generates position encoding and adds to patch_embeddings.
Transformer Encoder	[2, 180, 256]	Transformer encoder section with 4 TransformerEncoderLayers, with each layer containing num_heads = 4. Performs encoding on input patches.
CBMA Attention	[2, 180, 256]	Multi-head self-attention mechanism (CBMA): input is patch embeddings after Transformer encoding; dimensions remain unchanged
MLP	[2, 180, 256]	Multi-layer perceptron, including GELU activation.
Mean Pooling	[2, 256]	Pooling operation: computes mean along num_patches dimension, resulting in [batch_size, embed_dim].
Fully Connected	[2, 4]	Classification layer: outputs num_classes = 4 categories, representing the classification results.

:

3. Data

The research data used in this paper derived from the public dataset Vibration, Acoustic, Temperature, and Motor Current Dataset of Rotating Machine under Varying Operating Conditions for Fault Diagnosis, published by Jung Wonhee, Kim Seonghun, Yun Sunghyuk, et al. [20]. The dataset is divided into two parts, and this paper uses the second part, which is a fault diagnosis dataset containing vibration signals and current signals. Furthermore, the simultaneous acquisition of both vibration and current signals allows for the extraction of more effective features, which enhances the model’s ability to capture diverse fault characteristics. The large size of the dataset also improves the generalization and reliability of the model. The data information is organized in

Table 2. Data collection equipment and parameter information.

Data Type	Collection Location/Sensors	Collection Equipment	Sampling Frequency	Unit	Included Columns
Vibration Data	Two bearing housings (A and B) in x and y directions, using 4 accelerometers (PCB352C34)	Siemens SCADAS Mobile 5PM50	25.6 kHz	Gravity constant (g)	Timestamp, x-direction (Bearing A), y-direction (Bearing A), x-direction (Bearing B), y-direction (Bearing B)
Current Data	Current sensors (3 CT sensors, Hioki CT6700)	NI9775	100 kHz	Ampere (A)	Timestamp, R-phase current, S-phase current, T-phase current

.

The experimental equipment structure is shown in Figure 3.

This paper extracted vibration signals for various fault types. Figure 4 and Figure 5 show the time-domain waveforms of vibration signals and current signals, respectively. It can be observed that, under normal conditions, the vibration signal waveform is relatively stable, while under rolling element fault and inner-ring fault conditions, obvious periodic impact features appear in the signals. The three-phase current waveforms are basically symmetrical under normal conditions, but obvious current imbalance phenomena appear under outer-ring fault conditions, reflecting the impact of faults on motor performance.

4. Experimental Design and Results

4.1. Experimental Environment

The experiment used the PyTorch framework and Python programming, with Python version 3.11. The computer parameters used for network training and testing included an Intel(R) Core(TM) i7-8250U CPU and NVIDIA GeForce MX150 GPU, running on the Windows operating system.

4.2. Dataset Classification

The public dataset includes the following fault types: normal, inner-race fault, outer-race fault, and ball fault. After data preprocessing, the experimental data were processed in units of 256 data points, normalized, and converted through convolution into grayscale images. After adding labels, the data were randomly divided into training and testing sets in a 7:3 ratio. The algorithm parameters were adjusted using the training set, and the performance metrics, such as accuracy, precision, recall, and F1 score, along with the confusion matrices and classification results, were calculated using the test set.

4.3. Experimental Results

In this study, cross-entropy function (12) was employed, and after multiple iterations using the training set, the algorithm’s performance was evaluated, resulting in the confusion matrix shown in Figure 6. Additionally, data visualization using the t-SNE (t-distributed stochastic neighbor embedding) algorithm (Figure 7) confirmed an accuracy of 97.5% for the algorithm.

$$L=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_{i}log\left({\widehat{y}}_i\right)+(1-y_i)log(1-{\widehat{y}}_i)\right]$$

(12)

To further validate the feasibility of the proposed algorithm, comparisons were made with mainstream algorithms, including CBMA-ViT, ViT, Swin Transformer, and ConvNeXt. Four evaluation metrics were employed in this study: accuracy, precision, recall, and F1 score

Table 3. Evaluation indicators.

Evaluation Indicators	Prediction
		Positive	Negative
Actual	Positive	True Positive (TP)	False Negative (FN)
	Negative	False Positive (FP)	Ture Negative (TN)

[21].

(1) Accuracy: represents the proportion of correctly classified samples in the total samples

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FN+FP+TN}}$$

(13)

(2) Precision: represents the proportion of actual class i samples among samples predicted as class i

$$Precisio{n}_i={\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i}}$$

(14)

(3) Recall: represents the proportion of correctly predicted class i samples among actual class i samples

$$Recal{l}_i={\displaystyle \frac{T{P}_i}{T{P}_i+F{N}_i}}$$

(15)

(4) F1-score is a metric used to evaluate the performance of the classification models. It is the harmonic mean of the precision and recall. The formula is as follows:

$$F1-scor{e}_i={\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(16)

To demonstrate each parameter, confusion matrices were created, as shown in Figure 8, based on the results of the five algorithms used in the testing sets. The accuracy derived from the confusion matrices for the four algorithms is as follows: CBMA-ViT (Accuracy = 0.929), ViT (Accuracy = 0.907), Swin Transformer (Accuracy = 0.931), and ConvNeXt (Accuracy = 0.915). These results highlight the superior performance of the proposed FAN-BD algorithm compared to the other models. As seen in

Table 4. Performance comparison of algorithms used in test set.

Algorithm	Normal	Inner Fault	Outer Fault	Ball Fault
FAN-BD Precision	97.0%	97.5%	97.5%	97.0%
CBMA-VIT Precision	91.3%	92.2%	95.0%	91.9%
ViT Precision	93.0%	90.4%	91.0%	89.5%
Swin Transformer Precision	93.9%	93.5%	93.9%	92.1%
ConvNeXt Precision	92.0%	90.5%	91.1%	91.5%
FAN-BD Recall	97.5%	97.5%	97.5%	98.0%
CBMA-VIT Recall	90.0%	94.5%	95.0%	90.6%
ViT Recall	93.0%	89.5%	91.0%	89.5%
Swin Transformer Recall	93.0%	93.5%	93.0%	93.0%
ConvNeXt Recall	92.0%	90.5%	92.9%	91.5%
FAN-BD F1-score	97.0%	97.5%	97.5%	97.5%
CBMA-VIT F1-score	90.6%	93.3%	95.0%	91.2%
ViT F1-score	93.0%	89.9%	91.0%	89.5%
Swin Transformer F1-score	93.4%	93.5%	93.4%	92.5%
ConvNeXt F1-score	92.0%	90.5%	92.0%	91.5%

, the FAN-BD excels in terms of precision, recall, and F1 score, demonstrating the algorithm’s ability to effectively balance both the detection and classification of faults, ensuring high accuracy and reliability.

To verify the feasibility of the proposed FAN-BD algorithm, we compared it with mainstream algorithms, including CBMA-ViT, ViT, Swin Transformer, and ConvNeXt. The cross-entropy loss function was used to optimize the classification accuracy, chosen for its ability to effectively measure the difference between predicted probabilities and actual labels. A comparative analysis of accuracy and loss across multiple iterations highlights the superior performance of FAN-BD, as illustrated in Figure 9.

The changes in loss functions during the training process for FAN-BD, CBMA-ViT, ViT Model, Swin Transformer, and ConvNeXt algorithms are shown through line graphs. From this figure, we can see that the FAN-BD algorithm demonstrated the fastest convergence speed and the lowest loss value throughout the training cycle, indicating its superior performance in this experiment. While other algorithms also showed a downward trend in loss values, their final loss values were all higher than those of FAN-BD, with ConvNeXt and CBMA-ViT following closely behind, while ViT Model and Swin Transformer performed relatively worse. FAN-BD is an improved version of CBMA-ViT, which performs classification tasks by extracting all feature information. Compared to the multimodal feature fusion approach of FAN-BD, CBMA-ViT is less effective in fault diagnosis based on vibration and current signals.

Figure 10 visualizes the classification results of CBMA-ViT, ViT, Swin Transformer, and ConvNeXt when using the testing sets by mapping them into a two-dimensional space using t-SNE. Each subplot in Figure 10 corresponds to one algorithm, allowing for a direct comparison of their performance with the FAN-BD results shown in Figure 7. The t-SNE algorithm reduces the dimensionality of the data, enabling an intuitive observation of the classification results through the color and position of the points. As shown in the figure, FAN-BD achieved the highest accuracy, with fewer than ten misclassifications of six hundred samples. In contrast, CBMA-ViT and ConvNeXt misclassified more than thirty points, while both the ViT model and Swin Transformer misclassified at least fifty points each. This highlights the superior performance of FAN-BD compared to the other algorithms.

5. Conclusions

In conclusion, the multi-modal fusion algorithm proposed in this paper, which combines convolutional networks with self-attention mechanisms, provides an efficient and precise solution for the intelligent fault diagnosis of rotating machinery. The experimental results demonstrate that our method achieved an impressive classification accuracy of 97.5%, outperforming other mainstream methods, such as ViT, Swin Transformer, and ConvNeXt. In addition to its accuracy, our method has several advantages: it enhances the adaptability to complex operating conditions, exhibits strong robustness, and performs well across various fault types, ensuring reliable fault diagnosis under diverse scenarios. Additionally, the introduction of the Channel-Based Multi-Head Attention (CBMA) mechanism effectively leverages the complementary information from vibration and current signals, improving the fusion of multi-modal features.

However, there are some potential limitations to consider. The method’s performance may be influenced by the quality and consistency of the input data, and the increased complexity of the model may lead to increased computational costs in certain real-time applications. Despite these challenges, the proposed approach presents a significant advancement in the field of fault diagnosis and opens up new directions for future research. With further refinement, this method holds promise for broader applications in intelligent fault diagnosis systems, particularly in industrial environments with diverse operating conditions.

Future research will focus on several key areas. First, we plan to optimize the method used for real-time applications by improving its computational efficiency and reducing the model’s complexity. Second, we aim to enhance the robustness of the model by investigating strategies to handle noisy or incomplete data. Additionally, exploring the fusion of other data modalities, such as temperature and acoustic signals, will be a promising avenue to expand the model’s diagnostic capabilities. Finally, further refinements to the attention mechanisms, such as introducing more advanced attention strategies, could lead to improved performance in complex fault diagnosis tasks. With these advancements, the method holds promise for broader applications in intelligent fault diagnosis systems, particularly in industrial environments with diverse operating conditions.