Comparison of Deep Learning Architectures for Fault Diagnosis of Cross-Speed Rotor Unbalance Based on Leave-One-Speed-Out Validation

Abstract

Intelligent fault diagnosis of rotating machinery typically assumes that training and test data share the same operating speed, an assumption that rarely holds in industrial end-of-line testing, where a rotor must be certified across a range of shaft speeds. In this paper, we expose this assumption through a systematic benchmark of four deep learning architectures (TCN, 1D-CNN, BiLSTM, and CNN-BiLSTM) on a laboratory rotor testbench with three operating speeds (1000, 2000, and 3000 rpm) and four unbalance fault classes. Under within-speed 5-fold cross-validation, all four models achieve a perfect macro-F1 of 1.000, offering no basis for architecture selection. Under Leave-One-Speed-Out (LOSO) evaluation (train on two speeds, test on the held-out speed), performance drops substantially and diverges across models: BiLSTM 0.180, TCN 0.270, 1D-CNN 0.271, and CNN-BiLSTM 0.401. We trace the LOSO gap to the unbalance centrifugal force law F = meω², which makes speed-confounded features unreliable under cross-speed testing. CNN-BiLSTM improves the mean LOSO macro-F1 by 48% relative to the stronger single-module baseline, 1D-CNN. Although CNN-BiLSTM achieves the highest LOSO performance among the evaluated architectures, it still does not surpass the physics-informed LightGBM baseline of 0.487. Therefore, the primary contribution of this work is not to solve cross-speed diagnosis, but to demonstrate that conventional same-speed evaluation substantially overestimates model capability and that LOSO provides a more deployment-relevant benchmark for future algorithm development.

1. Introduction

Rotating machinery such as industrial compressors, turbines, and electric motors accounts for a significant portion of unexpected downtime in manufacturing facilities. Rotor unbalance occurs due to a non-uniform mass distribution with respect to the shaft center and is one of the most common fault modes. The centrifugal force generated by unbalance increases proportionally to the square of the rotational speed, induces resonance, accelerates bearing wear, and can lead to catastrophic failure if undetected [1,2]. Therefore, accurate early diagnosis is an essential prerequisite for predictive maintenance programs and industrial end-of-line (EOL) quality inspection.

A representative EOL scenario can be found in electric vehicle (EV) powertrain manufacturing. Electric air compressors must pass automated unbalance inspection at multiple shaft speeds before shipment. Since intentionally inducing faults in production compressors is practically infeasible, controlled laboratory rotor test rigs are widely used as surrogate platforms for the development and validation of diagnostic algorithms. The present study follows this approach: a laboratory rotor kit is used as a surrogate testbench, and the benchmark results are intended to characterize algorithm performance under controlled conditions rather than demonstrate readiness for a specific production line.

Data-driven fault diagnosis using vibration signals has advanced rapidly over the past decade. Convolutional neural networks (CNNs) [3,4], recurrent architectures [5,6], and their hybrids [7,8] have reported near-perfect accuracy on standard benchmarks. However, most studies employ a same-speed protocol, in which training and test data are collected at the same rotational speed and randomly partitioned. This approach implicitly allows the classifier to learn speed-specific features such as absolute amplitude, dominant frequency, and harmonic ratios, resulting in degraded performance when the shaft speed changes. In EOL inspection, models trained at one speed often need to be applied to different operating speeds. If a model relies on speed-specific amplitude patterns, its performance may deteriorate significantly under speed variation. Therefore, cross-speed generalization capability should be evaluated separately from conventional same-speed validation. Several strategies have been proposed to address the speed-domain gap, including physics-based feature extraction methods (order tracking, envelope spectrum analysis, and dimensionless indicators) [9,10], transfer learning and domain adaptation techniques [11,12], and multi-condition training that incorporates speed labels as auxiliary inputs [13].

In our previous study, the testbench was established first, and the high accuracy of same-speed deep learning was confirmed by using a Temporal Convolutional Network (TCN) algorithm. Building upon this work, the present study conducts a systematic comparison of four deep learning architectures (TCN, 1D-CNN, BiLSTM, and CNN-BiLSTM) using Leave-One-Speed-Out (LOSO) evaluation as the primary assessment criterion. Although cross-condition fault diagnosis has been extensively investigated, the primary objective of most existing studies is to improve diagnostic performance under varying operating conditions through domain adaptation, transfer learning, or physics-based signal preprocessing [14,15,16]. In contrast, the present work focuses on establishing a rigorous and deployment-relevant evaluation framework for cross-speed unbalance diagnosis and systematically quantifying the robustness of commonly used deep learning architectures under previously unseen operating speeds.

Many cross-condition diagnosis studies employ transfer learning or domain adaptation techniques that assume access to target-domain data during training or model adaptation. For example, adversarial domain adaptation and maximum mean discrepancy (MMD)-based approaches seek to reduce distribution discrepancies between source and target operating conditions [17,18]. While these methods often achieve high accuracy, their performance may depend on the availability of target-domain samples, which is not always feasible in industrial EOL inspection scenarios. Three important research gaps remain. First, most studies evaluate diagnostic models using same-speed training and testing protocols, which may substantially overestimate performance under practical operating conditions. Second, systematic benchmark comparisons of representative deep learning architectures under a unified cross-speed evaluation protocol are still limited. Third, the relationship between cross-speed diagnostic performance and the physical characteristics of rotor unbalance has not been sufficiently discussed. To address these gaps, this study performs a controlled benchmark comparison of four representative deep learning architectures using a deployment-oriented Leave-One-Speed-Out (LOSO) evaluation protocol. The contributions of this paper are as follows:

• It is demonstrated that same-speed validation is non-discriminative for architecture selection, and the LOSO criterion is introduced as a deployment-relevant cross-speed evaluation protocol. While all four architectures achieve a macro-F1 score of 1.000 under same-speed conditions, their LOSO performance ranges from 0.180 to 0.401.
• The LOSO performance of TCN, 1D-CNN, BiLSTM, and CNN-BiLSTM is compared, showing that CNN-BiLSTM achieves the highest cross-speed macro-F1 score (average F1 = 0.401), and that its AUC advantage over the other three models is statistically significant (DeLong test, p < 0.001).
• A physics-based interpretation of the LOSO performance gap is provided through the relation (F = meω²), and a practical baseline is established using physics-informed feature engineering with LightGBM (macro-F1 = 0.487).

The paper is organized as follows. In Section 2, some related works are briefly introduced. Next, the experimental setup and how to obtain the dataset are presented in Section 3. Then, construction of four deep learning architectures, feature-engineering baseline, and evaluation protocol are explained in detailed in Section 4. The results of four architectures are evaluated under LOSO criterion in Section 5, and some discussions are given in Section 6. At last, conclusions are drawn in Section 7.

2. Related Work

The study in the paper is to compare four deep learning algorithms for fault diagnosis of unbalance rotors. Thus, some related works are briefly reviewed in this section.

2.1. Deep Learning for Vibration-Based Fault Diagnosis

Convolutional neural networks (CNNs) applied directly to raw vibration waveforms or spectrograms have become the dominant paradigm in data-driven fault diagnosis. Zhang et al. [3] showed that 1D-CNNs eliminate the need for manual feature engineering and achieve state-of-the-art accuracy on multiple bearing benchmarks. Temporal Convolutional Networks (TCNs) [19], originally proposed for sequence modeling, capture long-range dependencies with linear complexity through dilated causal convolutions, providing an effective alternative to recurrent architectures for vibration signals [20]. Bidirectional Long Short-Term Memory (BiLSTM) networks [21] model temporal dependencies in both directions and are well suited for cyclic steady-state signals. Convolutional–LSTM hybrid architectures have also attracted attention by combining convolutional frontends for local spectral pattern extraction with recurrent backends for inter-cycle dynamics [7,8].

2.2. Cross-Speed Generalization

Despite the maturity of same-speed approaches, cross-speed generalization has received relatively limited attention. Domain adaptation techniques based on adversarial training [11] or maximum mean discrepancy (MMD) [12] have shown promising results when the source and target domains share the same fault classes but differ in operating conditions. Order tracking methods [9] reduce the speed dependency of spectral peak locations by resampling vibration signals with respect to shaft angle or rotational order prior to feature extraction. However, aligning the frequency axis alone cannot eliminate the speed-dependent amplitude scaling caused by rotor unbalance. Systematic LOSO benchmarks across multiple deep learning architectures are still lacking for controlled multi-speed rotor datasets.

A recurring issue highlighted in the fault diagnosis literature is the absence of standardized evaluation protocols that reflect real industrial deployment conditions [22]. Same-speed random splitting tends to overestimate accuracy because models can exploit speed-specific features rather than learning fault-specific invariants independent of rotational speed. LOSO evaluation is analogous to leave-one-domain-out cross-validation: a model trained on data from two shaft speeds must diagnose faults at a third, previously unseen speed. In this paper, this protocol is adopted as the primary evaluation criterion.

3. Experimental Setup and Evaluation Protocol

3.1. Rotor Testbench and Fault Configurations

The experiments were conducted using the laboratory rotor kit shown in Figure 1. The testbench consists of a steel shaft (diameter: 10 mm, span: 500 mm) supported by two ball bearings and driven by a variable-speed DC motor through a coupling. Two steel disks (diameter: 75 mm, mass: 150 g) are mounted at positions 165 mm and 335 mm from the left bearing support. Two unbalance masses (m = 52.8 g) are attached to the outer circumference of the discs at a radius of e = 29 mm to generate controlled unbalance conditions. Three piezoelectric accelerometers (ACC1–ACC3) are mounted on the right bearing support (ACC1 vertically and ACC2 horizontally) and the left bearing support (ACC3 vertically).

As summarized in Figure 2, one healthy condition and three representative unbalance conditions are considered. The balanced class (B) serves as the healthy reference condition. Two static unbalance classes (SL and SR) are generated by attaching an unbalance mass to a single left and right disk, respectively, producing a dominant first-order harmonic component (1×) near the corresponding bearing location. The dynamic unbalance class (D) is created by attaching identical masses to both disks with a 180° phase difference, thereby exciting a second-order moment that manifests as a characteristic vibrating mode.

3.2. Data Acquisition and Dataset

Experiments were conducted at three different shaft speeds—1000, 2000, and 3000 rpm—and data of 3 acceleration signals were acquired at a sampling rate of f_s = 25.6 kHz using a 24-bit data acquisition system. A total of 12 independent tests were conducted for three rotational speeds and four fault classes. In order to eliminate transient influence on rotors during start and stop procedures, data of each test was continuously acquired for more than 10 s after the rotor system came into a steady state. For an independent acquisition data record of each speed and each of the four fault classes, 124 non-overlapping windows of 2048 samples (approximately 80 ms) were extracted, resulting in a total of 1488 windows (372 per class). The dataset is perfectly class-balanced. Each sample is represented as a three-channel tensor of size (2048 × 3), corresponding to the ACC1, ACC2, and ACC3 channels. The dataset structure is summarized in

Table 1. Dataset summary.

Parameter	Value
Total windows	1488
Window length	2048 samples (80 ms)
Sensor channels	3 (ACC1, ACC2, ACC3)
Sampling frequency (fs)	25.6 kHz
Operating speeds	1000, 2000, 3000 rpm
Fault classes	B (Balance), SL, SR, D
Windows per speed	496
Windows per class	372
Windows per speed–class pair	124

.

Figure 3 illustrates the vibration characteristics of the dynamic unbalance class (D) at different operating speeds in both the time domain and frequency domain. As the rotational speed increases from 1000 to 3000 rpm, the time-domain amplitude increases significantly, as shown in the top row of Figure 3, and the dominant 1× harmonic component shifts from 16.7 Hz to 50.0 Hz in the bottom row of the figure. This amplitude scaling is quantitatively predicted by the unbalance force law, F = meω² = me(2πn/60)², which indicates a nine-times increase from 1000 to 3000 rpm. A model that relies on absolute amplitude patterns in high-speed data will encounter much weaker signals at low speed, and vice versa. Figure 4 demonstrates FFT spectra of ACC1 signal at 3000 rpm for all four fault classes. No window-wise amplitude normalization or whitening was applied prior to model input. By preserving the relative amplitude scale when converting the raw three-channel windows into float32 tensors, the models were intentionally exposed to natural speed-dependent amplitude variations.

4. Deep Learning Architectures and Evaluation Protocol

4.1. Deep Learning Architectures

Four deep learning architectures, TCN, 1D-CNN, BiLSTM, and CNN-BiLSTM, are evaluated in this study. All models take raw three-channel vibration segments (2048 × 3) as input and output a four-class probability vector through a Softmax layer. All models are trained using AdamW (learning rate = 10⁻³, weight decay = 10⁻⁴), cosine annealing over 40 epochs, cross-entropy loss, and gradient clipping (maximum norm 5.0). Figure 5 summarizes the architectural components, which will be described in detail below.

• Temporal Convolutional Network (TCN)

The TCN baseline follows Bai et al. [19] adapted for vibration signals. An initial depth-wise convolution (kernel size 4, stride 4) reduces the 2048-sample sequence to 512 time steps before four TemporalBlocks with dilation factors d = 1, 2, 4, 8 and kernel size 5. Channel widths are 32–64–64–128. Global average pooling followed by a fully connected layer produces the class logits.

• 1D Convolutional Neural Network (1D-CNN)

The 1D-CNN consists of three convolutional blocks, each comprising a 1D convolution (kernel sizes 9, 7, 5), batch normalization, ReLU, and MaxPool1d (4) downsampling. The first two blocks reduce 2048 × 3 → 512 × 32 → 128 × 64; adaptive average pooling col-lapses the temporal dimension to a 128-dimensional feature vector. Dropout (p = 0.3) and a linear classifier complete the network.

• Bidirectional LSTM with Self-Attention (BiLSTM)

To isolate the contribution of recurrent feature extraction, the standalone BiLSTM applies MaxPool1d (4) twice before the recurrent layers, reducing the 2048-sample input to 128 time steps. Two bidirectional LSTM layers (hidden size 64 per direction, 128 total) process the sequence. A self-attention layer computes a weighted sum of hidden states, emphasizing the most discriminative time steps. LayerNorm, dropout (p = 0.3), and a linear head complete the model.

• CNN-BiLSTM with Self-Attention

The CNN-BiLSTM integrates the 1D-CNN frontend (two convolutional blocks, 2048 × 3 → 128 × 64) with the BiLSTM backend (two bidirectional layers, hidden size 64 per direction). The CNN encodes local spectral patterns; the BiLSTM then processes the 128 time steps as a sequence, capturing inter-cycle dependencies that the CNN alone cannot access. Self-attention, LayerNorm, and a linear classifier follow. The motivation for evaluating CNN-BiLSTM is that convolutional spectral encoding followed by recurrent temporal modeling may yield features more robust to speed-induced amplitude changes than either module alone.

4.2. Feature-Engineering Baseline

As a non-deep learning reference, a LightGBM gradient-boosted classifier was trained using physics-informed features extracted from the same 2048 sample windows. The feature set included time-domain statistics (RMS, peak, crest factor, kurtosis); speed-normalized harmonic amplitudes at 1×, 2×, and 3× fundamental frequencies; and envelope power for each of the three channels. The same LOSO protocol (Section 5.2) was applied: the classifier was trained on two speeds and evaluated on the held-out speed. The LightGBM model used multiclass log-loss; hyperparameters (n_estimators = 300, learning_rate = 0.05, num_leaves = 31, random_state = fixed) were fixed across all LOSO folds without further tuning. This baseline serves as an empirical reference point for the end-to-end deep learning approach and is discussed further in Section 6.4.

4.3. Evaluation Protocol

In order to assess classification performance of the four deep learning algorithms mentioned above, the LOSO protocol was adopted for the evaluation criterion.

• Stage 1: Within-Speed Cross-Validation

To establish a reproducibility baseline consistent with prior work on the same testbench, we apply 5-fold stratified cross-validation using only data recorded at 3000 rpm, which contains 496 windows across four fault classes. Within each fold, 15% of training data is held out as a validation set for early stopping. Models are trained until validation loss does not improve for 10 consecutive epochs, and the best-validation checkpoint is evaluated on the test fold. Macro-F1 is the primary metric.

• Stage 2: Leave-One-Speed-Out (LOSO) Evaluation

The LOSO protocol serves as the primary evaluation criterion for cross-speed generalization. In each of the three LOSO folds, one shaft speed (1000, 2000, or 3000 rpm) is held out as the test set, while the remaining two speeds form the training pool. A stratified 15% split of the training pool is used as the validation set for early stopping. The models never observe data from the held-out speed during either training or validation. The LOSO mean F1 is computed as the simple average across the three held-out speeds. Prediction probabilities from the three LOSO folds are pooled for ROC analysis and pairwise DeLong testing.

This protocol is intentionally stringent: it evaluates whether a model can generalize to operating conditions that it has never encountered before, which is precisely the requirement in industrial EOL inspection. Higher LOSO performance suggests that the model has learned fault-related features that are at least partially invariant to the speed-induced amplitude scaling governed by F = meω².

For each LOSO fold, one operating speed was held out as the test domain, while the remaining two speeds were used for training and validation. The training pool was stratified into 85% training and 15% validation subsets. Each model was trained with early stopping based on validation loss, and the checkpoint with the lowest validation loss was used for prediction on the held-out speed. After repeating this process for all three held-out speeds, the predictions were pooled to compute macro-F1, ROC-AUC, and pairwise DeLong tests.

5. Results Analysis

5.1. Within-Speed Performance

Table 2. Within-speed 5-fold cross-validation results at 1000, 2000, and 3000 rpm. All four architectures achieve macro-F1 = 1.0, providing no basis for architecture selection under this protocol.

Model	1000 rpm		2000 rpm		3000 rpm
	Mean F1	Std F1	Mean F1	Std F1	Mean F1	Std F1
TCN	1.000	0.000	1.000	0.000	1.000	0.000
1D-CNN	1.000	0.000	1.000	0.000	1.000	0.000
BiLSTM	1.000	0.000	1.000	0.000	1.000	0.000
CNN-BiLSTM	1.000	0.000	1.000	0.000	1.000	0.000

reports the results of 5-fold cross-validation at each of the three operating speeds (1000, 2000, and 3000 rpm). All four architectures achieve a macro-F = 1.000 ± 0.000. This result should not be interpreted as evidence of deployment readiness; rather, it indicates that same-speed splitting is insufficiently discriminative for architecture selection. The cause of this performance saturation is discussed in Section 6.1.

5.2. LOSO Cross-Speed Performance

Table 3. LOSO macro-F1 by model and held-out test speed. CNN-BiLSTM achieves the highest mean LOSO macro-F1 of 0.401. The nominal chance level for a balanced four-class problem is 0.250.

Model	@1000 rpm	@2000 rpm	@3000 rpm	Mean F1	Std F1
BiLSTM	0.101	0.320	0.118	0.180	0.100
TCN	0.160	0.463	0.187	0.270	0.137
1D-CNN	0.177	0.512	0.125	0.271	0.172
CNN-BiLSTM	0.168	0.507	0.530	0.401	0.165

reports the LOSO results. Performance drops substantially for all models and varies across architectures: LOSO mean F1 spans 0.180 (BiLSTM) to 0.401 (CNN-BiLSTM), a 2.2× spread that is invisible under within-speed evaluation.

Table 4. Per-class LOSO macro-F1 scores (fold-averaged across three held-out speeds). CNN-BiLSTM achieves the highest F1 for balance (B) and dynamic (D) fault types. BiLSTM shows near-zero F1 for static right (SR) unbalance and dynamic (D) unbalance, indicating collapse under cross-speed conditions for these fault classes.

Model	F1				Mean Macro-F1
	Balance	SL	SR	D
BiLSTM	0.374	0.335	0.005	0.005	0.180
TCN	0.341	0.140	0.222	0.376	0.270
1D-CNN	0.470	0.004	0.225	0.385	0.271
CNN-BiLSTM	0.737	0.153	0.212	0.505	0.401

shows LOSO macro-F1 scores for each fault class.

Several patterns emerge from Table 3. First, all models perform best when the held-out speed is 2000 rpm (mid-range) and worst when it is 1000 rpm. The 1000 rpm condition produces the weakest vibration signals: the unbalance centrifugal force is nine times smaller than at 3000 rpm, so models trained primarily on 2000/3000 rpm data encounter near-noise-level signals at test time. Second, both CNN-BiLSTM and 1D-CNN surpass 0.50 F1 at their respective best held-out speed (0.530 at 3000 rpm and 0.512 at 2000 rpm, respectively). However, 1D-CNN collapses to 0.125 when 3000 rpm is held out, whereas CNN-BiLSTM maintains 0.530 at the same condition. This asymmetry suggests that the recurrent backend of CNN-BiLSTM learns temporal patterns related to inter-cycle waveform evolution, rather than relying solely on absolute amplitude.

Figure 6 visualizes the LOSO macro-F1 by model and held-out speed. CNN-BiLSTM achieves the highest mean LOSO macro-F1, although 1D-CNN slightly outperforms it at the 1000 and 2000 rpm held-out conditions. BiLSTM remains lowest throughout.

5.3. ROC Analysis

Figure 7 shows the receiver operating characteristic (ROC) curves computed by pooling LOSO predictions across all three held-out speeds. CNN-BiLSTM achieves the highest macro-AUC of 0.704, followed by 1D-CNN (0.654), TCN (0.622), and BiLSTM (0.616). The AUC ordering is consistent with the F1 ordering, providing convergent evidence that CNN-BiLSTM yields the most discriminative representation among the tested architectures.

5.4. Statistical Significance: DeLong Pairwise Test

To assess whether macro-AUC differences are statistically significant, we apply the DeLong nonparametric test [23] to each pair of models. LOSO predicted probabilities from all three held-out speeds are pooled into a single evaluation set (1488 samples). The multiclass problem is converted into pooled one-vs.-rest binary decisions before applying the DeLong test, and micro-averaged placement values are computed across the binarized four-class outputs. Figure 8 shows the −log₁₀(p) heatmap.

All pairwise comparisons involving CNN-BiLSTM reach statistical significance (p < 0.001;

Table 5. Pairwise DeLong test results (micro-averaged placement values, pooled 1488-sample evaluation set). AUC values are from the DeLong placement statistic and may differ slightly from the macro-AUC values in Figure 7, which are computed via scikit-learn one-vs.-rest ROC-AUC.

Comparison	AUC Values	p-Value	Interpretation
CNN-BiLSTM vs. BiLSTM	0.699 vs. 0.590	<0.001	Significant
CNN-BiLSTM vs. TCN	0.699 vs. 0.634	<0.001	Significant
CNN-BiLSTM vs. 1D-CNN	0.699 vs. 0.652	<0.001	Significant
1D-CNN vs. BiLSTM	0.652 vs. 0.590	<0.001	Significant
TCN vs. BiLSTM	0.634 vs. 0.590	<0.001	Significant
TCN vs. 1D-CNN	0.634 vs. 0.652	0.066	Not significant

Note: The significance threshold was set to α = 0.05. The AUC values correspond to the two models listed in each comparison.

). The difference between TCN and 1D-CNN does not reach significance (p = 0.066), consistent with their nearly identical mean F1 (0.270 vs. 0.271). These results support that the AUC advantage of CNN-BiLSTM is unlikely to be explained by random variation alone under the pooled LOSO analysis.

In order to address concerns about window-level non-independence arising from the limited number of speed domains, the DeLong AUC test with bootstrap confidence intervals (CIs) and an exact pairwise permutation test on macro-F1 were conducted as shown in

Table 6. Bootstrap 95% confidence intervals (2000 resamples, percentile method) for macro-F1 and macro-AUC under LOSO evaluation. Values in brackets are [lower, upper] of 95% CI bounds. Per-speed F1 CIs are computed within each held-out speed fold independently.

Model	Macro-F1 (Pooled)	Macro-AUC (Pooled)	F1 @ 1000 rpm	F1 @ 2000 rpm	F1 @ 3000 rpm
BiLSTM	0.224 [0.210, 0.237]	0.616 [0.598, 0.635]	0.101 [0.088, 0.113]	0.320 [0.296, 0.344]	0.118 [0.105, 0.132]
TCN	0.363 [0.338, 0.386]	0.623 [0.604, 0.642]	0.160 [0.137, 0.184]	0.463 [0.427, 0.495]	0.187 [0.159, 0.213]
1D-CNN	0.331 [0.311, 0.352]	0.654 [0.634, 0.672]	0.177 [0.155, 0.201]	0.512 [0.479, 0.543]	0.125 [0.105, 0.145]
CNN-BiLSTM	0.443 [0.420, 0.467]	0.704 [0.690, 0.717]	0.168 [0.149, 0.187]	0.507 [0.474, 0.536]	0.530 [0.501, 0.558]

. For each model, 2000 bootstrap resamples were drawn from the pooled LOSO predictions and from each held-out speed fold separately, and 95% CIs were computed via the percentile method. Table 6 reports macro-F1 and macro-AUC with their 95% bootstrap CIs. The AUC CIs of CNN-BiLSTM [0.690, 0.717] and BiLSTM [0.598, 0.635] are non-overlapping, corroborating the DeLong significance result. Similarly, the pooled macro-F1 CIs of CNN-BiLSTM [0.420, 0.467] and BiLSTM [0.210, 0.237] do not overlap, providing complementary evidence of architecture-dependent performance differences that is robust to the choice of significance framework. The DeLong tests and bootstrap confidence intervals are based on pooled window-level predictions and should, therefore, be interpreted as comparative measures within the present benchmark rather than definitive inferential statistics. Because multiple windows were extracted from the same vibration acquisition, statistical dependence among neighboring samples cannot be completely excluded. Consequently, the presented significance results should be regarded as indicative rather than conclusive evidence of performance differences among the evaluated architectures.

For macro-F1 significance, an exact pairwise permutation test is applied to the three per-fold macro-F1 values (one per held-out speed). Because only three paired observations are available, the minimum achievable two-tailed p-value is 0.250, and no pairwise comparison reaches p  <  0.05 under this test. This reflects a fundamental statistical limitation of three-fold LOSO evaluation rather than an absence of real performance differences; the non-overlapping bootstrap CIs and the DeLong AUC test—which leverage all 1488 window-level predictions—provide stronger evidence of genuine architectural differences. Regarding repeated random-seed evaluation: each LOSO fold requires independent model training, and full multi-seed replication (e.g., five seeds × four models × three folds = 60 training runs) is computationally intensive; the bootstrap CIs reported here serve as a variance estimate under the single-seed experimental protocol.

Figure 9 presents the LOSO confusion matrices of CNN-BiLSTM by held-out speed and in pooled form. In the pooled panel, the balance class is diagnosed reliably (84.4% accuracy), whereas the static left (SL) and static right (SR) classes show the largest off-diagonal entries, indicating that left/right static unbalance remains the primary source of confusion when the held-out speed changes. This is physically consistent: SL and SR differ mainly in the relative amplitude between ACC1 and ACC3, a ratio that shifts as vibration amplitude scales with speed.

Although the absolute LOSO F1 values remain modest, the key finding is not that cross-speed diagnosis is solved, but that LOSO evaluation exposes architecture-dependent robustness differences that are completely hidden under within-speed validation.

6. Discussion

Based on the above results analysis, several discussions are given below.

6.1. Why Within-Speed Evaluation Misleads

The unanimously perfect F1 = 1.000 across all four architectures under within-speed evaluation is not evidence of a solved problem. Rather, it indicates that the within-speed protocol may substantially overestimate deployment performance. At a fixed shaft speed, fault-class vibration patterns are consistent across time: the inter-class amplitude contrast is large, the dataset is perfectly balanced, and random splitting guarantees that test windows share the same speed-specific amplitude envelope, dominant frequency, and harmonic profile as the training set. The classifier can, therefore, solve the task by memorizing speed-specific cues rather than fault-invariant ones, and any architecture with sufficient capacity will saturate. This is why within-speed F1 provides no basis for choosing between BiLSTM and CNN-BiLSTM, yet the two differ by 2.2× under LOSO.

6.2. The Physics of Speed-Induced Feature Shift

The large per-speed variance in LOSO performance (Table 3) has a direct physical explanation. Rotor unbalance generates a centrifugal force F = meω² directed radially outward from the shaft axis, where m is the unbalance mass, e is the eccentricity radius, and ω = 2πn/60 is the angular velocity. With m = 52.8 g and e = 29 mm, the centrifugal force increases 9-fold across the operating range, as shown in

Table 7. Variation in centrifugal force with different rotational speeds.

Speed	ω (Rad/s)	F = meω2	Relative Scale
1000 rpm	104.7	5.3 N	1×
2000 rpm	209.4	21.1 N	4×
3000 rpm	314.2	47.5 N	9×

.

A model trained at 2000 and 3000 rpm learns to identify fault-class differences within a 21–48 N amplitude range. When tested at 1000 rpm, the signal is five to nine times weaker than anything seen during training. In both extreme cases, the learned amplitude thresholds are poorly calibrated for the test condition, explaining the near-chance performance at the extreme speeds.

The fact that CNN-BiLSTM achieves a higher mean LOSO F1 than 1D-CNN (0.401 vs. 0.271) suggests that the recurrent backend learns something beyond absolute amplitude. The BiLSTM processes the sequence of feature vectors produced by the CNN frontend, capturing how spectral patterns evolve across consecutive waveform cycles. These inter-cycle temporal patterns may be more consistent across speeds than absolute amplitude itself, providing a partial degree of speed-invariant representation. The self-attention mechanism further concentrates the classification signal on the most diagnostic time steps within the window.

6.3. Ablation Insight: CNN + BiLSTM Synergy

The LOSO results allow a natural ablation study. The standalone 1D-CNN achieves 0.271, and the standalone BiLSTM achieves 0.180. Their combination, CNN-BiLSTM, achieves 0.401, a 48% improvement over the stronger single-module baseline and more than double the BiLSTM alone. This improvement suggests that the two modules provide complementary representations rather than a simple increase in model capacity. The CNN frontend performs multi-scale frequency decomposition, progressively reducing temporal resolution while increasing channel depth. The BiLSTM backend then models temporal dependencies in this spectral representation, capturing how harmonic amplitudes evolve across consecutive cycles. The poor performance of standalone BiLSTM suggests that recurrent modeling alone is insufficient when the raw low-speed waveform has a weak signal-to-noise ratio. The convolutional frontend provides a frequency-selective representation that appears more suitable for the recurrent backend under cross-speed conditions.

It should be noted that the above explanation for the superior LOSO performance of CNN-BiLSTM is based on its architectural characteristics and the observed benchmarking results rather than on direct feature attribution or model interpretability analyses. Therefore, the suggested ability to capture more stable temporal representations should be regarded as a plausible interpretation rather than a verified mechanism. Future work will employ attention analysis, occlusion experiments, and embedding-space analysis to investigate which signal characteristics are primarily responsible for the improved cross-speed performance. Such analyses will provide stronger evidence regarding the physical and temporal features learned by different deep learning architectures.

6.4. Comparison with Physics-Informed Feature Engineering

As described in Section 4.2, a LightGBM classifier trained on physics-informed features under the same LOSO protocol achieved a macro-F1 of 0.487. CNN-BiLSTM (0.401) is the closest end-to-end deep learning result, but still 18% below that reference. The 18% gap defines a concrete target for future research. Architectures that incorporate physical prior knowledge (order normalization layers, harmonic attention, or speed-conditioned normalization) may close this gap without sacrificing the end-to-end learning paradigm.

Within the present controlled dataset, physics-informed feature engineering combined with gradient boosting provides a stronger practical reference than the tested raw-signal deep learning models. The two paradigms are complementary in scope: feature design versus architecture design.

6.5. Limitations and Future Work

Several limitations of this study should be noted. First, a limitation of the present study is that the evaluation was conducted on a single laboratory rotor testbench with fixed structural and sensor configurations. Although the test rig provides a controlled environment for comparing diagnostic architectures under identical operating conditions, the influence of mechanical factors such as frame stiffness, bearing support rigidity, structural damping, mounting conditions, and sensor attachment was not explicitly investigated. These properties may affect the measured vibration signatures, and consequently, influence the absolute performance of diagnostic models. The objective of this work, however, was not to characterize the dynamics of the test rig itself, but to establish a fair benchmark for comparing four deep learning architectures under a common experimental platform and a deployment-relevant LOSO protocol. Future research will extend the analysis to multiple rotor systems with different structural characteristics, bearing configurations, and sensor layouts in order to evaluate the robustness and transferability of the observed performance trends. In particular, the interaction between mechanical system dynamics and learned feature representations deserves further investigation, as it may provide valuable insights into the development of more physically generalizable fault diagnosis models. Such studies would help determine whether the relative ranking of diagnostic architectures remains consistent across platforms with different stiffness, damping, and modal properties.

Second, all experiments were conducted under controlled conditions with a limited number of fault classes (four unbalance configurations). Real industrial rotors exhibit additional complicating factors: bearing faults superimposed on unbalance, misalignment, speed transients, and measurement noise from nearby machinery. The dataset does not include load variations, temperature changes, or transient speed ramps, all of which affect vibration amplitude and frequency content in practice.

Third, no real production compressor EOLT data are used: the laboratory rotor kit serves as a surrogate testbed, and future validation on actual electric compressor end-of-line datasets is required before drawing conclusions about production deployment. Extending the LOSO benchmark to multi-fault, variable-load, and real-world datasets is an important direction for future work.

Fourth, another limitation of the present study is that the 1488 vibration windows were generated by segmenting continuous vibration recordings collected under 12 speed–class operating conditions. Consequently, although the windows are non-overlapping, they should not be interpreted as 1488 fully independent experimental observations for statistical inference. The primary purpose of the window segmentation is to provide standardized input samples for training and evaluating deep learning models under an identical benchmark protocol rather than to increase the number of independent experiments. Future studies will acquire multiple independent experimental runs for each operating condition, allowing statistical analyses to be performed using acquisition runs or speed–class conditions as the independent experimental units. Such an extension will provide a more rigorous assessment of confidence intervals and statistical significance while further strengthening the generalizability of the proposed benchmarking framework.

In addition, domain adaptation techniques that align representations across speed domains during training may substantially improve upon the LOSO numbers reported here, and their effectiveness on this dataset remains to be explored. The interpretability of the CNN-BiLSTM representations also warrants further investigation: understanding which temporal or spectral patterns the model attends to would support both engineering trust and targeted architecture design.

7. Conclusions

This paper presented a systematic LOSO benchmark of four deep learning architectures for cross-speed rotor unbalance fault diagnosis. The central finding is clear within this dataset: within-speed evaluation is insufficient for architecture selection. All four models achieve macro-F1 = 1.000 within speed but diverge under LOSO (BiLSTM 0.180, TCN 0.270, 1D-CNN 0.271, CNN-BiLSTM 0.401). The CNN-BiLSTM architecture, which combines convolutional spectral encoding with bidirectional recurrent temporal modeling and self-attention, achieves the highest mean LOSO macro-F1. In the pooled LOSO ROC analysis, CNN-BiLSTM shows statistically significant AUC advantages over the other three architectures (DeLong p < 0.001 vs. all three alternatives; Table 5).

The LOSO performance gap is physically grounded in the quadratic unbalance force law, which causes vibration amplitudes to scale 9-fold from 1000 to 3000 rpm. CNN-BiLSTM’s temporal modeling of repeated waveform patterns provides partial but incomplete protection. In contrast, the physics-informed LightGBM baseline is built upon engineered features derived from rotor dynamics knowledge, such as harmonic amplitudes, relative spectral indicators, and dimensionless statistics, which are inherently less sensitive to speed-dependent amplitude variations. These features explicitly encode fault-relevant information while suppressing operating-condition-specific effects, enabling better generalization to previously unseen speeds. Furthermore, LightGBM can effectively exploit these compact and physically meaningful representations without needing to learn speed invariance directly from raw waveforms. As a result, the physics-informed baseline achieves a higher LOSO macro-F1 score than all end-to-end deep learning models. The remaining gap relative to physics-informed feature engineering (0.401 vs. 0.487) motivates the development of hybrid architectures that embed physical priors (amplitude normalization, harmonic attention, or speed-conditioned batch statistics) within the deep learning framework.

The LOSO protocol applied in this study and the four-model benchmark provide a concrete and reproducible performance reference for future cross-speed fault diagnosis research.