Building on the econometric diagnostics and variable validation, which achieved an adjusted of 0.860 and resolved autocorrelation via the lagged Z-score (Durbin–Watson ≈ 1.915), we now assess out-of-sample performance using machine learning classifiers. The evaluation uses the DISTRESS label derived from the Altman Z-score threshold (1.8) defined in the cleaning stage, ensuring consistency between the econometric specification and the classification target. To avoid leakage, all preprocessing fits on the training were split and applied to the held-out test set.
In the present study, the empirical dataset consists of a panel of firm-year observations (N = 315) across 21 firms from 2010 to 2024, and the primary objective of the machine learning analysis is to compare the relative performance of different algorithms rather than to simulate a real-time deployment at a specific historical cutoff.
For this reason, we initially adopted a stratified 80–20 split at the observation level, ensuring that the class distribution of financially distressed and non-distressed firms was preserved in both the training and test sets. This approach is commonly used in the financial distress literature when the focus is on algorithmic discrimination and robustness, rather than on point-in-time forecasting.
4.3.2. Experimental Design (Data Split, Pipelines, Algorithms)
Objective. Build a classification model for firm–year financial distress using the cleaned panel and compare algorithm families under a consistent, leakage-aware validation protocol.
Data split and leakage control. We partition the data 80/20 into training and test sets with stratification on the DISTRESS label to preserve class balance. All preprocessing transformers (imputation, scaling) are fit on the training set only and then applied to the test set. Identifiers (Name) and Years are excluded from the feature matrix; ZSCORE is excluded when the target variable is DISTRESS.
Pipelines.
Linear/Kernel models (Logistic Regression, SVM-RBF): median imputation → z-score standardization (scale-sensitive).
Tree ensembles (Random Forest, Gradient Boosting): median imputation only (no scaling needed).
Class imbalance. We set class_weight = “balanced” for scale-sensitive and tree models to correct for any minority-class under-representation.
The machine learning algorithms considered in this study, along with their corresponding model families and hyperparameter settings, are summarized in
Table 7.
Evaluation protocol. We report Accuracy, Precision, Recall, F1, and ROC–AUC on the held-out test set. Confusion matrices and ROC curves provide error structure and threshold-free discrimination. Unless noted otherwise, the classification threshold is 0.50. For transparency, SHAP analysis is applied to the Gradient Boosting model (fit on the train set; SHAP computed and summarized on the test set) to indicate global feature influence and directionality.
Robustness add-on. We repeat the stratified 80/20 split 10× with different seeds {42…51} and summarize mean ± sd ROC–AUC to demonstrate stability of conclusions. Although the dataset includes 21 manufacturing firms, the empirical analysis is conducted on a panel of firm–year observations (N = 315) spanning 2010–2024. Accordingly, the train–test split is applied at the observation level rather than the firm level. An 80/20 stratified split is therefore appropriate and yields sufficiently large training and testing subsets for robust out-of-sample evaluation.
4.3.3. Evaluation Metrics
We evaluate all models on the held-out test set using Accuracy, Precision (TP/(TP + FP)), Recall (TP/(TP + FN)), and F1 (the harmonic mean of Precision and Recall), together with ROC–AUC.
Confusion matrices at a default decision threshold of 0.50 summarize Type-I errors (FP: false alarms) and Type-II errors (FN: missed distress)—with the latter generally being more costly from a financial-risk management perspective. We report ROC curves to compare models independent of any single threshold: the curve traces the trade-off between True Positive Rate and False Positive Rate across all thresholds, and AUC is the probability that a randomly chosen distressed firm is ranked above a healthy one (a threshold-free measure of discrimination). This makes ROC–AUC the appropriate primary metric when the operational cutoff may vary by user (e.g., bank vs. regulator).
Because the operating cutoffs used by practitioners may differ (e.g., bank vs. regulator), we adopt ROC–AUC as our primary, threshold-free comparator, and we complement it with fixed-threshold summaries (Accuracy, Precision, Recall, F1) and confusion matrices at 0.50 to show the concrete error profile. This protocol connects the ranking quality (ROC–AUC) to operational consequences (FN vs. FP), aligning the metrics with the assessment objective stated in our design.
Main finding. Ensemble learners outperform the linear and kernel baselines. Gradient Boosting attains the strongest discrimination (highest AUC), closely followed by Random Forest.
Table 8 and
Figure 7 summarize these differences across metrics, while the ROC curves in
Figure 8 compare models across all thresholds without committing to a single cutoff.
Figure 7 contrasts Accuracy, Precision, Recall, F1, and ROC–AUC for the four classifiers on the held-out test set. Reporting this bundle of metrics is intentional: ROC–AUC captures
threshold-free discrimination (how well a model ranks distressed above healthy firms across all cutoffs), while F1 summarizes the Precision–Recall trade-off at the default threshold (0.50). Accuracy is included for completeness but can be misleading when the costs of errors are asymmetric; in financial distress assessment, Recall (missed distress) and Precision (false alarms) matter more.
The figure shows a clear hierarchy. Gradient Boosting achieves the strongest overall discrimination (AUC ≈ 0.95) with competitive threshold performance (Precision/Recall/F1), indicating the best separation between distressed and healthy firms. Random Forest is a close second (AUC just below Gradient Boosting) and, in some metrics, matches or slightly exceeds it—together, these results highlight the advantage of tree-based ensembles in capturing non-linear interactions among financial ratios and macro indicators. SVM (RBF) performs solidly but trails the ensembles, and Logistic regression ranks last, consistent with the presence of non-linear effects that a linear boundary cannot fully model.
Practical implication. For practical implementation within a financial distress assessment framework, Gradient Boosting (or Random Forest) is the preferred base model. The operating threshold can then be tuned to the use case (e.g., favour higher Recall if missing a distressed firm is costlier than a false alert), but the AUC dominance in
Figure 8 indicates that ensemble methods provide the most reliable foundation.
Figure 8 plots the Receiver Operating Characteristic (ROC) for all classifiers on the held-out test set, showing True Positive Rate versus False Positive Rate across all decision thresholds; the diagonal denotes random guessing, and curves nearer the top-left indicate stronger discrimination. The Area Under the Curve (AUC) provides a threshold-free summary of ranking quality. In our results, Gradient Boosting consistently dominates the ROC space, achieving the highest AUC (≈0.95), with Random Forest a close second; SVM (RBF) performs respectably but remains below the ensembles, while Logistic regression lies closest to the diagonal. Notably, in the low-FPR region—the operating zone when false alarms are costly—both ensemble models sustain comparatively high recall, reinforcing their suitability for practical financial distress assessment. (Models were fit on the training split only; ROC and AUC were computed exclusively on the unseen test data).
Figure 9 displays each classifier’s confusion matrix on the held-out test set at the default threshold (0.50), showing the distribution of true and false positives and negatives to reveal the error profile at an operational cutoff.
In our results, the ensemble models (Gradient Boosting and Random Forest) achieve fewer false negatives (missed distress) without materially increasing false positives (unnecessary alerts), consistent with their higher Recall and F1 in
Table 8. Most residual errors cluster near the Altman Z boundary (~1.8–2.0), where firms are borderline and small shifts in ratios can flip the label—an economically plausible ambiguity. Practically, if minimizing missed distress is paramount, the decision threshold can be tuned below 0.50 (e.g., by maximizing Youden’s J or using a cost-based rule) to trade a modest rise in false alarms for a reduction in false negatives while retaining the ensembles’ superior discrimination.
To explain why the best model performs better, we apply SHAP to the Gradient Boosting classifier (fit on the training set; SHAP computed and summarized on the test set). Therefore, SHAP values and permutation importance are interpretability tools that describe model-based contributions rather than statistical significance tests. The SHAP summary (
Figure 10) ranks features by global impact and shows directionality.
Global insights. The SHAP summary plot (
Figure 10) ranks features by mean absolute contribution and shows directionality (colour: high vs. low feature value).
Stabilizers (negative SHAP): higher ROA, Liquidity (LIQ), and Financial Maturity (FinMat) reduce model-estimated distress.
Risk drivers (positive SHAP): higher Leverage (LEV), Operating Inefficiency (OPEFF, OPEFF2), and (in some regimes) PM raise model-estimated distress.
These patterns are economically coherent and align with signs from the econometric analysis.
Figure 10, reports the SHapley Additive Explanations (SHAP) summary for the Gradient Boosting classifier, providing both a global feature ranking (by mean |SHAP|) and the direction of their contributions to the model-estimated distress score. Each dot represents a firm–year; colour encodes feature magnitude (red = high, blue = low), and horizontal spread reflects each feature’s contribution size. The plot shows that higher ROA, Liquidity (LIQ), and Financial Maturity (FinMat) are associated with negative SHAP values—lower model-estimated distress score—whereas higher Leverage (LEV), Operating Inefficiency (OPEFF/OPEFF2), and (in some regimes) Profit Margin (PM) contribute positively to distress risk. These patterns are economically coherent and mirror the signs from our econometric analysis, indicating that the ensemble model’s decisions align with financial intuition. (Model fitted on the training split; SHAP computed and summarized on the held-out test set to avoid leakage. SHAP explains model-based associations, not causal relationships.)
Figure 11 reports permutation importance for the Gradient Boosting model on the held-out test set (50 repeats). The ranking shows FinMat as the most influential factor by a wide margin, followed by LIQ and PM; LEV and SLV have moderate impacts, while ROE/ROA/SIZE contribute less once the leading variables are included. IDEBT and INF are near zero. This measure reflects each feature’s contribution to the model’s output magnitude, not its directional effect; directionality is provided by the SHAP results (higher LIQ/FinMat lowers risk; higher LEV/OPEFF raises risk).
To verify stability, we repeat the stratified 80/20 split 10 times with seeds {42…51} and re-estimate all models under the same pipelines. We report mean ± sd ROC–AUC per algorithm; dispersion is small, indicating conclusions do not hinge on a specific split.
All steps (cleaning, preprocessing, training, figures, SHAP, repeated runs) are scripted in Python version 3.13 (pandas, scikit-learn, matplotlib, shap).
Table 9 reports additional standardized evaluation metrics recommended for financial distress and credit-risk modelling (Altman et al., 2023) [
40]. These metrics complement traditional measures by accounting for class imbalance, probability calibration, and distributional separation between distressed and non-distressed firms. The results confirm that ensemble-based models, particularly Gradient Boosting and Random Forest, consistently outperform benchmark models across discrimination (ROC–AUC, Gini, KS), balanced performance (Balanced Accuracy, MCC), and probability accuracy (Brier Score, Log Loss, ECE).
As shown in
Table 8 and
Table 9 of Additional Metrics, the ensemble-based models achieve the highest specificity values, with
Gradient Boosting (0.857) and
Random Forest (0.821) outperforming SVM and Logistic regression. This indicates that ensemble models reduce false alarms while maintaining high recall, which supports the construction of a rigorous distress-assessment score.
Specificity values are derived from the confusion matrices reported in
Figure 9, where ensemble models exhibit a lower number of false positives compared to benchmark models. Gradient Boosting achieves the highest MCC value (
0.712), followed closely by Random Forest (
0.679), indicating superior overall classification quality when accounting for true positives, true negatives, false positives, and false negatives simultaneously. In addition, the MCC results further reinforce the superiority of ensemble models under imbalanced financial distress conditions, consistent with Altman et al. (2023) [
40].
We have incorporated the
Brier Score to evaluate the accuracy of the model-estimated probabilities rather than only class labels. As shown in
Table 9, Random Forest (0.098) and Gradient Boosting (0.113) achieve substantially lower Brier scores than Logistic Regression (0.167), indicating better-calibrated and more reliable probability estimates. This is further visualized through the calibration analysis presented in
Figure 12, where ensemble models closely follow the ideal diagonal calibration line.
As reported in
Table 9, ensemble models yield lower log loss values, particularly
Random Forest (0.318) and
Gradient Boosting (0.486), compared to Logistic Regression (0.506). This indicates that ensemble methods assign higher confidence to well-estimated cases while penalizing incorrect probability estimates more effectively. The
Gini coefficient, computed as
, has now been explicitly reported in
Table 9.
Gradient Boosting achieves the highest Gini coefficient (
0.906), followed by Random Forest (
0.895), indicating excellent discriminatory power between distressed and non-distressed firms. These values are consistent with the ROC curves shown in
Figure 13, where ensemble models dominate across all thresholds.
As reported in
Table 9, Random Forest achieves the highest KS value (
0.821), followed by Gradient Boosting (
0.800), indicating strong separation between distressed and non-distressed probability distributions. This separation is visually illustrated in
Figure 14, which plots the empirical cumulative distribution functions (CDFs) of both classes and highlights the maximum KS distance.
As shown in
Table 9 and
Figure 15, both Gradient Boosting (0.963) and Random Forest (0.963) achieve very high PR-AUC values, indicating strong performance in identifying distressed firms while controlling false positives. This confirms that ensemble models remain effective even under imbalance-sensitive evaluation.
We have added a comprehensive
calibration analysis, including
calibration curves and Expected Calibration Error (ECE). As illustrated in
Figure 12, ensemble models demonstrate better alignment between their probability estimates and observed distress frequencies. Logistic Regression exhibits the highest ECE (0.156), while ensemble models show lower calibration error, indicating more reliable probability estimates. This analysis is particularly important for monitoring and risk-assessment systems where estimated probabilities provide more informative guidance than binary classification alone.