Temporal and Cost-Sensitive Evaluation Framework for Credit Risk Modeling Under Distributional Shifts

Abstract

Machine learning-based credit risk models are commonly assessed using discrimination metrics alone. Such evaluation, however, does not fully capture economic consequences, temporal deployment conditions, or changes in the underlying risk environment. This study develops a decision-aligned, temporally consistent evaluation framework for real-world deployment. Using loan-level data, model performance is examined under a rolling forward validation scheme. A coverage-based alert policy is implemented to reflect operational resource constraints. Predictive discrimination is measured using PR-AUC, while economic performance is evaluated through a cost-sensitive saving function. The false-negative cost parameter (λ) is varied between 5 and 25 to assess sensitivity. Performance is also compared across high- and low-default regimes, and alternative alert budgets of 5%, 10%, and 20% are considered to examine policy stability. The results indicate no systematic decline in PR-AUC over time. Changes in λ do not alter predictive ranking, although economic returns scale proportionally with the cost parameter. Economic gains are higher in high-default regimes, yet no structural deterioration is observed in low-default periods. Increasing coverage improves recall but reduces economic benefit due to higher false-positive costs. To consolidate these stability dimensions, the Unified Policy Stability Index (UPSI) is proposed as a composite measure integrating predictive variability, economic volatility, and regime differences. The index indicates sustained overall stability during the study period. The findings suggest that credit risk model evaluation should extend beyond accuracy-centered metrics and incorporate decision consistency, temporal robustness, and policy stability within a deployment-oriented framework.

1. Introduction

Credit risk assessment lies at the core of financial intermediation theory. Financial institutions operate under information asymmetry and must allocate limited capital while distinguishing between high- and low-risk borrowers. Akerlof (1970) showed that asymmetric information can distort market efficiency. Stiglitz and Weiss (1987) explained how adverse selection may lead to credit rationing. In such settings, misjudging borrower risk affects not only individual loans but also portfolio performance and financial stability.

Traditional credit-scoring models estimate default probability within a statistical classification framework. Foundational work by Altman (1968) and Ohlson (1980) established empirical approaches based on financial ratios. These models are interpretable and structurally stable. However, more recent research has questioned whether this holds universally across different portfolio types and economic conditions. Historically, model evaluation in this domain has primarily focused on predictive accuracy rather than on decision-oriented performance metrics. From a decision-theoretic perspective, however, predicted probability is not the final objective in isolation. While lenders do track default probabilities over time, the ultimate goal of credit risk evaluation is to support decisions that minimize expected costs and maximize portfolio value. Elkan (2001) demonstrated that when misclassification costs are asymmetric, accuracy-centered evaluation may be economically suboptimal. Concretely, a model that maximizes overall accuracy may still approve many high-risk loans if false negatives are costlier than false positives, a configuration typical in lending. Credit approval is, therefore, not merely a classification task. It is a decision problem involving gains and losses.

In recent years, machine learning methods have become widely adopted in credit risk modeling, owing to their capacity to capture nonlinear relationships and high-dimensional feature interactions that linear models cannot adequately represent. Brown and Mues (2012) showed that tree-based algorithms outperform traditional methods on imbalanced datasets. Lessmann et al. (2015) provided evidence from large-scale benchmarking that machine learning models systematically outperform logistic regression in discrimination performance. More recent work corroborates these findings. Suhadolnik et al. (2023) and Bhandary and Ghosh (2025) confirmed that gradient boosting and ensemble methods continue to outperform baseline statistical models across diverse credit portfolios. Yet improved statistical accuracy does not automatically translate into economic value. Verbraken et al. (2014) introduced the Expected Maximum Profit metric and showed that a high ROC-AUC does not necessarily imply higher profitability. This distinction highlights a gap between predictive discrimination and decision utility.

Financial data also evolve over time. Gama et al. (2014) formalized the concept of drift and showed that distributional changes may degrade model performance. Random train–test splits may, therefore, misrepresent real deployment conditions. Credit default is typically a rare event in retail consumer credit portfolios, though default frequency varies considerably across asset classes and economic cycles (Basel Committee on Banking Supervision 2006). Under class imbalance, metric choice becomes critical. Saito and Rehmsmeier (2015) demonstrated that precision–recall curves provide a more informative evaluation than ROC curves when the positive class is rare. Despite this evidence, AUC-based assessment remains common practice.

These considerations suggest the need for evaluation frameworks that better reflect operational reality. Credit risk models should be assessed under temporal ordering, asymmetric cost structures where the cost of missing a high-risk borrower (false negative) substantially exceeds the cost of a false alert (false positive), and varying default environments. Although each of these dimensions has been studied separately, integrated empirical evaluation across them remains limited.

This study addresses that gap. A rolling forward validation design is implemented to reflect sequential deployment, the operational scenario in which a model trained on historical loan data is applied month-by-month to newly originated loans. A coverage-based alert policy aligns model outputs with operational constraints. Sensitivity to the false-negative cost parameter is examined systematically. Default regimes are separated to compare economic performance across risk environments. Predictive discrimination and decision-level stability are evaluated jointly. The objective is to assess credit risk models across multiple dimensions. These include temporal consistency, cost structure, and environmental variation, rather than relying on a single statistical metric. To operationalize this objective, a Unified Policy Stability Index (UPSI) is introduced as a composite measure integrating predictive variability, economic variability, coverage consistency, and regime sensitivity into a single evaluative construct.

2. Literature Review

2.1. From Statistical Foundations to Machine Learning Approaches

Credit risk modeling originated in statistical methods based on financial ratios. Altman (1968) applied discriminant analysis to predict bankruptcy, while Ohlson (1980) introduced logistic regression as a probabilistic framework. These models offered interpretability and structural transparency, a constraint that motivates the use of machine learning approaches reviewed below.

The development of machine learning expanded the capacity to model complex relationships. Brown and Mues (2012) showed that tree-based algorithms outperform traditional methods on imbalanced credit datasets. Lessmann et al. (2015) provided evidence from large-scale benchmarking that machine learning models systematically outperform logistic regression in predictive discrimination. This marked a structural shift in credit-scoring research.

Subsequent studies reinforced these findings. Chen et al. (2020) and Suhadolnik et al. (2023) reported that random forests and gradient boosting outperform baseline statistical models. Yet predictive performance alone does not guarantee robustness. Addo et al. (2018) observed that tree-based methods may be more stable than deep learning approaches, such as feedforward neural networks and recurrent architectures, for learning hierarchical feature representations from raw data in credit applications. Model interpretability remains a central concern. Noriega et al. (2023) noted that boosting methods, ensemble techniques that combine many shallow decision trees in a sequential, error-correcting manner, are widely adopted but still pose challenges for explainability. Bussmann et al. (2020) proposed SHAP-based explainability frameworks, which decompose model predictions into additive feature contributions using cooperative game theory.

Taken together, recent work suggests that credit risk modeling should not be assessed solely by discrimination metrics. Stability, interpretability, and alignment with operational conditions have become equally relevant.

2.2. Class Imbalance, Rare Events, and Evaluation Metrics

In retail consumer credit portfolios, default is statistically rare, typically 1–10% of originated loans, depending on segment and economic conditions. The majority class (non-defaulters) typically exceeds the minority class (defaulters) by ratios of 10:1 to 50:1 in retail portfolios. Under such an imbalance, high accuracy may conceal poor detection of risky borrowers. Abdulrazaq (2023) emphasized that accuracy can be misleading in rare-event settings. Minority-focused metrics such as precision, recall, F1-score, AUPRC, and MCC provide a more informative assessment. Shyalika et al. (2024) argued that evaluation design is as critical as algorithm selection in rare-event prediction. Saito and Rehmsmeier (2015) demonstrated that precision–recall curves better reflect minority-class performance than ROC curves when positive cases are scarce. Despite this evidence, ROC-AUC remains widely used in practice.

Methodological adjustments have been proposed to address the imbalance. Wang (2022) combined class weighting with boosting techniques. Zhao et al. (2024) systematically evaluated resampling strategies for class imbalance in credit risk prediction. Udu et al. (2025) reviewed GAN-based augmentation approaches for imbalanced machine learning tasks. Bhandary and Ghosh (2025) reported that machine learning methods outperform traditional statistical models under imbalanced conditions. Nevertheless, resampling or weighting strategies do not fully resolve the challenge of aligning evaluation with real decision environments. Most studies still assess predictive discrimination without explicitly integrating operational cost structures or deployment constraints. This gap suggests that imbalance-aware modeling should be complemented by decision-aligned evaluation frameworks.

2.3. Cost-Sensitive Evaluation and Decision-Theoretic Foundations

Although credit scoring is often framed as a classification problem, it is fundamentally a decision problem involving gains and losses. Lieli and White (2010) modeled loan approval as an explicit decision framework. Finlay (2010) argued that profitability-oriented scoring is more appropriate than purely accuracy-based assessment. Elkan (2001) provided formal evidence that when misclassification costs are asymmetric, accuracy-centered evaluation may lead to economically suboptimal choices.

Subsequent research has incorporated cost considerations more directly. Verbraken et al. (2014) introduced the Expected Maximum Profit measure to align evaluation with financial outcomes. Petrides et al. (2022) incorporated variable misclassification costs into profit-driven scoring. Wang et al. (2020) proposed cost matrices for multi-class credit rating contexts. Zhang et al. (2025) developed a GEV-based gradient boosting approach with emphasis on extreme imbalance. Safi et al. (2022) applied cost-sensitive neural networks to predict financial distress. Khalili and Rastegar (2023) proposed hybrid performance metrics combining discrimination and cost considerations. Despite these advances, many studies assume a fixed cost parameter. Limited empirical work examines how variations in the false-negative cost affect predictive ranking and decision-level outcomes. Systematic sensitivity analysis of cost structures remains relatively underexplored.

2.4. Temporal Drift, Regime Sensitivity, and Financial Cycles

Credit data do not follow an independent and identically distributed process. Their distribution evolves over time. Žliobaitė et al. (2015) provided a conceptual overview of drift phenomena. Gama et al. (2014) distinguished concept among covariate shift changes in the distribution of borrower characteristics, prior-probability shift changes in the baseline default rate, and real-concept shift changes in the relationship between borrower features and default risk. In credit markets, all three may coincide: macroeconomic downturns alter borrower profiles, increase default rates, and simultaneously reshape the mapping between risk and features.

Empirical evidence supports this concern. Sun and Li (2011) documented drift in financial distress prediction, showing that models trained on pre-crisis data experienced significant performance degradation when applied to post-crisis periods due to structural changes in the economy. Sousa et al. (2016) proposed dynamic modeling based on sequential snapshots, an approach in which separate models are re-estimated for each consecutive time window, allowing the scoring function to adapt to distributional shifts without requiring a complete model rebuild. Shen et al. (2020) examined imbalance and drift jointly. Drift refers to statistical changes in the data-generating process that degrade model performance over time. Drift detection methods, algorithms that monitor model outputs or data statistics to flag when retraining is needed, have also evolved. Pugliese et al. (2021) compared detection algorithms in financial applications. Dehghan et al. (2016) proposed ensemble-based detection strategies, methods that maintain a pool of diverse classifiers and weight them by their recent accuracy, thereby detecting drift through changes in relative model performance. Sun et al. (2024) showed that rolling window random forests may enhance stability under dynamic conditions, that is, environments characterized by ongoing distributional shifts in borrower behavior and macroeconomic fundamentals.

Portfolio-level monitoring is equally important. Gandy (2012) introduced survival-based monitoring techniques for credit portfolios. Moiseev et al. (2021) modeled credit risk dynamics using time-varying DCC-GARCH parameters. Wang and Xiao (2025) examined credit risk forecasting under macroeconomic stress scenarios. However, much of the drift literature continues to rely on AUC as the primary performance measure. Few studies systematically evaluate decision utility across different risk regimes.

2.5. Research Gap and Positioning of This Study

The literature highlights several unresolved issues. First, frameworks that jointly integrate temporal rolling validation with decision-level policy evaluation remain limited. Second, predictive discrimination and economic utility are rarely assessed together under deployment-consistent conditions. Third, empirical evidence on the sensitivity of cost parameters and on ranking stability is sparse. Fourth, regime-based evaluation of decision stability is largely absent in applied machine learning studies on credit risk.

Dietterich (1998) emphasized the importance of statistical stability when comparing classifiers. Yet an integrated stability measure combining predictive variability, economic variability, and regime sensitivity has not been systematically developed in credit risk modeling. This study proposes an evaluation framework that integrates temporal consistency, decision alignment, and regime robustness. It separates predictive structure from economic scaling under varying cost parameters. To synthesize multiple dimensions of stability, a Unified Policy Stability Index is introduced. Following the multi-criteria composite index tradition (OECD 2008), each stability component is normalized via its coefficient of variation and assigned equal weight, consistent with the symmetry assumption across dimensions. This approach builds on Dietterich (1998), who advocated variance-based stability criteria for comparing classifiers. The index combines predictive and decision-level stability into a single evaluative construct. Based on this framework, the following hypotheses are examined:

H1. 

Under rolling temporal validation, PR-AUC does not exhibit a systematic decline over time.

H2. 

Changes in the false-negative cost parameter alter the scale of economic utility but do not affect the underlying risk ranking.

H3. 

Decision utility is, on average, higher in high-default regimes.

H4. 

Variations in coverage modify the precision–recall trade-off, while overall policy stability remains preserved.

3. Data and Methodology

3.1. Data Structure and Leakage Control

The empirical analysis is based on 15,186 loan-level observations issued between 30 October 2020 and 1 January 2023. The data originate from a Mongolian digital consumer lending institution (Moni brand) and consist of retail short-term loan contracts issued to individual borrowers. The dataset covers 16,824 unique customers across 557 loan product variants, predominantly standard consumer microfinance products. The unit of observation is an individual loan contract. The target variable is a binary indicator of high risk. The positive class $y_i\in \left\{\mathrm{0,1}\right\}$ represents loans that require early policy-level attention. The dataset can be expressed as

$${\left\{\left(X_i,y_i,t_i\right)\right\}}_{i=1}^N$$

(1)

where $X_i$ denotes the feature vector, $y_i$ the risk label, and $t_i$ the loan origination date. All observations are ordered chronologically. Each feature is constructed to ensure that no information from periods after $t_i$ is included. Formally,

$$Feature time\le t_i$$

(2)

This constraint prevents data leakage. The training environment, therefore, reflects the temporal sequence encountered in actual deployment. Thirteen features are used across three categories: (i) borrower demographics (age, gender); (ii) loan contract terms (amount, term, interest rate, penalty rate, product type, and log-transformed amount); and (iii) borrower credit history (prior loan count, historical default rate, and any-prior-default indicator) and temporal indicators (issue month, quarter, and day of week). Features such as age and gender may be subject to legal and regulatory restrictions in some jurisdictions in credit risk assessment. Therefore, they are used in this study for analytical purposes only, and their application in practice should comply with relevant legal and ethical standards. All history-derived features are strictly lagged to $t_i$ to prevent leakage. Categorical variables are encoded prior to modeling. Missing values are imputed using the median. Extreme values are treated through winsorization to reduce the influence of outliers. Because the positive class rate varies over time, random train–test splitting is avoided. Temporal ordering is preserved throughout the evaluation design.

3.2. Temporal Rolling Evaluation Design

To reflect real deployment conditions, a forward rolling evaluation scheme is implemented. The months from January to August 2022 are selected as sequential test windows. For each test month, only information available prior to that month is used for model training. For a given test month $T_k$, the training and test sets are defined as

$${\mathrm{D}}_{train}^{\left(k\right)}=\left\{\left(X_i,y_i\right):t_i<T_k\right\}$$

(3)

$${\mathrm{D}}_{test}^{\left(k\right)}=\left\{\left(X_i,y_i\right):t_i{\in T}_k\right\}$$

(4)

This structure ensures that training data are strictly limited to observations from the period preceding the evaluation period. Look-ahead bias is, therefore, avoided. The procedure simulates the sequential nature of operational decision making. Hyperparameter tuning is conducted only within each training window. No cross-sectional random validation is applied. The primary modeling approach is gradient boosting classification implemented via CatBoost, which natively handles categorical features and class imbalance. Logistic regression with balanced class weights is evaluated under the same rolling design as a benchmark model. Performance differences can, therefore, be attributed to model characteristics rather than data-partitioning effects.

3.3. Cost-Sensitive Decision Framework

After model estimation, each loan is assigned a predicted risk score:

$${\widehat{p}}_i=f\left(X_i\right)$$

(5)

where ${\widehat{p}}_i$ denotes the estimated probability of high risk. A coverage-based decision policy is applied within each test month. The top ccc percent of loans with the highest predicted probabilities are assigned to the alert category. This structure reflects operational constraints in which only a limited share of cases can receive enhanced monitoring. Economic performance is defined as

$$Saving\left(\mathsf{\lambda },\mathrm{c}\right)=\times \mathrm{T}\mathrm{P}\left(\mathrm{c}\right)-\mathrm{F}\mathrm{P}\left(\mathrm{c}\right)$$

(6)

where $TP\left(\mathrm{c}\right)$ represents correctly identified high-risk loans within the coverage level c, and $\mathrm{F}\mathrm{P}\left(\mathrm{c}\right)$ denotes false positives. The parameter $\mathsf{\lambda }$ captures the relative cost of a false negative. Sensitivity analysis is conducted by varying $\mathsf{\lambda }$ between 5 and 25. Coverage levels of 5%, 10%, and 20% are evaluated separately. This design allows predictive discrimination and decision utility to be examined as distinct but related dimensions. The saving function adopts a simplified linear cost specification, following Verbraken et al. (2014), where λ represents a fixed scalar cost ratio between false-negative and false-positive misclassifications. Extension to borrower-specific variable cost matrices represents a direction for future work.

3.4. Stability and Robustness Metrics

Predictive performance is evaluated using PR-AUC. The monthly value for the test window $k$ is denoted as ${PR}_k$. The average predictive performance across $K$ test months is defined as

$$\overline{PR}={\displaystyle \frac{1}{K}}\sum _{k=1}^K{PR}_k$$

(7)

Temporal variability is measured using the coefficient of variation:

$${CV}_{PR}={\displaystyle \frac{SD\left({PR}_k\right)}{\overline{PR}}}$$

(8)

Economic variability is computed analogously:

$${CV}_S={\displaystyle \frac{SD\left({Saving}_k\right)}{Sa\overline{v}ing}}$$

(9)

To assess regime robustness, test months are divided into high-default and low-default groups based on the median positive rate. The regime gap is defined as

$${Gap}_{regime}=\left|{\overline{Saving}}_{high}-{\overline{Saving}}_{low}\right|$$

(10)

These measures capture predictive stability and decision-level stability as separate dimensions. To formally evaluate Hypothesis H1, a Mann–Kendall non-parametric trend test is applied to the monthly PR-AUC series; this test detects monotonic trends without assuming normality. To compare model discrimination performance, a Wilcoxon signed-rank test is conducted on paired monthly PR-AUC values across the gradient boosting and logistic regression models.

3.5. Unified Policy Stability Index

To integrate multiple dimensions of stability, a Unified Policy Stability Index is defined. The index combines predictive variability, economic variability, coverage stability, and regime sensitivity into a single measure:

$$UPSI={\displaystyle \frac{1}{4}}\left[\left(1-{CV}_{Alert}\right)+\left(1-{CV}_S\right)+\left(1-{CV}_{PR}\right)+(1-{\displaystyle \frac{{Gap}_{regime}}{\left|\overline{Saving}\right|+ϵ}})\right]$$

(11)

where $ϵ>0$ is a small constant that prevents division by zero.

UPSI follows the multi-criteria composite index approach (OECD 2008). Each component captures a distinct stability dimension and is normalized using the coefficient of variation, consistent with Dietterich (1998). Components are bounded to [0, 1] and receive equal weights, reflecting symmetric importance across dimensions. The index takes values in the interval [0, 1]. Values closer to one indicate higher overall policy stability. A high UPSI is associated with the following conditions:

• PR-AUC remains stable across time.
• Economic variability is limited.
• The coverage mechanism exhibits minimal fluctuation.
• Regime-related performance differences are small relative to overall economic returns.

The index synthesizes predictive and decision-level stability within a unified evaluative structure. Each component is computed as (1 − CV), clipped to [0, 1], and the four components receive equal weight, reflecting the assumption of symmetric institutional importance across stability dimensions. This construction follows the multi-criteria composite index tradition (OECD 2008) and builds on the variance-based stability criterion of Dietterich (1998). It provides a compact summary of multi-dimensional performance without reducing evaluation to a single discrimination metric.

4. Results

4.1. Rare-Event Structure and Baseline Conditions

Credit risk prediction is inherently an imbalanced classification problem. It is, therefore, necessary to describe the sample structure before examining model performance. The analysis relies on rolling temporal evaluation, and pooled training and test statistics are reported in Table 1.

Table 1. Rare-event structure and sample characteristics.

Dataset	N	Positives	Positive Rate (%)	Imbalance (1:X)
Train (pooled)	7966	1323	16.608	5.02
Test (pooled)	7220	810	11.219	7.91

The positive rate in the test period is 11.22%, compared with 16.61% in the training data. The default frequency declines over time. Under such conditions, random train–test splitting could lead to overly optimistic evaluation. The imbalance ratio increases from 5.02 in the training sample to 7.91 in the test sample. High-risk cases become relatively rarer in the evaluation period. Descriptive statistics by risk class are reported in Table 2. The elevated positive rate in the training period (16.61%) reflects the early portfolio phase during which originated loans had sufficient seasoning for risk classification to be confirmed, while the declining rate through August 2022 (3.25%) reflects a maturing portfolio entering more stable operating conditions. High-risk borrowers show significantly higher interest rates (mean: 9.53% vs. 7.33%, t = −85.8, and p < 0.001) and are younger on average (30.3 vs. 32.1 years, t = 8.4, and p < 0.001). The loan amount does not differ significantly between risk classes (p = 0.067). In this setting, the ROC-AUC alone may provide an incomplete assessment. For this reason, PR-AUC and economic saving are treated as primary performance measures. Because the rolling design captures monthly variation in default rates, subsequent results are reported at the month-by-month level.

Table 2. Descriptive statistics by risk class (Welch t-test).

Feature	N (Low Risk)	Mean	SD	N (High Risk)	Mean	SD	t-Stat	p-Value	Sig.
Age (years)	25,434	32.12	8.32	1387	30.33	7.66	8.448	<0.001	Yes
Loan amount (MNT)	25,434	209,228	329,497	1387	231,044	435,709	−1.836	0.067	No
Loan term (days)	25,434	40.64	14.99	1387	39.93	16.37	1.575	0.116	No
Interest rate (%)	25,434	7.33	1.72	1387	9.53	0.87	−85.794	<0.001	Yes
Penalty rate (%)	25,434	0.05	2.03	1387	0.00	0.05	3.999	<0.001	Yes

Note: Welch t-test (unequal variances). All features are constructed from loan application data at origination time, free of look-ahead bias. MNT = Mongolian Tögrög. Sig. = statistically significant at α = 0.05.

4.2. Month-by-Month Predictive and Economic Performance

The monthly performance under a fixed 20% alert policy is reported in Table 3. Because a rolling evaluation scheme is applied, each month is assessed separately.

Table 3. Rolling month-by-month predictive and economic performance (fixed 20% policy).

Test Month	Pos. Rate (%)	Alert Rate (%)	ROC_AUC	PR_AUC	Precision	Recall	F1	Saving
2022-01	15.29	20.06	0.611	0.284	0.246	0.323	0.279	215.0
2022-02	9.69	20.07	0.691	0.296	0.241	0.500	0.326	192.0
2022-03	19.87	20.00	0.641	0.426	0.351	0.353	0.352	432.0
2022-04	11.42	20.06	0.701	0.402	0.257	0.451	0.327	263.0
2022-05	22.20	20.00	0.671	0.408	0.386	0.348	0.366	412.0
2022-06	16.08	20.00	0.767	0.491	0.412	0.513	0.457	523.0
2022-07	8.04	20.03	0.811	0.406	0.231	0.575	0.330	407.0
2022-08	3.25	20.02	0.777	0.288	0.100	0.617	0.172	37.0

The ROC-AUC ranges from 0.611 to 0.811. The PR-AUC varies between 0.284 and 0.491. Performance is not uniform across months. This variation is consistent with changes in the underlying default environment. Months with higher default rates tend to generate higher economic returns. In June 2022, the PR-AUC reached its maximum value of 0.491, and saving was also highest at 523.0. This pattern suggests a positive association between predictive discrimination and economic benefit. In contrast, August 2022 exhibits the lowest positive rate at 3.25%. Both PR-AUC and saving decline in that month. The economic return drops sharply to 37.0. When defaults are rare, the marginal value of ranking appears to decrease.

The logistic regression benchmark achieves a mean PR-AUC of 0.400 and mean saving of 332.1 across the eight test months, compared with 0.375 and 310.1 for the gradient boosting model under the same 20% alert budget. A Wilcoxon signed-rank test does not detect a statistically significant difference (W = 1; p = 0.996), which is consistent with the limited statistical power associated with the small number of evaluation periods (n = 8). The gradient boosting model demonstrates greater robustness to categorical features without requiring preprocessing and provides interpretable feature importance estimates for each rolling window. The month-by-month comparison between the CatBoost and logistic regression models is presented in Table 4.

Table 4. Month-by-month model comparison: CatBoost vs. logistic regression (20% budget; λ = 10).

Month	Pos. Rate (%)	ROC-AUC GBM	PR-AUC GBM	Saving GBM	ROC-AUC LR	PR-AUC LR	Saving LR
2022-01	15.29	0.611	0.284	215.0	0.586	0.315	193.0
2022-02	9.69	0.691	0.296	192.0	0.710	0.347	181.0
2022-03	19.87	0.641	0.426	432.0	0.682	0.472	542.0
2022-04	11.42	0.701	0.402	263.0	0.750	0.407	296.0
2022-05	22.20	0.671	0.408	412.0	0.674	0.417	445.0
2022-06	16.08	0.767	0.491	523.0	0.782	0.496	468.0
2022-07	8.04	0.811	0.406	407.0	0.834	0.458	484.0
2022-08	3.25	0.777	0.288	37.0	0.816	0.286	48.0
Mean	13.24	0.709	0.375	310.1	0.729	0.400	332.1

Note: GBM = CatBoost gradient boosting; LR = logistic regression with balanced class weights. Both evaluated under identical rolling forward validation. Wilcoxon signed-rank test: PR-AUC W = 1 (p = 0.996); ROC-AUC W = 5 (p = 0.973).

Figure 1 illustrates the monthly trajectories of the PR-AUC and saving. Periods of lower PR-AUC tend to coincide with reduced economic return. This pattern indicates that predictive quality influences decision-level outcomes. The strength and stability of this relationship are examined further in the subsequent stability analysis.

Figure 1. Monthly predictive performance and economic saving (fixed 20%). The line with circular markers represents PR-AUC (left axis), while the line with square markers represents economic saving (right axis) across the monthly test periods.

Feature importance estimates from the gradient boosting model are computed for each rolling window. Age (mean importance: 17.4%; CV = 0.112), issue month (14.8%; CV = 0.088), and day of week (9.5%; CV = 0.086) are the dominant predictors. Twelve of fourteen features exhibit a coefficient of variation in importance below 0.30, classified as high stability. This indicates that the model’s predictive structure remains consistent across deployment periods, addressing the concern that variable weights may shift substantially under rolling evaluation (Figure 2).

Figure 2. Top-5 feature importance across rolling windows (CatBoost; 20% budget). Importance values are plotted for each of the eight monthly test windows. Age and issue month are consistently the dominant predictors across all periods.

The stability of feature importance across rolling windows is summarized in Table 5.

Table 5. Feature importance stability across rolling windows (CatBoost; 20% budget).

Feature	Mean Imp. (%)	SD	CV	Stability
Age (Нас)	17.36	1.94	0.112	High (CV < 0.30)
Issue month	14.75	1.30	0.088	High
Day of week	9.51	0.82	0.086	High
Log loan amount	8.70	0.49	0.056	High
Loan amount	8.68	0.48	0.055	High
Interest rate (%)	8.12	1.23	0.152	High
Prior loan count	7.00	0.39	0.056	High
Loan term	6.91	0.56	0.081	High
Gender (Хүйс)	6.28	1.04	0.166	High
Prior default rate	3.53	0.51	0.143	High
Any prior default	3.42	0.63	0.185	High
Loan product type	2.95	0.89	0.300	Medium
Issue quarter	2.69	0.58	0.216	High
Penalty rate (%)	0.09	0.04	0.413	Medium

Note: CV = coefficient of variation across 8 rolling windows. High stability: CV < 0.30; medium: 0.30–0.60. In total, 12 of 14 features (86%) are classified as high stability. Features such as age and gender may be restricted in certain jurisdictions due to anti-discrimination regulations and are included for analytical purposes only.

4.3. Stability of the Fixed 20% Policy

Monthly results are aggregated to evaluate policy stability. The stability of the fixed 20% alert policy is summarized in Table 6. The mean, standard deviation, and coefficient of variation are computed for both the alert rate and economic savings.

Table 6. Stability of the fixed 20% alert policy.

Months	Mean Alert Rate	SD Alert Rate	CV Alert Rate	Mean Saving	SD Saving	CV Saving
8	0.2003	0.0003	0.001	310.12	160.25	0.517

The coefficient of variation for the alert rate is approximately 0.001. This indicates that the coverage mechanism is implemented consistently across months. The threshold structure does not introduce mechanical instability. The mean saving equals 310.12, with a standard deviation of 160.25. The resulting coefficient of variation is 0.517. Economic returns fluctuate across months. This variation likely reflects changes in the underlying default rate.

A Mann–Kendall trend test applied to the monthly PR-AUC series yields Kendall’s τ = 0.214 (S = 6; p = 0.536 (two-sided)), providing no statistical evidence of a monotonic trend over the eight-month evaluation horizon. Hypothesis H1 is, therefore, formally supported. The results of the statistical significance tests are presented in Table 7.

Table 7. Statistical significance tests.

Test	Metric	Statistic	p-Value	Sig. (α = 0.05)
Mann–Kendall trend	PR-AUC (GBM)	S = 6, τ = 0.214, Z = 0.619	0.536 (two-sided)	No
Wilcoxon signed-rank	PR-AUC: GBM > LR	W = 1	0.996 (two-sided)	No
Wilcoxon signed-rank	ROC-AUC: GBM > LR	W = 5	0.973 (two-sided)	No

Note: Mann–Kendall test is non-parametric and distribution-free. Wilcoxon signed-rank test applied to n = 8 paired monthly values. Due to the small sample size (n = 8), statistical power is limited, and results should be interpreted with caution.

Importantly, saving remains positive in all months. The fixed 20% alert policy does not generate aggregate losses under rolling evaluation. While economic variability is observed, there is no evidence of structural instability in the policy mechanism. These results suggest that a 20% coverage policy is operationally feasible under forward temporal conditions. The next section examines the influence of regime differences and cost parameter variation in greater detail.

4.4. Stability Across Risk Regimes and Cost Structures

This section evaluates performance under varying default environments and alternative false-negative cost parameters. The objective is to assess whether decision utility remains stable when economic conditions and cost assumptions change.

4.4.1. Performance Across Default Regimes

Test months are divided into high-default and low-default groups based on the median monthly positive rate. The performance across default regimes is summarized in Table 8.

Table 8. Stability across default regimes.

Regime	Months	AvgPosRate	Mean PR_AUC	SD PR_AUC	Mean Saving	SD Saving
High default	4	0.1836	0.402	0.087	395.50	129.67
Low default	4	0.0810	0.348	0.065	224.75	153.84

The mean PR-AUC equals 0.402 in high-default months and 0.348 in low-default months. The difference of 0.054 suggests slightly improved discrimination when the overall risk is elevated. The economic return is higher in high-default periods. The mean saving reaches 395.50 compared with 224.75 in low-default months. The gap equals 170.75. When defaults are more frequent, the alert policy generates greater economic benefit. Importantly, saving remains positive even in low-default environments. The model does not generate losses under normal conditions. No structural deterioration is observed. This supports regime robustness.

4.4.2. Cost Sensitivity Analysis

The false-negative cost parameter is varied between 5 and 25. The cost sensitivity analysis results are summarized in Table 9.

Table 9. Cost sensitivity stability (lambda grid).

Lambda_FN	Months	Mean_PR_AUC	Mean_Saving	SD_Saving	CV_Saving
5	8	0.375	155.06	80.12	0.517
10	8	0.375	310.13	160.25	0.517
15	8	0.375	465.19	240.37	0.517
20	8	0.375	620.25	320.49	0.517
25	8	0.375	775.31	400.62	0.517

The PR-AUC remains unchanged across values of the cost parameter. Predictive discrimination is, therefore, independent of $\mathsf{\lambda }$. Saving increases proportionally with $\mathsf{\lambda }$. Both the mean and standard deviation scale linearly, which explains why the coefficient of variation remains constant. The cost parameter rescales economic magnitude without altering risk ranking. These findings are consistent with H2. Cost variation affects economic scale but does not modify predictive structure.

4.4.3. Interaction Between Regime and Cost

The joint effect of regime and cost parameter is summarized in Table 10.

Table 10. Regime × lambda interaction.

Lambda_FN	High-Default Mean Saving	Low-Default Mean Saving
5	197.75	112.38
10	395.5	224.75
15	593.25	337.13
20	791.00	449.50
25	988.75	561.88

Saving grows more rapidly with $\mathsf{\lambda }$ in high-default regimes. In low-default environments, saving remains positive but increases more moderately. The regime gap widens as $\mathsf{\lambda }$ increases. Under higher cost assumptions, stress environments generate larger marginal returns. However, saving does not become negative under any configuration. No structural breakdown is detected. The decision framework remains economically viable across regimes and cost scenarios.

4.5. Budget Sensitivity and Integrated Stability Assessment

4.5.1. Budget Sensitivity

Policy stability is evaluated under alternative alert budgets of 5%, 10%, and 20%. The predictive and economic stability under alternative alert budgets is summarized in Table 11.

Table 11. Predictive and economic stability under alternative fixed alert budgets.

Budget	Months	Mean Precision	Mean Recall	Mean Saving	CV Saving
Top 5%	8	0.811	0.915	293.75	0.189
Top 10%	8	0.462	1.000	291.25	0.416
Top 20%	8	0.229	1.000	200.75	0.783

As coverage increases, recall rises, while precision declines. This pattern reflects the higher incidence of false positives at larger alert budgets. Mean saving is similar at 5% and 10% coverage. At 20%, saving declines. The additional false-positive cost outweighs the marginal benefit of higher recall. Economic variability increases with coverage. The coefficient of variation rises from 0.189 at 5% to 0.783 at 20%. Lower coverage, therefore, appears to generate more stable economic outcomes.

4.5.2. Unified Policy Stability Index

To summarize policy stability across multiple dimensions, the Unified Policy Stability Index is computed. The index incorporates coverage stability, economic variability, predictive variability, and regime sensitivity. The results for the baseline configuration (λ = 10; 20% coverage) are presented in Table 12.

Table 12. Unified Policy Stability Index.

CV_AlertRate	CV_Saving	SD_PR_AUC	Regime_Gap	UPSI
0.001	0.517	0.076	170.75	68.18

The coefficient of variation for the alert rate equals 0.001, indicating that coverage implementation is mechanically stable across months. The normalized alert rate component contributes 0.999 to the index, reflecting near-perfect coverage consistency. The PR-AUC component contributes 0.796 (CV_PR = 0.204), reflecting moderate predictive fluctuation. The economic saving component contributes 0.483 (CV_S = 0.517), and the regime component contributes 0.449 (normalized gap = 0.551). The four components average to UPSI = 68.18/100. The detailed breakdown of the Unified Policy Stability Index is presented in Table 13. The standard deviation of PR-AUC is 0.076, suggesting moderate predictive fluctuation without abrupt deterioration. The coefficient of variation for saving equals 0.517. Economic return is, therefore, more sensitive to environmental variation than predictive ranking.

Table 13. UPSI component breakdown (Paper Formula, Equation (11)).

Component	Raw Value	Normalized (1 − CV)	Contribution (×25)
CV AlertRate	0.0015	0.9985	24.96
CV Saving	0.5167	0.4833	12.08
CV PR-AUC	0.2039	0.7961	19.90
Regime Gap/|Mean Saving|	0.5506	0.4494	11.24
UPSI (0–100)	---	---	68.18

Note: UPSI = (1/4) [(1 − CV_Alert) + (1 − CV_S) + (1 − CV_PR) + (1 − Gap/|Saving|)]. Regime gap = 170.75 (high = 395.50; low = 224.75). Mean saving = 310.12. Each component is clipped to [0, 1].

The regime gap equals 170.75, reflecting differences in economic benefit between high- and low-default periods. When scaled relative to average saving, this gap does not indicate structural instability. The computed UPSI equals 68.18 on a 0–100 scale (0.68 on a 0–1 scale), indicating moderate overall policy stability. When predictive and decision dimensions are evaluated jointly, the policy maintains structural consistency under temporal, cost, and regime variations. The weight sensitivity analysis of the UPSI is presented in Table 14.

Table 14. UPSI weight sensitivity analysis.

Weight Scheme	w_Alert	w_Saving	w_PR_AUC	w_Regime	UPSI (0–100)	Δ from Baseline
Equal (baseline)	0.25	0.25	0.25	0.25	80.79	0.00
Saving-heavy	0.10	0.50	0.20	0.20	74.43	−6.36
PR-AUC-heavy	0.10	0.20	0.50	0.20	82.52	+1.73
Regime-heavy	0.10	0.20	0.20	0.50	74.00	−6.79
Alert-heavy	0.50	0.20	0.15	0.15	86.72	+5.93
Discriminative-focused	0.15	0.15	0.50	0.20	84.21	+3.42
Economic-focused	0.15	0.50	0.15	0.20	74.78	−6.01

Note: UPSI range across all weight schemes: 12.72 points. Maximum deviation from equal-weight baseline: 6.79 points. A narrow range indicates UPSI is robust to weight specification.

To evaluate whether the composite index captures information beyond any single component, the UPSI is benchmarked against three simpler alternatives: mean PR-AUC, saving stability (1 − CV_S), and PR-AUC stability (1 − CV_PR). While each individual metric captures one dimension of performance, none jointly accounts for predictive variability, economic volatility, and regime differences. The UPSI integrates these dimensions into a single evaluative construct, providing a richer characterization of deployment stability than any univariate measure alone.

5. Discussion

The findings indicate that credit risk models should be assessed beyond predictive discrimination alone. Decision outcomes, temporal consistency, and regime sensitivity require joint consideration. The results align with decision-theoretic reasoning and principles of temporal robustness. Unlike prior rolling regression studies that focus primarily on goodness-of-fit, interpretability, or calibration (e.g., Moiseev et al. 2021; Wang and Xiao 2025), this study integrates cost-sensitive economic evaluation with temporal validation and introduces the UPSI as a composite stability measure, a combination that has received limited attention in the credit risk evaluation literature.

H1 examined whether PR-AUC exhibits a systematic decline under rolling temporal validation. Monthly values fluctuate, yet no downward trend is observed. Variability remains moderate. The rolling design, therefore, provides a realistic evaluation framework. The evidence does not imply immunity to concept drift. However, no abrupt deterioration is detected during the observed period. A Mann–Kendall trend test confirms the absence of a statistically significant monotonic trend (τ = 0.214; p = 0.536). H1 is, therefore, supported. H2 evaluated whether variation in the cost parameter alters predictive quality. The PR-AUC remains stable across values of λ. Saving increases proportionally with λ. The cost parameter rescales economic magnitude but does not affect ranking structure. This distinction clarifies the separation between predictive performance and decision utility. H2 is supported.

H3 addressed regime differences. In high-default periods, both PR-AUC and savings are higher on average. When baseline risk rises, ranking carries greater economic value. This pattern is consistent with decision-theoretic expectations. In low-default periods, saving remains positive. The model does not operate only under stress conditions. H3 is supported. As coverage increases, recall improves while precision declines. At 20% coverage, economic return decreases, and variability rises. False-positive cost expands more rapidly at higher alert budgets. The Unified Policy Stability Index integrates predictive variability, economic variability, and regime differences. The index yields 68.18/100, indicating moderate overall stability. No structural instability emerges under alternative coverage settings. H4 is supported.

The contribution is fourfold. First, predictive discrimination and decision utility are treated as distinct dimensions. Second, regime robustness is evaluated using economic measures rather than relying solely on classification metrics. Third, multi-dimensional stability is summarized through a unified index. Fourth, a benchmark comparison with logistic regression is conducted using formal statistical testing, providing evidence of the marginal value of gradient boosting over a linear baseline in this deployment context. The coverage-based alert policy appears operationally feasible. Lower coverage levels generate more stable economic outcomes. Adjustment of the cost parameter may serve strategic objectives, yet it does not alter the underlying ranking.

Several limitations remain. The most significant concerns external validity. The analysis is based on a single market dataset, specifically a retail consumer lending portfolio in Mongolia, which limits generalizability across institutional settings, credit product types, and macroeconomic environments. The framework has not been validated on datasets from other countries or lending segments, and the reported UPSI value reflects conditions specific to this portfolio during the 2020–2023 period. Replication across multiple markets and credit cycle phases represents the most important direction for future research. The cost function Saving(λ,c) = λ·TP(c) − FP(c) adopts a simplified linear specification following Verbraken et al. (2014); extension to borrower-specific or portfolio-level cost structures represents an important direction for future research. The benchmark comparison is limited to logistic regression; future work should incorporate additional models such as random forests, XGBoost, and neural networks, as well as adaptive algorithms designed for drift environments. Calibration stability and uncertainty assessment also warrant further investigation. Finally, the statistical power of the Wilcoxon signed-rank test is constrained by the eight-month evaluation horizon; a longer deployment window would enable a more conclusive model comparison.

6. Conclusions

This study evaluates credit risk models beyond predictive discrimination alone. Decision utility, temporal consistency, and regime sensitivity are considered jointly. Predictive accuracy remains important, yet it does not fully represent policy-level effectiveness.

The use of rolling temporal validation avoids optimistic bias associated with static random splits. Monthly PR-AUC does not exhibit a systematic downward trend. Temporal consistency is maintained over the observed period. This does not imply the absence of future drift. It indicates stability within the evaluation horizon. Under a cost-sensitive framework, varying the false-negative parameter across a wide range does not alter PR-AUC. Economic return scales proportionally with the cost parameter. Predictive ranking and economic magnitude, therefore, represent distinct dimensions.

Regime-based analysis shows higher economic value during high-default periods. In low-default environments, performance remains positive. No structural deterioration is observed. The model does not appear limited to stress conditions. Budget sensitivity reveals a clear trade-off between precision and recall. Higher coverage increases the cost of false positives and reduces economic return. The Unified Policy Stability Index integrates predictive variability, economic variability, and regime differences. The index yields 68.18/100, indicating moderate overall policy stability. No systematic instability is detected during the study period

From a theoretical perspective, this study integrates decision alignment, temporal robustness, and regime awareness within a unified framework. Methodologically, it combines rolling-forward validation, cost–parameter sensitivity analysis, and coverage-based policy design. From a practical standpoint, lower coverage levels may generate more stable economic outcomes under resource constraints. As noted, external validity remains the primary limitation; the framework requires replication across diverse markets before broader generalization is warranted. Future research should validate the framework across multiple lending markets and macroeconomic cycles, compare adaptive drift-aware algorithms, examine calibration and uncertainty stability, incorporate borrower-specific cost specifications, and extend the benchmark comparison to include XGBoost, random forest, and neural network classifiers.

Overall, credit risk model evaluation should extend beyond accuracy-centered metrics. Deployment-oriented, decision-consistent, and stability-aware assessment appears necessary for applied machine learning contexts. Unlike existing rolling regression approaches that focus on goodness-of-fit or interpretability in isolation, the proposed framework explicitly integrates economic decision outcomes, regime sensitivity, and a composite stability index, offering a richer evaluation paradigm for operational credit risk modeling. The framework presented here contributes to that direction.