PC-Audit: A Decision-Reliability Framework for Auditing Validation-Based Machine-Learning Model Selection in Weak-Signal Financial Time Series

Featured Application

PC-Audit provides a reproducible decision-reliability record for validation-selected models in weak-signal financial time series. It distinguishes selection-time evidence from completed-period audit and next-cycle governance, helping risk and model-governance teams document when a numerical validation winner should be interpreted cautiously rather than treated as deployment authorization.

Abstract

Validation-based machine-learning selection can name a numerical winner in temporally dependent weak-signal data without establishing that the decision will remain reliable in a later period. We present PC-Audit (Prequential-Calibration Audit) as a decision-reliability architecture—not a new forecasting model or accuracy-enhancement method—that records the evidence surrounding a validation-selected candidate. It combines target-date partitioning, prequential non-negative calibration, Model Confidence Sets (MCS), transfer and refit diagnostics, dependence sensitivity, and forward-only governance records. In 45 walk-forward cases for nine Chinese index ETFs (2020–2024), signed-return validation ranking transfers weakly (mean rank correlation 0.090), PC-Select achieves the lowest test MAE in 15 of 45 cases, and the prequential MCS retains 4.80/5 candidates on average. L1 calibration sensitivity, alternative validation horizons, and dependence sensitivity do not support a strong signed-return selection claim. Base-model refitting adds ranking instability to already weak validation-to-future transfer. A frozen controlled simulation shows that the protocol is not mechanically abstention-only under clearly separable stationary oracle risk, while an unobserved future reversal remains a prospective limitation. Daily information-aligned econometric comparators and QLIKE analyses show that audit evidence is target- and loss-dependent: squared-return results can improve under QLIKE-aligned selection, whereas absolute-return and range-proxy results do not improve uniformly. Fallback uncertainty intervals cross zero. PC-Audit therefore supports transparent qualification of validation-selected candidates and completed-period audit/next-cycle governance, rather than deployment authorization or claimed predictive improvement.

1. Introduction

Validation-based machine-learning model selection is a routine step in deployment: several candidate models are trained on historical data, one is chosen using a finite validation interval, and the selected candidate is then carried forward to a later operating period. In temporally dependent and weak-signal settings, however, the existence of a numerical validation winner does not by itself establish that the choice is reliable. Candidate errors may be close, observations are serially dependent, regimes can change, and a short validation segment may appear decisive even when it contains little information about the next period.

Financial time series provide a particularly demanding setting for this problem. Daily index-ETF returns have low signal-to-noise ratios, broad-market information is incorporated quickly, and small apparent performance differences can be sensitive to calibration, execution assumptions, and market regime. These features make financial prediction a useful stress test for a broader evaluation question: whether the evidence behind a validation-selected machine-learning action is strong enough to support a deployment interpretation.

Most empirical studies naturally emphasize predictive accuracy, model architecture, hyperparameter choice, or trading outcomes. Considerably less attention is given to the reliability of the selection act itself: whether the model chosen on validation data is statistically distinguishable from alternatives, whether the ordering transfers out of sample, and whether the resulting interpretation survives dependence, multiplicity, cost, execution, and temporal stress. Without such checks, a routine validation winner can be overinterpreted as stable deployment evidence.

This paper addresses that narrower methodological gap. The aim is not to search for a universally strongest predictor or to claim an investable ETF timing rule, but to construct an auditable framework that determines how strongly a validation-selected candidate can be interpreted. The central idea is to treat model selection as a decision object: the selected model name is reported with selection-time calibration and confidence-set evidence, while validation-to-test transfer, regret, cost/execution stress, and temporal-holdout behavior are completed-period audit evidence that can inform the next review cycle. These stages have different information sets and are not merged into a putative deployment authorization.

Time-aware partitioning is necessary but not sufficient. A next-day target must be assigned to a split according to the date of the predicted return, not only the date of the feature row. Prediction-scale mismatch also complicates model comparison: a recurrent or tree model may have useful ordering information but an unstable amplitude. Calibration can correct amplitude, yet fitting and scoring a calibration coefficient on the same validation outcomes creates optimistic losses. Finally, selecting the numerically lowest validation error hides uncertainty when several candidates are statistically indistinguishable.

To address these issues, PC-Audit (Prequential-Calibration Audit) evaluates validation-driven model selection as a decision-reliability protocol. The framework combines target-date partitioning, chronological prequential calibration, a moving-block-bootstrap Model Confidence Set (MCS), completed-period transfer diagnostics, dependence sensitivity, transaction-cost and execution checks, computational-cost reporting, and a retrospective 2025 temporal holdout. PC-Select is the operational rule that identifies one candidate using the audit-defined validation loss; it is not an additional forecasting model. The remaining components quantify the uncertainty and practical limits surrounding that action. Thus, the framework is useful when evidence is insufficient for a strong interpretation.

The framework imposes three requirements. First, the selected action must arise from a leakage-controlled validation rule. Second, selection-time uncertainty and later transfer evidence must be reported without confusing their information sets. Third, any economic or cost-related interpretation must remain conditional on multiplicity, dependence, execution, sensitivity, and temporal-stress evidence. These requirements turn model selection into a falsifiable decision protocol: the evidence may support, qualify, or reject an interpretation of the validation-selected candidate. PC-Audit contributes a decision-reliability architecture for validation-based model selection. It combines leakage-controlled selection, prequential calibration, uncertainty-set assessment, completed-period audit, and forward-only governance within a common information boundary. Its novelty lies in the ordering, traceability, and explicit separation of these established components around the validation-selected action; no individual calibration, bootstrap, or confidence-set procedure is claimed as new.

The study answers three audit questions:

• Does a leakage-controlled validation rule provide sufficient signed-return evidence when confidence-set uncertainty, selection margin, later rank transfer, hit rate, regret, and competing baselines are jointly audited?
• Are volatility-like risk-surveillance targets more auditable than signed returns under the same protocol, and do they remain bounded under strong rolling/EWMA references and 2025 stress?
• Can completed audit outputs be carried forward as monitoring variables and fallback actions for the next deployment review without turning the benchmark into an online learner?

The contribution is threefold. First, the paper treats the validation-selected candidate as an explicit audit object and integrates calibration timing, MCS uncertainty, margins, transfer diagnostics, dependence sensitivity, and practical stress checks into one reproducible information-bounded protocol. Second, it documents that signed-return selection evidence is weak in this panel, whereas target-asymmetry evidence is conditional on the target, loss, comparator pool, and stress period. Third, it distinguishes validation-time selection from completed-period audit and from forward-only next-cycle governance. The latter is an organizational/traceability contribution, not an empirical claim of a more accurate fallback or of deployment authorization. Raw snapshots, features, prediction caches, scripts, intermediate tables, and file hashes are supplied for verification.

Table 1 summarizes how the design addresses common evaluation risks in weak-signal time-series machine learning. The table is included to make the claim boundary explicit: each protocol component either removes a possible source of optimism or defines when a positive interpretation is not warranted.

Table 1. Audit components and their corresponding risk-control objectives.

Evaluation Risk	Protocol Response	Interpretation Consequence
Target leakage from next-day returns	Split membership is assigned by target date, not feature-row date	Train, validation, test, and 2025 stress outcomes are chronologically separated
Optimistic calibration	Every scored validation block uses a coefficient fitted only on earlier validation blocks	Validation losses used for selection do not reuse the same labels for fitting and scoring
Overreading a validation winner	PC-Select is reported with MCS size, candidate-pool minimum coverage, rank transfer, and regret	A single selected model is an action, not proof of statistical separation
Multiple testing and dependence	Holm-adjusted tests are supplemented with ETF-cluster and year-block intervals	Positive claims require both multiplicity-adjusted and dependence-aware support
Trading-performance overclaiming	Cost, turnover, one-day execution delay, sensitivity variants, and 2025 stress results are reported	Economic metrics are interpreted as diagnostics unless signs remain consistent

2. Related Work

Machine learning has been applied extensively to financial time-series prediction, from regularized linear models and tree ensembles to recurrent and deep architectures [1,2,3,4]. Recent benchmark studies also examine larger time-series and foundation-model families [5,6]. These studies establish the candidate-model background for this work, but architectural breadth does not resolve the evaluation problem created by validation-based selection. PC-Audit therefore focuses on the reliability of the decision produced by model selection rather than on proposing another forecasting architecture.

Evaluation protocol is a central issue for temporally dependent data. Random cross-validation can mix temporal information and misrepresent deployment conditions, whereas rolling-origin or walk-forward evaluation preserves the forward-looking structure of the task [7,8,9]. Leakage-controlled financial benchmarks further emphasize train-only preprocessing, causal feature construction, strong naive references, and conservative conclusions when signed-return prediction remains close to baseline performance [10]. PC-Audit extends this evaluation logic by asking whether the deployment interpretation created by validation-based selection is supported by uncertainty, transfer, and stress evidence. Its audit object is the selection decision itself, not only the forecast error of each candidate.

A second stream studies model-selection risk. Even when several models are reasonable on average, a short validation segment may have low probability of identifying the best next-period forecaster [11]. This problem is related to backtest overfitting, research-protocol discipline, and multiple-testing risk in financial research, where repeated searching can create attractive but fragile results [12,13,14,15]. The present study therefore reports top-one hit rate, validation-to-test rank correlation, selection regret, and permutation evidence rather than treating the lowest validation MAE as self-justifying.

Uncertainty-aware model comparison provides a complementary foundation. The Model Confidence Set retains models that cannot be rejected as inferior under a specified loss and bootstrap scheme [16]. Its financial forecasting applications show why confidence-set reporting can be more informative than simply naming a best model when the data cannot sharply separate alternatives [17]. PC-Audit uses this property as an audit component: MCS membership constrains the strength of a deployment interpretation, but it is not treated as a forecasting model and does not imply economic profitability.

Resource-aware forecasting and multi-objective model selection have also been studied for time-series systems [18]. PC-Audit is related but distinct. It does not optimize a combined accuracy-cost score or search for a new model family. Instead, it evaluates a fixed candidate pool in a controlled CPU-only setting and asks whether the validation-selected action remains defensible after calibration timing, statistical uncertainty, transfer diagnostics, costs, execution delay, sensitivity variants, and temporal stress are audited.

Related work on Machine Learning Operations (MLOps) and ML lifecycle management broadens the evaluation perspective from model development to reproducible deployment, monitoring, versioning, and feedback across the operational lifecycle [19,20]. PC-Audit is complementary to this literature but targets a narrower upstream decision: before a validation-selected candidate is interpreted as deployable, how strong is the evidence supporting that selection? Its audit-state mechanism records completed evidence for the next scheduled review while preserving the information boundary of the current test period. The framework does not replace end-to-end MLOps infrastructure, continuous integration, or production monitoring; it supplies a prespecified model-selection audit layer that can be embedded within such lifecycle processes.

The remaining gap is therefore not the absence of forecasting architectures or individual statistical tools. Existing work provides time-aware evaluation methods, model-selection diagnostics, MCS uncertainty procedures, and warnings about overfitting. PC-Audit does not claim to replace them: it makes their distinct roles explicit around a validation-selected action, records conflicting evidence as mixed audit evidence rather than collapsing it into a scalar index, and separates selection-time information from completed-period audit and next-cycle governance.

3. Materials and Methods

Figure 1 summarizes the order in which information enters the audit. It separates model fitting and validation selection from completed-period evidence used to judge that selection, and it shows that completed test outcomes are carried forward only as governance inputs for the next review cycle. Algorithm 1 states the case-level protocol used for each ETF-window split. The 2025 split repeats the same fixed procedure as a retrospective temporal holdout.

Algorithm 1 PC-Audit for one ETF-window case.

1: Assign train, validation, and test membership by target date. 2: Train the fixed CPU-only candidate pool on the training window. 3: Generate validation and test prediction caches without test-year tuning. 4: Divide the validation year into chronological blocks. 5: Estimate non-negative calibration slopes only from earlier validation blocks. 6: Compute prequential validation losses for scored validation blocks. 7: From the same prequential loss matrix, independently select PC-Select and estimate the moving-block-bootstrap MCS uncertainty set. 8: Refit the selected base model on train plus validation and re-estimate its non-negative scalar on the full validation year. 9: Evaluate the selected model on the next test year. 10: Report transfer, regret, MCS coverage, paired inference, and stress checks. 11: After the test year ends, log audit-state variables for the next review.

Figure 1. Methodological workflow of PC-Audit. Validation-time information passes through target-date partitioning, fixed candidate models, prequential calibration, validation selection, and uncertainty assessment. Later test outcomes are completed-period audit evidence, not input to the prior selection. The 2025 split is a retrospective temporal holdout rather than an independent prospective test. Completed audit outcomes can affect only the next review cycle. Solid arrows denote within-cycle information flow; the dashed arrow denotes feedback available only for the next review cycle; the dashed-outline box marks the interpretation boundary; and gray shading is used only for visual grouping and carries no additional semantic meaning.

3.1. Audit Design, Data Provenance, and Target-Date Partitioning

PC-Audit is organized as a fixed decision-reliability protocol for validation-based model selection. The dataset is treated as an immutable benchmark input so that the Section 3 can focus on the audit logic. Although the public data vendor facilitates reproduction, it is not the exchange’s official historical database [21]. To prevent later vendor revisions from changing the experiment, the package stores the exact snapshots and file hashes. This supports computational replication while leaving one-vendor dependence and vendor-revision risk as limitations. We do not claim that the dataset is survivorship-free.

Throughout the paper, t indexes the date on which a target is realized. Let $P_t$ denote the ETF closing price on trading date t. The close-to-close return realized on that date is

$$r_t={\displaystyle \frac{P_t-P_{t-1}}{P_{t-1}}}.$$

(1)

At the preceding forecast origin $t-1$, the future price $P_t$ and hence $r_t$ are not yet observed; they become available only when the target is labeled for evaluation. Let ${\mathbf{x}}_{t-1}$ denote the feature vector constructed solely from information observed no later than date $t-1$. It contains lagged returns, rolling moments, momentum, price ranges, volume changes, moving-average ratios, and the technical indicators listed in the configuration file. Missing values introduced by rolling windows are removed deterministically.

Let $y_t$ denote the target realized on date t. For the primary signed-return audit, $y_t=r_t$. Candidate model m maps the lagged feature vector to a one-step-ahead prediction,

$${\widehat{y}}_{m,t}=f_m\left({\mathbf{x}}_{t-1}\right),$$

(2)

where $f_m$ is the prediction function fitted for candidate m and ${\widehat{y}}_{m,t}$ is its raw prediction of $y_t$. Thus, no quantity dated t is used as an input when predicting the target at date t.

The signed-return target remains the primary audit case because it corresponds to the common deployment question of whether a validation-selected model supports a directional interpretation. As a risk-surveillance extension, the same fixed pipeline is also run on three volatility-like targets:

$$v_t=\left|r_t\right|,s_t=r_t^2,q_t=log\left({\displaystyle \frac{H_t}{L_t}}\right),$$

(3)

where $v_t$ is absolute return, $s_t$ is squared return, $q_t$ is a high–low range proxy, and $H_t$ and $L_t$ are respectively the high and low prices on target date t. For each extension, $y_t$ is replaced by the corresponding target. These targets are evaluated with prediction-scale losses, rank transfer, MCS size, and selection regret. They are not used for directional accuracy, transaction-cost signal returns, or trading-performance claims. In the robustness scripts, target-derived columns are explicitly excluded from ${\mathbf{x}}_{t-1}$ before model fitting, so the alternative targets remain prediction targets rather than predictors.

Each observation stores both its feature date $t-1$ and target date t. Split membership is assigned by the target date, preventing a final training feature row from carrying a validation-period outcome and a final validation row from carrying a test-period outcome. Formally,

$$({\mathbf{x}}_{t-1},y_t)\in \left\{\begin{array}{cc}{\mathcal{D}}_{\mathrm{train}},\hfill & t\le T_{\mathrm{train}},\hfill \\ {\mathcal{D}}_{\mathrm{val}},\hfill & T_{\mathrm{train}}<t\le T_{\mathrm{val}},\hfill \\ {\mathcal{D}}_{\mathrm{test}},\hfill & t>T_{\mathrm{val}},\hfill \end{array}\right.$$

(4)

where ${\mathcal{D}}_{\mathrm{train}}$, ${\mathcal{D}}_{\mathrm{val}}$, and ${\mathcal{D}}_{\mathrm{test}}$ are the training, validation, and test sets, while $T_{\mathrm{train}}$ and $T_{\mathrm{val}}$ are the target-date cutoffs ending the training and validation windows. This target-date convention is used in all subsequent equations.

The outer design uses five years for training, one year for validation, and one year for testing. Test years 2020–2024 produce $9\times 5=45$ ETF-window cases. A sixth fixed split trains on 2019–2023, validates on 2024, and tests on 2025. Because its observations were already available during manuscript development, the 2025 split is reported only as a retrospective temporal holdout/stress assessment, not as an independent market, vendor, or prospective sample.

Auditability is operationalized as a multidimensional evidence profile, not a scalar score or a validated classifier. The profile includes validation separation and MCS membership, calibration stability, validation-to-future rank transfer, top-one hit/regret, baseline competitiveness, dependence sensitivity, and temporal stress. These indicators can conflict. For example, a target can have favorable rank transfer but weak top-one performance or competitive historical references; this is reported as mixed audit evidence rather than averaged into an overall “auditable” label.

Table 2 records the information boundary that separates validation-time selection from completed-period audit and next-cycle governance.

Table 2. Information boundary of the PC-Audit protocol.

Stage	Available Information and Action	Prohibited Interpretation
Validation-time selection	Training/validation observations, fixed models, prequential calibration, validation loss, MCS, margin, and chronological block agreement select a candidate and characterize validation uncertainty.	Test regret, future rank transfer, stress outcomes, and post-period audit state are unavailable and cannot authorize deployment.
Completed-period audit	The completed test period supplies transfer, hit, regret, baselines, dependence sensitivity, and stress diagnostics.	These post-period diagnostics cannot retrospectively change the prior selection or prove a prospective filter.
Next-cycle governance	Completed-period records may trigger a scheduled review or illustrative fallback mapping for the subsequent cycle.	This retrospective policy simulation does not establish online learning, performance improvement, or validated deployment authorization.

3.2. Candidate Models and Prequential Calibration

The candidate pool contains ElasticNet, Ridge, LightGBM, Random Forest, and a small GRU [22,23,24,25,26]. Hyperparameters are fixed in model_configs.json and are not retuned on test years. The GRU receives causal 10-trading-day sequences; validation and test sequences use only preceding context. All models run on CPU.

The pool represents common low-cost prediction models with different inductive biases: regularized linear models, tree ensembles, boosting, and a compact recurrent network. It is not an exhaustive architecture benchmark. The object of study is the audited selection protocol, not a new model architecture or a search for the globally strongest predictor. To avoid attributing weak separation only to the use of individual models, the extended analysis also includes simple historical-prediction and prediction-aggregation references. Larger alternatives such as transformer-style sequence models, additional boosting variants, or broad hyperparameter searches are left outside the default pool because they would change the study from a CPU-only deployment audit into a model-zoo search and would increase selection and multiple-testing risk.

Let $\mathcal{M}=\{1,\dots ,M\}$ denote the fixed candidate-model pool, where $M=5$. For target date t, ${\widehat{y}}_{m,t}$ is the raw prediction of candidate $m\in \mathcal{M}$. To address prediction-scale mismatch, calibration uses a non-negative scalar estimated from an explicit information set. For a calibration index set $\mathcal{C}$, define

$${\widehat{a}}_m\left(\mathcal{C}\right)=arg\underset{a\ge 0}{min}\sum _{t\in \mathcal{C}}{(y_t-a{\widehat{y}}_{m,t})}^2=max\left\{0,{\displaystyle \frac{{\sum }_{t\in \mathcal{C}}{\widehat{y}}_{m,t}{y}_t}{{\sum }_{t\in \mathcal{C}}{\widehat{y}}_{m,t}^2}}\right\}.$$

(5)

The set $\mathcal{C}$ is the slope-estimation information set. Non-negativity permits shrinkage to zero, but no arbitrary upper cap; a zero slope means that the available sample supports the zero forecast among non-negative scalar multiples. The numerical tolerance ${10}^{-15}$ matches the supplied implementation.

Using the same validation outcomes both to estimate ${\widehat{a}}_m$ and score calibrated validation predictions would be optimistic. PC-Audit therefore follows a prequential evaluation logic in which predictions are scored only after the information used for calibration is available [27]. It divides the validation year into four contiguous blocks ${\mathcal{V}}_1,{\mathcal{V}}_2,{\mathcal{V}}_3,{\mathcal{V}}_4$. Block 1 is a warm-up. For each scored block $b\in \{2,3,4\}$, the available calibration history is

$${\mathcal{H}}_b={\displaystyle \bigcup _{j=1}^{b-1}}{\mathcal{V}}_j,$$

(6)

and the block-specific prequential slope is

$${\widehat{a}}_{m,b}={\widehat{a}}_m\left({\mathcal{H}}_b\right).$$

(7)

Thus, ${\widehat{a}}_{m,b}$ is estimated only from validation labels that occur before block b, and the corresponding validation prediction is ${\tilde{y}}_{m,t}^{\left(b\right)}={\widehat{a}}_{m,b}{\widehat{y}}_{m,t}$. The prequential absolute loss for model m at scored validation date t is

$${\ell }_{m,t}^{PC}=\left|y_t-{\widehat{a}}_{m,b\left(t\right)}{\widehat{y}}_{m,t}\right|,t\in {\mathcal{V}}_2\cup {\mathcal{V}}_3\cup {\mathcal{V}}_4,$$

(8)

where $b\left(t\right)$ is the validation block containing date t. PC-Select is the validation-selection rule: it names the candidate with the lowest mean ${\ell }_{m,t}^{PC}$ but does not by itself imply statistical separation or deployment authorization. Let

$${\overline{\ell }}_m^{PC}={\displaystyle \frac{1}{|{\mathcal{V}}^{PC}|}}\sum _{t\in {\mathcal{V}}^{PC}}{\ell }_{m,t}^{PC},{\mathcal{V}}^{PC}={\mathcal{V}}_2\cup {\mathcal{V}}_3\cup {\mathcal{V}}_4,$$

(9)

where ${\overline{\ell }}_m^{PC}$ is the mean prequential validation MAE for model m over the scored validation blocks. The first, selection-stage decision is

$$m^{*}=arg\underset{m\in \mathcal{M}}{min}{\overline{\ell }}_m^{PC}.$$

(10)

Exact ties are resolved by the fixed candidate order recorded in the supplied implementation. After the validation year has fully elapsed, the selected candidate is retrained on the combined training and validation window, and its calibration scalar is re-estimated from the complete validation set $\mathcal{V}={\bigcup }_{b=1}^4{\mathcal{V}}_b$ before its test prediction is formed:

$${\tilde{y}}_{m^{*},t}^{\mathrm{test}}={\widehat{a}}_{m^{*}}\left(\mathcal{V}\right){\widehat{y}}_{m^{*},t},t\in \mathcal{T}.$$

(11)

The first stage selects without same-label calibration bias; the second refits the selected base model and estimates its final scalar after validation is complete, before the test year begins. It neither changes the selected model name nor reuses test outcomes. This timing creates a distinct final refitted predictor, so the prequential ranking is not assumed to be its stable ranking. The Q10 decomposition explicitly measures this additional source of instability.

3.3. Confidence Sets, Transfer Diagnostics, and Information-Boundary Checks

For each case, a moving-block-bootstrap MCS is computed from the candidate loss matrix

$$\ell ={\left[{\ell }_{m,t}\right]}_{m\in \mathcal{M},t\in {\mathcal{V}}^{PC}},$$

(12)

where each row corresponds to a candidate model and each column corresponds to a scored validation date. The default prequential-loss MCS uncertainty audit uses ${\ell }_{m,t}={\ell }_{m,t}^{PC}$, the prequential absolute loss defined above. The MCS is estimated using the maximum-t statistic, $\alpha =0.10$, 5000 bootstrap repetitions, and block length approximately equal to the square root of the evaluated validation length [16,17,28]. The 10% level is used as a screening-oriented confidence-set threshold rather than as a declaration of model separation; 5% and 20% alternatives and block-length multipliers of 0.5, 1, and 2 are included in the sensitivity audit. Because the calibration scalar is updated sequentially across prequential blocks, the scored loss-differential sequence is piecewise generated rather than produced by one fixed forecasting rule over the entire validation year. The strict stationarity conditions supporting a conventional moving-block-bootstrap interpretation therefore need not hold exactly. We use MCS as a validation-stage uncertainty screen, not as a source of nominal prospective coverage; the stated block/significance sensitivities and the controlled simulations assess empirical stability of that screening role, while formal coverage guarantees for adaptive or piecewise-calibrated forecasting are outside this study’s scope. The lowest prequential-MAE model remains the validation-selection rule; MCS membership quantifies uncertainty. Pairwise MCS comparisons are based on loss differences

$$d_{ij,t}={\ell }_{i,t}-{\ell }_{j,t},$$

(13)

where $d_{ij,t}<0$ means that model i has lower loss than model j on date t. The audit therefore asks whether observed differences among candidates are statistically distinguishable under the specified dependent-data bootstrap.

Two controls isolate calibration timing. The raw-loss MCS uncertainty audit uses uncalibrated errors on the same post-warm-up validation observations. The same-label calibrated MCS fits one coefficient on the complete validation year and evaluates it on those same labels. The MCS equal-weight comparator averages the individually final-calibrated test predictions of the prequential-loss MCS members:

$${\tilde{y}}_{c,t}^{\mathrm{MCS}-\mathrm{EW}}={\displaystyle \frac{1}{|{\mathcal{S}}_c|}}\sum _{m\in {\mathcal{S}}_c}{\tilde{y}}_{m,c,t}^{\mathrm{test}}.$$

(14)

No test outcome enters either member calibration or the average. The raw-loss selector, same-label selector, individually calibrated candidates, zero-return predictor, historical-mean predictor, and full five-model ensemble average are comparators. The last two are low-cost reference checks derived from available historical targets and saved candidate predictions; they are not tuned on the test period.

For any test index set $\mathcal{T}$, prediction-scale errors are computed from the final-calibrated prediction ${\tilde{y}}_{m,t}^{\mathrm{test}}={\widehat{a}}_m\left(\mathcal{V}\right){\widehat{y}}_{m,t}$. The root-mean-square error and mean absolute error for model m are

$${RMSE}_m\left(\mathcal{T}\right)={\left({\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}{(y_t-{\tilde{y}}_{m,t}^{\mathrm{test}})}^2\right)}^{1/2},$$

(15)

and

$${MAE}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}|y_t-{\tilde{y}}_{m,t}^{\mathrm{test}}|,$$

(16)

where $\mathcal{T}$ is the test-period date set for one ETF-window case. Direction is evaluated using directional accuracy (DA), balanced accuracy (BA), and Matthews correlation coefficient (MCC). Define the realized sign label and predicted sign label as

$$z_t=\mathbf{1}(y_t>0),{\widehat{z}}_{m,t}=\mathbf{1}({\tilde{y}}_{m,t}^{\mathrm{test}}>0),$$

(17)

where $\mathbf{1}(·)$ equals one when the condition is true and zero otherwise. Then

$${DA}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}\mathbf{1}\left(z_t={\widehat{z}}_{m,t}\right).$$

(18)

BA and MCC are computed from true positives ($TP$), true negatives ($TN$), false positives ($FP$), and false negatives ($FN$):

$${BA}_m\left(\mathcal{T}\right)={\displaystyle \frac{1}{2}}\left({\displaystyle \frac{TP}{TP+FN}}+{\displaystyle \frac{TN}{TN+FP}}\right),$$

(19)

and

$${MCC}_m\left(\mathcal{T}\right)={\displaystyle \frac{TP·TN-FP·FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}.$$

(20)

These directional metrics prevent an always-zero or always-down forecast from appearing skillful solely because of class imbalance. The supplied implementation uses the deterministic conventions of scikit-learn: if MCC has a zero denominator, it is returned as zero; if only one observed class is present, BA is the recall of that observed class. Both conventions are recorded rather than silently replacing an undefined value.

For each ETF-window case c, validation candidate ranks are compared with test-MAE ranks. Let $V_{m,c}={\overline{\ell }}_{m,c}^{PC}$ be the prequential validation loss of model m, and let $Q_{m,c}={MAE}_{m,c}\left({\mathcal{T}}_c\right)$ be its test MAE. Let $R_{m,c}^V$ be the rank of $V_{m,c}$ among $\{V_{j,c}:j\in \mathcal{M}\}$, and let $R_{m,c}^Q$ be the rank of $Q_{m,c}$ among $\{Q_{j,c}:j\in \mathcal{M}\}$, with rank 1 assigned to the smallest loss. The case-level validation-to-test rank correlation is then the ordinary Pearson correlation applied to these two rank vectors:

$${\rho }_c={\displaystyle \frac{{\sum }_{m\in \mathcal{M}}(R_{m,c}^V-{\overline{R}}_c^V)(R_{m,c}^Q-{\overline{R}}_c^Q)}{\sqrt{{\sum }_{m\in \mathcal{M}}{(R_{m,c}^V-{\overline{R}}_c^V)}^2}\sqrt{{\sum }_{m\in \mathcal{M}}{(R_{m,c}^Q-{\overline{R}}_c^Q)}^2}}},$$

(21)

where ${\overline{R}}_c^V=M^{-1}{\sum }_{m\in \mathcal{M}}{R}_{m,c}^V$ and ${\overline{R}}_c^Q=M^{-1}{\sum }_{m\in \mathcal{M}}{R}_{m,c}^Q$. This is the Spearman rank correlation written explicitly rather than treated as a named black-box function. Annual summaries average ${\rho }_c$ over the ETF-window cases completed in year k.

The top-one hit indicator for case c is

$${\delta }_{\mathrm{hit},c}=\mathbf{1}\left(m_c^{*}\in arg\underset{m\in \mathcal{M}}{min}{Q}_{m,c}\right),$$

(22)

so an exact tie for the lowest test MAE counts as a hit. The reported top-one hit rate is the average of ${\delta }_{\mathrm{hit},c}$ over the evaluated cases. A uniform five-model reference is 20%; a label-permutation audit additionally preserves the observed frequencies of selected and lowest-test-loss model labels, with a within-ETF permutation as a dependence-oriented control. Selection regret is the selected model’s test loss minus the best attainable test loss within the candidate pool:

$${\mathcal{R}}_c=Q_{m_c^{*},c}-Q_{min,c},Q_{min,c}=\underset{m\in \mathcal{M}}{min}{Q}_{m,c},$$

(23)

where $Q_{min,c}$ is an ex post candidate-pool minimum, not an additional candidate model. Regret is non-negative and equals zero only when PC-Select chooses a lowest-test-MAE candidate. MCS coverage records whether a lowest-test-MAE candidate was retained by the validation MCS.

Paired comparisons use two-sided Wilcoxon signed-rank tests. Holm correction is applied over the prespecified predictive comparison family. Since the 45 cases share ETFs, years, and overlapping training intervals, mean-difference intervals are also obtained by resampling ETFs as clusters and test years as blocks. The interpretation rules are applied conservatively and are allowed to return a negative or inconclusive audit outcome.

Signal diagnostics use the sign of the prediction to form an unlagged long-short proxy and one-day-lagged long-cash and long-short controls. Turnover costs of 0, 5, 10, and 20 bps are deducted. These are stress diagnostics, not implementable trading claims. Because the study is motivated by resource-constrained deployment, predictive and signal metrics are reported together with computational cost rather than collapsed into a single optimized score. No utility-cost score is tuned on the test period, and no favorable cost summary is allowed to override weak predictive, dependence-aware, or temporal-stress evidence.

Sensitivity analysis varies validation blocks (3–6), MCS significance (0.05, 0.10, 0.20), loss (absolute or squared), block-length multiplier (0.5, 1, 2), and calibration rule (non-negative, unconstrained, or the legacy clipped rule). Saved prediction caches are used without retraining so that sensitivity differences reflect the audit protocol rather than changes in fitted candidate models. The default four-block absolute-error non-negative protocol was fixed before sensitivity analysis and is retained even when another variant gives a more attractive economic proxy.

To make the audit falsifiable, each positive interpretation requires evidence beyond a favorable average. A lower-error claim requires a multiplicity-adjusted comparison and dependence-aware intervals pointing in the same direction. Reliable selection requires both nontrivial top-one evidence and validation-to-test rank-transfer intervals excluding zero. Any operational interpretation beyond the validation-selection result requires consistent signs under transaction costs, one-day execution delay, protocol sensitivity variants, and the 2025 retrospective stress split. When these conditions are not met, the manuscript reports the result as weak, mixed, or unsupported rather than treating a favorable single metric as a discovery.

3.4. Next-Cycle Audit-State Construction and Reproducibility

This study was not prospectively preregistered, and we do not claim that every original model, feature, hyperparameter, candidate-pool, or audit-rule choice was fixed before inspection of all historical outcomes. To reduce new researcher degrees of freedom in this revision, the additional simulation controls, sensitivity analyses, alternative losses, and expanded comparator pools were fixed before their formal batch runs; superseded technical trials remain archived and explicitly labeled.

PC-Audit is not implemented as an online learner in this benchmark. No model is retrained, no hyperparameter is retuned, and no reported test-year result is revised using realized test outcomes. To distinguish the annual review-cycle index from the target variable $y_t$, let k index a completed test year. After year k has ended, its realized audit outputs are logged as monitoring variables for the next scheduled review in year $k+1$. This construction represents a feedback mechanism at the audit-governance level rather than at the model-fitting level.

Signed-return and volatility-proxy audits are separate. Let $u\in \{\mathrm{ret},\mathrm{vol}\}$ identify the target and define the completed-case state

$$A_c^{\left(u\right)}=\left(m_c^{*,\left(u\right)},K_c^{\left(u\right)},I_c^{\left(u\right)},{\rho }_c^{\left(u\right)},{\mathcal{R}}_c^{\left(u\right)},C_c^{\left(u\right)}\right),$$

(24)

where $K_c^{\left(u\right)}$ is MCS size, $I_c^{\left(u\right)}=\mathbf{1}(m_c^{*,\left(u\right)}\in {\mathcal{S}}_c^{\left(u\right)})$ records selection–MCS membership, ${\rho }_c^{\left(u\right)}$ is rank transfer, and ${\mathcal{R}}_c^{\left(u\right)}$ is regret. The stress components use opposite favorable directions: larger is favorable for the unlagged 20-bps signed-return proxy $C_c^{\left(\mathrm{ret}\right)}$, whereas smaller is favorable for $C_c^{\left(\mathrm{vol}\right)}$, PC-Select MAE minus EWMA(0.97) MAE. For completed year k, let ${\mathcal{E}}_k$ denote its nine ETF-window cases. Its annual components are

$$\begin{array}{c}{\overline{K}}_k^{\left(u\right)}={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{K}_c^{\left(u\right)},{\overline{I}}_k^{\left(u\right)}={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{I}_c^{\left(u\right)},\hfill \\ {\overline{\rho }}_k^{\left(u\right)}={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{\rho }_c^{\left(u\right)},{\overline{\mathcal{R}}}_k^{\left(u\right)}={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{\mathcal{R}}_c^{\left(u\right)},\hfill \\ {\overline{C}}_k^{\left(u\right)}={\displaystyle \frac{1}{|{\mathcal{E}}_k|}}\sum _{c\in {\mathcal{E}}_k}{C}_c^{\left(u\right)}.\hfill \end{array}$$

(25)

The annual audit-state summary is

$${\overline{A}}_k^{\left(u\right)}=({\overline{K}}_k^{\left(u\right)},{\overline{I}}_k^{\left(u\right)},{\overline{\rho }}_k^{\left(u\right)},{\overline{\mathcal{R}}}_k^{\left(u\right)},{\overline{C}}_k^{\left(u\right)}).$$

(26)

The selected-model identity is retained in each case record but is categorical and therefore not numerically averaged into ${\overline{A}}_k^{\left(u\right)}$. The next-cycle governance state is then a target-specific deterministic function of the completed-year audit state:

$$\begin{array}{cc}\hfill D_{k+1}^{\left(u\right)}& =g^{\left(u\right)}({\overline{A}}_k^{\left(u\right)}),\hfill \\ \hfill D_{k+1}^{\left(u\right)}& \in \{\mathrm{Green},\mathrm{Yellow},\mathrm{Red}\}.\hfill \end{array}$$

(27)

The reported simulation uses four warning indicators for each target:

$$\begin{array}{cccc}\hfill W_{1,k}^{\left(u\right)}& =\mathbf{1}({\overline{K}}_k^{\left(u\right)}\ge 4.5),\hfill & \hfill W_{2,k}^{\left(u\right)}& =\mathbf{1}({\overline{\rho }}_k^{\left(u\right)}\le 0),\hfill \\ \hfill W_{3,k}^{\left(u\right)}& =\mathbf{1}({\overline{\mathcal{R}}}_k^{\left(u\right)}>5\times {10}^{-5}),\hfill & \hfill W_{4,k}^{\left(u\right)}& =\left\{\begin{array}{cc}\mathbf{1}({\overline{C}}_k^{\left(\mathrm{ret}\right)}\le 0),\hfill & u=\mathrm{ret},\hfill \\ \mathbf{1}({\overline{C}}_k^{\left(\mathrm{vol}\right)}>0),\hfill & u=\mathrm{vol}.\hfill \end{array}\right.\hfill \end{array}$$

(28)

The four warnings flag weak separation, nonpositive transfer, regret above a fixed tolerance, and target-specific adverse stress. ${\overline{I}}_k^{\left(u\right)}$ is a descriptive selection–MCS coherence measure, intentionally excluded from the traffic-light count because no threshold was prespecified. The warning count is

$$N_k^{\left(u\right)}={\displaystyle \sum _{j=1}^4}{W}_{j,k}^{\left(u\right)}.$$

(29)

The traffic-light rule is then

$$g^{\left(u\right)}\left({\overline{A}}_k^{\left(u\right)}\right)=\left\{\begin{array}{cc}\mathrm{Green},\hfill & N_k^{\left(u\right)}=0,\hfill \\ \mathrm{Yellow},\hfill & N_k^{\left(u\right)}\in \{1,2\},\hfill \\ \mathrm{Red},\hfill & N_k^{\left(u\right)}\ge 3.\hfill \end{array}\right.$$

(30)

Green maintains a cautious selected-model interpretation, Yellow calls for review, and Red recommends fallback or re-audit. These conservative thresholds are fixed, not fitted, retrospective audit rules: ${\overline{K}}_k^{\left(u\right)}\ge 4.5$ means that nearly all candidates remain indistinguishable; ${\overline{\rho }}_k^{\left(u\right)}\le 0$ means no positive ordering evidence; and ${\overline{\mathcal{R}}}_k^{\left(u\right)}>5\times {10}^{-5}$ marks loss relative to the candidate-pool minimum. For signed returns, ${\overline{C}}_k^{\left(\mathrm{ret}\right)}\le 0$ flags a nonpositive high-cost proxy; for volatility, ${\overline{C}}_k^{\left(\mathrm{vol}\right)}>0$ means PC-Select is worse than EWMA(0.97). Requiring three warnings avoids a full re-audit after one noisy diagnostic. All variables are carried forward only after the corresponding test year ends.

The fallback simulation converts this interpretation state into a fixed next-year action within the reported retrospective exercise. Let ${\pi }_{k+1}^{\mathrm{ret}}$ and ${\pi }_{k+1}^{\mathrm{vol}}$ denote the actions applied in year $k+1$ for the signed-return and volatility-proxy audits, respectively. For signed returns,

$${\pi }_{k+1}^{\mathrm{ret}}=\left\{\begin{array}{cc}\mathrm{PC-Select},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Green},\hfill \\ \mathrm{MCS}\mathrm{EqualWeight},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Yellow},\hfill \\ \mathrm{Zero}\mathrm{return},\hfill & D_{k+1}^{\left(\mathrm{ret}\right)}=\mathrm{Red}.\hfill \end{array}\right.$$

(31)

For the volatility-proxy audit, where strong naive volatility references are available,

$${\pi }_{k+1}^{\mathrm{vol}}=\left\{\begin{array}{cc}\mathrm{PC-Select},\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Green},\hfill \\ \mathrm{Naive-Select},\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Yellow},\hfill \\ \mathrm{EWMA}\left(0.97\right),\hfill & D_{k+1}^{\left(\mathrm{vol}\right)}=\mathrm{Red},\hfill \end{array}\right.$$

(32)

where Naive-Select chooses among Roll-10, Roll-21, Roll-63, EWMA(0.94), and EWMA(0.97) using only the preceding validation year. These fallback rules are not optimized on the test years; they test whether audit states can guide conservative next-cycle governance without claiming to improve current-year accuracy.

To test whether the audit could instead support retaining or replacing the current-year PC-Select action, a separate falsification analysis uses only information available at validation time: prequential validation MCS size, the relative validation margin, and the frequency with which the final selected model wins the three scored chronological validation blocks. The validation-only score components are defined in Supplementary Equation (S1); the information boundary, dependence-sensitive context, and nested-ablation evidence are reported in Supplementary Tables S1, S5 and S6. Test regret, validation-to-test transfer, candidate-pool minimum coverage, and current test outcomes are excluded from score construction and are used only for evaluation. The analysis compares the three individual signals with the equal-weight score over their full risk–coverage paths. This is a falsification of hypothetical prospective filtering, not an additional model-selection stage.

The experiment used Linux x86_64, Python 3.13.5, NumPy 2.3.5, pandas 2.2.3, scikit-learn 1.8.0, LightGBM 4.6.0, and CPU-only PyTorch 2.10.0+cpu. The container exposed 56 logical CPU interfaces and 4.0 GB RAM. Exact raw files, feature tables, prediction caches, scripts, configurations, results, audit-state tables, and hashes are included. The supplied checks verify the current headline values against the CSV result files, validate raw-data inventories, test the deterministic MCS implementation, and recompute file hashes.

4. Results

The results are organized to answer the three audit questions rather than to select the most attractive metric. They first establish the controlled simulation and case-study boundary, then examine signed-return selection reliability, target-specific evidence, and the feedback and fallback governance mechanism. Supplementary Tables S1–S11, Supplementary Figure S1, and Supplementary Equation (S1) supply the detailed tables, window-level records, robustness outputs, and validation-only score definition.

4.1. Controlled Known-Truth Simulation

The frozen simulation retains 16 prespecified base scenarios and adds only two fixed diagnostic controls. Across the base scenarios, the DGP varies model equivalence and signal separation, AR dependence, calibration scale, regime shift/reversal, and validation-window length. The stationary-separable control has one clearly lower validation expected risk; the material-reversal control has a validation winner whose future expected regret exceeds the fixed 10% false-reassurance future-regret threshold. Parameters were fixed using an independent DGP/oracle-risk pilot before the formal batch run; this study was not prospectively preregistered. The primary simulation uses L2/NNLS calibration, four chronological blocks, the fixed strong-positive rule, 1200 repetitions, and 300 MCS resamples. Positive calls are reported together with correct-positive, false-reassurance, and abstention rates rather than as a standalone success metric. Table 3 reports the frozen known-truth simulation controls.

Table 3. Frozen known-truth simulation controls. Positive-call, correct-call, false-reassurance, and abstention rates are unconditional proportions over 1200 repetitions. In the material-reversal control, every strong-positive call is false reassurance, so the conditional false-reassurance rate among positive calls is 1.0.

Scenario	Positive Call	Correct Call	False Reassurance	Abstention	Mean Future Regret
Weak-signal baseline	0.0083	0.0067	0.0000	0.9917	0.0039
Stationary separable control	0.5525	0.5525	0.0000	0.4475	0.0024
Material reversal control	0.5458	0.0000	0.5458	0.4542	0.1818

The stationary control shows that the rule is not mechanically abstention-only when oracle separation is clear. The reversal control gives the complementary limitation: the validation-time winner is called positive in 54.58% of repetitions, but all such calls are false reassurance because its future expected regret is about 18.18%, above the fixed threshold. The weak-signal baseline mostly abstains. These results separate evidence-sensitive behavior from any claim that validation-time evidence can anticipate an unobserved future regime reversal.

4.2. Data Universe and Predictive Baselines

Table 4 reports the fixed ETF universe used in the audit. The benchmark uses Yahoo Finance daily OHLCV snapshots for nine mainland Chinese index ETFs from 2 January 2014 through 31 December 2025. The objective is not to claim universal applicability across all financial markets, but to provide a controlled case study for deployment-oriented audit methodology. Each raw file contains 2916 unique trading dates. The universe was defined before the PC-Audit analysis using three operational requirements: continuous coverage from the beginning of 2014, no duplicate dates, and representation of broad-market, size, style, growth, and sector exposures. It is a controlled audit benchmark rather than an exhaustive or market-wide sample of Chinese ETFs.

Table 4. ETF universe. The exact files and SHA-256 hashes are supplied in the reproduction package.

Ticker	Descriptive Fund Identifier	Main Exposure	Exchange	Rows
510050.SS	SSE 50 ETF	Large-cap SSE 50	Shanghai	2916
510300.SS	CSI 300 ETF	Broad large-cap CSI 300	Shanghai	2916
510500.SS	CSI 500 ETF	Mid/small-cap CSI 500	Shanghai	2916
159915.SZ	ChiNext ETF	ChiNext growth board	Shenzhen	2916
159902.SZ	SME 100 ETF	SME 100	Shenzhen	2916
510880.SS	SSE Dividend ETF	Dividend style	Shanghai	2916
159928.SZ	CSI Consumer Staples ETF	Consumer staples	Shenzhen	2916
159929.SZ	CSI Medical and Health ETF	Health care	Shenzhen	2916
159901.SZ	SZSE 100 ETF	SZSE 100	Shenzhen	2916

The equal row counts in Table 4 matter for interpretation: differences among models are not driven by unequal sample coverage or ticker-specific missing-history rules. At the same time, the table also defines the external-validity boundary. The benchmark is deliberately narrow enough to support a reproducible audit, but it is not a claim about all Chinese ETFs or all market regimes.

Table 5 reports 2020–2024 averages at zero transaction cost. In all 45 cases, the model named by PC-Select is also retained by the prequential-loss MCS. Thus the validation-selected candidate is not excluded by the confidence-set audit, although the set is usually large; this is not deployment authorization. PC-Select records RMSE 0.015821, MAE 0.010593, balanced accuracy 0.49998, and MCC $-0.00061$. The directional metrics are effectively at chance.

Table 5. Signed-return prediction and directional summaries at 0 bps, averaged over 45 ETF-window cases. Individual candidates use full-validation non-negative calibration before test prediction. DA denotes directional accuracy; BA denotes balanced accuracy. Economic stress values are omitted from the main table and reported in the reproduction package listed in Supplementary Table S1.

Method	RMSE	MAE	DA	BA	MCC
Historical mean	0.015806	0.010546	0.5316	0.5000	0.0000
Zero return	0.015798	0.010563	0.5181	0.5000	0.0000
Calibrated ElasticNet	0.015901	0.010654	0.5115	0.5015	0.0022
Calibrated Ridge	0.015887	0.010648	0.5065	0.4965	−0.0072
Calibrated LightGBM	0.015977	0.010694	0.5094	0.4953	−0.0097
Calibrated Random Forest	0.015829	0.010598	0.5207	0.4985	−0.0045
Calibrated GRU-small	0.015809	0.010568	0.5059	0.4984	−0.0030
Ensemble Average	0.016269	0.010965	0.4929	0.4933	−0.0133
Raw-loss MCS selector	0.015823	0.010603	0.5082	0.4964	−0.0097
Same-label calibrated selector	0.015881	0.010650	0.5116	0.4973	−0.0062
PC-Select	0.015821	0.010593	0.5169	0.5000	−0.0006
MCS equal weight	0.015840	0.010603	0.5007	0.4932	−0.0141

The naive references are strong on prediction-scale error. The historical-mean reference has the lowest average MAE (0.010546), and the zero-return predictor has slightly lower RMSE (0.015798) and MAE (0.010563) than PC-Select. The paired PC-Select versus zero-return MAE comparison has unadjusted $p=0.0748$ and Holm-adjusted $p=1.000$; ETF-cluster and year-block mean-difference intervals favor the zero reference. Therefore, PC-Select does not establish lower prediction error than naive benchmarks under the present sample. The historical-mean reference also produces a nearly constant sign exposure, so its return proxy is interpreted only as a stress diagnostic rather than as timing skill.

No PC-Select comparison is significant after Holm adjustment. Relative to the same-label calibrated selector, the unadjusted MAE test favors PC-Select ($p=0.0080$), and both dependence-aware intervals have the same orientation, but the Holm-adjusted value is 0.431. The auxiliary ensemble-average row is deliberately included because it is a common robustness device; here it performs worse than the calibrated single-model references on RMSE and MAE, so the mixed conclusion cannot be dismissed as merely a failure to average candidates. This is suggestive evidence that protocol choices alter selection, not evidence of a universal lower-error advantage. Table 5 therefore functions as a guardrail for the rest of the Section 4: subsequent transfer and feedback analyses are interpreted as audit diagnostics, not as proof that PC-Select beats naive prediction.

4.3. Validation Transfer and Statistical Separation

Figure 2 and Table 6 quantify validation-to-test transfer. Under prequential calibration, mean rank correlation is 0.090 and both dependence-aware 95% intervals include zero. PC-Select nevertheless chooses the lowest-test-MAE candidate in 15 of 45 cases (33.3%). The exact binomial diagnostic against a uniform 20% rate gives $p=0.0379$, but the more conservative global label-permutation audit gives $p=0.0521$; the within-ETF permutation gives $p=0.0433$. We therefore treat the hit rate as limited, borderline transfer evidence rather than a confirmatory discovery. It does not extend to a stable ranking of all candidates and falls to 22.2% in 2025.

Figure 2. Signed-return validation-to-test candidate-rank correlations over 45 cases. The solid orange line within each box denotes the median, the dashed green line denotes the mean, and the dashed blue horizontal line marks zero correlation. The prequential mean is positive but small, and its dependence-aware intervals include zero.

Table 6. Signed-return validation-to-test transfer. Relative regret is measured against the best calibrated candidate in each test case. The exact p column is a descriptive uniform-reference diagnostic; the more conservative permutation checks are reported in the text and in the reproduction package listed in Supplementary Table S1.

Period and Protocol	Mean Rank $\mathit{\rho }$	Top-One Hit	Exact p vs. 0.20	MCS Minimum Coverage	Mean Regret	Relative Regret (%)
2020–2024 raw loss	−0.019	0.156	0.577	0.644	0.000049	0.431
2020–2024 same-label calibrated	−0.126	0.356	0.014	1.000	0.000096	0.740
2020–2024 prequential calibrated	0.090	0.333	0.038	0.933	0.000039	0.329
2025 raw loss	−0.242	0.222	1.000	0.667	0.000446	4.033
2025 same-label calibrated	0.426	0.444	0.086	1.000	0.000044	0.652
2025 prequential calibrated	0.220	0.222	1.000	0.889	0.000031	0.379

The same-label control has a 35.6% hit rate in 2020–2024 but negative mean rank correlation and larger regret. Its apparent top-one success therefore does not imply better global ranking. Across protocols, no method consistently dominates both the main period and 2025.

The weak transfer result is also visible before testing through small validation margins. The audit records the validation margin between the first- and second-ranked candidates under the default prequential loss. Let $V_{\left(1\right)}$ and $V_{\left(2\right)}$ denote the smallest and second-smallest values in the set $\{{\overline{\ell }}_m^{PC}:m\in \mathcal{M}\}$ for one ETF-window case. The selection-margin statistic is

$${\Delta }_{\mathrm{sel}}=V_{\left(2\right)}-V_{\left(1\right)}.$$

(33)

In 2020–2024, the median ${\Delta }_{\mathrm{sel}}$ is $8.0\times {10}^{-6}$, the interquartile range is 0 to $2.74\times {10}^{-5}$, and 93.3% of cases have ${\Delta }_{\mathrm{sel}}<{10}^{-4}$. These small margins explain why naming a single validation-selected candidate is operationally possible but statistically fragile. Figure 3 shows the same issue visually: a selection action can exist even when the scientific claim remains weak, because the selector is forced to name the smallest validation loss but the distance between the first and second candidates is often too small to justify strong language about superiority.

Figure 3. Distribution of the signed-return prequential validation MAE margin between the first- and second-ranked candidates in the 45 main ETF-window cases. Small margins indicate that many validation selections are made among nearly tied candidates.

Calibration timing changes the apparent statistical admissibility of candidates. The default prequential MCS contains 4.80 of five candidates on average and retains all five in 38 of 45 cases (84.4%). The raw-loss MCS contains 3.67 models on average and never retains all five. Same-label calibration retains 4.98 and includes all five in 44 cases. Raw and prequential memberships differ in 43 of 45 cases (95.6%); same-label and prequential memberships differ in 17.8%.

Table 7 shows that the more selective raw-loss protocol is not necessarily the more reliable protocol. It removes more candidates, but it also has lower coverage of a candidate that attains the minimum test MAE and almost always changes the confidence set relative to the prequential design. The omitted plot gives the same trade-off visually; the table is kept in the main text because it carries the numerical audit evidence. This is why PC-Audit reports confidence-set composition as uncertainty evidence rather than treating exclusion as automatic evidence of transferable inferiority.

Table 7. Effect of calibration timing on signed-return MCS composition and selection. Sel. change is the selected-model change rate relative to PC; set change is the MCS membership-change rate relative to PC. The selectivity–coverage plot is reported in Supplementary Figure S1.

Protocol	Size	All-Five	Sel. Change	Set Change	Min. Cov.
Raw loss	3.667	0.000	0.622	0.956	0.644
Same-label	4.978	0.978	0.511	0.178	1.000
Prequential	4.800	0.844	–	–	0.933

The raw-loss MCS excludes GRU-small in every main-period case, whereas prequential and same-label calibration retain it in 95.6% and 100% of cases. This contrast shows that apparent model elimination can reflect amplitude mismatch rather than transferable inferiority. PC-Select chooses LightGBM 14 times, ElasticNet and GRU-small 10 times each, Random Forest 6 times, and Ridge 5 times; no candidate dominates the validation periods.

Full-validation non-negative slopes equal zero in 41.8% of the 225 model-case combinations. The rate is highest for GRU-small (53.3%) and lowest for Ridge (28.9%). Zero slopes are not treated as failed fits: they state that no positive multiple of the raw forecast lowers validation squared error relative to the zero forecast.

4.4. Validation Reliability Across Prediction Targets

The same audit was also run on volatility-like risk-surveillance targets rather than directional signals. Table 8 reports the original five-model comparison; revised comparator and QLIKE evidence is reported below and in Supplementary Tables S8–S10. The more favorable evidence concerns target choice rather than superiority of any individual model. Signed-return selection has weak main-period rank transfer (0.090) and a nearly uninformative MCS size (4.80 of five models). The absolute-return proxy $|r_t|$, squared-return proxy $r_t^2$, and range proxy $log(H_t/L_t)$ show stronger rank transfer in this original pool, but that observation is not treated as a universal target-level advantage.

Table 8. Validation-reliability comparison across signed-return and volatility-like targets. Volatility-like targets are evaluated only through prediction-scale and validation-transfer diagnostics; they are not used for directional or trading-return claims.

Target	Main Rank $\mathit{\rho }$	Mean MCS Size	Top-One Hit	2025 Rank $\mathit{\rho }$	Interpretation
Signed return $r_t$	0.090	4.80	0.333	0.220	Weak validation-selection evidence
Absolute return $|r_t|$	0.560	3.73	0.356	−0.056	Stronger original-pool transfer; stress-sensitive
Squared return $r_t^2$	0.400	3.76	0.578	0.178	Conditional evidence; comparator/loss dependent
Range proxy $log(H_t/L_t)$	0.631	3.38	0.178	0.078	Strong original-pool transfer; low top-one hit

The volatility-like evidence remains bounded. The absolute-return proxy has a lower original-pool average MAE than the historical-mean reference in 2020–2024 (0.007201 versus 0.007392), a descriptive comparison rather than a general performance claim. Squared return and range proxy also produce stronger original-pool rank transfer than signed returns. However, all three target families weaken in the retrospective 2025 holdout, and the range proxy’s strong rank transfer does not translate into a high top-one rate. The result is not a stable volatility-modeling breakthrough. Its role is narrower: PC-Audit distinguishes unsupported signed-return selection evidence from conditional, stress-sensitive risk-surveillance evidence.

Table 9 prevents overinterpretation of the volatility-proxy result; Supplementary Table S8 gives the corresponding comparator-only reference checks. PC-Select improves on zero and historical references, and several paired signed-rank comparisons are favorable. Strong rolling and EWMA volatility references remain competitive, however, and EWMA(0.97) has a slightly lower mean MAE than PC-Select. Therefore, the volatility-proxy result is not reported as universal model-level superiority. In the original five-model comparison, several volatility-related targets show stronger ranking transfer and more selective confidence sets than signed returns; under expanded pools and alternative losses, that pattern is not uniform across targets.

Table 9. Absolute-return volatility-proxy comparison with stronger naive volatility references, 2020–2024. Positive improvement means lower MAE for PC-Select than for the reference. Naive-Select chooses among Roll-10, Roll-21, Roll-63, EWMA(0.94), and EWMA(0.97) using only the validation year. The signed-rank test reflects paired case-level differences and can be significant even when the mean percentage improvement is small.

Reference	Reference MAE	PC-Select Improvement (%)	PC Better Cases	Holm-Adjusted Wilcoxon p
Zero proxy	0.010563	31.87	44/45	$1.69\times {10}^{-9}$
Historical mean	0.007392	3.75	34/45	0.0038
Roll-10	0.007367	0.76	33/45	0.0041
Roll-21	0.007327	0.00	32/45	0.0392
Roll-63	0.007360	0.52	32/45	0.0161
EWMA(0.94)	0.007202	−1.46	29/45	0.2456
EWMA(0.97)	0.007187	−1.54	28/45	0.2456
Naive-Select	0.007279	−0.54	30/45	0.0710

4.5. Sensitivity, Costs, and Temporal Stress

The common-support horizon audit holds the 2021–2024 test years fixed at 36 ETF–year cases. With a five-year training window, changing validation length from 3 to 24 months gives top-one hit rates of 0.222–0.417, mean rank correlations of $-0.040$ to $0.280$, and MCS sizes of 4.611–4.833. The separately prespecified three-year-training/12-month-validation sensitivity has top-one hit 0.306, mean rank correlation $-0.278$, and MCS size 4.778. Thus, neither validation-window length nor the shorter training history supports a stable signed-return winner (Supplementary Table S4).

Across 11 prespecified sensitivity configurations, agreement with the default selection ranges from 55.6% to 100%, and mean MCS size ranges from 4.53 to 4.91. RMSE ranges from 0.015821 to 0.015878, MAE from 0.010593 to 0.010647, balanced accuracy from 0.4964 to 0.5054, and MCC from $-0.0082$ to 0.0123. Thus, the identity of the validation-selected candidate is specification-sensitive, while directional evidence remains weak.

Economic stress diagnostics are more sensitive. Across the prespecified sensitivity configurations, gross signal diagnostics vary materially and the 5 bps diagnostics include both positive and negative outcomes. The full values are retained in the reproduction package listed in Supplementary Table S1 to document sign and cost sensitivity, not economic profitability. Unconstrained calibration gives the strongest gross diagnostic but permits sign reversal; the legacy clipped rule gives a negative 5 bps result. The default non-negative rule is retained because it has a clear interpretation and was fixed before comparing these alternatives, not because it maximizes any economic proxy.

Under the default unlagged diagnostic, the PC-Select signal proxy turns negative once the transaction-cost assumption is increased to the highest stress level. One-day-lagged controls remain positive under some specifications, whereas the MCS equal-weight control is negative under others. In 2025, PC-Select is positive under lagged long-cash but negative under the unlagged 5 bps specification. This split is treated exclusively as a retrospective temporal stress/holdout assessment and is not included in statistical inference. Since sign and magnitude depend on calibration, execution, cost, and period, these stress diagnostics are not interpreted as trading strategies or investable evidence. Table 10 reports the compact 2025 stress signs; detailed execution and Sharpe tables are provided in the accompanying reproduction package.

Table 10. Signed-return 2025 temporal stress evaluation using the unlagged diagnostic. Stress signs summarize the annual signal-return proxy at 0 and 5 bps; full return and Sharpe values are reported in the accompanying reproduction package.

Method	RMSE	MAE	BA	MCC	Stress Sign 0 bps	Stress Sign 5 bps
Historical mean	0.011512	0.008132	0.5000	0.0000	+	+
Zero return	0.011533	0.008158	0.5000	0.0000	0	0
Ensemble Average	0.011763	0.008388	0.4968	−0.0063	−	−
Raw-loss MCS selector	0.012146	0.008591	0.4965	−0.0083	+	−
Same-label calibrated selector	0.011572	0.008188	0.5038	0.0095	+	−
PC-Select	0.011565	0.008175	0.5039	0.0082	+	−
MCS equal weight	0.011624	0.008234	0.4984	−0.0026	−	−

The historical-mean reference has the lowest 2025 mean MAE, while PC-Select remains close to the zero-return reference. The historical-mean signal proxy mainly reflects a nearly constant exposure determined by the preceding validation year; it is not evidence of timing ability. The 2025 sample contains only nine ETF cases, so its statistics are descriptive stress evidence rather than a confirmatory study.

4.6. Next-Cycle Feedback and Computational Boundary

Table 11 converts completed annual audit outputs into next-cycle monitoring states. The table is not an online-learning result: realized test-year outcomes are used only after each year is complete and only to define review information for the following cycle. The traffic light is an illustrative governance heuristic, not a validated reliability classifier. A threshold grid gives state agreement of 0.600 for signed return and 0.759 for the volatility proxy, so the labels are sensitivity-dependent and are not promoted to a calibrated decision rule.

Table 11. Next-cycle audit-state and audit-triggered fallback simulation. Panel A reports signed-return annual outputs; mean rank $\rho$ is the case-level validation-to-test candidate-rank correlation averaged within a test year, and mean regret is selected-model test MAE minus the candidate-pool minimum test MAE. Panel B starts in 2021 because the 2020 action has no previous completed audit state; each target uses its own completed-year audit state, and negative worst-year change means lower worst-year MAE under the fallback rule. Boldface is used only for panel titles and column headers.

Panel A: Completed-Year Audit States
Completed Test Year	Cases	Mean MCS Size	All-Five Rate	Mean Rank $\mathbf{\rho }$	Mean Regret	20 bps Stress	Next-Review State and Warnings
2020	9	4.67	0.78	0.322	0.000050	−0.03547	Red: large MCS, regret above tolerance, stress nonpositive
2021	9	5.00	1.00	0.207	0.000045	−0.08584	Yellow: large MCS, stress nonpositive
2022	9	4.56	0.67	−0.113	0.000008	−0.00507	Red: large MCS, nonpositive rank transfer, stress nonpositive
2023	9	4.89	0.89	0.270	0.000006	−0.04131	Yellow: large MCS, stress nonpositive
2024	9	4.89	0.89	−0.185	0.000086	0.00461	Red: large MCS, nonpositive rank transfer, regret above tolerance
Panel B: Next-Year Fallback Simulation
Target	Cases	Blind PC MAE	Fallback MAE	Mean Change (%)	Fallback Better Cases	Worst-Year Change (%)
Signed return	36	0.010248	0.010251	0.024	13/36	−0.35
Volatility proxy $|r_t|$	36	0.007050	0.007087	0.517	13/36	−0.90

The audit-state table is a governance-organizing illustration rather than a predictive result. Red years combine several warnings, such as large confidence sets with weak rank transfer, regret above tolerance, or nonpositive high-cost stress. Yellow years still require manual review. The illustration makes the definitions, timing, and sensitivity of a next-cycle review record explicit; it does not validate the numerical thresholds or improve already completed outcomes.

Panel B of Table 11 applies each target’s completed-year traffic-light state as a next-year fallback rule, with detailed values and uncertainty in Supplementary Table S11. The comparison starts in 2021 because the 2020 action has no previous completed audit state. For signed returns, Green would continue PC-Select, Yellow falls back to the MCS equal-weight control, and Red falls back to the zero-return reference. For the volatility-proxy target, the separately computed volatility audit state maps Yellow to validation-year Naive-Select and Red to EWMA(0.97). Conditional-on-observed-policy-path paired intervals for both mean and worst-year MAE cross zero under the structured ETF/year sensitivity. No statistically reliable predictive improvement is established. The fallback remains only an illustrative forward-only governance/containment mechanism; four next-cycle years are too few to estimate full policy-generation uncertainty stably.

The validation-time falsification does not support promoting the validation signals to a current-cycle filtering rule. For signed returns, the equal-weight score has essentially no association with subsequent candidate-pool regret ($\rho =0.040$, $p=0.792$), while the margin-only score fails to provide the intended monotone assurance ($\rho =0.498$, $p<0.001$); the latter association is descriptive, not causal. For the absolute-return proxy, MCS selectivity is weakly favorable ($\rho =-0.251$, $p=0.096$), but the full score is weaker ($\rho =-0.164$, $p=0.281$). Full-score associations are also small in the separate 2025 cases ($-0.075$ and $-0.083$, respectively). Prespecified threshold, cluster, leave-one-group, and risk–coverage checks are reported in Supplementary Table S6. The evidence supports next-cycle use of completed audits, not same-cycle filtering.

Finally, PC-Select averages 0.525 s of candidate training time and 0.0018 s of selected-model inference per case in the stated environment. The MCS equal-weight control retains multiple models and averages 2.37 s training and 0.0090 s inference. The full five-model ensemble average is reported as an additional low-cost diagnostic in statistical terms but would require retaining all five candidate predictions at operational use. These timings are therefore relative indicators under one controlled CPU environment rather than universal edge-device latency claims; a single-core or hardware-constrained implementation would require separate timing.

5. Discussion

5.1. Why PC-Audit Is a Decision-Reliability Framework

PC-Audit should be read as a decision-reliability framework rather than a return-prediction model. Its input is a fixed candidate pool and a chronological validation decision; its output is a selected model name plus an uncertainty and robustness record explaining whether that selected action deserves a strong interpretation. This distinction matters because the candidate with the lowest validation MAE can always be named, even when the validation window provides little evidence that the choice will transfer. PC-Audit makes that gap visible through confidence-set size, candidate-pool minimum coverage, rank transfer, regret, multiplicity-adjusted paired tests, dependence-aware intervals, cost and execution checks, sensitivity variants, and the 2025 temporal stress evaluation.

This framing also explains why PC-Select is not evaluated as a new predictive architecture. Large confidence sets, weak rank transfer, or unstable economic signs are informative audit outcomes because they limit the claim that can be attached to the selected candidate. A weak audit outcome is therefore not a failure of PC-Audit; it is decision-relevant evidence that the audit is designed to expose. The relevant comparison is between an unaudited selection workflow and one that reports the uncertainty and practical constraints surrounding the same validation-selected action.

5.2. What the Audit Establishes

The evidence is deliberately mixed. PC-Select chooses the test-year MAE winner more often than a uniform five-model reference in 2020–2024, but the result is borderline under label permutation and is interpreted only as limited transfer in the top-ranked candidate. Yet the average rank correlation is small and uncertain, the full ordering does not transfer, the selected model changes under reasonable protocol variants, and the 2025 top-one rate returns to 22.2%. The defensible conclusion is therefore not that validation is useless, but that one year supplies insufficient evidence for a stable winner.

Table 12 summarizes the change in interpretation caused by the audit layer. It is placed here rather than among the numerical result tables because its role is interpretive: it shows how a conventional validation-selection workflow could overstate the evidence and how PC-Audit weakens or rejects that interpretation.

Table 12. Unaudited validation interpretation versus PC-Audit interpretation.

Evidence Item	Possible Unaudited Interpretation	PC-Audit Interpretation
Lowest prequential validation MAE	Select the validation winner as the preferred candidate	Selection is allowed as an action, but its interpretation depends on uncertainty and transfer diagnostics
PC-Select top-one hit rate of 15/45	Treat the hit rate as evidence that validation selection beats a random five-model choice	Borderline only; the global label-permutation diagnostic gives $p=0.0521$
Mean prequential MCS size of 4.80/5	Ignore confidence-set size and report only the selected model name	Report weak statistical separation because nearly all candidates remain admissible
Zero-return and historical-mean references	Treat naive references as trivial checks secondary to the selected model	No significant lower-error advantage over naive references is detected after multiplicity adjustment
Validation reliability across targets	Treat the absolute-return result as proof that PC-Select is a superior volatility model	Report the target-level result: several volatility-like targets show stronger transfer than signed returns under the specified candidate pool, while rolling and EWMA references remain competitive
Cost, execution, and 2025 stress signs	Report favorable gross diagnostics as economic evidence	Do not claim investable evidence because signs change under costs, execution modes, sensitivity variants, and temporal stress
Completed annual audit states	Carry the latest validation winner into the next review without qualification	Carry forward Green/Yellow/Red states; fallback is an illustrative next-cycle containment mechanism, not a predictive-improvement rule

Calibration timing is consequential. Raw-loss MCS appears more decisive but has lower candidate-pool minimum coverage and almost never agrees with the prequential set. Same-label calibration retains nearly everything and reuses outcomes for fitting and scoring. The prequential design avoids label reuse and exposes uncertainty more transparently, even though it cannot create predictive information that is absent from the data.

The result also clarifies the role of MCS. PC-Select always names one validation-selected candidate, while the prequential-loss MCS usually retains all five. Reporting only the selected name would conceal the lack of statistical separation. Conversely, averaging candidates does not automatically provide a more useful selection signal: both the MCS equal-weight control and the full five-model ensemble average have weaker direction or prediction-scale diagnostics than several single-model references.

The comparison across prediction targets provides a conditional empirical result. Under the same target-date partitioning and prequential-selection logic, several volatility-related targets show stronger transfer or lower regret than signed returns under specified candidate pools and losses. This pattern is not uniform across targets or loss definitions: rolling, EWMA, HAR, and GARCH references remain competitive, and the 2025 retrospective stress split weakens some target-level evidence. The framework can therefore compare risk-surveillance evidence without converting any target into a deployment authorization claim.

5.3. Implications for Applied Deployment

First, studies that screen multiple models on short financial validation periods should report selection hit rates, regret, and confidence-set size in addition to the winning score. Second, calibration belongs inside the comparison protocol. Applying it only after model selection can penalize candidates with amplitude mismatch and produce misleading elimination. Third, naive zero and historical-mean references remain essential when the target is close to unpredictable; they should be reported even when their sign exposure is not interpreted as timing skill. Fourth, economic metrics should be treated as stress outcomes rather than used to choose among many protocol variants. In applied automated-decision settings, the practical value of such a protocol is not only to name a validation-selected candidate, but also to document when the available evidence is too weak for a strong operational claim.

The next-cycle audit-state simulation extends this implication from one-off reporting to scheduled model governance. A completed test year can be converted into monitoring variables for the next review: confidence-set size, rank transfer, regret, and stress outcomes. The warning-count rule is intentionally illustrative rather than validated. The fallback analysis reports conditional policy-path uncertainty intervals for both mean and worst-year MAE; because these intervals cross zero, no statistically reliable predictive improvement is established. Its permitted role is a forward-only governance/containment demonstration, not an accuracy booster. The validation-time falsification further shows why this distinction matters: validation-only MCS, margin, and block-stability signals do not form a stable same-cycle authorization score in this sample. PC-Audit therefore functions as a reliability layer around model selection rather than as an adaptive trading engine.

The non-negative calibration rule is intentionally simple. It prevents sign reversal, allows evidence-based shrinkage to zero, and avoids an arbitrary upper cap. The frequent zero slopes are substantively informative: the validation data often do not support carrying a model’s raw directional signal forward at any positive scale.

The value of PC-Audit is not to identify a universally superior forecasting model, but to quantify how strongly a validation-selected candidate should be interpreted given the available evidence. In this sense, the framework provides an auditable decision layer for model-selection governance rather than another prediction model.

5.4. Scope, Limitations, and Future Work

The study uses nine ETFs from one market and one public data vendor. The inclusion rule favors instruments with continuous historical coverage and introduces availability and survivorship bias; this design supports a controlled benchmark but not a market-wide claim about all Chinese ETFs. The study demonstrates protocol feasibility and evidence diagnosis in this panel; it does not establish cross-market generalizability. Because the vendor is not the official exchange record, replication on exchange-licensed data would be a useful external validity check. Future work should extend the audit to multi-market ETF panels, individual stocks, intraday horizons, and alternative vendors.

The candidate pool is small and fixed by design, and additional baseline comparisons are introduced to test whether the observed instability is attributable to model-pool weakness or to the underlying signal environment. In particular, a leakage-controlled historical-mean reference and an equal-weight ensemble of the five candidate predictions are evaluated without additional model training. More powerful architectures, fundamental data, cross-sectional learning, or additional hyperparameter searches could produce different results, although broader searches would also increase selection and multiple-testing risk. The five-model pool should therefore be interpreted as a controlled CPU-only audit panel spanning different computational regimes, not as an exhaustive model universe. Future work can apply the same audit layer to larger architecture sets, including foundation or transformer-style time-series models, provided that the expanded search space is accompanied by stronger multiplicity and selection-risk controls. PC-Audit evaluates selection among the stated CPU-only models; it is not an impossibility result for ETF prediction and should not be read as evidence that Chinese ETF returns are intrinsically unpredictable.

The calibration corrects only global amplitude. It does not address nonlinear conditional bias, volatility-dependent calibration, or probability calibration. Long-term regime changes may also require adaptive monitoring rather than a fixed annual validation rule. The audit-state carry-over and fallback rules reported here are lightweight governance simulations; a fully adaptive implementation that changes the model pool, validation length, fallback policy, or hyperparameter schedule would require a separately prespecified online protocol and additional statistical checks. The volatility-like target extension is also limited: it uses the same five-model CPU-only pool and tests whether risk-oriented targets give more auditable validation transfer, not whether PC-Select dominates all volatility references. The robustness scripts explicitly remove target-derived columns from the feature matrix, but the targets still share the same ETF panel and OHLCV data source. The sensitivity variants reuse the same dataset, and the exact top-one test treats cases as Bernoulli trials even though dependence remains; the cluster and year-block summaries are therefore reported alongside it. Finally, signal calculations omit bid-ask variation, market impact, taxes, short-sale constraints, and fund-specific implementation details. They are not investable backtests.

6. Conclusions

This paper presents PC-Audit, a reproducible decision-reliability protocol for auditing validation-based machine-learning model selection under temporal dependence. It is evaluated here in weak-signal financial time series. The framework is tested in a controlled CPU-only benchmark of nine Chinese index ETFs. It enforces target-date partitioning, evaluates non-negative scale calibration prequentially, separates validation-time ranking from MCS uncertainty, measures validation-to-test transfer and regret, audits dependence, costs, execution, sensitivity, and temporal stress behavior, and records completed-year audit states for the next scheduled review.

The three audit questions posed in the Introduction are answered as follows. First, the leakage-controlled validation rule does not justify a strong signed-return selection claim: PC-Select has only borderline top-one evidence and no Holm-adjusted lower-error advantage over naive or individual-model references. Directional metrics remain near chance, while weak 2025 transfer, large confidence sets, small validation margins, weak average rank transfer, and sensitivity to calibration timing further limit the interpretation. Second, several volatility-related targets show stronger audit evidence than signed returns under specified pools and losses, but this difference is not uniform across targets or loss definitions and remains bounded by competitive econometric references and weaker 2025 stress results. Third, completed audit outputs can be carried forward as monitoring variables and illustrative fallback actions in a rule-based retrospective simulation without converting the benchmark into an online learner; the uncertainty analysis does not establish a reliable predictive improvement.

These answers lead to a conservative practical conclusion. In this setting, ETF signed-return model selection remains a weak-signal problem, validation ranking does not guarantee future stability, and a one-year validation interval provides insufficient evidence for declaring a stable winning return model. The main empirical finding is not that PC-Select dominates naive forecasts, but that ordinary validation-based model selection would be overinterpreted without an audit layer. The favorable methodological finding is that PC-Audit can distinguish evidence profiles: signed returns do not support a strong interpretation beyond the validation-selection result, whereas several volatility-related risk-surveillance targets show stronger evidence under specified conditions while remaining sensitive to target, loss, pool, and temporal stress. Rather than arguing that financial prediction has become reliable, PC-Audit argues that selection decisions themselves should be audited before they are interpreted. This distinction is the principal contribution of the present work. Future research may extend the framework to broader asset universes, alternative data sources, and adaptive audit schedules while preserving strict out-of-sample evaluation.