Reliability-Aware Temporal Multiview Learning Under Distribution Shift: Direct-Identifier-Free Prediction of Unsuccessful Public Tenders

Abstract

Longitudinal administrative prediction is vulnerable to temporal leakage, institutional shortcut learning, and distribution shift. We evaluate a reliability-aware multiview framework for predicting unsuccessful public tenders using 176,470 finalized Mongolian procurement procedures (2016–2025). Initial training covers 2016–2022, development 2023, and forward evaluation 2024–2025. Current-announcement predictors are separated from historical inputs with conditional public-availability assumptions. Prior buyer-announcement histories improved PR-AUC by 0.02635 in 2024 and 0.02236 in 2025 over current structured attributes, while buyer-disjoint retraining did not establish a reliable benefit from explicit institutional identifiers. Expanded-feature domain classifiers distinguished 2023–2024 from 2024–2025 (ROC-AUC 0.896 versus 0.670), but retained a possible procurement-coding proxy; a stricter coding-excluded diagnostic showed substantially weaker discrimination. Conditional Structured + Text fusion attained PR-AUC values of 0.3269 (2024) and 0.3236 (2025). Transformer augmentation improved the 2024 ROC-AUC by 0.00749 (95% buyer-cluster CI 0.00297–0.01222), whereas its PR-AUC increments were not statistically established in either forward year. Separate calibration, selective-deferral, and conformal analyses revealed reliability limitations under temporal change. Independent replays verified saved-model inference, development fusion coefficients, displayed forward metrics, and clustered bootstrap conclusions. The Structured + Text reporting reference was selected retrospectively for stability, not confirmatory architectural superiority; historical first-publication constraints remain essential for deployment.

1. Introduction

Machine-learning systems developed from longitudinal administrative data face reliability challenges that are not adequately captured by conventional random-split evaluation. Temporal leakage, persistent institutional identifiers, recurrent behavioral patterns, and distributional change can all inflate apparent predictive performance while reducing transportability to future periods. These concerns are particularly important in public-sector applications, where predictive systems must be evaluated not only for discrimination but also for calibration, uncertainty, robustness, accountability, and suitability for human-supported decision-making.

Public procurement provides a demanding setting for studying these issues because procurement systems generate large-scale administrative records while operating within changing legal, procedural, organizational, and market environments. Reviews of artificial intelligence (AI) and machine learning (ML) in procurement describe a fragmented body of literature spanning supplier selection, tendering, forecasting, analytics, and decision support [1,2]. In the public sector, however, predictive performance must also be considered alongside accountability, contestability, data stewardship, and public value [3,4].

Prior procurement studies show that administrative records can predict adverse or operational outcomes, although their targets differ from unsuccessful tendering. Existing work includes prediction of corruption-related investigations, implementation inefficiency and breach [5], operational oversight tools [6], procedure failure or non-performance [7], and collusion or cartel screening [8,9]. These studies establish the predictive value of procurement data but provide less evidence on whether models remain reliable when institutional conditions, coding practices, or behavioral patterns change over time.

Historical relationships and textual information may provide a substantial predictive signal, but both can also create fragile dependencies. Evidence from European procurement data shows that a prior buyer–supplier network structure can strongly predict future awards in some domains [10], while procurement text can add information beyond conventional structured variables [11]. At the same time, recurrent institutional identities, repeated project patterns, and persistent linguistic cues may allow models to exploit shortcuts that perform well within familiar data environments but generalize poorly after temporal or institutional change [12]. This creates an important distinction between useful behavioral history and direct identity-based memorization.

Information timing constitutes a separate validity problem. Bid opening, bidder participation, bid values, procurement outcomes, and repeated buyer–supplier relationships may become observable only after the intended prediction point. If such information enters model development or feature construction, predictive performance can be optimistically biased [13,14]. We therefore define the tender announcement as the prediction boundary and require historical outcome-dependent information to have been observable before the focal announcement. Where field-level first-publication timestamps cannot be verified, analyses using such historical information are treated as conditional rather than strictly announcement-time admissible.

Temporal distribution shift is another central challenge. Time-based tabular benchmarks demonstrate that model rankings can change materially when realistic temporal splits replace random validation [15]. Classifier two-sample tests and feature-level distance measures provide complementary approaches for quantifying whether adjacent periods are distributionally distinguishable [16]. These methods allow temporal instability to be measured directly rather than inferred solely from calendar differences or changes in aggregate prediction accuracy.

Reliability also requires more than ranking performance. Precision–recall measures are informative under class imbalance [17], whereas calibration assesses whether predicted probabilities correspond to observed frequencies [18,19]. Conformal prediction and selective prediction address different aspects of uncertainty: conformal methods evaluate prediction-set coverage under explicit statistical assumptions, while selective prediction examines whether model error can be reduced by deferring uncertain cases [20,21,22]. Under temporal shift, these reliability diagnostics are especially important because a model may retain moderate discrimination while its uncertainty estimates or probability calibration deteriorate.

Mongolia provides a useful natural setting for examining these questions because the procurement environment experienced a major institutional transition during the study period. The revised Law on Procurement of Goods, Works and Services with State and Local Funds became effective on 1 January 2024, and Government Resolution No. 02 of 3 January 2024 established new maximum budget-cost thresholds while repealing the previous 2021 threshold resolution [23,24]. These changes provide institutional context for evaluating temporal transportability, but they are not treated as a causal intervention or instrument in the present analysis.

Against this background, this study investigates whether an announcement-time prediction system remains informative across successive future periods and how different information representations perform under temporal change. The framework combines a strict prediction-time information boundary for conservative primary models, conditional analyses of historical behavioral information, direct-identifier-free representations, distribution-shift diagnostics, rolling-origin evaluation, and multiview learning based on structured, textual, and sequential information. Reliability is assessed through discrimination, calibration, conformal coverage, selective prediction, procuring-entity cold-start analysis, and historical-sequence-depth stress tests.

The methodological contribution is a unified reliability-oriented evaluation framework that jointly addresses five problems that are often examined separately: temporal leakage, institutional shortcut learning, distribution shift, probability reliability, and uncertainty-aware deferral. Rather than treating predictive accuracy from a single static split as sufficient evidence of model quality, the study evaluates whether representations and model components continue to provide value under successive out-of-time tests. The empirical application therefore serves as a test bed for a broader question relevant to non-stationary administrative machine learning: how can the predictive contribution and temporal admissibility of historical information be assessed separately, and how should reliability be evaluated when discrimination, calibration, and uncertainty evolve differently over time?

2. Related Work and Research Gap

2.1. Machine Learning for Public-Procurement Risk and Outcomes

Public-procurement ML research has largely developed around oversight and adverse-outcome detection. Gallego et al. [5] demonstrated that contract-level administrative data can be used to construct early-warning scores for malfeasance, breach of contract, and implementation inefficiency. Salazar et al. [6] translated a related logic into a deployed oversight tool that combines predictive models of cost over-runs and delays with risk indices. In a more directly comparable outcome domain, Denisova et al. [7] modeled the probability that medicine-procurement procedures would fail to close using gradient-boosted decision-tree ensembles. These studies show that administrative information can forecast public-procurement outcomes, but their outcome definitions, validation designs, and deployment objectives differ from predicting unsuccessful tenders at announcement time.

A second line of research uses ML to screen competition and integrity risks. García Rodríguez et al. [8] compared eleven collusion-detection algorithms across public-procurement auction datasets from several countries. Fazekas et al. [9] extended cartel-screening evidence to 73 confirmed cartels across seven European countries and showed that combining multiple behavioral screens can outperform single indicators. This literature is methodologically relevant because it shows the value of historical bidding and competition features, but it targets confirmed or suspected collusion rather than unsuccessful procurement. The distinction is essential: the present target is a procedural outcome and should not be interpreted as fraud, corruption, or illegal conduct.

2.2. Behavioral History, Text, and Relational Structure

Repeated buyers and suppliers create a relational structure that standard tender-level covariates may not capture. Fountoukidis et al. [10], using longitudinal EU procurement data, found that historical buyer–supplier relationships and network measures strongly predicted future awards in several procurement domains. Their result supports the value of relational history but also highlights a potential memorization problem: direct relationship persistence can generate high predictive performance without necessarily providing robust signals for a different outcome or a changed institutional environment.

Text offers a distinct information view. Katona and Fazekas [11] showed in a working-paper analysis of Hungarian tenders that text fields added predictive information beyond structured procurement variables when studying restrictions to competition. Because the present study uses only tender titles rather than full tender documents, its text branch is intentionally modest. The purpose is not to claim full semantic understanding, but to test whether announcement-time text provides a complementary signal after accounting for structured and historical information.

These findings motivate the study’s separation between raw identity and behavior. Direct buyer identifiers can act as high-capacity shortcuts if they encode stable historical associations that do not survive future institutional change. Shortcut learning is a broader ML problem in which models exploit decision rules that perform well on benchmark data but fail under more challenging transfer conditions [12]. The direct-identifier-free design therefore excludes raw buyer and top-level institutional identifiers from predictive inputs while allowing behavioral summaries to be examined separately. Historical outcomes, competition, and supplier-concentration measures remain conditional on prediction-time information availability, and aggregated buyer histories may still preserve indirect institutional signals.

2.3. Leakage Control, Temporal Validation, and Distribution Shift

Leakage is a central validity threat in predictive studies because information unavailable at deployment can enter training or feature construction and inflate apparent performance [13,14]. Procurement data contain several obvious post-announcement fields, including bid opening, bidder participation, bid values, winning suppliers, and final outcomes. The present study therefore defines tender announcement as the prediction boundary and requires historical outcome-dependent information to have become observable before that boundary. Because field-level first-publication times could not be verified for all historical records, models using such information are interpreted as conditional historical-input analyses rather than fully verified announcement-time predictors.

Temporal validation is equally important. TabReD shows that method rankings can change when realistic time-based splits replace random splits and that GBDT and relatively simple MLP-like models remain highly competitive in industrial tabular settings with distribution drift [15]. This evidence directly motivates the present rolling-origin design and the decision not to treat architectural complexity as valuable unless gains persist across forward years.

We measure distribution shift rather than assume it. Classifier two-sample testing frames distribution comparison as a predictive task in which periods are labeled and a classifier distinguishes them [16]. The study complements this diagnostic with feature-level distances and separate categorical and numeric-only domain-classification specifications. Because the categorical specification may retain procedure-coding proxies, the numeric-only analysis provides a narrower sensitivity check of whether temporal separation persists without categorical inputs.

2.4. Calibration, Conformal Uncertainty, and Selective Prediction

Deployment-oriented risk models require reliable probabilities, not only discrimination. Niculescu-Mizil and Caruana [18] showed that supervised classifiers can differ substantially in calibration and that Platt scaling and isotonic regression can improve some probability estimates. Guo et al. [19] later demonstrated widespread miscalibration in modern neural networks and popularized simple post hoc calibration strategies. For the present binary outcome, we assess predictive discrimination using PR-AUC and ROC-AUC, and probability reliability using Brier score, log loss, and expected calibration error within a separately identified chronological calibration protocol.

Conformal prediction provides model-agnostic prediction sets with finite-sample marginal coverage under exchangeability [21]. However, covariate shift requires additional assumptions or weighting to preserve coverage [20]. The present study therefore treats split-conformal coverage under the temporally separated calibration experiment as an empirical exchangeability diagnostic rather than an unconditional guarantee for the retained deployment model.

Selective prediction addresses a different operational question: whether a model can abstain on uncertain cases so that error declines among the cases it retains. The risk–coverage perspective formalized in selective classification [22] suits public procurement, where algorithmic scores should support prioritization and human review rather than automatically determine legal or administrative outcomes. Taken together, these strands motivate a reliability-first framework that jointly evaluates leakage control, temporal transportability, probability calibration, uncertainty behavior, and deferral.

3. Methods

3.1. Data Source, Unit of Analysis, and Analytical Cohort

Sensitivity analysis identified alternative earliest/latest announcement endpoints and a 27-procedure reassignment between the 2016 and 2017 annual cells under a different date rule. This is a sensitivity to date selection, not a basis for assuming a uniform endpoint-selection rule; annual totals must therefore be interpreted conditional on the stated reconstruction rule (Supplementary S2).

The source administrative extract contains 443,178 bidder–procurement-level rows from Mongolia’s electronic public-procurement system. Records were consolidated to the unique tender-procedure level, with inclusion limited to procedures announced during 2016–2025 and classified as finalized contract-awarded (y_i = 0) or unsuccessful (y_i = 1). The resulting cohort comprises 176,470 tender procedures, including 44,549 unsuccessful outcomes. Cross-source cohort comparison confirmed all 176,470 retained procedure keys and binary labels. Historical supplier/winner and bid-derived aggregates are reported as conditional representations; their use at the announcement boundary requires event-level publication-time evidence beyond administrative result dates (Supplementary S1–S3 and S16). A separate all-status audit found 30,599 procedures announced in 2025, of which 30,113 (98.4%) had a finalized binary outcome; reported quarterly finalization rates were 98.6%, 97.9%, 99.1%, and 98.9%, with monthly rates from 97.3% to 99.6%. These maturity checks are separate from the finalized-cohort analysis, and their denominators require the restricted all-status extract for independent reproduction.

The unit of analysis is one unique tender procedure i; b(i) denotes its procuring entity, a_i the canonical administrative announcement date, and y_i its finalized binary label. Finalization is an outcome-based inclusion criterion, not a feature observable at announcement.

3.2. Prediction Boundary, Leakage Control, and Feature Construction

For a focal tender i announced at a_i, a historical event j is temporally eligible only when its information was publicly observable strictly before a_i. For a lookback window of W calendar days, the operational event-time test is a_i − W ≤ a_j < a_i; events exactly W days old are included, while same-day events without verified publication order are conservatively excluded. Event timestamps are proxies where first-publication timestamps are unavailable; outcome-, bid-, supplier-, and winner-derived history is not confirmed announcement-time admissible solely because an administrative date precedes a_i. The complete 82-feature admissibility and discrepancy register is discussed in Supplementary S3 and S18.

The prediction timestamp is the tender-announcement date, denoted T₀. Predictors may include only information observable at T₀ or information from prior tenders whose relevant outcome or relational data had already become observable before the focal announcement. This explicit observability rule follows standard safeguards against target and temporal leakage in ML evaluation [13,14].

For focal tender i, predictors exclude its bid-opening and result-publication dates, bidder identifiers or names, bidder status, submitted bids, winners, and final outcome. The conservative announcement-only specification uses announced structured fields and title; historical feature specifications are evaluated separately subject to information-availability restrictions. Historical event eligibility requires b(h) = b(i), h ≠ i, and a_h < a_i; same-day events are excluded. Conditional historical outcome, bidder, winner, price, and relational summaries additionally require the relevant field to have been publicly observable before a_i. Administrative result dates are only proxies for first publication; where field-level first-publication timestamps are absent, these historical views are conditional and cannot establish leakage-free deployment performance. Historical lookbacks include ALL, 365-, 730-, and 1095-day windows; event-boundary inclusion, supplier/winner deduplication, missing and zero bid prices, and population-versus-sample dispersion conventions are reported only where the source-backed operational specifications support them. No information after the focal announcement enters the conservative announcement-only specification.

Announcement-date reconciliation was required for 420 of 204,632 unique procedures with multiple raw announcement dates. The observed reference-cohort endpoints were heterogeneous: among the 370 multi-date cohort procedures, 197 aligned with the earliest recorded date and 173 with the latest. Opening date was excluded from predictor inputs. Of the 370 multi-date procedures in the primary cohort, 173 had a selected date different from the earliest raw announcement. A separate 15-record boundary check (14 in the 2024 test set and one record with an invalid 1970 placeholder) used a different comparison scope from the 27-record 2016/2017 annual reassignment. Neither count is interchangeable with the 370 multi-date cohort total. Multi-date procedures represented 0.27% of the 2024 test set and 0.33% of the 2025 test set. This is a structural timestamp and split-membership sensitivity check, not a full re-estimation under alternative announcement dates (Supplementary Section S1). Figure 1 summarizes the complete leakage-control and multiview workflow.

Figure 1. The leakage-aware temporal multiview prediction framework. Current-announcement inputs define the conservative primary boundary at T₀; historical and sequence views are conditional on the verification of prior public availability. Reference development-period fusion is evaluated in 2024–2025, and auxiliary reliability protocols are separate.

The representation is organized into five information views. Current-announcement attributes and tender-title text define the conservative primary information set. A wider inventory of 82 historical candidate features encompasses institutional activity, competition, outcomes, relational characteristics, and temporal recurrence. Historical information is assessed subject to the field-specific public-availability requirements described in Sections S3, S16 and S18 of the Supplementary Materials.

• Current-tender structured attributes: estimated value, current-year budget, lot structure, procurement type, funding source, and announcement-calendar variables.
• Procuring-entity history: prior tender volume, prior finalized outcomes, historical failure rates, activity recency, and procurement-duration summaries.
• Competition and relational history: prior bidder counts, single-bid frequency, bid-price dispersion, supplier and winner diversity, concentration, and repetition measures.
• Text view: the project/tender title represented by character n-gram term frequency–inverse document frequency (TF-IDF), learned only from the training-period corpus.
• Temporal sequence view: up to the 20 most recent prior completed tenders known before T₀, represented by event-level budget, outcome, competition, supplier/winner, duration, and time-gap features.

A normalized-title project-family proxy was explored for recurrence analyses, but it is not a ground-truth retender identifier. A verified novel-project subgroup result is not claimed; the executed exact-title recurrence sensitivity is distinguished in Supplementary Section S15.

Here, “identity-free” refers to the predictive representation: the model excludes raw procuring-entity and top-level institutional identifiers from its inputs. Identifiers may be used only as grouping keys for strictly prior behavioral summaries; they are not supplied as predictive categorical features to the retained model. Because the text branch uses tender-title character n-grams, this operational definition does not imply that all indirect institutional information potentially encoded in text or behavioral histories is absent.

3.3. Temporal Development, Prediction Models, and Multiview Fusion

The evaluation chronology distinguishes 2016–2022 initial training, 2023 development, and 2024 and 2025 forward evaluations. The 2024 coefficient-refit experiment is a retrospective sensitivity analysis distinct from the development-fitted forward evaluation. Fusion coefficients, fitted preprocessing and model hyperparameters are claimed as reproduced only when their artifacts and training provenance can be independently verified (Supplementary S5, S6 and S10A).

Model development follows a rolling-origin chronology: 2016–2022 for initial training, 2023 for development, and 2024 and 2025 for successive forward evaluations [15]. Model settings were documented where supported by available records; available hyperparameter and implementation details are provided in Supplementary S5, S16 and S21; full search histories are outside the documented inference-replay scope.

• 2024 test: models are retrained using information available through 2023.
• 2025 test: models are retrained using information available through 2024.

For the forward evaluations, branch models use expanding pre-test training periods, while fusion coefficients are fitted on the 2023 development period and held fixed for 2024 and 2025. Structured + Text was adopted as the reporting reference after examining both forward years for stability. This retrospective model choice is descriptive rather than a preregistered confirmatory selection. Average precision (PR-AUC) is the primary ranking metric; ROC-AUC, Brier score, log loss, and separately specified uncertainty diagnostics are secondary. Available model configurations and evaluation protocols are detailed in Supplementary Sections S5, S6, S16 and S17.

Within this temporal design, the model set is deliberately limited to complementary statistical, machine-learning, and deep-learning families.

• Penalized logistic regression is retained as an initial statistical baseline in the Supplementary Materials.
• CatBoost [25] is the principal structured/history benchmark.
• A TabM-style BatchEnsemble/weight-sharing reproduction [26] provides a modern tabular deep-learning benchmark.
• Character TF-IDF with logistic loss provides the text-only benchmark.
• A gated recurrent unit (GRU) [27] and a compact temporal Transformer [28] encode ordered prior tender sequences.

Late fusion combines branch-level prediction scores for Structured + Text, +GRU and +Transformer systems using development-fitted logistic coefficients. For compatible branch logits z_j, a generic representation is $p_{\mathrm{i}} = \sigma ({\beta }_0 + {\sum }_{\mathrm{j}} {\beta }_{\mathrm{j}} z_{\mathrm{i}\mathrm{j}})$; with coefficients estimated using development-period predictions. The 2024 coefficient-refitting sensitivity in Supplementary Section S10A is evaluated separately from the development-fitted fusion used in the main rolling-origin comparison (Section 4.4). Preprocessing, text-vocabulary fitting, and branch alignment are restricted to their respective training periods.

3.4. Feature-Family Ablation and Direct-Identifier-Free Stress Test

Feature-family ablation compares seven executed feature specifications (A–G), whose results are presented in Section 4.2. These are not a uniformly nested cumulative series, so differences other than explicitly matched contrasts are descriptive rather than isolated marginal effects. A uses current-announcement attributes; B adds strictly earlier buyer announcements under administrative-date eligibility; and richer C–G comparisons use historical outcome, competition, relational or title recurrence proxies with the stated observability caveats.

Feature-family inventory (not a uniformly nested progression): current-tender attributes; procuring-entity announcement history; historical competition; supplier/winner relational summaries; project-recurrence proxies.

Direct-identifier-free matched comparisons remove explicit procuring-entity identifiers from predictors but may retain aggregated buyer histories, which can encode indirect identity. Separate buyer-disjoint retraining evaluates held-out buyer identities rather than simply subgroup-scoring an existing forward model; saved-prediction cold-start and history-depth comparisons answer a different question. A verified row-level project-novelty group aligned to the reported 2024 fusion predictions is unavailable, so no novel-project effect is inferred (Supplementary S11, S11B and S15).

For specified paired model contrasts, 2000 buyer-cluster bootstrap draws resample procuring entities with replacement and evaluate paired prediction/label records under common cluster draws within each year. The reported percentile confidence intervals are conditional on fitted predictions; they exclude model-retraining, selection and tuning uncertainty, and unadjusted exploratory contrasts are not confirmatory. Subgroup-specific clustered CIs have not been established for the 2024 Transformer-depth decomposition (Supplementary S8 and S11B).

3.5. Distribution-Shift Quantification

The expanded domain classifier should be interpreted as a procedure-coding-sensitive diagnostic rather than a fully coding-independent measure: procurement_type may encode institutional procedural categories. Numeric-only diagnostics are reported separately. Classifier estimates use fold-local preprocessing, with coding-excluded sensitivity reported separately (Supplementary S9 and S9B).

As a more restrictive check of procedural coding, a separate five-fold, fold-safe logistic domain classifier was evaluated using only numeric announcement information without procurement_type, funding_source or categorical procedure codes. Its mean held-out ROC-AUC was 0.5407 (SD 0.0030) for 2023–2024 and 0.5393 (SD 0.0044) for 2024–2025. This narrower diagnostic does not reproduce the high separability of the expanded-feature classifier: the large expanded-model AUC should not be attributed solely to coding-independent covariates. Both analyses remain descriptive rather than causal reform tests (Supplementary S9B and S21).

We assess adjacent-year domain separability using a three-fold CatBoost classifier two-sample diagnostic with fold-local imputation and categorical missing-value handling [16]. The numeric specification uses log_budget, log_current_budget, and current_year_budget_ratio; the expanded specification also includes lot_flag, procurement_type, funding_source, and national_investment. Although direct institutional identifiers and explicit calendar-year and raw procedure-code inputs are excluded, procurement_type may retain procedure-coding information. The categorical diagnostic is therefore not evidence of a completely coding-independent shift. Fold-level, pooled, and numeric-only results are detailed in Supplementary Section S9B.

Feature-level shift is summarized using standardized Wasserstein distance for numeric variables and squared base-2 Jensen–Shannon divergence for pooled numeric bins or empirical categorical distributions (Supplementary S9A). Numeric Wasserstein uses pooled-SD standardization; it is inapplicable to categorical features. These distances measure descriptive distribution changes, not causal attribution to the 2024 procurement reform.

Domain-classifier fold means and sample standard deviations (ddof = 1) are distinguished from pooled out-of-fold AUCs. No domain-classifier result is interpreted as proof of a causal reform effect or as a coding-proxy-free estimate.

3.6. Reliability, Uncertainty, and Generalization Stress Tests

Calibration is evaluated in a separate chronological auxiliary experiment using Brier score, log loss, and 10-bin expected calibration error (ECE). The auxiliary model is trained before the evaluation year, with distinct pre-test periods for Platt scaling and split-conformal calibration. The auxiliary experiment is reported separately from the forward fusion comparisons in Section 4.4; calendar cutoff and binning specifications are interpreted within the documented auxiliary protocol (Supplementary Sections S12 and S14).

We evaluate selective prediction on the native probabilities from the retained structured/history + text rolling model. Observations are ranked by the absolute probability margin from 0.5, and classification error, Brier score, and log loss are reported as automatic-decision coverage is reduced. This analysis is kept separate from the auxiliary Platt-calibration experiment. It is a generic symmetric uncertainty-based deferral diagnostic, not a cost-optimized procurement decision rule; deployment thresholds would require explicit false-positive, false-negative, and review-capacity costs. This risk–coverage framing follows the selective-classification literature [22].

The auxiliary uncertainty analysis reports split-conformal prediction-set coverage at nominal 90% and 95% levels using temporally separated calibration data. Available summaries include calibration sample sizes, empirical coverage, and set-size diagnostics. Empirical coverage is reported as an observed diagnostic under the auxiliary calibration protocol, not as a deployment guarantee under temporal distribution shift [20,21].

3.7. Metrics, Software, and Reproducibility

The 82-column retained historical feature matrix was compared at all 176,470 tender IDs with an archived matched feature file, with no value or missingness disagreements in that file-to-file comparison. Independently specified raw-history and announcement-window diagnostics were also executed, but those alternatives are not asserted to be identical to the archived feature-construction process; source population, date selection and first-publication time can change individual values. In addition, recoverable CatBoost, GRU and Transformer checkpoints and the development-period fusion coefficients were identified and independently exercised on their retained derived inputs. The distinct validation layers—stored-feature consistency, checkpoint inference, fusion reconstruction, metric recalculation and temporal feature admissibility—are specified in Supplementary S3, S5, S6, S16–S18 and S21.

PR-AUC is average precision. When top-decile measures are reported, predictions are ranked descending and n_0.10 = max(1, ⌈0.10N⌉) top-ranked records are retained; Lift@10% is their outcome prevalence divided by evaluation-year prevalence. Information-set boundaries must be preserved in model comparisons. Available model configurations are summarized in S5 and S17–S18; fusion provenance is described in S6, while 2024-refit sensitivity belongs to Supplementary S10A. Complete hardware records, hyperparameter-search histories, and all fitting coefficients were not available for independent verification. Reported performance estimates are distinguished from fully repeatable model-training experiments.

The evaluation unit is a finalized tender identified by a unique procedure key and binary outcome. Within each chronological training period, preprocessing and text-vocabulary estimation are restricted to training observations. Model comparisons respect their stated information sets; additional historical predictors are not assumed observable at the focal announcement without field-level publication evidence. Feature-family contrasts are not uniformly nested, and paired buyer-cluster confidence intervals quantify uncertainty conditional on fitted predictions rather than retraining variability. Experimental settings and limitations are documented in Supplementary Sections S16–S18.

The computational verification environment included Python 3.13.5, NumPy 2.3.5, pandas 2.2.3, scikit-learn 1.8.0, CatBoost 1.2.8 and PyTorch 2.10.0 (CPU build). The independent checkpoint evaluations and numerical checks report model inputs, alignment keys, package versions and error tolerances (Supplementary S21). These details make inference and selected estimator refits independently checkable, while full training hardware records, historical search trials and an end-to-end raw-extract retraining environment are not available for every model branch. These boundaries define the scope of reproducibility established by checkpoint inference and prediction verification.

4. Results

4.1. Analytical Cohort and Historical-Information Availability

The analytical dataset comprised 176,470 finalized public-procurement procedures announced in Mongolia between 2016 and 2025, including 44,549 unsuccessful procedures (25.24%). The tender procedure was the unit of analysis, with unsuccessful procurement defined as the positive outcome and contract award as the negative outcome. The original administrative extract contained 443,178 source records, which were consolidated at the tender level before applying the study-period and binary-outcome eligibility criteria.

The temporal distribution of the analytical cohort is presented in Table 1. Annual tender volume increased from 336 procedures in 2016 to 30,113 in 2025, although this pattern should be interpreted in the context of changing administrative coverage, procurement activity, and data-recording practices. The unsuccessful-tender prevalence increased from 17.56% in 2016 to a peak of 28.49% in 2022, before declining to 25.09% in 2024 and 23.94% in 2025. These patterns describe observed annual outcome distributions and do not establish causal effects of institutional or regulatory changes.

Table 1. Analytical cohort and unsuccessful-tender prevalence by announcement year, 2016–2025.

Year	Tender Procedures, N	Unsuccessful Tenders, N	Unsuccessful Prevalence, %
2016	336	59	17.56
2017	3817	642	16.82
2018	9335	2196	23.52
2019	16,256	4011	24.67
2020	21,549	5426	25.18
2021	19,673	5089	25.87
2022	21,499	6124	28.49
2023	24,286	6365	26.21
2024	29,606	7429	25.09
2025	30,113	7208	23.94
Total	176,470	44,549	25.24

Note: The unit of analysis is one finalized tender procedure. The positive class denotes an unsuccessful procurement outcome, whereas the negative class denotes contract award. Annual prevalence is calculated as the proportion of unsuccessful tenders among finalized binary outcomes in the corresponding year. The 2024 and 2025 cohorts are the forward temporal evaluation samples. The aggregate prevalence is calculated from pooled tender counts, not as the unweighted average of annual percentages.

The two out-of-time evaluation cohorts contained 29,606 procedures in 2024, of which 7429 were unsuccessful, and 30,113 procedures in 2025, of which 7208 were unsuccessful. The corresponding outcome prevalences were 25.09% and 23.94%, respectively. These cohorts formed the forward-evaluation periods used to assess predictive performance under temporal change.

Historical-information availability was assessed separately from the current-announcement predictors. The feature-construction framework distinguished current-tender attributes, prior buyer announcements, historical procurement outcomes, competition-related information, supplier and winner relationships, and project-recurrence indicators. These feature families differ in their information requirements and in the extent to which historical public availability can be established from the administrative records.

For the conservative primary prediction specifications, only current-announcement structured attributes and tender-title information were retained. Buyer-announcement histories were evaluated separately using prior administrative announcement dates, while outcome-dependent, competition-related, supplier/winner, and project-recurrence histories were treated as conditional or exploratory information. Administrative event chronology alone could not establish the first-publication timestamp of every historical field. Consequently, historical feature availability was not interpreted as verified real-time observability, and the richer historical-input models were not treated as equivalent to the strict announcement-only predictors.

4.2. Incremental Value of Historical Information and Identity-Free Representation

The feature-family ablation examined the incremental predictive contribution of historical procurement information under the 2024 and 2025 forward-evaluation protocols. The current-announcement structured CatBoost specification provided the conservative baseline, while progressively richer specifications incorporated prior buyer announcements, historical outcomes, competition-related variables, supplier and winner relationships, and project-recurrence indicators. Because the administrative records do not establish the first-publication timestamps of all historical fields, the historical-input comparisons are interpreted as conditional predictive evidence rather than verified leakage-free deployment performance.

The current-announcement specification achieved PR-AUC values of 0.2833 in 2024 and 0.2899 in 2025 (Table 2). Adding historical buyer-announcement information increased PR-AUC to 0.3097 and 0.3123, respectively. The corresponding paired buyer-cluster differences were +0.02635 (95% CI: 0.00460–0.04662) in 2024 and +0.02236 (95% CI: 0.01056–0.03309) in 2025. Both intervals excluded zero, providing evidence that prior buyer-announcement information contributed incremental ranking information under the administrative-date eligibility protocol. These improvements should not be interpreted as fully verified real-time gains because first-publication timing could not be established for every historical record.

Table 2. Feature-family ablation under the 2024 and 2025 forward temporal evaluations.

Feature Specification	Information Boundary	2024 PR-AUC	2025 PR-AUC
A. Current-announcement attributes	Conservative primary	0.2833	0.2899
B. +Prior buyer announcements	Conditional announcement history	0.3097	0.3123
C. +Historical outcome proxies	Exploratory; publication timing unverified	0.3255	0.3158
D. +Competition proxies	Exploratory; bid-related timing unverified	0.3204	0.3137
E. +Supplier/winner relationship proxies	Exploratory; relational timing unverified	0.3207	0.3195
F. +Exact-title recurrence proxies	Exploratory; title-history timing unverified	0.3205	0.3190
G. Prior announcements + historical title	Conditional historical title information	0.3114	0.3063

Note: PR-AUC denotes area under the precision–recall curve. Specification A uses current-announcement information only. Specifications B–G incorporate additional historical information under administrative-date eligibility rules; their reported predictive performance does not independently establish that all inputs were publicly observable at the corresponding prediction timestamps. The A–G labels identify the executed feature-family specifications; rows should not be interpreted as a uniformly nested sequence of cumulative additions. Confidence intervals for the A-to-B PR-AUC contrasts were obtained using 2000 paired buyer-cluster bootstrap replicates.

Further feature-family additions produced heterogeneous results. Incorporating past-outcome proxies increased PR-AUC to 0.3255 in 2024 and 0.3158 in 2025. Adding competition-related information yielded 0.3204 and 0.3137, while supplier/winner relationship proxies yielded 0.3207 and 0.3195, respectively. Exact-title recurrence features produced PR-AUC values of 0.3205 in 2024 and 0.3190 in 2025. These patterns suggest that richer administrative histories can contain additional predictive information, but improvements were neither uniform nor necessarily cumulative across feature families. In particular, outcome-dependent, bidding-related, and relational features remain subject to unverified historical information-availability conditions.

A more conservative augmentation combining prior buyer announcements with historical title information achieved PR-AUC values of 0.3114 and 0.3063 in 2024 and 2025, respectively. This specification illustrates that incorporating additional historical variables does not automatically improve out-of-time ranking performance. The results therefore support selective evaluation of information families rather than assuming that greater feature complexity necessarily improves predictive accuracy.

Figure 2 summarizes the feature-family ablation results across the two forward-evaluation years. The figure should be interpreted as a comparison of conditional feature specifications rather than a demonstration that all historical inputs are admissible for real-time prediction.

Figure 2. Feature-family ablation across the 2024 and 2025 forward temporal evaluations. PR-AUC values are shown for current-announcement attributes and conditional historical-input specifications. Asterisks indicate historical feature families subject to unverified information-availability timing.

The contribution of direct institutional identifiers was examined separately using matched identity-bearing and identity-free CatBoost specifications. In ordinary forward evaluation, the buyer-ID specification achieved higher PR-AUC than the identity-free comparator: 0.3271 versus 0.3044 in 2024 and 0.3291 versus 0.3091 in 2025. However, these ordinary forward-test differences do not isolate the generalization value of institutional identity because previously observed buyers may appear in both the training and evaluation periods.

To assess transfer across institutional entities, a buyer-disjoint evaluation was also conducted. In 2024, PR-AUC was 0.2808 for the buyer-ID specification and 0.2795 for the identity-free specification; the corresponding values in 2025 were 0.2707 and 0.2739. The paired buyer-cluster confidence intervals for the buyer-ID-minus-identity-free differences included zero in both years (2024: +0.0013, 95% CI −0.0423 to +0.0313; 2025: −0.0032, 95% CI −0.0269 to +0.0146). Thus, the ordinary forward-test advantage associated with direct buyer identifiers was not statistically established under the buyer-disjoint protocol.

Importantly, excluding raw buyer identifiers does not eliminate all institutional information. Aggregated buyer histories, procurement characteristics, recurrent project descriptions, and historical sequence patterns may still encode indirect institutional signals. The direct-identifier-free specification is therefore interpreted as a restriction on explicit identity inputs and a means of evaluating shortcut risk, rather than proof of complete institutional invariance. More detailed cold-start and historical-depth analyses are presented in Section 4.6 and the Supplementary Materials.

4.3. Distribution Shift Across the 2024 Institutional Transition

The expanded-feature domain classifier yielded fold-mean ROC-AUC values of 0.895707 for 2023–2024 and 0.669704 for 2024–2025 under fold-local preprocessing. Because procurement_type can encode procedural categories, these estimates are not coding-independent. Numeric-only and coding-excluded specifications provide complementary distributional sensitivity analyses; none establishes a causal regulatory effect (Supplementary S9 and S9B).

To characterize temporal distribution shift, three-fold domain-classification tests were conducted for the 2023–2024 and 2024–2025 transitions. Explicit calendar-year, raw procedure-code, and direct institutional-identifier inputs were excluded. Nevertheless, the expanded categorical specification includes procurement_type, which may encode procedural or method information. A second specification based on quantitative attributes alone provides a narrower sensitivity analysis without categorical inputs (Supplementary Sections S9A and S9B).

Under this specification, a fold-safe CatBoost domain classifier distinguished 2023 tenders from 2024 tenders with a mean cross-validated ROC-AUC of 0.896 (fold SD = 0.001) and PR-AUC of 0.921 (Table 3). For the 2024–2025 transition, the corresponding values were lower but remained clearly above chance, with ROC-AUC 0.670 (fold SD = 0.001) and PR-AUC 0.677. These results indicate a pronounced distributional break between 2023 and 2024 and a weaker but still detectable shift between 2024 and 2025.

Table 3. Expanded-feature and numeric-only domain-classification tests for temporal distribution shift.

Transition	Feature Set	N	Domain ROC-AUC	Fold SD	PR-AUC
2023–2024	Coding-robust current-tender features	53,892	0.896	0.001	0.921
2023–2024	Numeric structural features only	53,892	0.616	0.001	0.667
2024–2025	Coding-robust current-tender features	59,719	0.670	0.001	0.677
2024–2025	Numeric structural features only	59,719	0.606	0.009	0.621

Note. Domain classification evaluates temporal separability, not causal effects. The categorical specification retains procurement_type, which may proxy procedure/method coding; numeric-only results are separately reported. Table entries are fold means (and sample SD where shown), not pooled out-of-fold metrics. See Supplementary S9A and S9B for inputs, fold settings and limitations.

To determine whether the shift remained detectable using only structural quantitative information, we repeated the analysis with numeric structural features only. The resulting ROC-AUC values were 0.616 for 2023–2024 and 0.606 for 2024–2025, with corresponding PR-AUC values of 0.667 and 0.621, respectively. Thus, temporal separation weakened after excluding coding-related predictors and restricting the analysis to numeric structural information, but it did not disappear.

Feature-level divergence diagnostics identified differences in procurement characteristics, including funding-source composition and current-year budget allocation (Supplementary S9A). The observed separation cannot be causally attributed to the 2024 reforms; moreover, because procurement_type may retain coding-related information, the categorical domain-classifier result cannot by itself establish that the shift persists after removal of every procedure/method coding proxy. The numeric-only sensitivity provides the narrower evidence for separation without categorical inputs.

Overall, the domain-classification results indicate stronger temporal distribution shift between 2023 and 2024 than between 2024 and 2025, without establishing that the observed differences were caused by the 2024 procurement reforms.

Figure 3 illustrates the domain-classification ROC-AUC results for the 2023–2024 and 2024–2025 transitions.

Figure 3. Fold-safe domain-classification diagnostics for the 2023–2024 and 2024–2025 transitions. The plotted estimates describe temporal separability and do not establish a causal effect of procurement reform.

4.4. Rolling-Origin Performance Across Model Families

Rolling-origin evaluation compared standalone predictors and conditional multiview models in the 2024 (N = 29,606) and 2025 (N = 30,113) forward cohorts (Table 4). Under the conservative current-announcement information boundary, title-based character n-gram TF-IDF achieved PR-AUC values of 0.3036 and 0.3007, respectively, compared with 0.2833 and 0.2899 for the current-announcement CatBoost specification reported in Table 2. The title-only results provide a directly observable announcement-time benchmark; the richer historical-input models are interpreted separately.

Table 4. The rolling-origin performance of standalone and conditional multiview models in the 2024 and 2025 forward evaluations.

Model	2024 PR-AUC	2024 ROC-AUC	2024 Brier ↓	2025 PR-AUC	2025 ROC-AUC	2025 Brier ↓
Structured/history	0.3111	0.5647	0.1861	0.3111	0.6053	0.1784
TabM-style	0.2898	0.5455	0.1874	0.3068	0.6022	0.1783
Title TF-IDF	0.3036	0.5624	0.1886	0.3007	0.5826	0.1805
GRU	0.2929	0.5670	0.1868	0.3000	0.5858	0.1798
Compact Transformer	0.2991	0.5725	0.1863	0.2974	0.5811	0.1799
Structured + Text	0.3269	0.5789	0.1849	0.3236	0.6165	0.1768
Structured + Text + GRU	0.3292	0.5843	0.1844	0.3269	0.6189	0.1764
Structured + Text + Transformer	0.3294	0.5864	0.1843	0.3250	0.6158	0.1767

Note. The downward arrow (↓) indicates that lower Brier scores are better. Brier scores are better when lower. Structured/history, GRU, Transformer and fusion models use conditional historical inputs. Title TF-IDF uses current-announcement text. The current-announcement CatBoost benchmark (PR-AUC 0.2833/0.2899) is reported in Table 2 and is not interchangeable with the Structured/history row. Values are rounded to four decimals.

Recalculation of the average precision, ROC-AUC and Brier score from the aligned tender-level probability file agreed with all 48 Table 4 metric cells at their displayed precision. This calculation verifies the stored-probability-to-metric mapping and the tender/year/target alignment; it does not imply that every model family was retrained. Independently executed inference checks recover the retained GRU and Transformer branch predictions from archived checkpoints and 20-event sequence inputs (six architecture–year comparisons, maximum absolute discrepancy 1.0543 × 10⁻⁷). A separately identified three-seed, 16-input CatBoost buyer-announcement model also passed six checkpoint-to-probability tests (179,157 seed-level scores, maximum discrepancy 9.71 × 10⁻¹⁷); that CatBoost experiment must not be substituted for the broader Structured branch in Table 4 (Supplementary S5 and S21).

Among conditional historical-input standalone models, Structured/history achieved PR-AUC 0.3111 in both years. TabM-style yielded 0.2898 in 2024 and 0.3068 in 2025, while the GRU obtained 0.2929 and 0.3000 and the compact temporal Transformer obtained 0.2991 and 0.2974. No standalone neural architecture consistently outperformed the structured/history model across the two forward periods.

Combining structured and text information yielded PR-AUC values of 0.3269 (2024) and 0.3236 (2025). Adding the GRU branch produced 0.3292 and 0.3269; adding the Transformer branch produced 0.3294 and 0.3250. Differences in ROC-AUC and Brier score were also modest (Table 4). Figure 4 displays the corresponding PR-AUC point estimates across model families and years.

Figure 4. The forward PR-AUC of standalone and conditional multiview prediction models in 2024 and 2025. The points represent year-specific PR-AUC estimates; the chart does not imply statistical significance of pairwise differences. The historical-input results are conditional on information-availability assumptions.

Paired buyer-cluster bootstrap contrasts against Structured + Text did not establish a statistically reliable incremental PR-AUC benefit from either sequence branch. For the Transformer augmentation, the differences were +0.00254 (95% CI −0.00148 to +0.00687) in 2024 and +0.00140 (95% CI −0.00491 to +0.00655) in 2025. For the GRU augmentation, the differences were +0.00234 (95% CI −0.00031 to +0.00517) and +0.00327 (95% CI −0.00205 to +0.00768), respectively. All four intervals included zero.

Overall, conditional multiview models showed higher PR-AUC point estimates than the announcement-only benchmarks, but this contrast should not be interpreted as a verified real-time deployment gain: the first-publication times of all historical inputs could not be established. The evidence supports cautious interpretation of sequence augmentation and prioritizes information admissibility alongside discrimination and calibration.

4.5. Multiview Fusion and the Incremental Value of Temporal Sequence Models

The 2024 coefficient-refit experiment evaluated on 2025 is a retrospective coefficient-sensitivity exercise; it is distinct from a 2023-development-frozen fusion specification. Independently retrained neural sensitivity-analysis results are reported separately in Supplementary S19 and cannot replace the retained model estimates in Table 4 and Table 5.

Table 5. Paired procuring-entity-clustered bootstrap contrasts in the 2024 and 2025 forward evaluations.

Year	Comparison	Δ PR-AUC (95% CI)	Δ ROC-AUC (95% CI)	Δ Brier (95% CI)
2024	+GRU vs. Structured + Text	+0.00234 [−0.00031, +0.00517]	+0.00545 [+0.00231, +0.00881]	−0.00043 [−0.00111, +0.00028]
2024	+Transformer vs. Structured + Text	+0.00254 [−0.00148, +0.00687]	+0.00749 [+0.00297, +0.01222]	−0.00052 [−0.00113, +0.00009]
2024	TabM-style vs. CatBoost	−0.02125 [−0.04010, −0.00519]	−0.01918 [−0.03694, −0.00152]	+0.00134 [−0.00014, +0.00288]
2025	+GRU vs. Structured + Text	+0.00327 [−0.00205, +0.00768]	+0.00245 [−0.00087, +0.00577]	−0.00035 [−0.00079, +0.00003]
2025	+Transformer vs. Structured + Text	+0.00140 [−0.00491, +0.00655]	−0.00062 [−0.00482, +0.00337]	−0.00008 [−0.00059, +0.00040]
2025	TabM-style vs. CatBoost	−0.00435 [−0.01168, +0.00744]	−0.00308 [−0.01131, +0.00539]	−0.00004 [−0.00074, +0.00068]

Note. The differences equal the first-listed model minus its comparator. The intervals are 95% paired buyer-cluster bootstrap confidence intervals; the negative differences favor the first-listed model for Brier only. The results use conditional historical inputs where applicable. The values are reported to five decimal places to preserve small differences; rounding of the Table 4 model metrics may not exactly reproduce these contrasts.

Incremental sequence-model contributions were assessed using paired, procuring-entity-clustered bootstrap contrasts against the Structured + Text fusion (Table 5). Fusion coefficients were estimated on 2023 development predictions and held fixed for the 2024 and 2025 forward evaluations. The fusion analyses use conditional historical inputs; consequently, they should not be interpreted as fully verified real-time, announcement-only deployment results.

Independent 2000-replicate paired procuring-entity-clustered resampling of the retained forward probabilities reproduced the twelve Table 5 point differences (tolerance 5 × 10⁻⁶). All twelve independently obtained 95% percentile intervals overlapped the retained intervals and gave the same zero-inclusion decisions, despite different resampling seeds. These checks corroborate the fitted-prediction uncertainty interpretation, not variation from model retraining; neither forward-year Transformer PR-AUC increment was statistically established by the reported buyer-cluster intervals (Supplementary S8 and S21).

The development-fitted fusion specification was verified by independently refitting three penalized logistic combiners on 24,286 aligned 2023 development records (C = 0.1; lbfgs; maximum 1000 iterations). All three coefficient and intercept sets agreed numerically with the fitted values, provided the architecture-specific matched 2023 GRU and Transformer predictions were used. The fixed coefficients then reproduced all six 2024/2025 model–year fusion probability sets (179,157 scores) to a maximum absolute error of 2.22 × 10⁻¹⁶. The matched 2023 analysis is a numerical reproducibility check and does not convert the stability-oriented reference choice into a prospectively prespecified architecture test (Supplementary S6 and S21).

In 2024, Structured + Text achieved PR-AUC 0.3269, ROC-AUC 0.5789, and Brier score 0.1849 (Table 4). Adding the GRU branch yielded PR-AUC 0.3292, ROC-AUC 0.5843, and Brier 0.1844. The paired PR-AUC difference was +0.00234 (95% CI: −0.00031 to +0.00517), whereas the ROC-AUC difference was +0.00545 (95% CI: +0.00231 to +0.00881). The Brier difference was −0.00043 (95% CI: −0.00111 to +0.00028). Thus, a positive ROC-AUC increment was detected, but the PR-AUC and Brier intervals included zero.

For the Transformer-augmented fusion, the 2024-point estimates were PR-AUC 0.3294, ROC-AUC 0.5864, and Brier 0.1843. Relative to Structured + Text, the paired PR-AUC difference was +0.00254 (95% CI: −0.00148 to +0.00687), the ROC-AUC difference was +0.00749 (95% CI: +0.00297 to +0.01222), and the Brier difference was −0.00052 (95% CI: −0.00113 to +0.00009). The evidence therefore indicates an increment in ROC-AUC, without establishing corresponding PR-AUC or Brier improvements at the 95% confidence level.

In the 2025 forward evaluation, Structured + Text achieved PR-AUC 0.3236, ROC-AUC 0.6165, and Brier 0.1768. GRU-augmented fusion yielded PR-AUC 0.3269, ROC-AUC 0.6189, and Brier 0.1764, whereas Transformer-augmented fusion achieved PR-AUC 0.3250, ROC-AUC 0.6158, and Brier 0.1767. All three paired confidence intervals for each sequence augmentation included zero in 2025 (Table 5). Thus, the forward-year results do not establish consistent gains from sequence augmentation in the primary class-imbalance metric, PR-AUC.

The TabM-style benchmark was compared with the corresponding structured CatBoost model as a separate architectural contrast. In 2024, TabM-style minus CatBoost was −0.02125 in PR-AUC (95% CI: −0.04010 to −0.00519) and −0.01918 in ROC-AUC (95% CI: −0.03694 to −0.00152), whereas its Brier contrast was not clearly different from zero. In 2025, the intervals for all three TabM-style-versus-CatBoost contrasts included zero. These comparisons do not support stable superiority of the tabular neural benchmark.

Taken together, the paired analyses support a descriptive, stability-oriented interpretation of multiview learning. Structured + Text provides a parsimonious reference for the conditional fusion comparisons, while the GRU and Transformer branches are exploratory temporal extensions. This reference specification was selected after inspection of the two forward periods and is not a separately confirmed optimal architecture. A retrospective sensitivity refitted fusion coefficients on 2024 out-of-sample branch predictions and evaluated them on 2025; Supplementary S10A reports the resulting 2025 PR-AUCs (Structured + Text: 0.3249; +GRU: 0.3301; +Transformer: 0.3166). This retrospective comparison uses separately estimated coefficients and should not be combined with the development-fitted reference results. Full coefficient-to-prediction replication is limited by the available fitting records. Neither raw identifier exclusion nor administrative-date availability establishes the complete absence of indirect identity information or historical leakage.

4.6. Generalization Across Procuring-Entity History and Historical Sequence Depth

The retained 2024 sequence-depth decomposition in Table 6 is descriptive unless its original aligned predictions and clustered uncertainty can be independently verified. The additional 2024–2025 buyer-history-depth and project-family subgroup bootstrap uses separately trained sensitivity-analysis predictions, buyer-name proxy clusters, and exploratory strata; it is therefore not an inferential replication of Table 6 (Supplementary S20).

Table 6. Transformer augmentation by procuring-entity announcement history and historical-sequence length, 2024 forward evaluation.

Subgroup	N	Positive, n	S + T PR-AUC	S + T + Tr PR-AUC	Δ PR-AUC
A. Buyer-announcement history
No prior announcement	719	142	0.2032	0.2012	−0.00203
Prior announcements available	28,887	7287	0.3284	0.3309	+0.00255
B. Historical sequence length
0	719	142	0.2032	0.2012	−0.00203
1–2	553	126	0.3182	0.2953	−0.02290
3–5	914	166	0.2593	0.2455	−0.01381
6–10	1192	222	0.2680	0.2716	+0.00352
11–20	26,228	6773	0.3324	0.3354	+0.00298

Note. S + T = Structured + Text; S + T + Tr = Structured + Text + Transformer; Δ PR-AUC = S + T + Tr minus S + T. The positive class denotes an unsuccessful tender. A positive Δ PR-AUC indicates higher PR-AUC for S + T + Tr than for S + T. Panels A and B partition the same 29,606 procedures along different historical dimensions and must not be pooled across panels. The results were obtained from saved aligned 2024 predictions without retraining. Subgroup-specific clustered confidence intervals were not computed; differences are descriptive and do not establish statistically reliable heterogeneity. The models require conditional historical inputs whose first-publication times were not independently verified.

The additional neural subgroup sensitivity is supported by independent row-identity, label and evaluation-year checks for all 59,719 pilot forward tender records. Its 300-draw buyer-name-proxy bootstrap pertains to a separately trained sensitivity model and cannot be interpreted as the missing cluster intervals for the retained Table 6. The retrospective 2024-refit-to-2025 sensitivity likewise evaluates fusion-weight transport rather than determining that obsolete coefficients caused the 2025 attenuation (Supplementary S19–S21).

We examined heterogeneity in the 2024 forward evaluation using aligned predictions from the retained Structured + Text fusion and its Transformer-augmented counterpart. This analysis complements the buyer-disjoint experiment in Section 4.2 and the pooled paired contrasts in Section 4.5: it characterizes performance among tenders with different amounts of available buyer-announcement history and different historical-sequence lengths, rather than re-estimating either model. The subgroup results are conditional on the historical-information availability assumptions described in Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5.

Among 719 tenders with no prior buyer announcement, Structured + Text achieved PR-AUC 0.2032 and the Transformer-augmented fusion achieved 0.2012 (difference −0.00203; Table 6). For the 28,887 tenders with prior buyer announcements, the corresponding PR-AUC values were 0.3284 and 0.3309 (difference +0.00255). These subgroup comparisons indicate no observed Transformer benefit in the no-history group, whereas the point estimate was slightly positive when historical announcements were available.

The sequence-depth decomposition showed a heterogeneous pattern. Transformer augmentation had negative PR-AUC differences for histories of 1–2 (−0.02290) and 3–5 (−0.01381) prior sequence elements, but small positive differences for histories of 6–10 (+0.00352) and 11–20 (+0.00298) elements. The zero-length stratum coincided with the no-prior-announcement subgroup in this executed grouping. Because the strata vary substantially in sample size and unsuccessful-tender prevalence, their PR-AUC levels should not be interpreted as directly comparable measures of institutional difficulty.

These analyses therefore suggest a practical constraint: additional sequence complexity did not consistently improve ranking in data-sparse historical strata, while small positive point estimates appeared in the deepest groups. The pooled 2024 Transformer PR-AUC increment and its buyer-clustered confidence interval remain as reported in Section 4.5; the present subgroup differences should not be treated as independently significant. A project-family novelty comparison was considered in the archival analyses, but aligned, point-in-time project-novelty membership was not independently verified; consequently, no project-novelty performance claim is advanced here. Excluding direct institutional identifiers remains distinct from establishing complete invariance to indirect buyer-history signals.

4.7. Calibration, Selective Prediction, and Conformal Uncertainty

Calibration and conformal results in the auxiliary chronological evaluation are not computed from the same probability pipeline as the retained-model selective risk–coverage analysis. The panels of Table 7 and Figure 5 and Figure 6 should therefore be read as protocol-specific diagnostics, not a single jointly calibrated model evaluation (Supplementary S12–S14).

Table 7. The principal reliability values from the two executed protocols, with their provenance distinct.

Diagnostic/Protocol	2024	2025
A.
Auxiliary native Brier	0.1849	0.1773
Auxiliary native ECE (10 bins)	0.0181	0.0164
Auxiliary Platt ECE (10 bins)	0.0238	0.0283
B.
Retained-model error, 100%	25.03%	23.96%
Retained-model error, 50%	19.58%	17.55%
Retained-model error, 40%	18.36%	16.74%
C.
Auxiliary conformal, nominal 90%	86.44%	91.38%
Auxiliary conformal, nominal 95%	90.09%	94.96%

This table is a summary of calibration, selective prediction, and empirical conformal coverage. Panels A and C derive from the temporally separated auxiliary experiment; Panel B derives from saved native probabilities of the retained rolling model. Percentages are rounded from saved outputs; no uncertainty intervals are inferred.

Figure 5. Temporally separated auxiliary calibration and split-conformal diagnostics. The native and Platt-scaled ECE are shown by evaluation year, alongside nominal and empirical conformal coverage. The results refer to the auxiliary chronological protocol and should not be conflated with the retained rolling-model probabilities.

Figure 6. The selective-prediction risk–coverage trade-off for the retained structured/history + text rolling model in 2024 and 2025. Performance is evaluated among procedures retained for automatic prediction at each coverage level; deferred procedures are excluded from the displayed retained-set risks.

We evaluated probability reliability using a temporally separated auxiliary calibration protocol and assessed selective prediction using the retained structured/history + text rolling model. These analyses address distinct questions and must not be interpreted as outputs from one identical fitted probability pipeline. In particular, the auxiliary calibration experiment should not be equated with the rolling-origin performance estimates in Table 4.

In 2024, the temporally separated auxiliary native probabilities yielded Brier score 0.1849, log loss 0.5561, and 10-bin ECE 0.0181. The corresponding Platt-scaled values were 0.1851, 0.5565, and 0.0238. Thus, Platt scaling did not improve these three diagnostics in this executed auxiliary protocol. This is a within-protocol comparison, not a claim about the superiority of a calibration procedure across all model configurations.

In 2025, the temporally separated auxiliary native probabilities yielded Brier score 0.1773, log loss 0.5376, and 10-bin ECE 0.0164. The corresponding Platt-scaled values were 0.1775, 0.5379, and 0.0283. The results do not indicate a benefit from Platt scaling in this auxiliary comparison.

Figure 5 summarizes the temporally separated auxiliary experiment. The conformal component was estimated on a separate pre-test chronological partition rather than on the forward-test labels. At nominal 90% coverage, empirical coverage was 86.44% in 2024 and 91.38% in 2025; at nominal 95%, it was 90.09% and 94.96%, respectively. The shortfall relative to nominal coverage in some settings illustrates that the exchangeability conditions underpinning conventional split-conformal guarantees need not hold across temporal shifts. These are empirical diagnostics for the auxiliary Platt-plus-conformal protocol, not guaranteed coverage for the retained native deployment probabilities.

Selective prediction was evaluated separately by ranking the retained rolling-model predictions according to their uncertainty and deferring the least-certain procedures. In 2024, classification error decreased from 25.03% at 100% coverage to 19.58% at 50% and 18.36% at 40%. In 2025, the corresponding error rates were 23.96%, 17.55%, and 16.74%. Brier score and log loss also declined as coverage was reduced in the saved evaluation. These comparisons characterize the retained subsets; they do not imply that errors were eliminated for deferred tenders or that an operational abstention threshold has been validated.

Taken together, these diagnostics describe probability reliability and retrospective deferral under two explicitly different saved-output protocols: temporally separated auxiliary native/Platt calibration with split conformal, and retained rolling-model selective prediction. The reported auxiliary empirical coverage does not establish distribution-free coverage under temporal shift; 2024 fell below both nominal levels, while 2025 exceeded 90% nominal coverage and was slightly below 95%. Individual auxiliary predictions, calibration scores, and prediction sets are not publicly available; complete independent replication of these auxiliary analyses is therefore not established. No operational abstention threshold or review-capacity optimum is established. Historical-input observability remains conditional as described in Section 4.1.

5. Discussion

Forward evaluations distinguish whether a feature carries predictive information in a given sample from whether that information remains admissible and useful in later periods. Prior buyer-announcement history improved ranking under administrative-date eligibility, but historical public availability was not verified for all fields. Direct institutional identifiers improved ordinary forward-test PR-AUC in matched comparisons, yet this advantage was not established in buyer-disjoint evaluations. The 2023–2024 transition showed stronger distributional separation than 2024–2025. Sequence augmentation generated small point-estimate changes, without a statistically established improvement in the primary PR-AUC metric in either forward year. Calibration, selective prediction, and conformal analyses further demonstrate why ranking, probability reliability, and uncertainty must be assessed separately. The interpretation emphasizes transportability and evidence boundaries rather than universal architectural superiority.

5.1. Historical Representation vs. Institutional Memorization

Historical buyer-announcement information was the clearest supported ablation increment relative to current-announcement structured attributes. Adding prior buyer announcements increased PR-AUC from 0.2833 to 0.3097 in 2024 (paired Δ = +0.02635; 95% buyer-cluster CI [0.00460, 0.04662]) and from 0.2899 to 0.3123 in 2025 (Δ = +0.02236; CI [0.01056, 0.03309]). These intervals exclude zero under the fitted-model, administrative-date protocol. They do not establish that every historical feature was publicly observable before the corresponding focal announcement. Richer outcome, competition, supplier/winner, and recurrence inputs generated heterogeneous results and remain conditional or exploratory rather than verified strict announcement-time predictors.

Removing raw institutional identifiers did not uniformly improve prediction. In the matched ordinary forward comparisons, the buyer-ID model achieved higher PR-AUC than the direct-identifier-free comparator (0.3271 versus 0.3044 in 2024; 0.3291 versus 0.3091 in 2025). In separate buyer-disjoint retraining, however, the respective ID-bearing and ID-free PR-AUC values were 0.2808 versus 0.2795 in 2024 and 0.2707 versus 0.2739 in 2025; paired buyer-cluster intervals for their differences included zero. Thus, the ordinary setting reveals predictive identity-associated information, whereas an advantage from explicit identity was not established for previously held-out procuring entities. These distinct training/evaluation designs should not be interpreted as the same ablation.

Buyer-grouped histories may still encode institutional patterns even when categorical buyer identifiers are excluded. Persistent procurement mix, prior activity, and textual recurrence can transmit an indirect identity-related signal, consistent with the broader shortcut-learning concern [12]. The evidence supports the evaluation of explicit identifiers and behavior-based representations under forward and buyer-disjoint tests, not a causal attribution of overfitting to identity or proof of complete institutional invariance. Exact-title recurrence is an exploratory proxy; no independently verified, prediction-aligned novel-project performance result is claimed.

5.2. Prediction Under Institutional Distribution Shift

The fold-safe domain-classification diagnostic found stronger separation between 2023 and 2024 than between 2024 and 2025. Fold-mean ROC-AUC was 0.8957 and 0.6697, respectively, for the expanded-feature categorical specification; the numeric-only sensitivity yielded 0.6162 and 0.6058. The categorical diagnostic retains procurement_type, which may proxy procedure or method coding, and therefore does not establish that all coding-related information has been removed. The narrower three-fold numeric-only results retained modest separability; a distinct five-fold coding-excluded logistic sensitivity yielded ROC-AUC near 0.54 in both transitions (Supplementary S9B). Neither analysis constitutes a causal decomposition. Feature-level Jensen–Shannon and standardized Wasserstein diagnostics give complementary descriptive evidence about changes in funding composition and budget structure.

The stronger 2023–2024 separation coincides with the procurement-system transition but cannot isolate the effects of reform from changes in funding, coding, market composition, data-recording practices, or buyer activity. The documented fold-safe classifiers measure temporal distinguishability under their specified feature sets rather than a causal effect of procurement reform.

Forward validation consequently adds information that a random split would not supply. Structured + Text achieved conditional PR-AUC values of 0.3269 in 2024 and 0.3236 in 2025, while the strictly announcement-observable title-only baseline obtained 0.3036 and 0.3007. The higher conditional fusion point estimates should not be represented as verified announcement-time deployment improvements because the publication-time eligibility of all historical inputs remains uncertain. The 2023 development and expanding-origin protocol is preserved, but the reported primary fusion was selected for descriptive stability after viewing both forward periods rather than in an independent confirmatory model-selection exercise.

5.3. When Does Additional Deep Temporal Complexity Help?

The results do not establish a general advantage for deeper temporal architectures. The TabM-style benchmark had lower PR-AUC than the corresponding structured CatBoost model in 2024 (paired difference −0.02125, 95% CI [−0.04010, −0.00519]); the 2025 contrast was −0.00435 with an interval including zero. Standalone GRU and Transformer models also did not consistently outperform structured/history across the forward years. Architectural complexity therefore needs to be evaluated against matched baselines and its additional information requirements, not assumed to confer an advantage.

In 2024, adding a Transformer to Structured + Text increased ROC-AUC by +0.00749 (95% CI [+0.00297, +0.01222]), but the PR-AUC increment was only +0.00254 (CI [−0.00148, +0.00687]) and the Brier difference was −0.00052 (CI [−0.00113, +0.00009]). Thus, an ROC-AUC difference was detected while the intervals for the primary PR-AUC measure and Brier included zero. The GRU PR-AUC difference was +0.00234 (CI [−0.00031, +0.00517]), also not established as positive. In 2025, Transformer and GRU PR-AUC differences were +0.00140 (CI [−0.00491, +0.00655]) and +0.00327 (CI [−0.00205, +0.00768]); neither interval excluded zero. The 2025 Transformer ROC-AUC difference was slightly negative (−0.00062), not its PR-AUC difference.

The 2024 descriptive sequence-depth analysis found negative Transformer PR-AUC differences for histories of 1–2 and 3–5 events (−0.02290 and −0.01381) and small positive differences for 6–10 and 11–20 events (+0.00352 and +0.00298). No subgroup-specific clustered intervals were established, so these patterns do not demonstrate statistically reliable heterogeneity. In an explicitly retrospective sensitivity, fusion coefficients refitted on 2024 out-of-sample predictions yielded 2025 PR-AUC of 0.3249 for Structured + Text, 0.3301 with GRU, and 0.3166 with Transformer. Those retrospectively fitted coefficients are distinct from the 2023 development fusion coefficients used in the forward comparisons. The retrospective refits therefore do not support a simple explanation in which updating fusion coefficients restores a Transformer advantage in 2025. Nevertheless, these descriptive comparisons cannot identify whether reduced sequence informativeness, coefficient instability, or other temporal changes caused the observed performance differences. Taken together, the evidence supports stability-driven, post hoc selection of the simpler Structured + Text reporting reference, without confirmatory claims about the superiority or obsolescence of sequence architectures.

5.4. Reliability, Uncertainty, and Human Review

The reliability analyses use two distinct saved-probability protocols and cannot be collapsed into a single model calibration assessment. Within the auxiliary chronological native/Platt experiment, native Brier score and 10-bin ECE were 0.1849 and 0.0181 in 2024, versus 0.1851 and 0.0238 after Platt scaling. In 2025, the corresponding native values were 0.1773 and 0.0164, versus 0.1775 and 0.0283 after Platt scaling. Platt scaling therefore did not improve these diagnostics within the executed auxiliary comparison. Calibration slopes and intercepts are not advanced as independently verified numeric endpoints, and incomplete original ECE bin-edge provenance limits interpretation of that measure.

Selective prediction instead used retained native rolling-model probabilities. Classification error among retained cases fell from 25.03% at 100% coverage to 19.58% at 50% coverage in 2024; the corresponding 2025 figures were 23.96% and 17.55%. These are retrospective retained-subset comparisons, not prospective evidence of a cost-optimal abstention threshold or reduced error among deferred cases. Review-capacity constraints and the asymmetric costs of false positives and false negatives were not modeled.

The separate auxiliary conformal experiment achieved 86.44% and 90.09% empirical coverage for nominal 90% and 95% targets in 2024, respectively; in 2025, the corresponding values were 91.38% and 94.96%. Undercoverage in 2024 cautions against transporting conventional exchangeability-based coverage guarantees across temporal shift without further assumptions or corrections [20,21]. These empirical prediction-set diagnostics do not provide coverage guarantees for the separate retained selective-prediction pipeline. In public procurement, the model should therefore support prioritization and accountable human review rather than determine legal or administrative decisions. An unsuccessful tender is not, by itself, evidence of corruption, collusion, fraud, or unlawful conduct.

5.5. Generalization Boundaries and Deployment Constraints

Generalization depends on how much genuinely prior information is available. In the 2024 saved-prediction subgroup analysis, 719 tenders had no prior buyer announcement (142 unsuccessful), compared with 28,887 with prior announcements (7287 unsuccessful). The Transformer-minus-Structured + Text PR-AUC differences were −0.00203 and +0.00255, respectively. These differences are descriptive, are not buyer-disjoint retraining results, and lack subgroup-specific clustered confidence intervals. No verified row-level project-novelty indicator could be aligned to the 2024 fusion predictions; consequently, the earlier numeric novel-project generalization claim is not retained.

External validity remains limited to a single national procurement system during 2016–2025, with no reported cross-country replication. Changes in reporting practices, tender categories, market structure, and outcome definitions may alter both the predictor distribution and the relationship between inputs and unsuccessful procedures. The target measures a finalized procurement outcome, not post-award contract quality, lifecycle cost, implementation delays, supplier performance, or integrity violations. The 2025 all-status audit reported a 98.4% finalized binary inclusion rate, reducing a simple unresolved-case concern but not eliminating selection effects associated with finalization. Multi-announcement-date adjudication affected a small share of the forward cohorts; its sensitivity audit addressed chronology and cohort assignment rather than re-estimating all models under alternative dates.

For deployment, these limitations imply that historical fields need first-publication verification, models require monitoring and periodic forward re-evaluation, and review policies require locally validated calibration and cost-sensitive thresholds. The central contribution is an evidence-bounded assessment of prediction under non-stationarity, not proof of universally leakage-free histories, identity invariance, or conformal guarantees under drift.

6. Conclusions

The model comparisons and subgroup findings are interpreted in light of the chronological evaluation design and the retrospective selection of the preferred configuration. Further validation should examine independent buyer populations, external procurement systems, and prospective data-availability controls before deployment.

This study examined the reliability of predicting unsuccessful public tenders at announcement time under temporal and institutional change, using 176,470 finalized procedures from Mongolia during 2016–2025. The conservative prediction setting used current-announcement attributes and tender-title text, while buyer-announcement histories and richer outcome, competition, relational, and sequence-based representations were evaluated separately according to their information-availability constraints. Earlier buyer announcements improved PR-AUC relative to current structured attributes in both forward years under the administrative-date eligibility protocol, but first-publication timestamps were insufficient to verify all historical inputs as observable at the focal announcement. Excluding direct institutional identifiers did not consistently improve ordinary forward-test performance; instead, the buyer-disjoint comparisons found no statistically established advantage from those identifiers. These results distinguish predictive usefulness from temporal admissibility and direct identifier exclusion from the removal of indirect institutional signals.

Fold-safe diagnostics showed a stronger distributional break between 2023 and 2024 than between 2024 and 2025, although the categorical specification retained a possible procedure-coding proxy and neither diagnostic identifies a causal effect of procurement reform. The conditional Structured + Text fusion attained PR-AUC of 0.3269 in 2024 and 0.3236 in 2025. Although Transformer augmentation increased ROC-AUC in 2024 (paired difference +0.00749; 95% CI [+0.00297, +0.01222]), its incremental PR-AUC point estimates were small (+0.00254 in 2024 and +0.00140 in 2025), and both paired buyer-cluster confidence intervals included zero. No general architectural superiority is inferred. The Structured + Text reporting reference was selected retrospectively for stability across the two forward tests, not through a preregistered confirmatory comparison. Separate reliability analyses showed that Platt scaling did not improve the reported auxiliary calibration diagnostics, that retrospective deferral reduced error among retained predictions, and that auxiliary split-conformal coverage fell below both nominal levels in 2024. These results describe distinct evaluation protocols rather than one validated deployment pipeline.

Several limitations indicate priorities for future research. First, despite the executed feature, checkpoint, fusion and numerical verification exercises (Supplementary S21), historical competition and outcome variables require independently verifiable first-publication timestamps to establish their availability at tender announcement. Second, broader buyer-disjoint and cross-country evaluations are needed to assess transferability beyond the observed institutions. Third, prospective testing should compare sequence representations across history-depth strata and project types using prespecified models and cluster-aware uncertainty. Finally, calibration, conformal coverage, and selective-review thresholds should be evaluated under changing procurement conditions before operational use.