A Bayesian-Optimized XGBoost Approach for Money Laundering Risk Prediction in Financial Transactions

Abstract

The rapid expansion of global commerce has escalated the complexity of money laundering schemes, making the detection of illicit transfers an urgent but highly challenging research problem. In operational anti-money laundering (AML) systems, the extreme rarity of illicit transactions often overwhelms compliance teams with false positives, leading to severe “alert fatigue.” To address this critical bottleneck, this paper introduces an enhanced, probability-driven risk-prioritization framework utilizing an XGBoost classifier integrated with Bayesian Optimization (BO-XGBoost). By optimizing directly for the Area Under the Precision–Recall Curve (PR-AUC), the model is specifically tailored to rank high-risk anomalies under severe class imbalance. We validate the proposed approach on a rigorously resampled transaction dataset simulating a realistic 5% laundering rate. The BO-XGBoost model demonstrates exceptional prioritization capability, achieving an ROC-AUC of 0.9686 and a PR-AUC of 0.7253. Most notably, it attains a near-perfect Precision@1%, meaning the top 1% of flagged transactions are 100% true illicit activities, entirely eliminating false positives at the highest priority tier. Comparative and SHAP-based interpretability analyses confirm that BO-XGBoost easily outperforms sequence-heavy deep learning baselines. Crucially, it matches computationally expensive stacking ensembles in peak predictive precision while significantly surpassing them in operational efficiency, indicating its immense promise for resource-optimized, real-world compliance screening.

1. Introduction

Illegal financial transactions—referring to operations involving illicitly obtained funds intended to be integrated into the legitimate financial system—have a long global history [1]. With the rapid development of financial markets, financial products and transaction methods have become increasingly complex, creating greater space and opportunities for illicit operations [2]. Creating a fair and transparent financial transaction environment is a core component of promoting high-quality financial development. However, anti-money laundering (AML) efforts have faced numerous challenges in recent years (particularly from 2018 to 2025). The People’s Bank of China, in its explanatory note on the draft revisions, noted that some institutions have shortcomings in fulfilling their AML obligations, their supervisory coordination mechanisms are incomplete, information sharing is limited, and the ability to monitor and respond to emerging money laundering risks requires strengthening [3]. Furthermore, efforts to address illegal activities such as corruption, cross-border gambling, and “underground banks” urgently need to be strengthened. Complex financial structures not only increase regulatory complexity but also make it more difficult to identify and prevent illicit transactions [4].

In the operation of financial markets, profit-driven behavior is a major driver of illicit transactions [5]. In pursuit of higher profits and performance, some financial institutions and practitioners are willing to violate laws, regulations, and professional ethics and engage in illicit activities. Furthermore, moral hazard is a significant factor contributing to illicit behavior [6]. With the acceleration of economic globalization, illicit financial transactions are becoming more diverse. In addition to traditional insider trading and market manipulation, new practices such as false advertising, illegal fundraising, and promises of high returns have emerged [7]. These practices not only seriously infringe upon the legitimate rights and interests of investors, but also disrupt market order and increase systemic financial risks.

Traditional financial money laundering risk prediction primarily relies on rule-based engines and expert systems [8]. These approaches typically identify suspicious transactions using pre-set rules, primarily drawn from historical cases, laws and regulations, and industry experience. Expert systems assess transaction risk based on the knowledge and judgment logic of domain experts. While these approaches can identify typical money laundering patterns to a certain extent, their adaptability and scalability are limited [9]. On the one hand, they struggle to cope with increasingly complex new money laundering methods, and on the other hand, they fail to fully leverage the advantages of big data and nonlinear feature modeling [10].

In this context, the introduction of machine learning (ML) and artificial intelligence technologies to predict financial violations and money laundering risks is of great significance [8]. ML can identify potential violation patterns and risk factors by mining historical transaction data, thereby improving prediction accuracy [11,12]. Artificial intelligence can process multi-source, heterogeneous, and large-scale data, enabling dynamic updates of risk assessments and providing more scientific decision-making support for regulators [13,14]. However, a critical operational gap remains: most conventional ML approaches frame AML strictly as a binary classification problem (illicit vs. legitimate). In real-world financial systems, rigid binary classifiers generate an overwhelming volume of false-positive alerts. Facing strictly limited investigative resources, compliance officers struggle with “alert fatigue” because binary outputs cannot indicate which suspicious transactions should be investigated first [9,15,16].

Therefore, an effective AML system requires more than just a binary label; it requires a calibrated, continuous risk score that enables the effective ranking and prioritization of high-risk transactions. Furthermore, modern money laundering schemes deliberately mimic legitimate behaviors, embedding illicit activities within complex, non-linear interactions among limited transaction features (e.g., amounts, timestamps, counterparties). While advanced ensemble models like XGBoost show great promise in capturing these highly non-linear patterns, their performance and stability are heavily constrained by manual hyperparameter tuning. Inadequate parameter configurations often hinder the model from reaching its full potential in complex financial environments.

To address these issues and bridge the gap between binary classification constraints and practical AML prioritization needs, this paper proposes an enhanced XGBoost framework optimized via Bayesian Optimization (BO-XGBoost). This method incorporates the Tree-structured Parzen Estimator (TPE) algorithm into the XGBoost framework to achieve efficient and adaptive hyperparameter search, outputting robust continuous risk scores for transaction prioritization.

The core novelty of this study lies not in the isolated application of a mathematical algorithm, but in framing the AML screening process as a highly calibrated risk-ranking task specifically prioritized for alert fatigue reduction. The main contributions of this paper are redefined as follows:

• Task Reformulation for Practical AML: Instead of framing the task as a conventional hard-threshold binary classification, we explicitly optimize the BO-XGBoost model to maximize the PR-AUC metric and output continuous risk probabilities. This approach directly addresses the real-world operational need for transaction-level risk ranking and prioritization.
• Automated Non-linear Pattern Extraction: By integrating the TPE algorithm with XGBoost, the proposed framework autonomously identifies optimal hyperparameter configurations. This significantly improves the model’s ability to capture complex, non-linear laundering patterns within tabular financial data without the biases and suboptimalities of manual tuning [17].
• Enhanced Operational Value: Through rigorous evaluation using risk-oriented and ranking metrics, this method demonstrates superior capability in prioritizing high-risk transactions while maintaining a near-perfect Precision@1% at the highest investigative tier, providing regulators with actionable technical support.

The remainder of this paper is organized as follows: Section 2 reviews the related work categorized by methodological paradigms. Section 3 details the dataset characteristics and preprocessing procedures. Section 4 outlines the mathematical foundation of the proposed BO-XGBoost framework. Section 5 presents the experimental results, including comparisons with diverse baseline models. Finally, Section 6 discusses the inherent limitations of the study and concludes with perspectives on future research directions.

2. Related Work

In the past decade (2015–2025), predicting financial money laundering risk has emerged as a prominent research focus, encompassing multiple dimensions including data quality, model development, technological approaches, and policy implications. To contextualize our proposed BO-XGBoost framework, we review the evolution of AML detection methodologies across four primary paradigms.

2.1. Rule-Based, Expert Systems, and Regulatory Frameworks

Early AML efforts heavily relied on static rules and policy-driven frameworks. From a policy and regulatory perspective, Premti et al. [18] examined the impact of the European Union’s Fourth Anti-Money Laundering Directive (4AMLD) on bank valuations, revealing the role of regulatory frameworks in balancing compliance costs with risk control. Considering regional contexts, Paul et al. [19] explored the relationship between AML compliance and financial inclusion, emphasizing proportionate measures. To operationalize these rules, Eddin et al. [20] introduced a multi-layered model combining rule-based systems with ML to enhance alert accuracy. While essential for regulatory compliance, purely rule-based systems struggle to adapt to novel evasion tactics.

2.2. Classical Machine Learning Approaches

To overcome the rigidity of rules, researchers turned to data-driven statistical learning. Gupta et al. [21] emphasized the critical role of high-quality input data and event definitions (e.g., suspicious transaction reports) in constructing effective detection models. Regarding model applications, Fan et al. [22] developed a credit rating model integrating support vector machines (SVM) with logistic regression analysis to identify clients with potential money laundering risk. Additionally, Wronka [23] analyzed the characteristics of cryptocurrency use in money laundering, applying anomaly detection to identify risk patterns in virtual currencies. However, these classical methods often underperform when confronted with massive, highly non-linear tabular datasets.

2.3. Deep Learning and Graph-Based Methods

With the advent of complex financial networks, deep representations gained traction. Colladon et al. [24] proposed using social network analysis (SNA) to assess risk, leveraging relational network structures to identify potential money laundering activities. Expanding on this, Usman et al. [25] applied graph-based ML models to analyze risk indices across countries, illustrating the advantages of graph models in complex financial networks. Recently, Mhammad et al. [26] employed generative AI, network analysis, and anomaly detection methods to improve the identification of suspicious transactions. Despite their expressive power, deep learning architectures often introduce unnecessary structural biases and severe inference latency when applied to independent, non-sequential tabular records [27].

2.4. Ensemble and Optimization-Based Approaches

To achieve state-of-the-art performance on tabular data, ensemble methods have become the industry standard. Xia et al. [28] proposed the ENKMRH ensemble prediction framework, integrating K-Means++ with optimization algorithms to handle complex data environments, enhancing fraud detection in digital financial systems. Similarly, gradient boosting frameworks like XGBoost have shown immense potential, though their success is highly contingent on optimal hyperparameter configurations.

While the aforementioned studies have significantly advanced AML detection, a critical gap remains: the majority of the existing literature optimizes for standard binary classification metrics (e.g., Accuracy or ROC-AUC) and relies on manual or grid-based hyperparameter tuning. Few studies explicitly integrate autonomous Bayesian hyperparameter optimization with risk-ranking metrics specifically tailored for alert fatigue reduction. Bridging this operational gap—transitioning from rigid binary detection to dynamic, calibrated risk prioritization—constitutes the core incremental contribution of the present work.

The research methods and technical implementations, performance evaluation metrics, achieved results, and key findings from the literature cited in this paper are summarized in Table 1.

Table 1. Summary of selected studies: methods, techniques, and principal conclusions.

Author(s) & Ref.	Method	Techniques	Principal Conclusion
Gupta et al. [21]	Empirical analysis	Data curation; event definition	Data quality and event definitions are primary determinants of AML model reliability.
Fan et al. [22]	Transaction modelling	SVM; logistic regression	Transaction-level scoring enables risk stratification and improves flagging of suspicious clients.
Wronka [23]	Crypto analysis	Anomaly detection	Cryptocurrency flows show distinctive anomalous patterns needing tailored detection.
Premti et al. [18]	Regulatory assessment	Policy evaluation (EU 4AMLD)	AML regulatory changes significantly affect bank valuation and compliance burden.
Colladon et al. [24]	Network analysis	Social network analysis (SNA)	Network topology uncovers suspicious relationships beyond single-transaction signals.
Eddin et al. [20]	Hybrid modelling	Rule-based filters + ML classifiers	Rule + ML hybrids reduce false positives while maintaining detection coverage.
Usman et al. [25]	Cross-border study	Graph ML; synthetic data	Graph-based models capture cross-border linkages and aid detection in complex flows.
Mhammad et al. [26]	Suspicion enhancement	Generative models; network analytics	Generative models with network context help surface subtle anomalies.
Xia et al. [28]	Ensemble design	K-means++ + optimization; ensemble fusion	Optimized clustering-based ensembles better handle high-dimensional financial features.
Paul et al. [19]	Policy vs. inclusion review	Literature synthesis; data analysis	Balancing AML tightening and financial inclusion requires localized, proportionate measures.

3. Methodology and Approach

3.1. Datasets

The study uses a transaction dataset consisting of 200,000 records covering the period 1 January 2024 to 31 December 2024. This full-year period was specifically selected to ensure comprehensive seasonal coverage of financial activities and to maintain alignment with the original mathematical modeling competition’s design framework. The file contains 15 attributes (13 numeric, 2 categorical), with one binary target variable indicating money laundering confirmation and 14 input features for modeling. We set a binary target variable where a TRUE value indicates confirmation of money laundering in the transaction operation. The dataset contains 100,760 positive examples and 99,240 negative examples of money laundering incidents, with a positive example ratio of 50.38% (see Table 2).

Table 2. Summary of the transaction dataset used in this study.

Item	Value	Note
Total transactions	200,000
Money laundering cases	100,760	TRUE-like values interpreted as positive.
Positive rate (%)	50.3800
Number of features	14	(13 numeric/1 categorical).
Transaction date range	1 January 2024–31 December 2024
Transaction amount (mean/median)	2.50 × 106/2.50 × 106	(min, max) = (1.00 × 104, 4.99 × 106). Values shown in scientific notation.

Key predictor variables include transaction value (Transaction amount), transaction timestamp, origin and destination countries, reporting status, currency source, an external risk score, number of shell companies involved, and the financial institution involved. Transaction amounts (USD) are highly skewed: mean = 2,501,818, median = 2,501,311, std = 1,424,297, min = 10,032, max = 4,999,812.

We applied a $log(1+\mathrm{amount})$ transformation for modeling; the log-transformed amounts have a mean of 14.45 and a median of 14.73. Comparing groups, non-confirmed cases have a mean of ≈$2.48\times {10}^6$ (median $2.46\times {10}^6$) while confirmed cases have a mean of ≈$2.52\times {10}^6$ (median $2.54\times {10}^6$). A Mann–Whitney U test comparing amount distributions between confirmed and non-confirmed cases yields $p<0.001$ (two-sided), indicating a statistically significant difference.

It should be noted that this artificial 50.4% positive rate, while intentionally balanced for initial modeling convenience, severely deviates from real-world AML scenarios, where illicit transactions typically constitute less than 0.5% of total volume. To rigorously simulate operational constraints, our hold-out test set was specifically undersampled to a more realistic 5% illicit rate. We intentionally opted for strict random undersampling rather than synthetic oversampling techniques (e.g., SMOTE or ADASYN), as generating artificial financial records can distort the complex, non-linear boundaries inherent in true laundering typologies, leading to unreliable real-world risk calibration.

Dataset Limitations

Although sourced from the 2024 National Undergraduate Mathematical Modeling Competition (Problem B), this simulated dataset may not fully capture the dynamically evolving adversarial evasion tactics or complex transaction networks prevalent in live proprietary banking systems. Consequently, the exceptionally high performance metrics observed in this study (such as a Precision@1% of 1.0000) likely reflect the model’s absolute proficiency in isolating static structural anomalies within a controlled, non-adversarial data environment, rather than a guaranteed plug-and-play performance in real-world AML deployments.

3.2. Data Preprocessing

(1) Feature cleaning and type conversion.

Non-numeric identifiers such as Transaction ID and institution names were excluded from analytical steps, as they provide no predictive signal for ML algorithms. Categorical attributes including Transaction type, Industry, Destination country, and Currency source were flagged for transformation into dummy variables (or other numerical encoding) during model development. This ensures that categorical information is numerically encoded without imposing artificial ordinal relationships. Continuous attributes (Transaction amount, Risk Score) were preserved in their original form. Furthermore, time-related features (Transaction date) were transformed into derived variables such as day of week, transaction hour, and month to capture temporal dynamics of suspicious behavior, as temporal regularities often play a critical role in money laundering schemes [29].

(2) Data Distribution Visualization and Interpretation.

Beyond conventional preprocessing, this study also employs exploratory visualization to assess data quality and distribution characteristics. The first composite visualization provides a detailed examination of the univariate distributions of the key features.The results obtained are illustrated in Figure 1. The boxplots identify notable outliers in transaction amount, participant count, and risk score, indicating that extreme or atypical transactions constitute a non-negligible portion of the dataset. The violin plots further characterize these distributions, revealing long-tailed behavior and multimodal patterns in transaction amount and participant count, which reflect heterogeneous transaction structures [30]. Histograms augmented with kernel density estimation (KDE) highlight a pronounced right skew in transaction amount, suggesting that a limited subset of high-value transactions may exert disproportionate influence on overall risk assessment. The standalone KDE curves yield smoothed density profiles, demonstrating that risk scores predominantly cluster within the mid-to-high range, with both very low and extremely high values being relatively rare. Collectively, these visualizations capture central tendency, dispersion, and skewness at the univariate level, while emphasizing the critical role of tail behavior and outlier dynamics in shaping financial risk characterization.Outlier analysis was deliberately descriptive rather than eliminative, as such anomalies are precisely the focus of money laundering detection.

Figure 1. Preprocessing: Overview of Key Variable Distributions.

After analyzing univariate distributions, multivariate visualizations were employed to uncover feature interactions and their influence on risk scores, enabling a progressive shift from single- to multi-dimensional characterization [31].

The analysis results are shown in Figure 2. The second category of visualizations focuses on joint variability and interrelationships among variables, highlighting the differentiated patterns of risk scores across the multidimensional feature space. Kernel density estimation (KDE) demonstrates that low-risk transactions exhibit narrow, peaked distributions in both transaction amount and participant count, whereas high-risk transactions display broader and less skewed distributions, indicating greater deviation from typical low-risk ranges. The bivariate scatter plots further reveal evident associations between risk levels and specific feature combinations, although the relationships are not strictly deterministic, as low-risk points are still observed in regions with high attribute values. This underscores the multidimensional nature of risk formation and suggests that no single feature can fully capture risk patterns. Moreover, the joint distribution of transaction amount and participant count shows an overall trend of increasing risk with larger values, yet anomalies highlight that these predictors, while informative, are not sufficient on their own. Collectively, these visualizations emphasize that transaction amount and participant count are critical dimensions for distinguishing risk levels, but robust risk prediction requires an integrative multi-feature modeling framework.

Figure 2. Preprocessing: Bivariate distribution of transaction amount and people involved (re-rendered to enhance legibility of data clusters).

(3) Correlation structure evaluation.

We computed the Pearson correlation matrix over numeric features and visualized the upper triangle via a masked heatmap Figure 3. The Pearson correlation heatmap (upper-triangle) demonstrates that the pairwise correlations among numeric features are uniformly weak, with coefficients concentrated between −0.05 and 0.07. No feature pair exhibits a statistically meaningful linear dependence, suggesting an absence of severe redundancy or multicollinearity within the dataset. For example, “Transaction amount” shows only marginal association with “People involved” (r $\approx 0.02$), and “Institution” displays the highest observed correlation (r $\approx 0.07$ ) with “Tax avoidance,” which remains negligible in practical terms. Such uniformly low coefficients indicate that the features capture largely independent dimensions of the data, thereby minimizing the risk of unstable parameter estimation in subsequent modeling. This structural property implies that feature selection should prioritize predictive relevance rather than correlation pruning, and dimensionality-reduction techniques (e.g., PCA) are not immediately required unless dictated by downstream performance considerations.

Figure 3. Preprocessing: Feature Heatmap.

3.3. Proposed Solution

The primary objective of this section is to describe a robust, data-driven pipeline for estimating transaction-level risk probabilities using an XGBoost classifier combined with automated hyperparameter optimization. Raw transaction records are first preprocessed to extract temporal attributes (hour, day, month) and to encode categorical predictors (one-hot encoding in the present implementation). After feature engineering, the dataset is partitioned into an 80% training–validation set and a 20% hold-out test set; the former is used for model fitting and internal hyperparameter tuning, while the latter is reserved for an unbiased assessment of final model performance.

Hyperparameter optimization was performed with the hyperopt library using the Tree-structured Parzen Estimator (TPE) sampler. The search space comprised learning rate (${10}^{-5}$–1.0), maximum tree depth (1–20), subsample ratio (0.5–1.0), minimum child weight (1–10), and column subsampling (0.5–1.0). For each candidate configuration, an XGBoost classifier was instantiated and fit to the training partition. During the Bayesian optimization process, the Area Under the Precision–Recall Curve (PR-AUC) computed on the validation partition served as the optimization objective—implemented by returning the negative PR-AUC as the loss for hyperopt (which performs minimization). The optimization routine was executed for up to 500 evaluations, and the best parameter set was used to re-instantiate a final XGBoost classifier.

After completing the hyperparameter optimization, the optimal configuration was adopted to instantiate and train the final XGBoost classifier. The model outputs a continuous probability score ${\widehat{p}}_i\in [0,1]$ for each transaction, serving as the prioritized risk score. Predictive and ranking performance was assessed using a robust combination of threshold-free metrics, namely ROC-AUC, PR-AUC, Precision@K, and the Brier Score. This metric combination provides a comprehensive perspective that strictly evaluates the model’s capacity to rank high-risk anomalies under severe class imbalance, thereby ensuring a rigorous and balanced assessment of generalization performance while enhancing the credibility of evaluation results [32].

3.4. BO-XGBoost Model Design

3.4.1. Bayesian Optimization for Hyperparameter Tuning

We employ Bayesian optimization to automate hyperparameter tuning of the XGBoost classifier and thereby improve predictive performance. Let $h\in H\subseteq {\mathbb{R}}^d$ denote the hyperparameter search space and define the objective function

$$f\left(h\right)=\mathrm{Eval}\left(\mathrm{XGBoost}(h;D_{\mathrm{train}},D_{\mathrm{val}})\right),$$

which evaluates model performance. In this work, to handle the extreme class imbalance inherent in money laundering detection, the Area Under the Precision–Recall Curve (PR-AUC) on the validation set is used as the primary metric. The optimization task is to find

$$h^{\ast }=\arg \underset{h\in H}{\min }f\left(h\right)$$

(or equivalently to maximize the chosen performance metric).

We adopt the Tree-structured Parzen Estimator (TPE) as the surrogate strategy. TPE models two conditional densities,

$$l\left(h\right)=P(h\mid f\left(h\right)<f^{\ast }),g\left(h\right)=P(h\mid f\left(h\right)\ge f^{\ast }),$$

where $f^{\ast }$ is a performance threshold (typically a low quantile of observed objectives). New candidates are proposed by maximizing the ratio $l\left(h\right)/g\left(h\right)$,

$$h_{t+1}=\arg \underset{h\in H}{\max }{\displaystyle \frac{l\left(h\right)}{g\left(h\right)}}.$$

Compared with grid or random search, TPE allocates evaluations adaptively to promising regions, naturally handles mixed parameter types (continuous, integer, categorical), and scales well to higher-dimensional spaces without requiring costly Gaussian-process inversions. By balancing exploitation of high-quality regions and exploration of lesser-visited areas, TPE improves both the efficiency and robustness of hyperparameter search for complex models such as XGBoost [33]. Data-driven predictive modeling has also been widely explored in cross-disciplinary engineering scenarios, including mechatronic [34] and aerodynamic systems [35].

3.4.2. XGBoost Algorithm

Extreme Gradient Boosting (XGBoost) implements an efficient variant of gradient-boosted trees designed for large-scale problems. The algorithm builds a strong predictor by incrementally adding classification trees and differs from classical GBDT through several practical enhancements: an explicit complexity penalty in the objective, second-order loss approximations, split procedures that handle sparse inputs, and optimized parallel tree construction. These features improve convergence stability and empirical generalization while keeping training costs manageable.

Formally, for a dataset of n examples and m features, the ensemble prediction for instance $x_i$ is written as:

$${\widehat{y}}_i=\sum _{k=1}^K{f}_k\left(x_i\right),f_k\in \mathcal{F}$$

where $\mathcal{F}$ denotes the space of trees. Each tree maps an input to a leaf index via a routing function $q(·)$ and assigns a real-valued score vector $w\in {\mathbb{R}}^T$ to its T leaves, so that $f\left(x\right)=w_{q\left(x\right)}$.

Learning minimizes a regularized objective that combines empirical loss with a tree-complexity penalty:

$$\mathcal{L}=\sum _{i=1}^{n}l({\widehat{y}}_i,y_i)+\sum _{k=1}^K\mathrm{\Omega }\left(f_k\right)$$

with a commonly used penalty defined as:

$$\mathrm{\Omega }\left(f\right)=\gamma T+{\displaystyle \frac{1}{2}}\lambda \sum _{j=1}^T{w}_j^2$$

where $l(·,·)$ is the pointwise loss, and the parameters $\gamma$ and $\lambda$ control tree complexity and leaf-weight regularization, respectively. Integrating $\mathrm{\Omega }\left(f\right)$ into the objective discourages overly complex trees and reduces the risk of overfitting.

For the binary classification task of money laundering detection, the pointwise loss function $l({\widehat{y}}_i,y_i)$ is explicitly defined as the binary cross-entropy (logarithmic loss). To map the ensemble’s raw continuous output ${\widehat{y}}_i$ into a calibratable risk probability, a logistic sigmoid transformation is applied:

$${\widehat{p}}_i={\displaystyle \frac{1}{1+e^{-{\widehat{y}}_i}}}$$

This ensures that the final output is a continuous probability score ${\widehat{p}}_i\in [0,1]$, which accurately reflects the transaction’s laundering risk level for downstream prioritization.

Training proceeds stage-wise: at iteration t, a new tree $f_t$ is fitted to optimize the log-odds, updating predictions by ${\widehat{y}}_i^{\left(t\right)}={\widehat{y}}_i^{(t-1)}+f_t\left(x_i\right)$. Approximating the loss by a second-order Taylor expansion at the current predictions, and denoting the first and second derivatives of the log-loss by $g_i$ and $h_i$, yields an incremental objective that depends on these gradient statistics. For a leaf j containing a sample set $I_j$, we define the aggregated gradient and Hessian as:

$$G_j=\sum _{i\in I_j}{g}_i,H_j=\sum _{i\in I_j}{h}_i$$

The optimal leaf weight $w_j^{\ast }$ minimizing the local objective admits the closed-form solution:

$$w_j^{\ast }=-{\displaystyle \frac{G_j}{H_j+\lambda }}$$

Substituting this optimal weight back into the objective function provides the structural score of a tree. Consequently, the exact gain for evaluating a candidate split (dividing a leaf into Left and Right child nodes) is calculated as:

$$\mathrm{Gain}={\displaystyle \frac{1}{2}}\left[{\displaystyle \frac{G_L^2}{H_L+\lambda }}+{\displaystyle \frac{G_R^2}{H_R+\lambda }}-{\displaystyle \frac{{(G_L+G_R)}^2}{H_L+H_R+\lambda }}\right]-\gamma$$

In practice, XGBoost applies both exact (greedy) and approximate (e.g., percentile-based) split-finding strategies. Candidate thresholds are generated and evaluated using the Gain statistic above, enabling the efficient identification of high-quality splits even on large or sparse tabular datasets. These fundamental components—regularized objectives, second-order updates, gradient-scored splits, and efficient split search—explain XGBoost’s strong empirical performance and motivate its selection as the primary classifier in this AML framework.

3.4.3. BO-XGBoost Prediction Model Design

In this study, the XGBoost classification framework was integrated with Bayesian optimization to develop the BO-XGBoost risk-scoring algorithm. This approach leverages the Tree-structured Parzen Estimator (TPE) strategy to automatically identify optimal hyperparameters tailored for imbalanced learning. Based on the MLRDTs (Money Laundering Risk Data in Transactions) dataset, the BO-XGBoost prediction pipeline was established, and the workflow of the algorithm is illustrated in Figure 4.

Figure 4. Overall Flowchart of the Algorithm.

By framing the detection task as a probability-based classification problem, the proposed BO-XGBoost framework transcends the limitations of rigid binary thresholds. Integrating advanced ensemble learning with adaptive hyperparameter optimization significantly enhances the model’s capacity to rank and prioritize high-risk anomalies. Ultimately, this framework presents a scalable, robust, and operationally interpretable solution capable of effectively adapting to the evolving complexity of illicit financial behaviors, thereby providing a practical tool to mitigate compliance alert fatigue.

4. Experimental Setup and Analysis

4.1. Experimental Platform and Configuration

All experiments were performed on a high-performance workstation equipped with an Intel Core i7-14700K processor operating at 5.60 GHz, providing robust computational capability for large-scale data analysis. The software environment was based on Windows 11 with MiniConda3 as the package and environment manager, ensuring flexible dependency control and reproducibility of the experiments. Python 3.8 was adopted as the primary programming language, supplemented with standard scientific and machine learning libraries for data preprocessing, model training, and performance evaluation. This configuration ensured both computational efficiency and experimental reliability.

4.2. Evaluation Metrics and Methodology

The primary objective of this study is to predict and prioritize money laundering risks in financial transactions. Unlike traditional studies that frame this as a standard regression or balanced classification problem, this task is fundamentally a highly imbalanced binary classification and risk-ranking challenge. The original dataset provided exhibits an artificial balance with a positive (illicit) rate of 50.38%. While this balanced distribution is beneficial for initial model convergence during training, it diverges significantly from real-world financial environments where money laundering activities constitute a minuscule fraction of total transactions. To ensure the operational validity and commercial relevance of our evaluation, the hold-out test set was systematically undersampled to simulate a realistic 5% illicit transaction rate. This strict resampling guarantees that the model’s performance is tested under conditions that mimic the severe class imbalance faced by actual AML compliance teams.

Given the imbalanced nature of the target application, traditional threshold-based accuracy metrics or standard regression error measurements (e.g., MSE, MAE) are structurally inadequate and heavily biased toward the majority class. Therefore, the predictive and ranking performance of the proposed BO-XGBoost framework is evaluated using a robust suite of threshold-free ranking metrics and probability-calibration scores. These metrics are specifically selected to assess the model’s capacity to identify high-risk anomalies without triggering severe alert fatigue. The specific definitions and formulas of the evaluation metrics used in this study are summarized in Table 3.

Table 3. Evaluation metrics for risk prioritization and classification.

Name	Formula
ROC-AUC	${\displaystyle \mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(t\right)d\mathrm{FPR}\left(t\right)}$
PR-AUC	${\displaystyle \mathrm{PR}\text{-}\mathrm{AUC}=\sum _{k=1}^n{\mathrm{Precision}}_k\Delta {\mathrm{Recall}}_k}$
Precision@K	${\displaystyle \mathrm{Precision}@K=\frac{1}{K}\sum _{i=1}^K\mathbb{I}\left(y_{\left(i\right)}=1\right)}$
Brier Score	${\displaystyle \mathrm{BS}=\frac{1}{n}\sum _{i=1}^n{\left({\widehat{p}}_i-y_i\right)}^2}$

Note: ${\widehat{p}}_i$ is the predicted risk probability, and $y_i\in \{0,1\}$ is the actual binary label. $y_{\left(i\right)}$ denotes the true label of the transaction with the i-th highest predicted risk score, and $\mathbb{I}(·)$ is the indicator function.

This combination of metrics provides a comprehensive perspective that strictly evaluates the model’s capacity to rank high-risk anomalies under severe class imbalance, thereby ensuring a rigorous and balanced assessment of generalization performance [36].

To ensure robust prediction performance, Bayesian optimization using the hyperopt Python library (implementing the Tree-structured Parzen Estimator, TPE sampler) was employed to tune the XGBoost hyperparameters. In accordance with the implemented code, nine core hyperparameters were included in the search: $n_{\mathrm{estimators}}$, $\eta$ (learning rate), $d_{\max }$ (maximum depth), subsample, $w_{\min }$ (minimum child weight), colsample_bytree, $\gamma$, $re{g}_{\alpha }$, and $re{g}_{\lambda }$. Integer-valued parameters (e.g., $n_{\mathrm{estimators}}$, $d_{\max }$, $w_{\min }$) are cast to integers during model instantiation. Reasonable search ranges for each parameter were specified to guide the optimization; the exact parameter domains and optimal values are reported in Table 4.

Table 4. Hyperparameter Configuration and Optimized Values for the XGBoost Model.

Parameter Name	Parameter Space (As in Code)	Default	BO Optimal Parameter
$n_{\mathrm{estimators}}$	Uniform (1, 100)	100	98
$\eta$	Uniform (1 × 10−5, 1)	0.1	0.7546
$d_{\max }$	QUniform (1, 20, 1) (cast int)	6	19
subsample	Uniform (0.5, 1.0)	1.0	0.9007
$w_{\min }$	QUniform (1, 10, 1) (cast int)	1	3
colsample_bytree	Uniform (0.5, 1.0)	1.0	0.6647
$\gamma$	Uniform (0, 20)	0	0.0042
$re{g}_{\alpha }$	Uniform (0, 20)	0	0.6918
$re{g}_{\lambda }$	Uniform (0, 20)	1.0	6.9043

Figure 5 compares the parameter distributions obtained from Bayesian optimization (blue points) and random search (gray points). Panels (a–d) plot the search history for selected hyperparameters, with the Bayesian-best and random-best configurations marked by a red star and a black star, respectively. The plots indicate that Bayesian optimization progressively concentrates samples in promising regions. As detailed in Table 4, the TPE algorithm converged on an unusually high learning rate ($\eta \approx 0.7546$) alongside a deep tree structure ($d_{\max }=19$). While typical manual tuning for XGBoost favors lower learning rates (e.g., 0.01–0.3), this aggressive step size was uniquely counterbalanced by the exceptionally strong L2 regularization ($re{g}_{\lambda }\approx 6.9043$) discovered simultaneously by the optimizer. This synergistic hyperparameter pairing allows the model to capture deep non-linear interactions rapidly while preventing algorithmic overfitting. Furthermore, as discussed in Section Dataset Limitations, the competition dataset’s relatively clean, non-adversarial structural boundaries permit such aggressive learning without sacrificing generalization capability. This robustness is empirically supported by the minimal train-test performance gap (cross-validation gap $<0.02$) and the highly calibrated Brier Score. Overall, the visual evidence indicates that Bayesian optimization leverages information from previous evaluations to reliably converge to high-performance hyperparameter regions [37].

Figure 5. Parameter distributions of two different parameter optimization methods. (a–d) Search history for selected hyperparameters (blue points: Bayesian optimization; gray points: random search; red star: Bayesian-best; black star: random-best).

4.3. Feature Visualization

The feature importance results derived from the BO-XGBoost classifier are visualized in Figure 6. As shown in the figure, transitioning to a probability-based classification framework refined the feature importance distribution. The top five most influential features—ranked in descending order—are: Currency source, Destination country, country (of transaction occurrence), industry, and Transaction amount. Other significant contributors include temporal features (month, hour), People involved, and involved institutions.

Figure 6. Global feature importance ranking derived from the BO-XGBoost classification model, highlighting the primary predictors of transaction-level laundering risk.

The Financial Action Task Force (FATF) and allied typology studies have synthesized a concise set of operational “red-flag” indicators that strongly align with our model’s findings [38]. Core among these are the provenance and unexplained irregularity of funds, which perfectly justifies the model’s heavy reliance on the top-ranked Currency source and Transaction amount. Complex layering and routing across multiple jurisdictions, which breaks straightforward audit trails and obscures origin and destination links, are repeatedly documented as hallmark mechanisms in trade-related and cross-border laundering typologies. This is accurately captured by the model’s prioritization of Destination country and country of occurrence [39].

Furthermore, transaction-level attributes that empirically improve detection—such as the specific industry recording the flow, the number of participants (People involved), the participation of specific institutions, and anomalous temporal patterns (month, hour) that depart from normal commercial behavior—are consistently flagged across FATF case studies and national risk assessments as high-value features for screening and risk-scoring [38]. Taken together, these empirically grounded typologies not only justify our feature selection but also demonstrate that the BO-XGBoost classifier successfully captures the most critical, real-world operational mechanisms of cross-border money laundering.

To provide deep interpretability and specifically address how the model distinguishes between different risk profiles, we conducted both local and global SHAP (SHapley Additive exPlanations) analyses. Figure 7 directly contrasts the feature contributions of a high-risk laundering transaction versus a low-risk legitimate transaction, supplemented by a global impact summary.

Figure 7. SHAP interpretability analysis of the BO-XGBoost classifier. (a,b) Local force plots detailing feature-level risk contributions for individual instances. (c) The global SHAP summary plot, illustrating the distribution of feature impacts across the entire resampled dataset (red dots denote higher feature values, while blue dots denote lower values).

In the local force plots, the baseline value represents the average model output across the dataset. As illustrated in Figure 7a, for a true laundering instance, specific anomalous values in features such as Currency source, Transaction amount, and Destination country exert strong positive SHAP values (red arrows), aggressively pushing the risk probability higher. Conversely, Figure 7b demonstrates a legitimate transaction where the standard nature of the involved industry and typical People involved exert negative SHAP values (blue arrows), effectively suppressing the risk score.

Furthermore, the global summary plot in the bottom panel reveals the overall distribution of feature impacts. For instance, high values in ’Currency source’ and ’Destination country’ (represented by red dots) systematically stretch the SHAP values to the right, indicating a strong positive correlation with laundering risk. This comparative analysis clearly maps feature importance to actual prediction behavior, validating that the BO-XGBoost evaluates complex contextual combinations of financial attributes rather than relying on isolated static rules.

5. Experimental Results and Discussion

The primary objective of this experimental phase is to rigorously evaluate the BO-XGBoost model’s capability to predict and prioritize money laundering occurrences. The dataset was divided into training and test sets in an 80:20 ratio. To address the inherent class imbalance of real-world financial systems, the test set was meticulously resampled to reflect a realistic 5% illicit transaction rate. During the training phase, random undersampling was applied to the training data, and Bayesian optimization utilized a 20% validation subset to iteratively optimize hyperparameters. The final optimized parameters were then used to retrain the model, which was subsequently applied to the highly imbalanced test set, ensuring an unbiased and commercially realistic evaluation of its detection capabilities.

To benchmark the effectiveness of the proposed BO-XGBoost approach, we compared it against a diverse spectrum of representative machine learning and deep learning architectures. The Convolutional Neural Network (CNN) is conventionally employed to capture spatial dependencies, while the Long Short-Term Memory (LSTM) network targets sequential dynamics. The Multi-Layer Perceptron (MLP) serves as a feed-forward baseline. For ensemble methods, the CatBoost Stacking model integrates multiple learners to leverage diverse model strengths, and the Gradient Boosting Regression combined with XGBoost (GBR-XGBoost) represents a hybrid approach. This selection covers linear, tree-based, and deep learning paradigms, providing a comprehensive framework to highlight the specific strengths and operational trade-offs of the proposed method in handling tabular financial data [40].

Table 5 presents the comprehensive performance of all evaluated models under the strictly resampled, highly imbalanced scenario (simulating a 5% illicit transaction rate). As anticipated, the linear baseline (Logistic Regression) entirely failed to capture the non-linear fraud typologies, yielding an ROC-AUC of 0.5115 and a PR-AUC of 0.0544. Similarly, deep learning architectures relying on strong spatial (CNN) or sequential (LSTM) inductive biases exhibited exceptionally poor ranking performance, failing to prioritize high-risk alerts effectively (LSTM achieved a 0.0000 Precision@1%). This empirically validates that forcing linear algorithms or complex sequential architectures onto independent, non-sequential tabular financial records introduces unnecessary noise and structural inadequacy.

Table 5. Predictive and ranking performance of evaluated models under realistic imbalanced scenarios.

Model	ROC-AUC	PR-AUC	Precision@1%	Precision@5%	Brier Score
Logistic Regression	0.5115	0.0544	0.0723	0.0537	0.1733
CNN	0.6165	0.0930	0.4000	0.1154	0.2589
LSTM	0.7044	0.1172	0.0000	0.1538	0.2414
MLP	0.9379	0.5157	0.8000	0.3462	0.0651
LightGBM	0.9226	0.6059	0.9719	0.5341	0.1295
GBR-XGBoost	0.9750	0.5649	0.4000	0.5385	0.1057
Cat-Boost Stacking	0.9443	0.7426	1.0000	0.6846	0.0290
Random Forest	0.9656	0.8100	1.0000	0.7562	0.0530
BO-XGBoost (Ours)	0.9686	0.7253	1.0000	0.6538	0.0796

Note: ${\widehat{p}}_i$ is the predicted risk probability, and $y_i\in \{0,1\}$ is the actual binary label. $y_{\left(i\right)}$ denotes the true label of the transaction with the i-th highest predicted risk score, and $\mathbb{I}(·)$ is the indicator function. Best performing metrics are highlighted in bold. Lower values indicate optimal performance for Brier Score. The test set was resampled to contain approximately 5% positive (illicit) cases to simulate real-world conditions. The near-perfect Precision@1% of 1.0000 is intrinsically tied to the specific structural characteristics of this competition dataset and should not be broadly generalized to live banking environments without future validation.

Conversely, tree-based ensemble methods demonstrated remarkable robustness in extracting complex non-linear interactions. As shown in Table 5, Random Forest achieved the highest PR-AUC of 0.8100 and a Brier Score of 0.0530, followed by the Cat-Boost Stacking ensemble. However, assessing operational viability solely through PR-AUC fundamentally ignores the engineering constraints of real-world AML deployments. In a real-time transaction screening environment, standard Random Forests (a bagging ensemble) and heavy stacking models require constructing, tuning, and concurrently traversing hundreds of deep, unpruned decision trees. This architectural complexity incurs prohibitive memory footprints and high inference latency that severely bottleneck high-throughput screening systems handling millions of daily transactions.

In stark contrast, our proposed BO-XGBoost demonstrates exceptional operational pragmatism. Most notably, despite having a lower PR-AUC than the heavy Random Forest, BO-XGBoost successfully matched it in the most critical business-oriented metric—achieving a near-perfect Precision@1%, while maintaining a superior overall ROC-AUC of 0.9686. In practical financial compliance operations, this signifies that BO-XGBoost delivers an identical zero-false-positive rate at the highest priority investigative tier. By achieving this critical business requirement through a structurally lighter gradient boosting architecture optimized by the TPE algorithm, BO-XGBoost provides the optimal trade-off between maximal predictive precision and system scalability.

While an exceptionally high 100% precision is rare in broad fraud detection, in this specific context, it signifies that the model successfully isolates the most extreme, unambiguous laundering typologies at the very top of its risk distribution. We acknowledge that this perfect precision partially reflects the inherent characteristics of the curated competition dataset, where decision boundaries lack the deep adversarial obfuscation typically encountered in proprietary banking networks. Nevertheless, by entirely eliminating false positives at the highest priority tier within this controlled framework, the model provides a highly reliable algorithmic engine for prioritizing regulatory intelligence without burying compliance officers in false alerts.

To further validate the model’s effectiveness in continuous risk profiling and prioritization, we conducted visual diagnostics of the prediction probabilities. Figure 8 visualizes the distribution of predicted risk scores. The BO-XGBoost model exhibits excellent calibration (supported by a Brier Score of 0.0796), heavily compressing legitimate transactions toward the low-risk spectrum while sharply isolating illicit transactions at the high-risk tail.

Figure 8. Distribution of predicted risk scores for legitimate (0) and money laundering (1) transactions. The continuous scoring mechanism effectively compresses normal transactions toward the low-risk spectrum while accurately isolating illicit transactions at the high-risk tail.

Furthermore, the Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves presented in Figure 9 confirm the exceptional reliability of the proposed framework.

Figure 9. Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves for the BO-XGBoost model. Evaluated under a rigorously resampled, highly imbalanced scenario, the high PR-AUC (0.725) validates the model’s reliability in ranking suspicious transactions.

Taken together, the diagnostic suite delineates a clear and consistent picture: BO-XGBoost outperforms or matches the most complex ensembles in per-transaction risk prioritization. Under extreme class imbalance, the high PR-AUC and near-perfect Precision@1% ensure that the model can confidently rank suspicious transactions without burying investigators in false positives. These properties are especially valuable in real-world AML applications because they maximize resource utilization and mitigate compliance alert fatigue.

In future work, this modeling pipeline could be extended beyond AML applications to cross-domain predictive tasks, for example in mechatronic system forecasting [41] and other engineering contexts that similarly involve high-dimensional nonlinear relationships and challenging optimization landscapes.

6. Conclusions and Perspectives

This study addresses the critical operational challenge of “alert fatigue” in anti-money laundering (AML) screening by transitioning from rigid binary classification to a continuous, probability-driven risk prioritization framework. We developed the BO-XGBoost classifier, integrating Tree-structured Parzen Estimators (TPEs) to autonomously identify optimal hyperparameters tailored specifically for severe class imbalance. Evaluated under a rigorously resampled scenario simulating a realistic 5% anomaly rate, BO-XGBoost consistently surpassed sequence-heavy deep learning architectures—which struggled with the inductive biases of non-sequential tabular data—and matched the predictive power of computationally expensive bagging (e.g., Random Forest) and stacking ensembles, all while maintaining the agile efficiency of a single model.

The operational effectiveness of this framework is highlighted by its exceptional ranking metrics, including an ROC-AUC of 0.9686, a PR-AUC of 0.7253, and a Brier Score of 0.0796 for robust probability calibration. Crucially, the model achieved a near-perfect Precision@1% of 1.0000. In a commercial context, this signifies the model’s robust capability to isolate the most extreme, unambiguous laundering typologies at the absolute top of the risk distribution, thereby delivering zero false positives within the highest-priority investigative tier. Furthermore, local and global SHAP analyses confirmed that the model’s decision logic closely aligns with established Financial Action Task Force (FATF) typologies—anchoring primarily on contextual anomalies in currency sources, destination countries, and transaction amounts—thus providing the interpretability required in strict regulatory environments.

Despite these promising results, a critical limitation of this study must be candidly addressed regarding the origin of the dataset and the inherent challenges of temporal validation. As the transaction data originates from a mathematical modeling competition (Problem B of the 2024 National Undergraduate New-Quality Productivity Competition), it was generated in a controlled, simulated environment without the continuous, adversarial dynamics inherent to live banking systems. Consequently, our primary evaluation relied on a standard randomized train–test split. Given the chronological nature of financial transactions, this random partitioning inevitably introduces look-ahead bias (information leakage). Because the synthetic dataset lacks genuine chronological dependencies—exhibiting extreme concept drift when strict Out-of-Time (OOT) validation is applied—the exceptionally high performance metrics, particularly the near-perfect Precision@1% of 1.0000, primarily reflect the model’s capacity to isolate static structural anomalies rather than its sequential predictive power.

Looking ahead, several research directions warrant deeper exploration to bridge the gap between algorithmic potential and production-level deployment. First, validating the BO-XGBoost framework on massive, proprietary commercial bank datasets is essential to confirm its robustness against real-world adversarial evasion and dynamically evolving laundering tactics. Second, since illicit financial typologies evolve rapidly, integrating dynamic data drift detection and continuous online learning mechanisms is crucial for ensuring long-term model stability post-deployment. Finally, investigating federated learning and privacy-preserving training paradigms will facilitate the broader adoption of this methodology, enabling cross-institutional AML collaboration without violating strict data privacy regulations across isolated financial institutions.

In summary, BO-XGBoost provides a highly interpretable, computationally efficient, and extraordinarily precise tool for transaction-level risk scoring. By successfully mitigating false positives at the highest risk tiers and aligning with international AML typologies, this approach holds significant promise as a core algorithmic engine for next-generation, resource-optimized financial compliance systems.