BI-GBDT: A Graph-Free Behavioral Interaction-Aware Gradient Boosting Framework for Fraud Detection in Large-Scale Payment Systems

Abstract

Detecting fraudulent and anomalous transactions in large-scale digital payment systems is significantly challenging due to severe class imbalance and the fact that transactional risk is tightly coupled to the historical interactions and behaviors of transacting parties. In this study, a scalable Behavioral Interaction-Aware Gradient Boosting (BI-GBDT) framework is proposed for anomaly detection in tabular transaction data to overcome these challenges. The methodology models sending and receiving behaviors separately through direction-specific clustering based on transaction frequency and amount. Each transaction is characterized by cluster-pair prevalence ratios, which capture the population-level prevalence of sender–receiver interaction patterns. To handle extreme class imbalance, all transactions are clustered, and a cluster-level risk score is computed as the ratio of anomalous transactions to the total number of transactions within each cluster. This score is incorporated as a feature, serving as a behavioral risk prior highlighting concentrated anomaly. These interaction-aware features are integrated into a GBDT in a big data environment. Experiments were conducted on a large masked real-world payment dataset spanning six months and containing more than 456 million transactions, with the prediction task defined as binary classification between fraudulent and non-fraudulent transactions. Unlike standard GBDT models trained only on transactional attributes and graph-based approaches, BI-GBDT captures sender–receiver interaction patterns in a graph-free manner and outperforms a baseline GBDT, reducing the false positive rate from 37.0% to 4.3%, increasing recall from 52.3% to 72.0%, and improving accuracy from 63.0% to 95.7%.

1. Introduction

Anomaly detection has become a critical requirement in modern digital payment infrastructures, especially in domains where rare anomaly events can cause disproportionate damage, such as cybersecurity, finance, industrial monitoring, and healthcare [1,2,3]. In large-scale payment systems, financial institutions process millions of money transfers per day, while fraudulent or anomalous transactions remain extremely rare and continuously evolving [4,5]. This combination of massive transaction volume, severe class imbalance, and strict latency requirements makes real-time fraud detection a challenging anomaly detection problem. Missed fraudulent events may lead to direct financial losses. Excessive false positives can overload analysts and disrupt legitimate transactions. In large-scale payment systems, even small increases in the false positive rate may result in high operational costs [5].

Traditional transaction screening has relied on manually crafted rule sets and scorecards [6], which are interpretable but struggle to adapt to new fraud patterns. Supervised machine learning methods such as logistic regression, random forests, and gradient boosting decision trees (GBDT) have therefore been widely adopted for fraud detection on tabular transaction data [7,8,9]. These models can handle heterogeneous features and support high-throughput deployment. However, they often rely on aggregate account statistics or simple transactional variables in real-world fraud detection settings [6,7,8,9]. This limits their ability to capture sender–receiver interaction structure and may lead to overfitting the majority class, causing rare or emerging fraud behaviors to be missed.

To better exploit relational information, recent work has explored graph-based representations of financial transaction networks and graph neural networks or random-walk-based embeddings for fraud detection [10,11,12,13]. While graph-based methods can effectively propagate risk signals across neighborhoods, their applicability to high-volume, real-time systems is constrained. In real-world financial institutions maintaining a large and continuously evolving transaction graph in production is often operationally challenging because it requires substantial computational resources and complex infrastructure for graph construction, storage, and real-time processing [14]. Furthermore, integrating such graph-based pipelines into existing Spark-based batch and streaming architectures commonly used in large-scale payment systems can introduce additional latency and system complexity. Financial institutions therefore need methods that remain compatible with tabular, feature-based infrastructures while still being sensitive to interaction patterns.

These challenges are addressed in this work by proposing a scalable, graph-free Behavioral Interaction-Aware Gradient Boosting (BI-GBDT) framework for anomaly detection in large-scale payment systems. Accounts are profiled with direction-specific behavioral clustering for send and receive channels along frequency and amount axes. For each transaction, cluster-pair prevalence ratios that quantify how typical the sender–receiver behavioral pairing is in the population are derived. Standard-score (z-score) deviations of the transaction amount relative to both parties’ historical distributions are computed. Furthermore, a robust mechanism is introduced to address the severe class imbalance. All transactions are clustered with gaussian mixture model (GMM). Then, a cluster-level risk measure is derived by computing the rate of labeled fraud transactions relative to the total number of transactions within each cluster. This risk score is then used as a supplementary feature, effectively establishing a behavioral risk prior that signals areas of heightened anomaly concentration. Finally, these newly generated behavioral metrics, interaction prevalence indicators, and localized anomaly signals are combined with traditional transactional attributes. This comprehensive feature set is then incorporated into a class-weighted Gradient Boosting Decision Tree (GBDT) model, optimized for a big data environment via a distributed PySpark and Spark MLlib deployment. The main contributions of this paper are as follows:

• Graph-Free Interaction Features: A novel feature engineering approach that captures sender-receiver relationships using direction-specific clustering, eliminating the need for an explicit graph.
• Scalable BI-GBDT with Imbalance Handling: The proposed BI-GBDT framework integrates new behavioral metrics and a cluster-level risk prior to robustly address severe class imbalance within a scalable, Spark-compatible GBDT model.
• Proven Superiority on Real Data: Empirical evaluation on large-scale real-world masked dataset demonstrates significant improvements in recall and false positive rate reduction compared to baseline models.

The remainder of this paper is organized as follows. Section 2 reviews related work on anomaly detection and fraud detection in financial transaction systems. Section 3 presents the proposed BI-GBDT framework. Section 4 describes the experimental setup and reports the results. Section 5 discusses practical considerations and limitations, and Section 6 concludes the paper.

2. Related Work

Fraud and anomaly detection in financial transaction systems has been widely studied with a variety of rule-based, machine learning, and graph-based methods. Early work focused on static expert rules and scorecards, while more recent approaches leverage supervised and unsupervised learning to cope with evolving attack patterns and large-scale data streams [1,2,3,4]. The main work in this area can be summarized into the following three aspects: (i) rule-based and classical machine learning approaches on tabular features, (ii) graph-based and interaction-aware anomaly detection methods, (iii) imbalanced learning techniques tailored to rare-event detection, and (iv) other methods.

2.1. Rule-Based and Classical Machine Learning Approaches

Traditional transaction monitoring in banks has long relied on manually crafted rules, threshold-based alerts, and heuristic scoring models [5,6]. These systems are interpretable and easy to align with regulatory requirements, but they require frequent manual tuning and are slow to adapt to new fraud strategies. To improve detection performance, many studies have applied supervised machine learning models such as logistic regression, random forests, and gradient boosting decision trees (GBDTs) to tabular transaction data [7,8,9,10]. Typical feature sets combine customer demographics, aggregate account statistics, and simple transactional variables such as amount, channel, and time of day [11]. Some works further incorporate hand-crafted risk scores or aggregated behavior profiles to enhance discrimination under class imbalance [12,13]. However, many operational detection pipelines model each transaction as an independent sample. This design choice limits the extent to which sender–receiver interaction structure is exploited and can reduce sensitivity to rare or emerging behaviors that appear primarily through unusual counterparty combinations [14].

2.2. Graph-Based and Interaction-Aware Anomaly Detection

To extract relational information effectively, some researchers have proposed graph-based representations of financial transaction networks, where accounts or cards are nodes and transfers are edges [15,16,17,18]. Graph neural networks, random-walk embeddings, and message-passing architectures have been used to propagate risk scores across neighborhoods and capture higher-order interaction patterns that are difficult to encode with simple tabular features [19,20,21]. These methods often achieve strong detection performance on benchmark datasets and internal transaction logs, especially for collusive or mule-account fraud. Nonetheless, maintaining a large, continually evolving transaction graph in production is operationally demanding. Graph construction, stateful neighborhood aggregation, and consistency management in streaming environments introduce additional latency and complexity, and many graph-based solutions are not directly compatible with existing Spark-based batch and streaming pipelines widely used in financial institutions [22]. Consequently, there is growing interest in interaction-aware methods that approximate relational signals while remaining deployable within standard feature-based machine learning infrastructures.

2.3. Imbalanced Learning

Severe class imbalance is a defining characteristic of fraud and anomaly detection problems, where the proportion of fraudulent or abnormal events can be orders of magnitude smaller than the legitimate class [23]. Naïve classifiers tend to favor the majority class, yielding high overall accuracy but poor recall on rare events. To address this, a range of imbalanced learning techniques has been explored, including class-weighted loss functions, cost-sensitive learning, and decision threshold calibration to optimize precision–recall trade-offs under operational constraints [24,25,26]. Resampling strategies such as random under sampling, random oversampling, and synthetic minority oversampling (SMOTE) have also been applied to rebalance training data and improve minority-class sensitivity [27]. Empirical studies have shown that SMOTE-based rebalancing can be effective in highly imbalanced real-world detection tasks, particularly when combined with complementary modeling or generative mechanisms [28]. More advanced methods combine clustering-based under sampling with ensemble models to reduce noise and overfitting in rare-event scenarios [29].

2.4. Other Methods

Some anomaly detection methods that treat fraud as a generic outlier detection task, such as one-class support vector machines, Isolation Forest, and autoencoder-based reconstruction models, have been proposed to reduce reliance on labeled fraud examples [30,31,32]. These approaches can be effective when labeled anomalies are scarce, but they often ignore domain-specific structure, such as directional account behavior and sender–receiver interactions, and may struggle to scale to hundreds of millions of transactions under strict latency requirements. Moreover, many imbalanced learning and anomaly detection studies evaluate on relatively small or static datasets, leaving open questions about their performance and robustness in production-scale, streaming financial systems.

In summary, rule-based and classical tabular machine learning approaches offer practicality but underutilize interaction structure, graph-based models exploit relational information at the cost of substantial operational overhead, and existing imbalanced learning methods only partially address the challenges of rare-event detection on a very large scale. The BI-GBDT framework proposed in this paper aims to bridge these gaps by encoding sender–receiver relationships through direction-specific behavioral clustering and cluster-pair prevalence ratios in a graph-free manner and by combining these interaction-aware features with class-weighted learning to enable scalable anomaly detection on large-scale payment data within a standard Spark environment.

3. The Proposed Framework

The BI-GBDT is a multi-stage fraud detection framework developed specifically for highly imbalanced and large-scale transaction data, designed to encode rich domain-specific knowledge and address computational challenges. Rather than relying on a straightforward combination of existing techniques, the BI-GBDT systematically introduces novel mechanisms at each stage to capture three critical dimensions: account-level behavioral patterns and interaction prevalence, local anomaly signals, and a robust behavioral risk prior for imbalance handling.

As illustrated in Figure 1, the framework is organized into four main sequential modules that transform raw transaction data into a comprehensive feature set for final classification:

• Interaction-Aware Feature Engineering: Focuses on deriving direction-specific account profiles and sender-receiver interaction prevalence indicators.
• Local Anomaly Signals (Z-Scores): Measures the standard-score deviation of transaction amounts against historical distributions of both transacting parties.
• Behavioral Risk Prior: Utilizes GMM clustering to establish a cluster-level risk measure, effectively creating a powerful imbalance-aware feature.
• Class-Weighted GBDT Classification: Integrates the comprehensive feature set into a scalable, distributed GBDT model for final anomaly detection.

The pipeline follows a strictly sequential structure, as shown in Figure 1. Account-level transaction statistics are first computed and used for direction-specific clustering. Sender–receiver interaction patterns are then encoded using cluster-pair prevalence ratios and propagated to transaction-level records. Transaction-level deviation features and the Behavioral Risk Prior are subsequently derived and integrated into the final GBDT model. Fraud labels are used only during training for risk prior estimation and model optimization.

The complete execution order of these stages and the separation between training and test data are summarized in Algorithm 1.

Algorithm 1: BI-GBDT Feature Generation and Training Pipeline

Input: Transaction records D, fraud labels y (available for training data only)

Output: Trained class-weighted GBDT model

1. Split dataset      Partition D into training set $D_{train}$ and test set $D_{test}$.

2. Account-level feature extraction      Compute direction-specific account statistics for sending and receiving roles.

3. Directional behavioral clustering      Learn GMM clusters on $D_{train}$ for frequency and amount-based features.

4. Cluster assignment      Assign learned cluster labels to accounts in $D_{train}$ and $D_{test}$.

5. Interaction prevalence encoding      Compute sender–receiver cluster-pair prevalence ratios.

6. Local deviation reference estimation      Estimate reference distributions using $D_{train}$.

7. Deviation feature computation      Compute transaction-level z-scores for $D_{train}$ and $D_{test}$.

8. Behavioral Risk Prior estimation      Estimate risk prior using $D_{train}$ and labels y.

9. Supervised model training      Train class-weighted GBDT using $D_{train}$ and labels y.

10. Inference      Apply the trained model to $D_{test}$ to generate anomaly scores.

The proposed BI-GBDT framework prevents both data leakage and label leakage. Deterministic features are computed directly from transaction records and do not involve any model fitting or label information. Components that require data-driven estimation, including direction-specific clustering, reference distributions for local deviation signals, and the Behavioral Risk Prior, are learned exclusively on the training data. The learned parameters are then applied to the remaining records without incorporating any information from the test set. Fraud labels are used only during training for risk prior estimation and GBDT optimization and are never accessed during feature generation or inference on the test data.

The following sections detail each of these modules and their specific contributions to the overall BI-GBDT framework.

3.1. Interaction-Aware Feature Engineering

The first module of the BI-GBDT framework is dedicated to modeling account-level behavioral patterns and sender–receiver interactions in a scalable and graph-free manner. The objective of this module is to transform raw historical transaction activity into structured behavioral representations that reflect direction-specific dynamics and interaction typicality within the payment system. To achieve this, behavioral patterns are first learned independently for sending and receiving roles through unsupervised clustering, and subsequently combined to encode interaction prevalence. This two-step formulation enables the framework to capture relational signals between accounts without explicitly constructing a transaction graph, while preserving directionality and population-level behavioral context.

Directional Behavioral Clustering

In the first stage of the proposed framework, we construct compact, direction-aware behavioral descriptors for each account using unsupervised clustering. In alignment with the architecture illustrated in Figure 1, historical behavioral statistics are extracted separately for sending and receiving roles along two dimensions: transaction frequency and transaction amount. This role-based separation captures asymmetric behavioral dynamics, as accounts may exhibit substantially different patterns when sending versus receiving funds. For example, an account may act as a high-frequency sender while remaining a low-activity receiver. Along the frequency axis, the following statistical features are computed for each account:

• Total number of transactions;
• Number of distinct counterparties;
• Unique transaction rate (the ratio of distinct counterparties to total transactions).

Along the amount axis, transaction value statistics are extracted:

• Mean, standard deviation, minimum, and maximum transaction amounts.

To map these behavioral features into discrete behavioral profiles, we employ the Gaussian Mixture Model (GMM) due to its soft clustering capability and probabilistic interpretability. Unlike hard clustering approaches such as K-means, GMM assigns posterior membership probabilities, enabling smoother behavioral boundaries and capturing uncertainty in account behavior [33,34,35]. The GMM density function is defined as

$$p(x)={\sum }_{\mathrm{k}} {\pi }_{\mathrm{k}} N(x | {\mu }_{\mathrm{k}}, {\Sigma }_{\mathrm{k}})$$

(1)

where ${\pi }_{\mathrm{k}}$ is the mixing coefficient of component $k ({\sum }_{\mathrm{k}} {\pi }_{\mathrm{k}} = 1)$, ${\mu }_{\mathrm{k}}$ and ${\Sigma }_{\mathrm{k}}$ are the mean and covariance of the k-th Gaussian component, and $N(x | {\mu }_{\mathrm{k}}, {\Sigma }_{\mathrm{k}})$ denotes the multivariate normal distribution. The parameters $\{{\pi }_{\mathrm{k}}, {\mu }_{\mathrm{k}}, {\Sigma }_{\mathrm{k}}\}$ are optimized via the Expectation-Maximization (EM) algorithm.

In the E-step of EM, we compute the posterior responsibility γ_nk that component k takes for data point x_n:

$${\gamma }_{\mathrm{n}\mathrm{k}} = [{\pi }_{\mathrm{k}} \mathcal{N}(x_{\mathrm{n}} | {\mu }_{\mathrm{k}}, {\Sigma }_{\mathrm{k}})]/{\sum }_{\mathrm{j}} [{\pi }_{\mathrm{j}} \mathcal{N}(x_{\mathrm{n}} | {\mu }_{\mathrm{j}}, {\Sigma }_{\mathrm{j}})]$$

(2)

Let ${X_{\mathrm{x}}}^{\mathrm{d}}$ represent a feature matrix for one of the four groups. For each matrix, clustering produces label assignments as follows:

$$C_s^f=\mathrm{G}\mathrm{M}\mathrm{M}\left(X_s^f\right),C_r^f=\mathrm{G}\mathrm{M}\mathrm{M}\left(X_r^f\right),C_s^a=\mathrm{G}\mathrm{M}\mathrm{M}\left(X_s^a\right),C_r^a=\mathrm{G}\mathrm{M}\mathrm{M}\left(X_r^a\right)$$

(3)

Each account a is then assigned behavioral cluster labels from each result:

$$\begin{array}{c}freq\_Send(a)=C_s^f\left(a\right), freq\_Recv(a)=C_r^f(a),\\ amt\_Send(a)=C_s^a\left(a\right), amt\_Recv\left(a\right)=C_r^a\left(a\right)\end{array}$$

(4)

These tags collectively define a direction-specific behavioral signature for each account, which is then used to derive sender–receiver interaction features in the next module.

3.2. Interaction Prevalence Encoding

In the second module of the BI-GBDT framework, we derive interaction-level prevalence features that quantify how frequently specific sender–receiver behavioral cluster pairs occur across the transaction corpus. As illustrated in Figure 1, these features encode population-level interaction regularities and provide contextual information indicating how statistically consistent or structurally rare a given transaction is within the broader behavioral landscape.

Interaction prevalence is computed separately along two dimensions: transaction frequency and transaction amount. Let $N$ denote the total number of transactions in the dataset. For each transaction, the sender and receiver accounts are identified together with their corresponding direction-specific behavioral cluster labels, obtained from Section Directional Behavioral Clustering. Let $c_i$ and $c_j$ denote the sender and receiver cluster labels, respectively.

Let $count\_freq(c_i, c_j)$ represent the number of transactions in which the sender belongs to frequency-based send cluster $c_i$ and the receiver belongs to frequency-based receive cluster $c_j$. Similarly, $count\_amt(c_i, c_j)$ represents the corresponding co-occurrence count for amount-based clusters. The cluster-pair prevalence ratios are defined as

$$r_{ij}^{freq}={\displaystyle \frac{count\_freq(c_i, c_j)}{N}} r_{ij}^{amt}={\displaystyle \frac{count\_amt(c_i, c_j)}{N}}$$

(5)

These normalized ratios measure the global prevalence of specific sender–receiver behavioral pairings. Cluster pairs with high prevalence correspond to statistically dominant and behaviorally consistent interaction patterns, such as recurrent transfers between similar account profiles. Conversely, low-prevalence cluster pairs reflect structurally infrequent and behaviorally uncommon interaction configurations, which may be indicative of elevated risk. Importantly, this formulation preserves directional structure, such that transfers from a high-frequency sender to a low-frequency receiver are treated differently from the inverse scenario.

By incorporating cluster-pair prevalence ratios as transaction-level features, the BI-GBDT framework embeds relational behavioral context in a compact, scalable, and graph-free manner. This representation avoids the computational overhead of explicit graph construction while remaining well suited for high-dimensional supervised learning pipelines.

3.3. Local Deviation and Risk Scoring

In the third stage of the BI-GBDT framework, we capture transaction-level deviation signals that quantify how strongly an individual transaction departs from the historical activity patterns of the involved accounts. While earlier modules focus on account-level behavioral profiling and interaction prevalence, this stage operates at the individual transaction granularity, providing complementary information that cannot be inferred from aggregate behavior alone.

For each transaction, we compute the standardized deviation (z-score) of the transaction amount relative to the historical distribution of the corresponding account, evaluated separately for sender and receiver roles. Let μ_s and σ_s denote the mean and standard deviation of past outgoing transfers for the sender, and μ_r and σ_r for the receiver. Given a transaction with amount A, the directional z-scores are defined as follows:

$$Z_s=(A-{\mu }_s)/{\sigma }_s$$

(6)

$$Z_r=(A-{\mu }_r)/{\sigma }_{r }$$

(7)

High absolute values of $Z_s$ or $Z_r$ indicate that the transaction amount is statistically inconsistent with the historical behavior of the sender or receiver, respectively. Such deviations may arise in scenarios involving account misuse, opportunistic fraud, or behavioral drift, particularly when observed in conjunction with structurally infrequent sender–receiver interaction patterns identified in earlier modules.

3.4. Behavioral Risk Prior

This module introduces a behavioral risk prior that represents fraud concentration at the cluster level within the transaction population. The objective is to incorporate population-level risk information derived from historical fraud labels as an additional feature for downstream classification. While previous components focus on interaction structure and transaction-level deviation signals, the behavioral risk prior provides complementary information reflecting systematic fraud accumulation in specific regions of the behavioral space.

All transactions are embedded into a feature space composed of direction-specific behavioral cluster assignments, interaction prevalence ratios, and amount-related statistics derived from the preceding modules. Transactions are then grouped using a Gaussian Mixture Model (GMM), enabling soft partitioning of the feature space and probabilistic cluster membership.

For each resulting cluster ($k$), a cluster-level risk score is computed as the ratio of labeled fraudulent transactions to the total number of transactions within that cluster:

$$RiskPrior\left(k\right)={\displaystyle \frac{F_k}{T_k}}$$

(8)

where $F_k$ denotes the set of confirmed fraudulent transactions assigned to cluster $k$, and $T_k$ represents the total number of transactions in the same cluster. This cluster-level risk score is assigned to each transaction according to its cluster membership and used as a supplementary feature, thereby establishing a behavioral risk prior. The resulting prior complements transaction-level deviation signals and interaction prevalence features and is jointly provided as input to the final class-weighted GBDT model.

3.5. Class-Weighted GBDT Classification

The final module of the framework performs supervised classification by integrating all features generated in the preceding stages into a class-weighted GBDT model. This module constitutes the decision layer of the framework, where interaction-aware behavioral representations, transaction-level deviation signals, and the behavioral risk prior are jointly evaluated to produce a transaction-level anomaly score.

The input feature set includes direction-specific behavioral cluster labels, cluster-pair interaction prevalence ratios, standardized deviation metrics, and the behavioral risk prior derived at the cluster level, together with conventional transactional attributes. By combining features extracted at the transaction, interaction, and population levels, the classifier captures both localized irregularities and broader behavioral structure within the payment system.

To explicitly handle the severe class imbalance characteristic of fraud detection, the GBDT model is trained using a class-weighted loss function. Formally, the optimization objective is defined as

$$L={\sum }_{i=1}^N{w}_{y_i}\cdot \mathcal{l}(y_i, {\widehat{y}}_i)$$

(9)

where $y_i$ denotes the true class label of transaction $i$, ${\widehat{y}}_i$ represents the corresponding model prediction, $\mathcal{l}(\cdot )$ is the base loss function, and $w_{y_i}$ is the weight associated with class $y_i,$. In this formulation, higher weights are assigned to fraudulent transactions relative to benign ones ($w_{fraud}\gg w_{normal}$), ensuring that misclassification errors on the minority fraud class exert a stronger influence on the learning process.

Gradient boosting constructs an ensemble of decision trees sequentially, where each new tree is trained to minimize the weighted loss by correcting the residual errors of the current ensemble. This iterative procedure enables the model to learn nonlinear decision boundaries and complex interactions among behavioral, interaction-based, and deviation-related features [17].

The output of this module is a transaction-level anomaly score that reflects the combined influence of behavioral patterns, interaction prevalence, deviation signals, and historical fraud concentration. This unified scoring mechanism enables effective discrimination between benign transactions and those exhibiting elevated fraud risk.

4. Experiments and Results

We conduct a series of experiments to evaluate the performance of the proposed BI-GBDT framework on a large-scale real-world payment transaction dataset characterized by high volume and severe class imbalance. All experiments are carried out in a distributed big data environment, and model training and evaluation are performed using PySpark ML on Apache Spark version 3.3.2 to support scalable processing of the full dataset.

4.1. Dataset

The dataset originates from a real-world financial transaction system, where all sensitive fields have been masked to preserve privacy and institutional confidentiality. Despite anonymization, the data retain the behavioral and structural properties of genuine payment flows. The corpus spans a continuous six-month observation period and consists of more than 456 million transactions, of which approximately 40,000 are labeled as confirmed fraud, corresponding to an extreme class imbalance below 0.01%.

Each record represents a single money transfer between two accounts and includes the transaction timestamp, sender and receiver account identifiers, sender and receiver bank codes, and transaction amount. These attributes enable the construction of direction-specific behavioral features, interaction-aware indicators, and transaction-level deviation signals used in the proposed BI-GBDT framework.

To capture temporal dynamics in account behavior, the six-month transaction history is processed using fixed-length monthly aggregation windows. Behavioral descriptors based on transaction frequency and transaction amount are computed independently within each monthly window and separately for sending and receiving roles. Standardized deviation (z-score) metrics are likewise calculated relative to the corresponding monthly behavioral distributions, ensuring consistency with the direction-aware modeling strategy.

The scale of the dataset necessitates distributed processing for efficient feature computation, while the severe class imbalance reflects realistic operational conditions in payment systems. For model development and evaluation, the dataset is partitioned into training and testing subsets using a stratified 70/30 split, preserving the original fraud-to-non-fraud ratio in both sets. The test set is reserved exclusively for final performance evaluation and is not used during feature learning, hyperparameter optimization, model selection, or decision threshold determination.

4.2. Model Parameters and Tuning

Hyperparameter selection is a critical component of the proposed BI-GBDT framework, as both the quality of behavioral representations and the effectiveness of the supervised decision model are sensitive to parameter choices. In this study, hyperparameters are determined using a grid search strategy and fixed consistently across all experiments to ensure a fair and reproducible evaluation.

For the direction-specific behavioral clustering component, clustering is applied independently across behavioral axes and transaction roles in accordance with the framework design described in Section 3. A unified cluster selection procedure is adopted for all clustering stages. The number of clusters is determined through a combination of silhouette analysis, the elbow method, and domain knowledge, which together balance cluster compactness, separation, and behavioral interpretability. This procedure avoids excessive fragmentation while preserving meaningful behavioral distinctions. The final clustering configurations used throughout the experiments are summarized in

Table 1. The clustering models configurations.

Axis	Direction	Model	Number of Clusters
Frequency	Send	GMM	13
Frequency	Receive	GMM	8
Amount	Send	GMM	11
Amount	Receive	GMM	9

.

For the supervised classification stage, hyperparameters of the class-weighted GBDT model are selected via grid search over a constrained parameter space informed by the scale and severe class imbalance of the dataset. Hyperparameter optimization is performed exclusively on the training data to prevent information leakage. Given the large-scale nature of the dataset and the use of a fixed and limited hyperparameter search space, a single stratified train–test split is adopted. The held-out test set is reserved solely for final performance evaluation and does not participate in any stage of model tuning or selection.

The decision threshold is determined based on training data performance and fixed prior to evaluation on the test set. The optimal hyperparameter configuration used across all experiments is summarized in

Table 2. The final hyperparameter configuration of the class-weighted GBDT model.

Hyperparameter	Value
Number of boosting iterations	320
Learning rate	0.06
Maximum tree depth	9
Minimum leaf size	10,000
Subsampling rate	0.7
Number of histogram bins	128
Minimum split-gain threshold	1 × 10−4

.

4.3. Evolution Metrics

This study evaluates performance using Precision, Recall, False Positive Rate (FPR), and Accuracy, computed from True Positive (TP), True Negative (TN), and False Negative (FN); the formulas are provided in Equations (10)–(13).

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(10)

Precision measures the proportion of correctly identified fraudulent transactions among all transactions classified as fraud by the model Equation (10).

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(11)

Recall measures the proportion of actual fraudulent transactions that are correctly identified by the model, as defined in Equation (11).

$$FPR={\displaystyle \frac{FP}{FP+TN}}$$

(12)

The False Positive Rate quantifies the proportion of legitimate transactions that are incorrectly predicted as fraud, as defined in Equation (12).

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(13)

Accuracy is the fraction of correct classifications among all transactions. In highly imbalanced settings such as fraud detection, this measure is less informative; the formula is given by Equation (13).

4.4. Results

The experimental evaluation is designed to quantify the incremental performance gains introduced by each component of the proposed BI-GBDT framework, rather than merely reporting absolute classification performance. Accordingly, a staged evaluation strategy is adopted to examine how progressively enriching a standard classifier with behavioral and interaction-aware components improves fraud detection effectiveness under severe class imbalance.

We evaluate three model configurations on an independent test set obtained via a stratified 70/30 train/test split, ensuring that the original fraud to non-fraud ratio is preserved across both subsets. The GBDT baseline represents a conventional supervised learning approach trained solely on raw transactional attributes. This configuration serves as a reference point for assessing how far the proposed framework advances beyond a standard classifier under realistic operational conditions.

The second configuration, GBDT + Behavioral Risk Prior, augments the baseline model with transaction-level deviation indicators and the proposed Behavioral Risk Prior score, which encodes cluster-level fraud concentration derived from historical labels. This setting isolates the contribution of population-level risk information that becomes particularly important under severe class imbalance and cannot be captured by raw transactional attributes alone.

The third configuration, referred to as the Proposed Method, represents the complete BI-GBDT framework by extending the second configuration with interaction aware feature engineering based on direction specific behavioral clustering and interaction prevalence encoding. Building on the Behavioral Risk Prior score and transaction level deviation signals, this configuration explicitly models sender–receiver interaction structure through cluster pair prevalence ratios, resulting in a graph-free relational representation of transaction behavior. As a result, it evaluates the full benefit of jointly incorporating behavioral context and interaction level signals beyond anomaly indicators and population-level risk information alone.

Performance results are summarized in

Table 3. Performance comparison of evaluated models on the test set.

Model	Accuracy (%)	Recall (%)	Precision (%)	FPR (%)
BI-GBDT	95.7	72.0	14.7	4.3
GBDT + Behavioral Risk Prior	76.0	66.7	2.4	24.0
GBDT	63.0	52.3	1.2	37.0

, reporting Accuracy, Recall, and False Positive Rate (FPR). The decision threshold is selected exclusively on the training split with the objective of reducing validation FPR while preserving high recall, and is subsequently applied unchanged to the independent test set. No oversampling or synthetic data generation techniques are used.

The results are consistently supported across all evaluation metrics, as well as by the confusion matrix and ROC analysis. The plain GBDT baseline attains an accuracy of 63.0%, a recall of 52.3%, a precision of 1.2%, and a high False Positive Rate (FPR) of 37.0%, confirming the limited effectiveness of raw transactional attributes under severe class imbalance.

Incorporating the Behavioral Risk Prior, which encodes cluster-level fraud concentration derived from historical labels, improves performance to 76.0% accuracy, 66.7% recall, and a precision of 2.4%, while reducing the FPR to 24.0%. This indicates that the Behavioral Risk Prior provides an effective class-imbalance-aware signal. However, the precision remains relatively low due to the extreme rarity of fraudulent transactions in the dataset, where even modest false positive rates correspond to a large absolute number of benign transactions.

The full BI-GBDT framework further improves performance, achieving 95.7% accuracy, 72.0% recall, and a precision of 14.7%, with an FPR of only 4.3%. As illustrated in Figure 2, the confusion matrix on the independent test set shows that the majority of fraudulent transactions are correctly identified while the number of false alarms remains tightly controlled relative to the scale of legitimate transactions. This balance is particularly critical in operational payment systems, where even small increases in FPR can translate into a large volume of unnecessary alerts.

The Receiver Operating Characteristic (ROC) curves presented in Figure 3 further reinforce these findings. The BI-GBDT model achieves an Area Under the Receiver Operating Characteristic Curve (AUC) of 0.92, significantly outperforming both the GBDT + Behavioral Risk Prior configuration (AUC = 0.78) and the plain GBDT baseline (AUC = 0.68). The steeper ascent of the BI-GBDT ROC curve at low false positive rates highlights its superior ranking capability in the operating region most relevant for real-world fraud detection, where maintaining high recall under strict false positive constraints is essential.

Overall, the results show that the observed performance improvements do not stem from a single modeling component. Performance improves through the stepwise integration of class-imbalance handling, behavioral context, and interaction-aware representations, with each component contributing complementary information. As a result, the proposed BI-GBDT framework clearly extends a standard GBDT classifier into a more effective and operationally suitable fraud detection approach for large-scale payment systems.

Although the evaluation is based on a single stratified train–test split, the extremely large scale of the dataset and the high absolute number of observed fraud cases contribute to stable and reliable performance estimates. In large-scale operational fraud detection systems, small relative variations in data composition tend to have a limited impact on aggregate metrics such as recall and false positive rate. Similar evaluation protocols have been widely adopted in prior large-scale fraud detection studies employing tree-based models [34,35,36].

5. Discussion

This study demonstrates that fraud detection performance in large-scale payment systems cannot be sufficiently improved using a standard GBDT classifier trained only on raw transactional attributes. The results show that such baseline models are strongly affected by severe class imbalance and lack the ability to capture interaction-driven fraud patterns, leading to high false positive rates and limited recall.

Incorporating the Behavioral Risk Prior, which encodes cluster-level fraud concentration derived from historical labels, improves performance by mitigating the effects of class imbalance. This component reduces false positives and increases recall compared to the plain GBDT baseline. However, the results indicate that the Behavioral Risk Prior alone is not sufficient to fully separate fraudulent and legitimate transactions, particularly in cases where fraud emerges through uncommon or coordinated sender–receiver interactions.

The substantial performance improvements are observed when interaction-aware feature engineering is incorporated through direction-specific behavioral clustering and cluster-pair prevalence ratios. These features allow the model to represent sender–receiver interaction structure in a graph-free manner, capturing relational patterns that are not visible at the transaction level. This interaction-aware representation substantially reduces false positives while preserving detection sensitivity.

From an operational perspective, this improvement is critical. Even small increases in the false positive rate can generate many unnecessary alerts and increase investigation costs in large-scale payment systems [28,31]. The confusion matrix and ROC analysis show that the BI-GBDT framework maintains a favorable balance between recall and false positive control, making it more suitable for deployment than baseline approaches.

Overall, the findings show that effective fraud detection is achieved by jointly incorporating class-imbalance handling, behavioral context, and interaction-aware representations within a unified framework. The proposed BI-GBDT approach provides a scalable and practical solution that improves upon standard GBDT-based models without relying on explicit graph construction.

It is also important to clarify the scope of the proposed approach in relation to graph-based fraud detection methods. Existing graph-based fraud detection approaches typically rely on explicit transaction graph construction and graph neural networks to model higher-order dependencies between accounts. Although effective in controlled settings, such methods often face scalability, latency, and maintenance challenges in production environments with hundreds of millions of transactions. In contrast, the proposed BI-GBDT framework captures sender–receiver interaction structure using compact, tabular representations that can be seamlessly integrated into existing distributed processing pipelines [37,38,39].

Limitations

Despite the strong empirical results, this study has several limitations. First, the evaluation is conducted on a masked real-world dataset, which restricts full reproducibility and public benchmarking. Second, the analysis relies on a six-month observation window with fixed monthly aggregation. This design choice may reduce sensitivity to short-term fraud bursts or rapidly evolving attack patterns. Third, the experimental evaluation is based on a single stratified train–test split. While this reflects realistic large-scale operational settings, additional statistical validation using multiple splits or resampling techniques could further quantify performance variability. Such validation is left as an important direction for future work. Finally, as with most data-driven fraud detection systems, the proposed framework may be affected by concept drift over time, which may require periodic model retraining and recalibration in long-term deployments.

Graph-based fraud detection methods are not included as baselines in this study, as the primary objective is not to outperform graph-based models, but to demonstrate that interaction-aware representations can be effectively captured in a graph-free manner that is more compatible with large-scale operational payment systems. While graph-based approaches can offer strong expressive power, their computational and operational overhead often limits their applicability in real-time or large-scale production environments [38,39,40].

6. Conclusions

This study presents BI-GBDT, a scalable and graph-free fraud detection framework designed for large-scale payment systems operating under severe class imbalance. The proposed approach combines direction-specific behavioral clustering, interaction prevalence encoding via cluster-pair ratios, transaction-level deviation signals, and a behavioral risk prior derived from cluster-level fraud concentration within a unified class-weighted GBDT model.

Experimental results on a large real-world masked transaction dataset demonstrate that a standard GBDT classifier trained on raw transactional attributes is insufficient to reliably detect fraud in this setting. While the Behavioral Risk Prior improves performance by explicitly addressing class imbalance, the results show that this component alone cannot fully separate fraudulent and legitimate transactions. Substantial and consistent performance improvements are achieved only when interaction-aware behavioral representations are incorporated into the learning process.

The full BI-GBDT framework achieves higher recall while significantly reducing the false positive rate, as confirmed by point metrics, confusion matrix analysis, and ROCs. This balance is particularly important for operational deployment, where excessive false alarms can impose substantial investigation costs. By capturing sender–receiver interaction structure without constructing an explicit transaction graph, BI-GBDT provides an effective compromise between expressive modeling and computational scalability.

Overall, the findings show that effective fraud detection in large-scale payment systems requires the joint integration of class-imbalance handling, behavioral context, and interaction-aware feature engineering. The proposed BI-GBDT framework demonstrates how these components can be combined in a practical and scalable manner, offering a strong alternative to both transaction-only classifiers and computationally intensive graph-based approaches.

Future work may extend the proposed framework along several practical directions. The current study relies on fixed monthly aggregation windows for constructing behavioral features; exploring adaptive or multi scale temporal windowing strategies could improve sensitivity to short-term behavioral shifts and emerging fraud patterns. In addition, incorporating complementary contextual signals such as transaction channels, device information, or session-level attributes may further enrich interaction-aware representations while preserving the graph-free and scalable design of the framework. Another important direction is extending BI-GBDT to online or streaming detection settings, enabling continuous model updates and faster adaptation to evolving fraud behavior in real-world payment systems. Finally, future studies may investigate cost-sensitive thresholding and decision strategies that explicitly account for operational constraints and investigation costs, allowing the framework to be tailored to different deployment scenarios and risk preferences.