Detecting Imbalanced Credit Card Fraud via Hybrid Graph Attention and Variational Autoencoder Ensembles

Abstract

Credit card fraud detection remains a major challenge due to severe class imbalance and the constantly evolving nature of fraudulent behaviors. To address these challenges, this paper proposes a hybrid framework that integrates a Variational Autoencoder (VAE) for probabilistic anomaly detection, a Graph Attention Network (GAT) for capturing inter-transaction relationships, and a stacking ensemble with XGBoost for robust prediction. The joint use of VAE anomaly scores and GAT-derived node embeddings enables the model to capture both feature-level irregularities and relational fraud patterns. Experiments on the European Credit Card and IEEE-CIS Fraud Detection datasets show that the proposed approach outperforms baseline models by up to 15% in F1-score, achieving values above 0.980 with AUCs reaching 0.995. These results demonstrate the effectiveness of combining unsupervised anomaly detection with graph-based learning within an ensemble framework for highly imbalanced fraud detection problems.

1. Introduction

Credit card fraud poses an escalating threat to financial institutions worldwide, with losses estimated to surpass $40 billion annually by 2027 [1,2,3]. This upward trajectory is driven largely by the accelerated adoption of digital payments, e-commerce, and mobile banking—platforms that fraudsters increasingly exploit due to their rapid expansion and inherent vulnerabilities [4]. Detecting fraudulent activities early and accurately is essential; however, this task remains inherently challenging due to the extreme imbalance in transaction datasets, where fraud typically constitutes fewer than 0.2% of all transactions [5,6]. In this context, fraudulent transactions represent the minority samples (positive class), while the overwhelming majority of legitimate transactions form the majority class. Distinguishing these rare minority samples is critical, as they drive the difficulty and importance of the fraud detection task.

Historically, supervised machine learning models, including logistic regression, random forests, and neural networks, have been employed extensively for fraud detection [4,7,8]. These approaches primarily rely on historical labeled data and often presume static fraud patterns, leading to degraded performance when encountering new fraud behaviors [9]. Moreover, they treat transactions independently, neglecting relational cues such as shared devices, accounts, or temporal correlations.

To address these shortcomings, anomaly detection methods—particularly autoencoders—have been widely explored. Autoencoders model legitimate transaction behavior and flag deviations as potential fraud [10,11,12]. However, these models operate at the individual transaction level and may overlook coordinated schemes spanning multiple related transactions [13].

The evolution of fraud detection methods can be broadly characterized in three stages. First, supervised learning dominated early studies, leveraging labeled datasets but struggling with class imbalance and evolving fraud tactics. Second, unsupervised and semi-supervised methods such as autoencoders emerged, focusing on capturing deviations from normal patterns without requiring extensive fraud labels. Third, recent advances in Graph Neural Networks (GNNs) have transformed the field by modeling inter-transaction dependencies explicitly. Architectures like GraphSAGE [14] and Relational Graph Convolutional Networks (RGCN) [15] demonstrated that neighborhood aggregation and relation-specific learning can uncover subtle, coordinated fraud rings that traditional methods miss. This trajectory highlights a clear research gap: while VAEs capture probabilistic feature-level anomalies and GNNs capture relational dependencies, few frameworks effectively integrate the two with ensemble meta-learning for robust fraud detection.

Motivated by these complementary strengths, this study introduces a novel hybrid architecture combining VAE-based anomaly detection and Graph Attention Network (GAT)-based relational modeling within an ensemble framework. Specifically, a VAE captures probabilistic latent representations of legitimate transactions, enhancing anomaly detection at the feature level, while a GAT processes the transaction graph to extract relational embeddings encoding fraud-relevant dependencies. Outputs from both models are fused through a stacking ensemble with XGBoost, enabling robust and generalizable predictions.

The primary contributions of this paper are summarized as follows:

• Development of a hybrid architecture combining VAE-based anomaly detection and GAT-based relational modeling for enhanced credit card fraud detection.
• Introduction of a stacking ensemble with XGBoost to effectively fuse anomaly scores and graph-derived embeddings, improving generalization under class imbalance.
• Extensive validation on two real-world, highly imbalanced datasets, demonstrating superior performance of the proposed framework compared to strong baselines and recent state-of-the-art methods.

The rest of this paper is structured as follows: Section 2 reviews related works in credit card fraud detection. Section 3 details the datasets and fundamental methodologies employed in this study, including autoencoders, Graph Attention Networks, and the XGBoost meta-learner, as well as the evaluation metrics. Section 4 describes the proposed hybrid ensemble framework, integrating probabilistic anomaly detection, relational modeling, and ensemble learning. Section 5 presents the experimental results and comparative evaluations, followed by Section 6, which concludes the paper and discusses directions for future research.

2. Related Works

Research into credit card fraud detection has evolved considerably over the past decade, driven by the urgent need to mitigate significant financial losses due to fraudulent activities. Current methodologies broadly encompass supervised machine learning, unsupervised and semi-supervised anomaly detection, graph-based relational modeling, and ensemble learning techniques.

Early research primarily utilized supervised classification methods such as logistic regression, decision trees, support vector machines (SVM), and random forests due to their interpretability and ease of implementation. However, these traditional supervised methods often struggled with imbalanced data scenarios, resulting in high false negatives and limited generalization to emerging fraud patterns [4,8]. To address this imbalance, oversampling methods and cost-sensitive learning techniques have been widely investigated [9]. Despite such improvements, these supervised methods remain limited by their dependency on historical labeled data, making them susceptible to novel fraudulent behaviors.

To overcome these limitations, researchers have turned to unsupervised and semi-supervised methods, particularly autoencoders. Autoencoders have gained prominence due to their effectiveness in capturing complex latent representations of legitimate transactions, enabling the detection of anomalous deviations indicative of fraud [16]. Tayebi and El Kafhali [6], for instance, employed temporal autoencoders using Long Short-Term Memory (LSTM) networks, demonstrating significant improvements in fraud detection by explicitly modeling temporal transaction dependencies. Additionally, Almansoori and Telek [17] presented a hybrid autoencoder-based approach integrating isolation forests to enhance anomaly detection effectiveness, while Alshameri and Xia [18] explored Variational Autoencoders (VAEs) for robust unsupervised anomaly detection, significantly improving fraud identification. Compared to standard autoencoders, VAEs provide a probabilistic latent space, which enhances their ability to generalize to unseen fraudulent behaviors, though at the cost of increased computational complexity.

Graph-based methods, especially GNNs, have been employed to address this gap by explicitly modeling inter-transaction relationships. Recent studies have validated the effectiveness of GNN-based architectures such as Graph Convolutional Networks (GCN) and GAT in capturing relational fraud patterns. Cherif et al. [9] recently proposed an encoder–decoder GNN model for fraud detection, demonstrating superior performance by effectively exploiting structural information within transaction graphs. Similarly, Gupta et al. [19] presented a GCN-based model, validating its capability to detect sophisticated fraudulent patterns through node embeddings derived from transaction graph structures. Furthermore, Khosravi et al. [20] demonstrated the effectiveness of attention-based GNN models in dynamically identifying complex relational fraud structures, indicating superior generalization and interpretability. In particular, GATs often outperform GCNs because their attention mechanism selectively emphasizes informative neighbors and downweights noisy or irrelevant edges, whereas GCNs aggregate information uniformly across neighbors, making them more sensitive to graph noise.

Additionally, ensemble methods have been widely used in fraud detection for their ability to combine diverse learners and improve robustness. Bagging approaches, such as random forests, enhance stability but may underperform in extreme imbalance scenarios, while boosting techniques (e.g., AdaBoost, XGBoost, CatBoost) incrementally focus on hard-to-classify fraud cases and often deliver superior recall. Stacking ensembles extend these methods by learning meta-level representations from multiple base models, a strategy particularly effective in imbalanced fraud detection tasks [21]. Beyond classical approaches, recent hybrid models integrating deep learning with GNNs have shown significant promise in domains such as intrusion detection and cybersecurity, demonstrating their potential for fraud detection [22,23,24,25]. For example, Zhang and Luo [26] proposed a hybrid model integrating GNN embeddings with deep neural network classifiers, demonstrating significant performance gains in real-world fraud detection tasks.

Despite these advancements, existing hybrid models often do not fully leverage the probabilistic modeling advantages of advanced autoencoder variants such as VAEs, nor do they extensively explore enriched graph representations capturing dynamic relational patterns. In response to these research gaps, our proposed approach integrates a Variational Autoencoder-based anomaly detection module with a GAT designed for relational modeling, employing an XGBoost meta-learner to unify and interpret the combined outputs. This framework aims to capitalize on the strengths of unsupervised probabilistic anomaly detection, attention-based relational learning, and ensemble fusion, thus offering improved accuracy and robustness to imbalanced and evolving fraud detection scenarios.

3. Materials and Methods

This section describes the dataset and summarizes the fundamental techniques used in this research, specifically autoencoders for anomaly detection, GAT for relational modeling, and XGBoost as a meta-learner.

3.1. Dataset Description

This study evaluates model performance on two widely used, real-world credit card fraud datasets: the European Credit Card dataset [27] and the IEEE-CIS Fraud Detection dataset. The experiments will be restricted to these two benchmark datasets, rather than multiple datasets, in order to provide a focused and rigorous evaluation of the proposed hybrid framework.

3.1.1. European Credit Card Fraud Dataset

This study uses the publicly available dataset introduced by Dal Pozzolo et al. [27], which consists of 284,807 transactions collected over two consecutive days in 2013. Out of these, only 492 transactions (0.172%) are labeled as fraudulent, resulting in a highly imbalanced binary classification problem. Each transaction is described by 30 numerical features: 28 have been anonymized through principal component analysis (PCA) to protect confidentiality, while the remaining two features (Time and Amount) retain their original form.

3.1.2. IEEE-CIS Fraud Detection Dataset

The second benchmark is the IEEE-CIS Fraud Detection dataset, made available as part of a Kaggle competition (https://www.kaggle.com/c/ieee-fraud-detection, accessed on 26 May 2025). This dataset was provided by Vesta Corporation, a global leader in fraud prevention for e-commerce platforms, making it highly representative of real-world online transaction environments. It contains over 1.1 million transactions, with approximately 3.5% labeled as fraud. Each record is described by 431 features, spanning transaction amount, device and browser information, payment card attributes, network details, and digital identity variables. The dataset has become a widely recognized benchmark in the research community due to its scale and complexity and the visibility it gained through the Kaggle competition, which attracted thousands of participants. This significance makes it particularly well-suited for evaluating the scalability and robustness of advanced fraud detection frameworks.

3.2. Autoencoder

Autoencoders are unsupervised neural networks that learn to encode input data into a compressed latent representation and then decode it back to reconstruct the original input [28,29]. In the context of fraud detection, an autoencoder is trained using only legitimate (non-fraudulent) transactions. This allows the model to learn the underlying patterns of normal behavior, so that, during inference, transactions that deviate significantly from these patterns, resulting in high reconstruction errors, can be flagged as potential fraud. The strength of autoencoders lies in their ability to capture subtle deviations from normality without requiring explicit fraud labels.

3.3. Graph Attention Networks

GATs [30] are deep learning models designed to work with graph-structured data. Here, each node represents a transaction, and edges indicate similarity or contextual relationships, such as shared card information or temporal proximity [31]. Unlike traditional graph neural networks, GATs introduce an attention mechanism that learns to assign different levels of importance to neighboring nodes, thus enabling the model to focus more on informative or suspicious relationships. This is especially useful for fraud detection, as it allows the network to identify complex, coordinated fraud activities that manifest as anomalous connectivity patterns within the transaction graph.

In this study, the construction of the transaction graph is a critical step, based on temporal windows, feature similarity, and shared identifiers. The full procedure for graph construction is detailed in Section 4, Algorithm 1, which illustrates how transactions are transformed into graph-structured input for the GAT.

Algorithm 1 Transaction graph construction

Require: Transaction dataset $\mathcal{D}$ with features $\left\{x_i\right\}$ and timestamps $\left\{t_i\right\}$

Ensure: Transaction graph $\mathcal{G}=(V,E)$   1: Initialize graph $\mathcal{G}$ with nodes $v_i\in V$ for each transaction $x_i$   2: for each pair of transactions $(x_i,x_j)$ do   3:       if $|t_i-t_j| \le 60$ s then   4:           Add edge $(v_i,v_j)$ to E   5:       else if shared identifiers exist (e.g., card ID, device) then   6:           Add edge $(v_i,v_j)$ to E   7:       else if $\parallel PCA\left(x_i\right)-PCA\left(x_j\right){\parallel }_2 \le \tau$ then   8:           Add edge $(v_i,v_j)$ to E   9:       end if 10: end for 11: Encode node features using normalized transaction attributes 12: return Graph $\mathcal{G}=(V,E)$

3.4. XGBoost

XGBoost is an efficient implementation of gradient-boosted decision trees that excels on structured, tabular data and is particularly well-suited for tasks with severe class imbalance, such as fraud detection [32]. It combines multiple weak learners—typically shallow decision trees—into a strong ensemble, iteratively correcting errors of previous trees while applying regularization to prevent overfitting [33]. In this study, XGBoost acts as a meta-classifier, aggregating the diverse information produced by both the autoencoder and the graph-based models to generate robust final predictions.

3.5. Evaluation Metrics

To rigorously assess model performance under extreme class imbalance, we employ a suite of metrics that captures both overall correctness and the model’s sensitivity to rare fraud cases; these metrics include accuracy, precision, recall, F-measure, receiver operating characteristics (ROC) curve, and area under the ROC curve (AUC). Accuracy measures the proportion of all transactions that are correctly classified but can be misleading in imbalanced datasets, since a model predicting only the majority class may still achieve high accuracy [34,35]. To provide a more meaningful evaluation, we also report precision, which measures the proportion of predicted fraud cases that are actually fraudulent and thus reflects the model’s ability to avoid false alarms [36]. Recall, also known as sensitivity, captures the proportion of actual fraudulent transactions that are correctly identified, making it critical for minimizing undetected fraud. The F1-score, which is the harmonic mean of precision and recall, balances the trade-off between these two metrics and provides a single value reflecting both false positives and false negatives [37]. These metrics are mathematically defined as follows:

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(1)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(2)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(3)

$$F1-Score=2\times {\displaystyle \frac{Precision·Recall}{Precision+Recall}}$$

(4)

where $TP$ denotes true positives (correctly detected frauds), $TN$ denotes true negatives (correctly detected non-frauds), $FP$ denotes false positives (genuine transactions incorrectly classified as fraud), and $FN$ denotes false negatives (fraudulent transactions missed by the model) [37,38]. Together, these metrics provide a holistic evaluation framework that captures both the correctness of fraud detection and the model’s discriminative ability in the presence of severe class imbalance.

4. Proposed Methodology

The proposed framework, shown in Figure 1, integrates probabilistic anomaly detection, relational graph modeling, and ensemble learning to robustly address imbalanced credit card fraud detection. The process begins with a VAE, which is trained exclusively on non-fraudulent transactions to learn the typical distribution of normal behavior. Given input vector $x\in {\mathbb{R}}^n$, the encoder parameterized by $\theta$ maps x to a latent representation z, and the decoder parameterized by $\varphi$ reconstructs x from z [28]. The VAE is optimized by minimizing a combined loss comprising the reconstruction error and the Kullback–Leibler (KL) divergence, as follows:

$${\mathcal{L}}_{VAE}\left(x\right)={\parallel x-g_{\varphi }\left(z\right)\parallel }_2^2+\mathrm{KL}[q_{\theta }\left(z\right|x)\parallel p\left(z\right)],$$

(5)

where $g_{\varphi }\left(z\right)$ is the reconstruction of the input, $q_{\theta }\left(z\right|x)$ is the approximate posterior, $p\left(z\right)$ is the prior over latent variables, and ${\parallel ·\parallel }_2$ denotes the Euclidean norm. The reconstruction error between the input x and its reconstructed output $g_{\varphi }\left(f_{\theta }\left(x\right)\right)$ is treated as an anomaly score, where higher values suggest deviations from normal behavior and potential fraudulent activity.

A central component of this framework is the construction of the transaction graph $\mathcal{G}=(V,E)$. Each transaction is represented as a node, and edges are formed when transactions share identifiers (e.g., card or device), occur within a temporal window of 60 s, or lie within a thresholded Euclidean distance in PCA-transformed feature space. Node features are encoded using normalized transaction attributes such as amount, device metadata, and anonymized principal components. The detailed procedure is outlined in Algorithm 1.

Meanwhile, in the GAT model, the constructed graph $\mathcal{G}$ is processed to update each node embedding $h_i\prime$ using a weighted aggregation of its neighbors’ features:

$$h_i\prime =\sigma \left(\sum _{j\in \mathcal{N}\left(i\right)}{\alpha }_{ij}W{h}_j\right),$$

(6)

where $h_j$ is the feature vector for node j, W is a learnable weight matrix, $\sigma$ is a non-linear activation, and $\mathcal{N}\left(i\right)$ is the set of neighbors for node i [39,40]. The attention coefficients ${\alpha }_{ij}$ are computed as

$${\alpha }_{ij}={\displaystyle \frac{exp\left(\mathrm{LeakyReLU}\right(a^{\top }[W{h}_i\left|\right|W{h}_j\left]\right))}{{\sum }_{k\in \mathcal{N}\left(i\right)}exp\left(\mathrm{LeakyReLU}\right(a^{\top }[W{h}_i\left|\right|W{h}_k\left]\right))}},$$

(7)

where a is a trainable attention vector and $[·||·]$ denotes vector concatenation [40]. This mechanism enables the GAT to learn the relative importance of each neighbor, allowing for effective identification of complex, context-driven fraud patterns. In this work, the GAT is trained in a semi-supervised setting where the available fraud labels guide learning. A cross-entropy loss function is optimized over labeled nodes, and class imbalance is mitigated by applying inverse-frequency class weights during training. The model is trained end-to-end rather than freezing embeddings at intermediate stages, ensuring that node representations remain fully adaptive to the fraud classification objective. The outputs from the VAE (anomaly scores), GAT (node embeddings and contextual fraud probabilities), and other relevant features are concatenated to form enriched feature vector $z_i$ for each transaction:

$$z_i=[e_i,s_i,h_i\prime ],$$

(8)

where $e_i$ is the VAE anomaly score, $s_i$ is the GAT-based fraud probability, and $h_i\prime$ is the learned node embedding. These representations are provided as input to an XGBoost meta-classifier, which learns to combine them for final fraud prediction:

$${\widehat{y}}_i=f_{\mathrm{meta}}\left(z_i\right),$$

(9)

where $f_{\mathrm{meta}}$ denotes the XGBoost model. The overall workflow is summarized in Algorithm 2.

Furthermore, model validation is performed using stratified 5-fold cross-validation to ensure the minority class is adequately represented. Early stopping is applied to all neural components, and ensemble training is carried out on out-of-fold predictions to prevent information leakage. This unified architecture capitalizes on the strengths of probabilistic anomaly detection, relational modeling, and ensemble fusion, resulting in improved performance and robustness for fraud detection in highly imbalanced and dynamic settings.

Algorithm 2 Hybrid VAE–GAT ensemble for fraud detection

Require: Transaction dataset $\mathcal{D}$ with labels $\{x_i,y_i\}$

Ensure: Fraud predictions ${\widehat{y}}_i$ for transactions $x_i$   1: Train VAE on non-fraud subset ${\mathcal{D}}_{norm}$ and compute anomaly scores $e_i$   2: Construct transaction graph $\mathcal{G}$ using Algorithm 1   3: Train GAT on $\mathcal{G}$ with weighted cross-entropy loss   4: for each node $v_i$ in $\mathcal{G}$ do   5:       Compute embeddings $h_i\prime$ and fraud probabilities $s_i$   6: end for   7: for each transaction $x_i$ do   8:       Form ensemble input $z_i=[e_i,s_i,h_i\prime ]$   9:       Predict ${\widehat{y}}_i=f_{\mathrm{meta}}\left(z_i\right)$ using XGBoost 10: end for

5. Results and Discussion

The evaluation was conducted on two benchmark datasets: the European Credit Card Fraud dataset and the IEEE-CIS Credit Card dataset, both exhibiting severe class imbalance. All preprocessing and modeling were implemented in Python 3.11 using scikit-learn v1.7.2, XGBoost v3.0.5, CatBoost v1.2.3, TensorFlow v2.20.0, and PyTorch-Geometric v2.6.1 for graph-based learning. For the graph-based components, transactions were connected based on shared identifiers, temporal proximity (within 60 s), and Euclidean distance in PCA space.

For model selection and hyperparameter tuning, a stratified 80/20 train–test split was employed, ensuring representation of the minority fraud class in both sets. A stratified 5-fold cross-validation strategy was used for hyperparameter optimization, ensuring balanced representation of fraud and non-fraud samples across folds. Early stopping was applied to all neural models to reduce overfitting, while the GAT employed a weighted cross-entropy loss to address class imbalance. To prevent information leakage, ensemble training was based on out-of-fold predictions from the base learners rather than reusing training outputs. The VAE was trained exclusively on non-fraud transactions. The model parameters are shown in

Table 1. Summary of key model hyperparameters.

Component	Hyperparameter	Value/Range
VAE	Hidden layers	[32, 16, 8]
	Latent dimension	8
	Activation	ReLU, sigmoid (output)
	Optimizer	Adam, lr = 0.001
	Epochs	50–100, early stopping
GAT	Layers	2
	Attention heads	4
	Hidden units	32
	Activation	LeakyReLU
	Dropout	0.2–0.5
XGBoost/CatBoost	n_estimators	100–200
	max_depth	4–8
	scale_pos_weight	[imbalance ratio]
	learning_rate	0.05–0.1

.

5.1. Experimental Results

Table 2. Comparison of model performance on the European Credit Card dataset.

Model	Accuracy	Precision	Recall	F1-Score
LR	0.9741	0.7579	0.9184	0.8305
Random Forest	0.9995	0.8610	0.8551	0.8580
XGBoost	0.9995	0.8469	0.8469	0.8469
CatBoost	0.9991	0.6774	0.8571	0.7567
Hybrid-XGBoost	0.9992	0.9741	0.9976	0.9857
Hybrid-CatBoost	0.9987	0.9738	0.9940	0.9838

and

Table 3. Comparison of model performance on the IEEE-CIS Credit Card dataset.

Model	Accuracy	Precision	Recall	F1-Score
LR	0.9802	0.6612	0.7087	0.6841
Random Forest	0.9892	0.8321	0.7213	0.7727
XGBoost	0.9693	0.8126	0.8320	0.8222
CatBoost	0.9690	0.6744	0.8971	0.7698
Hybrid XGBoost	0.9993	0.9862	0.9822	0.9842
Hybrid CatBoost	0.9990	0.9658	0.9701	0.9679

summarize the performance of all models across the two credit card fraud detection benchmarks. Several insights emerge from these results. Firstly, all traditional baseline models, including logistic regression (LR), random forest (RF), XGBoost, and CatBoost, exhibited high overall accuracy, a well-documented artifact of class imbalance in fraud detection. LR, despite its high recall (0.9184 and 0.7087 for the European and IEEE-CIS datasets, respectively), showed comparatively lower precision, indicating a tendency towards false positives.

RF, XGBoost, and CatBoost demonstrated more balanced trade-offs with improved precision–recall balances reflected in their respective F1-scores (Table 2). The hybrid models (Hybrid-XGBoost and Hybrid-CatBoost) significantly outperformed the baseline classifiers on both datasets, achieving F1-scores exceeding 0.97 on the European dataset and approximately 0.98 on the IEEE-CIS dataset. This improvement highlights the benefit of integrating VAE-generated anomaly scores with graph-based relational features captured by the GAT within an ensemble learning framework.

As illustrated in Figure 2 and Figure 3, the ROC curves for both hybrid models dominate those of baseline models across all thresholds. Hybrid-XGBoost achieved AUC values of 0.995 (European dataset) and 0.990 (IEEE CIS dataset), while Hybrid-CatBoost recorded 0.992 and 0.984, respectively, confirming strong discriminatory performance.

5.2. Comparison with Similar Studies

We benchmark our approach against state-of-the-art methods in Table 4. While deep ensembles such as GAN-LSTM [41] and CNN-LSTM hybrids [42] report strong F1-scores around 0.975–0.984, our Hybrid-XGBoost surpasses these with an F1-score of 0.9875. Similarly, encoder–decoder GNNs [43] achieve moderate gains by exploiting relational structures, but their performance (F1 = 0.86) falls short of our probabilistic–relational integration. Transformer-based models [44] show solid precision (0.958) but considerably weaker recall (0.795), highlighting their limitation in detecting rare fraud cases.

Compared with stacked deep learning ensembles [21] and LSTM ensembles [45], which report recall above 0.996 but without corresponding precision values, our hybrid approach achieves a more balanced performance, reducing false alarms while maintaining near-perfect recall. These findings shows that the joint use of VAE-derived anomaly scores and GAT-based embeddings within a stacking ensemble provides consistent improvements across all metrics, offering a more generalizable and practical solution for real-world fraud detection.

Table 4. Comparison with other recent studies. Results for our proposed models are highlighted in bold.

Reference	Technique	Recall	Precision	F1-Score
Mienye and Swart [41]	GAN-LSTM	0.990	0.979	0.984
Forough and Momtazi [46]	LSTM-conditional random fields	0.756	-	0.807
Alfaiz and Fati [47]	CatBoost-resampling	0.961	-	0.869
Alfaiz and Fati [47]	XGBoost-SVM SMOTE	0.930	-	0.865
Ileberi and Sun [42]	Deep learning ensemble	0.961	0989	0.975
Xinwei et al. [48]	RNN	-	-	0.473
Madhurya et al. [49]	LR	0.968	-	-
Khalid et al. [50]	LR-SMOTE	0.944	-	0.944
Varmedja et al. [51]	MLP-SMOTE	0.816	-	0.804
Alarfaj et al. [52]	XGBoost	-	-	0.845
Mrozek et. al. [53]	RF-undersampling	1.000	-	0.118
Jain et al. [54]	RF	0.780	-	0.858
Lin and Jiang [55]	Autoencoder-RF	0.814	-	-
Najadat et al. [56]	LR	0.720	-	0.220
Dighe et al. [57]	LR	0.816	-	-
Nadim et al. [58]	Classification and regression trees	0.885	-	-
Alwan et al. [59]	Naive Bayes	0.836	-	-
Asha and Kumar [60]	Artificial neural network	0.761	-	-
Dhankhad et al. [61]	Random forest + undersampling	0.950	-	0.950
Dhankhad et al. [61]	XGBoost with undersampling	0.950	-	0.950
Tang and Liu et al. [44]	Structured Data Transformer	0.795	0.958	0.867
Esenogho et al. [45]	LSTM Ensemble + SMOTE-ENN	0.996	-	-
Mienye and Sun [21]	Stacked deep learning ensemble	0.997	-	-
Cherif et al. [43]	Encoder–decoder GNN	0.920	0.820	0.86
Ileberi and Sun [42]	Hybrid CNN-LSTM ensemble	0.961	0.989	0.975
This Paper	Proposed Hybrid-XGBoost	0.9976	0.9741	0.9875
This Paper	Proposed Hybrid-CatBoost	0.9940	0.9738	0.9838

Table 4. Comparison with other recent studies. Results for our proposed models are highlighted in bold.

Reference	Technique	Recall	Precision	F1-Score
Mienye and Swart [41]	GAN-LSTM	0.990	0.979	0.984
Forough and Momtazi [46]	LSTM-conditional random fields	0.756	-	0.807
Alfaiz and Fati [47]	CatBoost-resampling	0.961	-	0.869
Alfaiz and Fati [47]	XGBoost-SVM SMOTE	0.930	-	0.865
Ileberi and Sun [42]	Deep learning ensemble	0.961	0989	0.975
Xinwei et al. [48]	RNN	-	-	0.473
Madhurya et al. [49]	LR	0.968	-	-
Khalid et al. [50]	LR-SMOTE	0.944	-	0.944
Varmedja et al. [51]	MLP-SMOTE	0.816	-	0.804
Alarfaj et al. [52]	XGBoost	-	-	0.845
Mrozek et. al. [53]	RF-undersampling	1.000	-	0.118
Jain et al. [54]	RF	0.780	-	0.858
Lin and Jiang [55]	Autoencoder-RF	0.814	-	-
Najadat et al. [56]	LR	0.720	-	0.220
Dighe et al. [57]	LR	0.816	-	-
Nadim et al. [58]	Classification and regression trees	0.885	-	-
Alwan et al. [59]	Naive Bayes	0.836	-	-
Asha and Kumar [60]	Artificial neural network	0.761	-	-
Dhankhad et al. [61]	Random forest + undersampling	0.950	-	0.950
Dhankhad et al. [61]	XGBoost with undersampling	0.950	-	0.950
Tang and Liu et al. [44]	Structured Data Transformer	0.795	0.958	0.867
Esenogho et al. [45]	LSTM Ensemble + SMOTE-ENN	0.996	-	-
Mienye and Sun [21]	Stacked deep learning ensemble	0.997	-	-
Cherif et al. [43]	Encoder–decoder GNN	0.920	0.820	0.86
Ileberi and Sun [42]	Hybrid CNN-LSTM ensemble	0.961	0.989	0.975
This Paper	Proposed Hybrid-XGBoost	0.9976	0.9741	0.9875
This Paper	Proposed Hybrid-CatBoost	0.9940	0.9738	0.9838

5.3. Discussion

The results across both datasets confirm that hybridizing VAE anomaly scores with GAT-based relational embeddings significantly enhances fraud detection performance. The F1-scores for Hybrid-XGBoost (0.9857 on the European dataset, 0.9842 on the IEEE dataset) and Hybrid-CatBoost (0.9838 on the European dataset, 0.9679 on the IEEE dataset) indicate an almost ideal balance between precision and recall. These models not only detect nearly all fraud instances but also significantly reduce false positives, which is critical in minimizing operational costs in fraud analytics systems.

The ROC curves in Figure 2 and Figure 3 corroborate this finding. Hybrid-XGBoost achieves AUC scores of 0.995 (European dataset) and 0.990 (IEEE dataset), while Hybrid-CatBoost follows with AUCs of 0.992 and 0.984. These high AUC values indicate that the hybrid models are effective across a wide range of thresholds. The improvements can be attributed to complementary strengths: the VAE identifies statistical outliers in high-dimensional feature space, while GAT captures latent topological patterns. Feeding these diverse insights into the meta-classifier enables more robust and informed decision-making.

5.4. Limitations of the Study

While highly effective, the proposed framework entails several practical trade-offs. First, constructing high-quality transaction graphs introduces additional computational overhead, especially for large-scale datasets. The performance of the GAT module is also sensitive to graph quality; noisy or sparse edges may limit its ability to capture meaningful relationships. Second, although class imbalance is mitigated using weighting schemes, extreme imbalance may still pose challenges for stable model training. Finally, this study does not include a complete ablation analysis (e.g., VAE-only, GAT-only, or linear fusion versus the full hybrid), which would provide further evidence of the contribution of individual modules. These aspects represent natural extensions for future research, particularly in the context of dynamic graph construction, scalable GNN variants, and visualization techniques for better interpretability.

6. Conclusions and Future Work

This paper presented a hybrid fraud detection framework that integrates VAE-based anomaly detection, GAT-based relational learning, and gradient-boosted decision ensembles (XGBoost and CatBoost) into a unified architecture. The method was evaluated on two highly imbalanced benchmark datasets, i.e., the European Credit Card and IEEE-CIS, and outperformed traditional models across all key metrics. In quantitative terms, the hybrid models achieved 15–20% higher F1-scores compared to baseline classifiers such as logistic regression, random forest, and standalone boosting methods. Hybrid-XGBoost in particular reached an F1-score of 0.9857 on the European dataset, while Hybrid-CatBoost achieved 0.9838, establishing new performance benchmarks on these datasets.

The results demonstrate that combining probabilistic feature-level anomalies from VAE with graph-based topological features from GAT significantly improves detection accuracy. Notably, the proposed Hybrid-XGBoost model achieved excellent performance, outperforming other baseline models. The Hybrid-CatBoost variant also performed strongly, highlighting the robustness of the stacking ensemble mechanism. These findings support the hypothesis that hybrid architectures leveraging both local feature irregularities and global relational structures are more effective in capturing complex fraud patterns.

Future work will also explore explainable AI (e.g., SHAP, LIME) to improve model interpretability and expand the framework to real-time streaming fraud detection environments using dynamic graph construction and online learning.