Windows Malware Detection via Enhanced Graph Representations with Node2Vec and Graph Attention Network

Abstract

As malware has become increasingly complex, advanced techniques have emerged to improve traditional detection systems. The increasing complexity of malware poses significant challenges in cybersecurity due to the inability of existing methods to understand detailed and contextual relationships in modern software behavior. Therefore, developing innovative detection frameworks that can effectively analyze and interpret these complex patterns has become critical. This work presents a novel framework integrating API call sequences and DLL information into a unified, graph-based representation to analyze malware behavior comprehensively. The proposed model generates initial embeddings using Node2Vec, which uses a random walk approach to understand structural relationships between nodes. Graph Attention Network (GAT) then enhances these initial embeddings, which utilizes attention mechanisms to incorporate contextual dependencies and enhance semantic representations. Finally, the enhanced embeddings are classified using Convolutional Neural Network (CNN) and Gated Recurrent Units (GRU)s, a custom hybrid CNN-GRU-3 deep learning-based model capable of effectively modeling sequential patterns. The dual role of GAT as a classifier and feature extractor is also analyzed to evaluate its impact on embedding quality and classification accuracy. Experimental results show that the proposed model achieves superior results with an accuracy rate of 0.9961 compared to state-of-the-art approaches such as ensemble learning and standalone GAT. This achievement highlights the framework’s ability to utilize contextual information for malware detection. The real-world dataset used provides a benchmark for future work, and this research lays a comprehensive foundation for advancing graph-based malware analysis.

1. Introduction

Malware is defined as malicious code that infiltrates computers or devices connected to the internet and steals sensitive information belonging to government, commercial, or private organizations [1]. Malware has become more sophisticated and advanced with time, using advanced techniques, making malware detection more difficult [2,3]. This dynamic structure makes malware detection difficult and also requires the development of detection methods. Malware detection is divided into two main categories: static and dynamic analysis methods. Static analysis allows examination of the code without executing it [4], and low-level features such as text patterns, opcodes, and byte sequences are obtained. Although these methods are fast and effective, they are vulnerable to methods such as encryption and code obfuscation. In this study, static analysis was performed using software behavioral features, such as API call sequences and DLL dependencies, to understand contextual relationships. On the other hand, dynamic analysis involves running the program in a secure environment and observing its behavior [5]. However, these methods require more computational resources. A key feature for malware detection is the Application Programming Interface (API) call sequences that define the program behaviors [6]. API calls cover important behaviors such as file operations, network access, and registry changes and often contain contextual patterns that are distinctive for malware. Recent studies have applied Machine Learning (ML) [7,8,9,10] and Deep Learning (DL) [11,12,13,14] approaches to analyze API call sequences for malware detection. While ML often overlooks semantic context, DL models offer improved accuracy by capturing complex patterns and structural dependencies, making them effective against advanced and increasingly sophisticated evasion techniques. DL also stands out for its success in modeling complex malware features such as API call sequences or structural dependencies. Examining the relationships between API call sequences and Dynamic Link Library (DLL) helps develop more comprehensive and effective detection techniques for malware behavior. For example, an API sequence like “CreateWindow->ShowWindow->CreateFile” may indicate a malicious behavior when analyzed in context, such as hiding a malware window and creating a malicious file on the system [3]. DLL and API call sequences provide semantic and contextual information crucial to accurate malware detection. Unlike frequency-based approaches, NLP-based (Natural Language Processing) methods can effectively understand these relationships by treating API calls and DLLs as sequential text-like data [15]. However, many existing models still overlook deeper semantic connections between API functions.

Graph representation has been widely used in language processing research for many years. In recent years, it has also come to the fore in malware analysis [16]. Integrating API call sequences and DLL usage into a unified graph analysis framework provides a more holistic view of software behavior, enabling more sensitive and robust malware detection. Inspired by the above arguments, we propose a new framework to detect increasingly sophisticated Windows malware in this paper utilizing the semantic analysis of API call sequences and DLLs. Unlike previous works in the literature that focus on analyzing API calls or DLLs separately, our approach addresses the joint analysis of these two critical components. In this paper, we propose a new framework for malware detection that uses graph-based methodologies using API call sequences and DLLs to construct meaningful graph representations. In the proposed framework, the Node2Vec algorithm generates initial embeddings (vector representations of nodes) in the graph. Node2Vec’s random walk approach provides a strong starting point for understanding the structural relationships of nodes. These initial embeddings are improved by GAT with attention coefficients and made more contextual by giving weight to the importance of information from neighboring nodes. To the best of our knowledge, this is the first application to integrate Node2Vec embeddings with GAT for contextual enhancement in malware detection. Finally, these improved embeddings are fed into our custom CNN-GRU-3, a DL-based hybrid classifier that can effectively model sequential and complex relationships. Multiple experimental setups are designed to evaluate the performance of the proposed framework and assess the effectiveness of graph embedding methods and classifiers. First, graph embedding techniques Node2Vec and GraphSAGE are compared regarding their ability to extract semantic and structural features from the generated graphs. These embeddings are then evaluated using two classification approaches: (1) a custom CNN-GRU-3 hybrid model designed to understand patterns and (2) an ensemble learning method that combines the strengths of multiple classifiers to improve performance. Next, we examine GAT in different roles. As a classifier, GAT is directly used to detect malware by processing graph embeddings and utilizing its ability to understand contextual dependencies within API and DLL graphs. Additionally, GAT is evaluated as an embedding enhancer, where it improves the semantic representations of Node2Vec and GraphSAGE embeddings by incorporating the contextual structure of the graph before feeding the improved embeddings to classifiers such as CNN-GRU-3 and ensemble learning models. Finally, GAT was used as a feature extractor, where it generated enriched embeddings from the graphs, which were then classified using CNN-GRU-3 and ensemble learning.

This new comprehensive evaluation and framework not only compares the performance of different embedding techniques and classifiers but also highlights the dual role of GAT as both a classifier and a feature extractor. The results of these experiments allowed us to propose the most effective framework for malware detection by balancing semantic richness and classification accuracy.

The key contributions of this research are as follows.

• A novel framework is proposed that enhances initial embedding methods, such as Node2Vec and GraphSAGE, with GAT by integrating API call sequences and DLL information through graph representations. This approach establishes a robust foundation for malware detection by enriching contextual semantic features.
• A comprehensive comparative analysis is performed to evaluate the effectiveness of various graph embedding techniques, including Node2Vec, GraphSAGE, and GAT, in the context of malware detection.
• The dual functionality of GAT as both a classifier and feature extractor is thoroughly examined, with a detailed assessment of its impact on semantic information extraction and classification performance.
• A real-world dataset is developed from practical scenarios, contributing to the literature while providing a reliable benchmark for future malware detection research.

The rest of the paper is organized as follows: Section 2 discusses some state-of-the-art research in the literature. Section 3 and Section 4 highlight theoretical backgrounds and the experimental results. Section 5 includes limitations and discussion. This study is concluded in Section 6 by addressing the research aspects.

2. Related Studies

Recent research explores innovative methodologies to overcome the above-mentioned challenges. Sequence-based methods for malware detection use DL models to extract sequential features from API call sequences. API-MalDetect is a framework that utilizes API call sequences for malware detection on Windows systems [13]. The framework combines a CNN and BiGRU-based hybrid feature extractor with NLP-inspired encoding to understand semantic relationships in API calls. API-MalDetect achieves high accuracy and robustness against evolving malware variants, outperforming state-of-the-art methods in various evaluation metrics. Another paper [17] presents a DL framework that detects malware through intrinsic features derived from API sequences, such as software behavior, semantic information, and relationships between API calls. By employing Bi-LSTM and convolutional layers, the proposed model achieves improved accuracy and F1-scores over existing baselines. The study highlights the limitations of traditional methods, which often overlook the contextual semantics of API calls, and proposes a more comprehensive approach for dynamic malware detection. Kumar et al. [18] introduced a transfer and ensemble learning method using CNNs to classify malware in enterprise networks. Their approach effectively handles polymorphic malware with an impressive detection accuracy of 0.9936 on benchmark datasets by visualizing malware as grayscale images. Similarly, Bensaoud and Kalita [1] used a CNN-LSTM hybrid model to classify malware based on opcode sequences and API calls, demonstrating the effectiveness of combining sequential and convolutional approaches for high accuracy. Darem et al. [19] introduced a novel DL-based framework utilizing feature engineering techniques to reduce feature space significantly and achieved a 0.9601 detection accuracy using the Microsoft malware dataset. Brown et al. [20] explored the potential of Automated Machine Learning (AutoML) for optimizing DL architectures, demonstrating superior performance in static and online malware detection compared to manually crafted models. Recent work in malware detection highlights the importance of integrating advanced analysis techniques and Graph Neural Networks (GNNs) against evolving threats. Bao et al. [21] proposed a multimodal Windows malware detection method combining static features from PE headers and dynamic Control Flow Graphs (CFG) using a hybrid analysis and DL framework, achieving 0.9925 accuracy. Zhang et al. [22] introduced a semantics-preserving reinforcement learning attack targeting GNN-based malware detection system, highlighting vulnerabilities in existing defenses while preserving malware functionality through semantic manipulation of CFGs. Lin et al. [23] explored the use of Graph Convolutional Networks (GCNs) and GATs for ransomware detection, utilizing API call graphs extracted from Cuckoo Sandbox reports, demonstrating the effectiveness of GCNs over GATs in analyzing structural and relational data in malware detection. These studies underline the critical role of GNNs and multimodal approaches in enhancing the accuracy and resilience of modern malware detection systems. Chen et al. proposed a multimodal approach combining structural and semantic features using Class Set Call Graph (CSCG) and GAT. They achieved state-of-the-art accuracy in Android malware detection by combining graph-based and permission features [24]. In another study, Feng et al. [2] presented a graph neural network-based framework called DawnGNN for Windows malware detection that utilize semantic embeddings derived from API documents to improve detection performance. Extending API behavior analysis, Wu et al. proposed the MINES framework, which combines graph contrastive learning and CNNs to extract and fuse API presence and transition features, significantly improving malware classification [25].

3. Theoretical Background

This section presents the theoretical and architectural foundations of the proposed malware detection framework. Before introducing the experimental setup, we first detail the construction of API and DLL-based graphs, the embedding techniques used to represent graph nodes, and the DL architectures employed for classification. Specifically, we describe the system components, including the graph construction process, the Node2Vec and GraphSAGE embeddings, the custom CNN-GRU-3 model, the ensemble learning strategy, and the use of GAT as both a classifier and a feature enhancer. Finally, we introduce the dataset utilized in the experiments and describe its structure and pre-processing.

3.1. Graph Construction

Graphs consist of nodes (or states) and edges (or relationships) that connect the nodes. The created nodes and edges can have specific attributes or properties associated with them [26]. In our graph representation, nodes correspond to both API calls and DLLs extracted from executable files. Each API or DLL encountered during static analysis is treated as a unique node in the graph structure. Relationships between API calls are represented as edges in the graph structure. Edges represent relationships between these nodes. For API nodes, a directed edge is drawn from one API to the next based on their sequential appearance in the call trace. Similarly, edges between DLLs or API and DLL nodes represent functional associations or co-usage patterns. In addition to temporal edges between API calls, we construct edges from API nodes to the corresponding DLLs that implement or invoke those functions. This dual relationship modeling enhances the semantic richness of the graph and enables more accurate behavior profiling. For example, if software calls the API “CreateFile” and then calls the API “WriteFile”, a directed edge is created between these two nodes. Edges are designed to reflect the temporal order of API calls. Alternatively, relationships can be established between APIs called within the same transaction. However, since static analysis data are used in this study, a control flow relationship based on temporal order is preferred. This approach is practical in understanding behavioral patterns that are frequently seen in malware. We can model DLL and API Calls using graph theory to represent software behavior. The rationale behind using a graph representation lies in its ability to reflect the behavioral flow of a program. By modeling API call sequences and DLL usages as connected nodes, the framework understands meaningful execution patterns. Each API Call and DLL is considered as a node, while the relationships between these nodes are represented by edges (Equation (1)). The following explains how to define this graph structure:

$$G=(V,E)$$

(1)

Node Set (V ): each API call is defined as a node (Equation (2)). The consecutive calls or operations performed in a specific order between the API calls constitute the node set. Similarly, DLL files are defined as nodes (Equation (3)). DLLs are dynamic libraries associated with specific API calls.

$$V_{\mathrm{API}}=\{v_1,v_2,\dots ,v_i\}$$

(2)

$$V_{\mathrm{DLL}}=\{d_1,d_2,\dots ,d_i\}$$

(3)

Here, $v_i$ represents the $i^{th}$ API call, and di represents the $i^{th}$ DLL. Edge Set (E): whenever an API call follows another API call, a directed edge is drawn between these two nodes (Equation (4)). For example, if the API call “CreateFile” is followed by “WriteFile”, an edge is created between these two nodes:

$$E_{\mathrm{API}}=\{(v_i,v_j)\mid v_i\to v_j\}$$

(4)

For instance, if a software makes three consecutive API calls “CreateFile” → “WriteFile” → “CloseHandle”, we represent these calls with three nodes ($v_1,v_2,v_3$) and two edges (($v_1,v_2$), ($v_1,v_3$)). If a DLL implements multiple API calls or is used in conjunction with another DLL, directional edges can be created between these DLLs (Equation (5)).

$$E_{\mathrm{DLL}}=\{(d_i,d_j)\mid d_i\to d_j\}$$

(5)

For example, in cases where the files “kernel32.dll” and “user32.dll” work together, an edge is drawn between these two DLLs. Figure 1 presents an example API call sequence graph constructed from a PE file. Each node corresponds to an individual API function, and directed edges indicate the chronological order in which these functions are invoked. For instance, GetFileSize is followed by PostMessage, which is then succeeded by FreeLibrary and subsequently by GetCurrentProcess. This sequence continues through various file and process-related API calls, capturing the behavioral flow of the executable. Such graph representations allow the model to understand not only the occurrence of specific API calls but also their contextual relationships within the execution chain, an essential factor in accurate malware detection.

3.2. Graph Node Embedding

After graphs are created, node features need to be converted to vectors. For this purpose, we used Node2Vec and GraphSAGE embedding techniques in our study. These techniques are explained in detail below.

3.2.1. Node2Vec

Node2Vec is an algorithm that converts graph nodes into vectors (embedding) [27]. These vectors represent the structural features of the graph (topology) and the relationships between nodes. Node2Vec performs random walks on the graph starting from a certain node. During these walks, the context of the nodes is determined. Node sequences created by random walks are treated as word sequences, and embeddings are created using Word2Vec. In practice, a walk length of 30 and 200 walks per node, and 128-dimensional embeddings based on experimental validation were used. Algorithm 1 shows the pseudocode of the Node2Vec algorithm.

Algorithm 1 Node2Vec Random Walk and Embedding Learning

Input: Graph $G=(V,E)$, embedding dimension d, walk length l, walks per node r, context window size k, return parameter p, in-out parameter q Output: Feature representations f   1: Compute transition probabilities $\pi$ using parameters p and q   2: Create weighted graph $G^{\prime }=(V,E,\pi )$   3: Initialize list all_walks as empty   4: for each node $u\in V$ do   5:     for $i=1$ to r do   6:         $walk$←RandomWalk($G^{\prime },u,l$)   7:         Append $walk$ to all_walks   8:     end for   9: end for 10: Train Skip-Gram model on all_walks using context window size k to obtain embeddings f 11: return f 12: function RandomWalk($G^{\prime },start\_nodeu,lengthl$) 13:     Initialize walk $\leftarrow [start\_node]$ 14:     while length of walk $<l$ do 15:         $curr\_node$← last node in walk 16:         $neighbors$←GetNeighbors($curr\_node$, $G^{\prime }$) ▹ GetNeighbors, returns the set of neighboring nodes directly connected to the given node in the graph G. 17:         $next\_node$←AliasSample($neighbors$, $\pi$) ▹ The AliasSample, samples the next node among neighbors according to their pre-calculated transition probabilities. 18:         Append $next\_node$ to walk 19:     end while 20:     return walk 21: end function

3.2.2. GraphSAGE

GraphSAGE creates embeddings for nodes using node features and the local structure of the graph [28]. Only a certain number of neighboring nodes are selected for a node. The features of the selected neighboring nodes are combined with a specific function (e.g., mean, maximum, LSTM). Embedding is created by combining the information from neighboring nodes with the target node’s features. GraphSAGE uses aggregation functions to aggregate the information accumulated in the node’s local neighborhood. The graphic illustration of the GraphSAGE sample and the aggregate approach is shown in Figure 2.

Specifically, at each iteration or layer depth k, each node v collects feature information from its $N\left(v\right)$ neighbors. This collection process can be performed by different methods such as mean pooling, LSTM-based aggregation, or max pooling. The representation of each node v at depth k is expressed as $h_v^k$ (Equation (6)) [28].

$$h_v^k=\sigma \left(W_k·\mathrm{CONCAT}\left(h_v^{k-1},{\mathrm{AGGREGATE}}_k\left(\{h_v^{k-1},{\forall }_u\in N\left(v\right)\}\right)\right)\right)$$

(6)

where $W_k$ is the trainable weight matrix in layer k, $\sigma$ is the non-linear activation function, $h_v^{k-1}$ is the representation of the node features from the previous layer, and AGGREGATE is the aggregator function that collects information from neighbors of v.

3.3. Custom CNN-GRU-3 Model

In this study, a hybrid CNN-GRU-3 architecture was selected to utilize the strengths of both convolutional and recurrent layers. The CNN layers effectively understand local patterns and spatial features from the API-DLL embeddings, while the stacked GRU layers are well-suited for modeling sequential dependencies across time steps. This combination enables the model to learn low-level patterns and long-term behavioral dependencies, which is essential to identifying complex malware behaviors accurately. The model has three convolutional layers, three grouping layers, three GRU layers, and one fully connected layer. The size of the convolutional layer used for feature extraction is 3 × 3. We use the Exponential Linear Unit (ELU) activation function [29]. Max-pooling layers with 2 × 2 kernels reduce the feature map dimensions, followed by a smoothing layer that transforms the output into a one-dimensional vector. The model architecture includes one GRU layer with 512, 256, and 128 neurons, a depth of 3, and a dropout rate of 0.3. Finally, a fully connected dense layer with 64 neurons and a Rectified Linear Unit (ReLU) activation function is added, followed by a dense layer with a single neuron and a Sigmoid activation function. This GRU component increases the ability of the model to distinguish complex patterns in sequential data. The proposed architecture that mixes two DL models is shown in Figure 2. The optimization process for the model included 150 epochs during training and a batch size of 64. The Adam optimizer with a learning rate of 0.001 was preferred for adaptive learning rate computations, making it a suitable choice for efficient optimization. The dataset is split into 80% training and 20% test sets and trained with 5-fold cross-validation. The pseudocode for the custom CNN-GRU-3 model is given in Algorithm 2.

Algorithm 2. Pseudocode for Custom CNN-GRU-3 Model

Input: $X_{\text{train\_input}},X_{\text{train\_output}}$ Output: Trained CNN-GRU-3 model and classification performance metrics   1: function ModelTraining(X_train_input, X_train_output)   2:     Apply convolutional layers with ELU activation:   3:         Conv1D(filters=64, kernel_size=3), activation=ELU   4:         MaxPooling1D (pool_size=2)   5:         Conv1D(filters=64, kernel_size=3), activation=ELU   6:         MaxPooling1D (pool_size=2)   7:         Conv1D(filters=64, kernel_size=3), activation=ELU   8:         MaxPooling1D (pool_size=2)   9:     Apply a Flatten layer 10:     GRU Layers: 11:         GRU Layer 1: 512 units, dropout=0.3 12:         GRU Layer 2: 256 units, dropout=0.3 13:         GRU Layer 3: 128 units, dropout=0.3 14:     Fully Connected Output Layers: 15:         Dense: 64 units, dropout=0.3, activation=ReLU 16:         Dense: 1 unit, activation=Sigmoid 17:        Output layer with sigmoid activation for 2 classes 18:     Training and Evaluation: 19:        Train model using Adam optimizer and binary_crossentropy loss 20:        Evaluate model performance 21: end function

The embeddings developed by GAT are arranged in a time series-like sequence to provide data input to the CNN-GRU model. The embedding of each node is placed in the sequence according to the original order of the API calls. This arrangement allows CNN to understand spatial relationships between adjacent nodes and GRU to understand sequential patterns. For example, if the API call order in a software is [“CreateFile”, “WriteFile”, “CloseHandle”], the embeddings of these calls are fed into the model in the same order. This approach allows graph data to be aligned with time-series models.

3.4. Ensemble Learning Approach

Ensemble learning can be particularly effective in malware classification because it combines multiple models to catch complex patterns and situations that a single model might miss [30]. Malware data present unique challenges, such as high dimensionality, diverse patterns across different malware families, and imbalanced class distributions (malicious vs. benign). Each model provides insight into different aspects. For example, neural networks like CNNs and LSTMs are good at identifying sequential patterns in API call sequences. At the same time, tree-based methods like Random Forest (RF) or XGBoost (XGB) can better handle data with complex interactions [31,32]. Combining these models allows an ensemble to benefit from the fact that each model better analyzes different features of malware data. In this study, an ensemble learning approach is applied with a soft voting classifier of the predictions of the best-performing models. The probabilities generated by each model are averaged to determine the final prediction [33,34]. The best results from the XGB, RF, Decision Tree (DT), LightGBM, and Adaboost models are combined with the voting mechanism to form a final classification model [35,36,37]. The dataset is split into 80% training and 20% test sets and trained with 5-fold cross-validation.

3.5. Graph Attention Network Classifier

Graph Neural Networks (GNNs) are DL models that perform learning on data in a graph structure at the node, edge, or graph level [38]. Unlike traditional neural networks, GNNs extract features for each node by considering the topological relationships and contextual information between nodes and represent these nodes as vectors [39]. GNNs generally use the message-passing method [2]. In this method, each node receives information from its neighbors and combines it with its features to create an updated node representation. In recent years, GNNs have achieved significant performance gains in many tasks such as Graph Convolutional Networks (GCNs) [40], GraphSAGE [28], and GAT [41]. Therefore, many studies have started to rely on GNNs for malware analysis. GAT is an improved version of GNNs and dynamically determines the importance of neighbors by giving different weights to the information from the neighbors of each node [42]. Using the attention mechanism, GAT learns the importance of each node relationship and thus obtains better node representations [43]. GAT calculates an attention coefficient $\left(e_{vu}\right)$ for each node v and its neighbor u. This coefficient is calculated using the feature vectors ($h_v$ and $h_u$) of node v and its neighbor u as follows:

$$e_{vu}=\mathrm{LeakyReLU}\left(a^T·\left[W{h}_v\Vert W{h}_u\right]\right)$$

(7)

where a is the learnable weight vector, W is the learnable weight matrix, $\left[W{h}_v\Vert W{h}_u\right]$ is the concatenation of the feature vectors of nodes v and u. LeakyReLU is an activation function that provides a small slope for negative values. The attention coefficients are normalized by the softmax function, so that the sum of the attention coefficients among the neighbors of node v is 1:

$${\alpha }_{vu}={\displaystyle \frac{exp\left(e_{vu}\right)}{{\sum }_{k\in N\left(v\right)}exp\left(e_{vk}\right)}}$$

(8)

where ${\alpha }_{vu}$ determines the influence of node u on node v. This normalization process expresses the relative importance of neighbors. Each node v creates a new node representation $h_v$ by weighting the features from its neighbors with their normalized attention coefficients:

$$h_v^{\prime }=\sigma \left(\sum _{u\in N\left(v\right)}{\alpha }_{vu}W{h}_u\right)$$

(9)

This formula allows the node to be updated by considering its neighbors’ information. Thus, each node has richer information, not only about itself but also about its neighbors. GAT generally uses the multi-head attention mechanism. This mechanism learns the representation of a node using multiple independent attention heads and combines the results of these heads. The multi-head attention mechanism is formulated as:

$$h_v^{\prime }={\parallel }_{k-1}^K\sigma \left(\sum _{u\in N\left(v\right)}{\alpha }_{vu}^k{W}^k{h}_u\right)$$

(10)

where K denotes the number of attention heads and ${\parallel }_{k-1}^K$ denotes the concatenation of heads. This method allows capturing different relationships between nodes and creating a more robust node representation. All notations and explanations in the study are given in

Table 1. Notations and explanations of the formulas.

Notations	Explanations
G	Graph
V	Node set
E	Edge set
u	Node
v	Neighbor of the node
y	Classifiers
W	Learnable weight matrix.
$\left[W{h}_v\Vert W{h}_u\right]$	Weighted concatenation of feature vectors of nodes v and u.
L	Loss function
$h_u$	The feature vector of neighbor node u obtained using Node2Vec or GraphSAGE.
$h_v$	The feature vector of node v obtained using Node2Vec or GraphSAGE.
$Z_v$	New node representation by GAT.
$\alpha$	Learnable weight vector.
${\alpha }_{vu}$	It shows the normalized influence of node u on node v.
$N\left(v\right)$	The neighbor set of node v. This includes the nodes to which v is directly connected.
$e_{vu}$	The attention coefficient between node v and its neighbor u.
p	Controls the probability of the walk to return to the previous node.
q	Determines the tendency of the walk to explore local neighborhoods or the wider area.

.

The role of GAT has been evaluated both as a standalone classifier and feature extractor in different scenarios. In Experiment 2, GAT was used as a standalone classifier, and its performance was compared with the combinations of Node2Vec + GAT and GraphSAGE + GAT. This experiment aims to examine the effectiveness of GAT in malware detection by directly exploiting the contextual relationships between nodes and neighbors. In Experiment 3, the node embeddings obtained using Node2Vec and GraphSAGE were further improved with GAT, and these improved embeddings were evaluated with custom CNN-GRU-3 and ensemble learning models. This experiment aims to demonstrate the potential of GAT to improve classification performance by making node embeddings more contextual. Finally, in Experiment 4, GAT was used only as a feature extractor, and the embeddings obtained from graph data were fed to CNN-GRU-3 and ensemble learning models. In this experiment, we examined how GAT provides a performance increase over other classifiers by improving the features obtained from the nodes. The details of these experiments are given in Section 4.

3.6. Dataset

The dataset, which was created using API Call and DLL sequences extracted from Windows PE files, initially consisted of 1000 benign and 1000 malicious software samples. In order to increase the performance of the dataset and provide a more robust evaluation, the amount of data for each category was increased to 4000 using the data augmentation method, as shown in

Table 2. The amount of data in the PEmalware dataset.

	Label	Number of Software
	Malicious Software	1000
Before Data Augmentation	Benign Software	1000
	Total Software	2000
	Malicious Software	4000
After Data Augmentation	Benign Software	4000
	Total Software	8000

. As a result of this process, a total of 8000 data samples (4000 benign and 4000 malicious) were obtained. This balanced dataset reduces bias during model training and increases model performance. The features and descriptions of the dataset are shown in

Table 3. Dataset features.

Features	Description	Values
File Size	Size of file in byte	Integer
Digital Signature	Whether there is a digital signature	Boolean
API Calls	The name of functions	String
DLLs	The name of DLLs	String
Label	Whether the file is malicious	Boolean

.

In our graph construction process, we integrate two distinct features extracted from each PE file: the sequence of API calls and the list of imported DLLs. Both API calls and DLLs are represented as nodes in a unified directed graph. Sequential API calls are connected with directed edges to understand the temporal execution flow. In addition, for every API call in the sequence, we establish directed edges pointing to all DLL nodes associated with the file. This way reflects potential dependencies or origins of the API calls, even in the absence of exact DLL-API pairings. Combining dynamic behavior (via API sequences) and static structural context (via DLL imports) into a single graph provides the model with a rich and comprehensive representation of each sample. This enables downstream models like GAT to learn behaviorally and contextually better relevant patterns for distinguishing between benign and malicious files. Different embedding techniques, such as Node2Vec, GraphSAGE, and GAT, were used to represent the graph nodes. Feature selection was performed to identify the highly significant features that could improve the model’s performance. For this purpose, features were ranked using SelectKBest and ANOVA (Analysis of Variance) F-value, and their significance was visualized. In Figure 3, feature names are shown on the x-axis, and significance scores are shown on the y-axis. For this purpose, features were ranked using the SelectKBest method and ANOVA F-value, and their significance was visualized. ANOVA F-value is a statistical method used to measure the discrimination power of each feature in a dataset on the target variable by calculating the ratio between the between-class variance and within-class variance. In this study, the SelectKBest method from Scikit Learn [44] was utilized to calculate each feature’s ANOVA F-value [45]. Features were then sorted in descending order based on their F-values, and the top k features with the highest impact on the target variable were selected. This process enabled the model to perform better using fewer features, prioritizing those with the most contextual and meaningful information. Using ANOVA F-value in selecting features played a critical role in improving classification performance by focusing on the most significant attributes in the dataset.

4. Proposed Approach and Experiments

This section presents the proposed malware detection framework in a formalized structure, followed by four distinct experimental configurations designed to evaluate the contribution and interaction of its core components. Before a detailed analysis of the experiments, we outline the structural relationship between the experimental setups and the proposed framework. The experiments were designed as follows: (i) graph-based node embedding, achieved through Node2Vec or GraphSAGE to understand topological and semantic features from API and DLL graphs; (ii) contextual refinement of embeddings through the GAT, which assigns importance weights to neighboring nodes using attention coefficients; and (iii) final classification using either a custom hybrid CNN-GRU-3 model, optimized for sequential pattern recognition, or an ensemble of traditional ML classifiers. The experiments were designed as follows:

• Experiment 1: to assess their baseline performance, Node2Vec and GraphSAGE embeddings were paired with CNN-GRU-3 and ensemble learning models.
• Experiment 2: GAT was used as a standalone classifier, and its effectiveness was compared with the combinations Node2Vec + GAT and GraphSAGE + GAT.
• Experiment 3: Node2Vec and GraphSAGE embeddings were refined by GAT, and enhanced embeddings were evaluated using CNN-GRU-3 and ensemble learning.
• Experiment 4: GAT was used solely as a feature extractor, and the resulting embeddings were classified using CNN-GRU-3 and ensemble learning.

Each of the four experiments explores different configurations of these components to assess their individual and combined contributions. Specifically, Experiment 1 evaluates the baseline performance of Node2Vec and GraphSAGE embeddings using CNN-GRU-3 and ensemble classifiers without GAT. Experiment 2 measures the standalone classification capability of GAT and its integration with initial embeddings. Experiment 3 corresponds to the proposed framework, where GAT refines the embeddings, and CNN-GRU-3 performs classification. Experiment 4 analyzes GAT solely as a feature extractor prior to classification. This experimental design allows for a modular and comparative evaluation of each architectural component and demonstrates how their integration leads to the performance gains achieved by the whole framework.

4.1. Experiment 1: Graph Embedding with Node2Vec and GraphSage Embedding—Comparison of Graph Node Embeddings

In Experiment 1, node embeddings obtained using Node2Vec and GraphSAGE are paired with CNN-GRU-3 and ensemble learning models to evaluate their baseline performance. This experiment aims to perform a baseline performance evaluation by comparing the contribution of both embedding methods to the classification process. For each node v, an embedding is created using Node2Vec or GraphSAGE:

$$h_v=\mathrm{Node}2\mathrm{Vec}\left(v\right)\mathrm{or}{h}_v=\mathrm{GraphSAGE}\left(v\right)$$

(11)

These embeddings are used as a vector representation of each node in low-dimensional space. Embeddings are given as input to CNN-GRU-3 and ensemble learning models. These classifiers perform malware detection using node representatives.

$$y=f\left(h_v\right),f\in \{\mathrm{CNN}\text{-}\mathrm{GRU}\text{-}3,\mathrm{Ensemble}\mathrm{L}.\}$$

(12)

4.2. Experiment 2: GAT as Classifier with Node2Vec and GraphSage

In Experiment 2, GAT is used as a standalone classifier, and the results are compared with the combinations of Node2Vec + GAT and GraphSAGE + GAT. This experiment analyzes the performance of GAT as a standalone classifier and its contribution when used with other embedding methods. In the same manner as Experiment 1, embedding is generated for each node using Node2Vec or GraphSAGE. Using these embeddings, GAT analyzes the relationships between nodes and creates a new representation $h_v$ for each node:

$$h_v^{\prime }=\sigma \left(\sum _{u\in N\left(v\right)}{\alpha }_{vu}W{h}_u\right)$$

(13)

In the last layer of GAT, a Sigmoid activation function is used to classify the resulting node representations. The sigmoid activation function used in the last layer of GAT makes it possible to estimate the probability (between 0 and 1) that a node belongs to a certain class in two-class malware detection. This function provides a non-linear activation, allowing the model to distinguish between classes more precisely.

$${\widehat{y}}_v=\sigma \left(W^{\prime }{h}_v^{\prime }\right)$$

(14)

Here, $W^{\prime }$ represents the learnable weights, and the Sigmoid activation function reduces the output of the model to the range [0, 1], which is especially suitable for binary classification. The model is trained using a loss function between the model outputs ${\widehat{y}}_v$ and the true labels $y_v$:

$$L=-\sum _v\left[y_{v}log\left({\widehat{y}}_v\right)+(1-y_v)log(1-{\widehat{y}}_v)\right]$$

(15)

This last layer and activation process is one of the necessary steps for using GAT as a classifier. This allows it to transform the node representations of the GAT model into a classification output that will decide on the presence or absence of malware. This allows GAT to be effectively used as a standalone classifier.

4.3. Experiment 3: Enhancing Node Embedding Using GAT with Node2Vec and GraphSAGE

In Experiment 3, GAT enhances the initial embeddings generated by Node2Vec and GraphSAGE with attention mechanisms. The initial embeddings understand the structural relationships of the nodes but may lack context information. GAT contextually enriches the embeddings by computing the contributions of neighboring nodes with attention weights. For example, in cases where an API call indicates malware behavior, GAT can assign a higher weight to this node. This process is achieved by directly feeding the embeddings from node2vec/GraphSAGE into the GAT layers. As a result, the resulting attention-enhanced embeddings become more meaningful and suitable for classification. This experiment evaluates the embedding improvement ability of GAT and the performance of these embeddings when used with more complex classifiers. The embedding $h_v$ for each node is generated with Node2Vec or GraphSAGE. GAT enhances these embeddings with information from neighboring nodes:

$$Z_v=\sigma \left(\sum _{u\in N\left(v\right)}{\alpha }_{vu}W\left(h_u\right)\right)$$

(16)

For each neighbor u of node v, the neighbor’s feature vector $h_u$ is first multiplied by the weight matrix (W). This transforms the node’s features to account for their influence on node v. This transformed feature vector $W\left(h_u\right)$ is then multiplied by the attention coefficient ${\alpha }_{vu}$. This expresses how much the neighbor node contributes to node v relative to its importance. The contributions of all neighbors of node v are summed N(v). This process allows node v to contextually obtain information from its neighbors and incorporate it into its representation. After the addition process, the resulting total value is inserted into an activation function $\sigma$.

4.4. Experiment 4: Extracting Features Using Single GAT

In Experiment 4, GAT is used solely as a feature extractor, and the resulting embeddings are given as input to CNN-GRU-3 and ensemble learning models. This experiment aims to analyze the effectiveness of GAT as a feature extractor and its classification success when combined with other classifiers.

4.5. Proposed Framework

This work presents a novel malware detection framework that integrates graph-based methodologies with DL approaches. The proposed model uses Node2Vec, GAT, and CNN-GRU-3 in a novel structure to exploit structural and contextual relationships in malware data. The overall proposed framework is shown in Figure 4, and the pseudocode is given in Algorithm 3. The model aims to extract and enhance semantic features from API call sequences and DLL information. First, the graph is constructed using API call sequences and DLL information, where nodes represent API calls or DLLs and edges represent interactions between them. Then, initial node embeddings are generated using Node2Vec via random walks that understand structural relationships in the graph. The embeddings are enhanced using GAT by applying attention mechanisms to improve contextual relationships. Finally, the GAT-enhanced embeddings are fed into the CNN-GRU-3 model, which performs the final classification to detect malware.

Algorithm 3. Pseudocode for Node2Vec + GAT + CNN-GRU-3 Training Pipeline

1: Input: Graph $G=(V,E,W)$, Dimensions d, Walks per node r, Walk length l, Context size k, Return parameter p, In-out parameter q, Number of epochs n   2: Output: Predicted class labels $\widehat{y}$   3: function Node2VecEmbedding($G,d,r,l,k,p,q$)   4:     $\pi$ = PreprocessModifiedWeights($G,p,q$)   5:     $G^{\prime }=(V,E,\pi )$   6:     Initialize $walks$ as Empty   7:     for $iter=1$ to r do   8:         for each node $u\in V$ do   9:            $walk$ = Node2VecWalk($G^{\prime }$, u, l) 10:            Append $walk$ to $walks$ 11:         end for 12:     end for 13:     f = StochasticGradientDescent(k, d, $walks$) 14:     return f 15: end function       16: function GAT(f, Adjacency Matrix A, Attention Parameters W, a) 17:     Initialize attention coefficients: $e_{vu}$ 18:     Normalize attention coefficients: ${\alpha }_{vu}$ 19:     Update node embeddings: $Z_v$ 20:     return Z 21: end function       22: function TrainCNNGRU3(Z, Y, n) 23:     Input Z 24:     Pass Z through convolutional layers: 25:         Conv1D: 64 filters, kernel size 3, activation ELU, then MaxPooling1D 26:         Conv1D: 64 filters, kernel size 3, activation ELU, then MaxPooling1D 27:         Conv1D: 64 filters, kernel size 3, activation ELU, then MaxPooling1D 28:     Flatten the output of convolutional layers 29:     Pass through GRU layers: 30:         GRU1: 512 hidden units, dropout 0.3 31:         GRU2: 256 hidden units, dropout 0.3 32:         GRU3: 128 hidden units, dropout 0.3 33:     Fully connected layers: 34:         Dense1: 64 units, dropout 0.3, activation ReLU 35:         Dense2: 1 unit, activation Sigmoid 36:     Compile model with Adam optimizer, binary crossentropy loss 37:     Train model for n epochs using Z and labels Y 38:     return trained model and predictions $\widehat{y}$ 39: end function

GAT’s attention mechanism improves Node2Vec embeddings by weighting the information from each node’s neighbors with attention coefficients (${\alpha }_{vu}$) to make them more contextual. While Node2Vec understands node relationships with random walks, these embeddings only carry structural information. GAT, on the other hand, provides semantic enrichment of embeddings by calculating the importance of nodes to their neighbors ($e_{vu}$). This process enables each node to create a more meaningful representation in its context. This contextual enrichment is evaluated with higher accuracy rates by DL classifiers. The effectiveness of the proposed model is evaluated through multiple experimental setups, where it is also compared with other graph embedding and classification techniques. The results show that the proposed Node2Vec + GAT + CNN-GRU-3 model achieves the highest accuracy of 0.9961 and outperforms other tested configurations. This achievement is due to the combination of Node2Vec’s effective structural embedding, GAT’s contextual enhancement of embeddings with an attention mechanism, and CNN-GRU-3’s sequential modeling capabilities.

4.6. Performance Metrics

This study uses various metrics to evaluate the performance of malware detection models. Accuracy (Equation (17)) determines the overall success of the model. At the same time, Recall (Equation (18)) and (Equation (19)) Precision metrics are used to measure how much malware was correctly detected and how much of the detected malware was malicious, respectively. F-measure (Equation (20)) is a measure obtained by combining recall and precision in a balanced manner, indicating the balanced success of the model. Cohen’s Kappa (Equation (21)) evaluates the reliability of the model against random guesses. At the same time, Mean Absolute Error (MAE) (Equation (22)) is used to measure how much the model’s predictions deviate from the true values. Using these metrics together provides a more comprehensive assessment of the model’s performance in malware detection.

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(17)

True Positive ($TP$ ) are true positive predictions and are cases where the model correctly detects malware; False Negative ($FN$) are cases where the model misses the malware, i.e., does not detect it despite being malicious; True Negative ($TN$) are cases where the model correctly classifies a benign software as benign; and False Positive ($FP$) are cases where a benign software is incorrectly classified as malicious.

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(18)

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(19)

$$\mathrm{F}\text{-}\mathrm{Measure}={\displaystyle \frac{2·\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(20)

$$\mathrm{Cohen}’\mathrm{s}\mathrm{Kappa}\left(\kappa \right)={\displaystyle \frac{P_0-P_e}{1-P_e}}$$

(21)

$P_0$ represents the observed agreement between raters or classifiers. $P_e$ represents the expected agreement between raters or classifiers based on chance. $\kappa =1$ indicates perfect agreement between raters or classifiers. $\kappa =0$ indicates agreement equivalent to what would be expected by chance. $\kappa <0$ indicates agreement worse than what would be expected by chance.

$$\mathrm{MAE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N|y_i-{\widehat{y}}_i|$$

(22)

where N is the total number of samples, $y_j$ is the actual value and ${\widehat{y}}_j$ is the value predicted by the model.

4.7. Results and Evaluation

In this study, the Node2Vec + GAT + CNN-GRU-3 model proposed for malware detection was developed and compared with other methods to analyze its performance. The hyperparameters for DL models (CNN and GRU) were determined empirically based on iterative experimentation and validation performance. The CNN block uses three convolutional layers with 64 filters and ELU activation, effectively capturing local spatial features. The GRU block consists of 3 layers with decreasing hidden units (512, 256, 128) and a dropout rate of 0.3 to prevent overfitting, optimized using the Adam optimizer with a learning rate of 0.0001. For the ensemble learning models (Random Forest and XGBoost), hyperparameter tuning was performed using grid search, exploring combinations of key parameters such as the number of estimators, maximum depth, and learning rate. The best-performing configurations, as shown in

Table 4. Hyperparameter configurations.

Model	Hyperparameter	Value
CNN	Convolution Layers	3
	Convolution filters	64
	Kernel Size	3
	Pool Size	2
	Activation	ELU
GRU	GRU Layers	3
	Hidden Units	512, 256, 128
	Dropout	0.3
	Optimizer	Adam
	Learning Rate	0.0001
	Loss	Binary Crossentropy
RF	n estimators	[50, 100, 200]—Best value: 100
	max depth	[3, 5, 7]—Best value: 3
	min samples split	[2, 5]—Best value: 2
	min samples leaf	[1, 2]—Best value: 2
XGB	n estimators	[50, 100, 150]—Best value: 100
	max depth	[3, 5, 7]—Best value: 7
	learning rate	[0.01, 0.1, 0.2]—Best value: 0.1
Node2Vec	Walk Length	30
	Walks per Node	200
	Embedding Dimension	128

, were selected based on cross-validation accuracy.

As a result of the experiments conducted with different embedding techniques (Node2Vec and GraphSAGE), contextual enhancement methods (GAT), and classifiers (CNN-GRU-3, Ensemble Learning), it is seen that our proposed model provides the highest accuracy compared to all other combinations. While implementing this study, the NetworkX [46] library was used to convert API calls and DLL information into a graph structure. While the graph nodes represent API calls and DLLs, the edges represent the relationships between these elements (e.g., call order or dependencies). We applied the Node2Vec and GraphSAGE methods to create embeddings of the nodes. For this purpose, we used node2vec and stellargraph [47] (Python 3.6 version was preferred for StellarGraph library support in the study) libraries. To further improve the obtained embeddings, we used the GAT model for contextual enrichment with attention mechanisms; the PyTorch Geometric [48] library was preferred for GAT. Finally, we designed a custom CNN-GRU hybrid model to classify the improved embeddings; we used TensorFlow/Keras libraries at this stage. During the whole process, we also used Pandas, NumPy, and Scikit-learn libraries extensively for data processing and analysis. Training and testing operations were performed in the Google Colab environment using L4 GPU, 53.0 GB system RAM, 22.5 GB GPU RAM, and 235.7 GB disk capacity. This integrated approach combined both graph-based features and the power of DL models to detect malware with high accuracy rates.

Node2Vec + GAT + CNN-GRU-3 model outperforms other Node2Vec and GraphSAGE-based models and models that use GAT as an independent classifier or use GAT for contextual enhancement, with an accuracy value of 0.9961. This result shows that the proposed model benefits from the random walk method of Node2Vec to provide a strong initial embedding, the attention mechanism of GAT to enhance contextual relationships, and the capabilities CNN-GRU-3 to model sequential data.

Table 5. Comparison of classification performance of Node2Vec and GraphSAGE embeddings with CNN-GRU-3 and ensemble learning.

Model	Accuracy	F-Measure	Precision	Recall	MAE	Cohen’s Kappa
Node2Vec + CNN-GRU-3	0.9675	0.9650	0.9660	0.9675	0.0325	0.94
Node2Vec + Ensemble L.	0.9603	0.9590	0.9585	0.9603	0.0397	0.92
GraphSAGE + CNN-GRU-3	0.9442	0.9430	0.9425	0.9442	0.0558	0.89
GraphSAGE + Ensemble L.	0.9563	0.9550	0.9545	0.9563	0.0437	0.91

presents the performance results of Node2Vec and GraphSAGE embeddings with CNN-GRU-3 and ensemble learning.

When Node2Vec and GraphSAGE node embedding methods were classified with custom CNN-GRU-3 and Ensemble Learning classifiers, respectively, the best result was obtained from the Node2Vec + CNN-GRU-3 hybrid model. The Node2Vec + CNN-GRU-3 model performed better with 0.9675 accuracy, indicating that the random walk approach of Node2Vec is more effective when combined with the powerful sequential and local information extraction capabilities of CNN-GRU-3.

Table 6. Performance comparison of using GAT as a standalone classifier and combining it with Node2Vec/GraphSAGE.

Model	Accuracy	F-Measure	Precision	Recall	MAE	Cohen’s Kappa
GAT	0.9388	0.9360	0.9375	0.9603	0.0612	0.87
Node2Vec + GAT	0.9711	0.9700	0.9715	0.9711	0.0289	0.94
GraphSAGE + GAT	0.9703	0.9685	0.9690	0.9703	0.0297	0.93

shows the performance results obtained using the GAT classifier and Node2Vec/GraphSAGE embeddings. The GAT model was evaluated both as a standalone classifier and combined with Node2Vec and GraphSAGE embeddings. According to the results, the Node2Vec + GAT model gave the best result with 0.9711 accuracy. When the results were examined, it was seen that the attention mechanism of GAT could process the contextual information more effectively when combined with Node2Vec embeddings. After improving the node embeddings of Node2Vec and GraphSAGE with GAT, they were classified with CNN-GRU-3 and ensemble learning classifiers. The results are shown in

Table 7. Classification performances of Node2Vec and GraphSAGE embeddings with CNN-GRU-3 and ensemble learning by improving with GAT.

Model	Accuracy	F-Measure	Precision	Recall	MAE	Cohen’s Kappa
Node2Vec + GAT + CNN-GRU-3	0.9961	0.9955	0.9960	0.9961	0.0039	0.99
Node2Vec + GAT + Ensemble L.	0.9871	0.9860	0.9875	0.9871	0.0129	0.97
GraphSAGE + GAT + CNN-GRU-3	0.9898	0.9890	0.9885	0.9898	0.0102	0.98
GraphSAGE + GAT + Ensemble L.	0.9873	0.9865	0.9860	0.9873	0.0127	0.97

. GAT was used as a contextual enhancer, and the resulting embeddings were given to CNN-GRU-3 and ensemble learning classifiers. Node2Vec + GAT + CNN-GRU-3 (proposed model) gave the best result among all groups with 0.9961 accuracy. In

Table 8. Using GAT as a feature extractor and comparing its classification performance with CNN-GRU-3/ensemble learning models.

Model	Accuracy	F-Measure	Precision	Recall	MAE	Cohen’s Kappa
GAT + CNN-GRU-3	0.9713	0.9700	0.9705	0.9713	0.0287	0.94
GAT + Ensemble L.	0.9677	0.9670	0.9665	0.9677	0.0323	0.93

, GAT embeddings are classified with CNN-GRU-3 and ensemble learning. The GAT + CNN-GRU-3 model showed better performance than ensemble learning with 0.9713 accuracy. This situation revealed that CNN-GRU-3 is more effective compared to ensemble learning.

The Node2Vec + CNN-GRU-3 model proved to be an effective initial embedding technique with an accuracy value of 0.9675 for Node2Vec, one of the components of the proposed model. However, when not combined with GAT and CNN-GRU-3, this model could not adequately handle contextual and sequential relationships. The Node2Vec + GAT model showed that GAT’s attention mechanism enriched the embeddings contextually with an accuracy of 0.9711. However, it performed less than the proposed model because it did not understand the sequential information with CNN-GRU-3. The GraphSAGE + GAT + CNN-GRU-3 model provided high performance with an accuracy of 0.9898 even when GraphSAGE was used instead of Node2Vec. This configuration was not as successful as the Node2Vec + GAT + CNN-GRU-3 configuration because Node2Vec can better understand contextual information with the random walk method. As seen in Figure 5, the models’ performance comparison reveals the proposed framework’s superiority. The performance comparison of the models, as shown in Figure 5, highlights the superiority of the proposed framework. The confusion matrix of the three models that achieved the highest accuracy is given in Figure 6.

The proposed Node2Vec + GAT + CNN-GRU-3 model achieved a highly balanced classification, with only 3 false positives and 3 false negatives out of 1600 test samples. In contrast, the GraphSAGE + GAT + CNN-GRU-3 configuration resulted in slightly more errors (8 false negatives and 9 false positives), while the GraphSAGE + GAT + Ensemble Learning model showed the highest number of misclassifications (10 false negatives and 11 false positives). The proposed Node2Vec + GAT + CNN-GRU-3 model achieved the highest accuracy (0.9961) in all experiments. This proves the effectiveness of the model’s integration of graph-based contextual information and DL. Recall and F-measure values were higher in the experiments where GAT was used as an embedding improvement method. This is because GAT optimizes contextual relationships and reveals meaningful relationships between nodes. The MAE value of the proposed model (0.02%) is lower than other models. This shows that the model’s error rate in malware detection is relatively low. The Node2Vec + GAT + CNN-GRU-3 model obtained a high value of 0.98 with Cohen’s Kappa. This supports the classification stability and overall accuracy of the model. Considering Experiment 1, it was observed that the accuracy and recall rates were lower in these experiments where Node2Vec and GraphSAGE embeddings were used without improvement with GAT. In particular, the generalization success of the GraphSAGE + Ensemble model was limited. When Experiment 2 was evaluated, contextual information extraction increased in these experiments, where GAT was used as a classifier. However, since the initial embeddings were not directly improved, the accuracy rates were lower compared to the embedding improvement method. When Experiments 3 and 4 were evaluated, the model’s accuracy and generalization success increased significantly in the experiments where GAT was used as the embedding improvement method. This result shows the effect of the contextual enrichment provided by the attention mechanism of GAT on Node2Vec and GraphSAGE embeddings. The graphs in Figure 7 show the performances of Node2Vec + GAT + CNN-GRU-3, GraphSAGE + CNN-GRU-3, Node2Vec + CNN-GRU-3, and GraphSAGE + GAT + CNN-GRU-3 models during the training and validation processes.

The performance evaluation of the models shows that the proposed Node2Vec + GAT + CNN-GRU-3 model outperforms other approaches in terms of accuracy and stability. Achieving an accuracy of 0.9961, this model benefits from the effective use of Node2Vec embeddings, which better understand structural relationships, and GAT’s attention mechanism, which improves the contextual relevance of these embeddings. When combined with the sequential modeling capabilities of the CNN-GRU-3 classifier, the proposed framework achieves a better generalization ability compared to alternatives such as GraphSAGE + GAT + CNN-GRU-3 and Node2Vec + CNN-GRU-3, which show slightly lower accuracy and more fluctuating validation loss. When we examine different GAT scenarios, the scenarios where GAT is used as an embedding enhancement method have shown superior results, especially in performance metrics such as accuracy, precision, recall, and MAE, compared to those used as a classifier. The main reason for this difference is that the initial embeddings created with Node2Vec and GraphSAGE are contextually enriched with attention coefficients by GAT. This improvement of the embeddings has increased the generalization ability of the classification models (CNN-GRU-3, ensemble learning) by making the complex relationships in API calls and DLL information more meaningful. In cases where GAT is used directly as a classifier, the contextual information is only processed during the classification. The initial embeddings are not improved, so MAE is higher, and recall is relatively low. The results show that GAT offers a more effective approach to malware detection when used as an embedding enhancement method.

5. Limitations and Discussion

This section outlines the key limitations of the proposed framework and discusses its practical implications, scalability, and areas for future improvement. To assess generalizability, the performance of the proposed framework is evaluated on three separate datasets: MalBehavD-V1, APIMDS, and the custom-built PEMalware dataset. These datasets differ significantly in sample distribution, API behavior patterns, and dataset scale. As shown in

Table 9. Comparison of detection results across different datasets and models.

Dataset	Malicious	Benign	Model	Detection Results (%)
MalBehavD-V1 [13]	1285	1285	API-MalDetect [13]	Accuracy: 98.0
			DawnGNN [2]	Accuracy: 96.38
			Proposed Framework	Accuracy: 97.05
APIMDS [49]	23,080	300	MINES [2]	Micro-F1: 97.03, Macro-F1: 93.70
			DawnGNN [2]	Accuracy: 99.75
			Proposed Framework	Accuracy: 95.41
PEMalware (Ours)	1000	1000	Proposed Framework	Accuracy: 99.61

, our framework achieved strong results on MalBehavD-V1 (97.05%) and PEMalware (99.61%) despite the high-class imbalance (23,080 malicious vs. 300 benign samples) and large-scale and showed competitive performance on and APIMDS (95.41%). The performance on APIMDS was slightly lower compared to other datasets. This can be attributed to several factors: (i) severe class imbalance in APIMDS (23,080 malicious vs. only 300 benign examples), complicating decision boundaries; (ii) the scale of the dataset, which may require deeper or more scalable architectures; and (iii) behavioral inhomogeneity of benign examples, which may lead to overfitting or limited generalization. All models were trained independently on each dataset using the same protocol: standard 80/20 train-test split and 5-fold cross-validation to ensure consistency and robustness. This approach allows for fair comparisons across datasets. Training was performed on a single NVIDIA L4 GPU for all experiments. Training durations were similar except for the APIMDS dataset, which took approximately 60 min longer due to the more significant number of examples. This increase is expected and acceptable in the context of real-world scalability. While Table 9 confirms that the proposed framework achieves high accuracy, evaluating these results beyond accuracy alone is important. Compared to state-of-the-art methods such as DawnGNN and MINES, our framework demonstrates comparable or slightly superior performance but has significant architectural differences. For example, DawnGNN uses pre-trained GNN layers and hand-crafted semantic features, which may reduce adaptability to unseen environments. In contrast, our method performs end-to-end feature learning via graph-based embeddings and a hybrid CNN-GRU design and offers greater flexibility across various API-DLL patterns. Despite including computationally intensive modules such as GAT and GRU, the framework remains efficient due to mini-batch training, gradient checkpointing, and early stopping optimizations. While our framework achieves high accuracy on all datasets, we acknowledge that real-world malware detection scenarios are inherently dynamic and more complex. To mitigate dataset-specific limitations, we designed the graph generation and feature extraction pipeline to be dataset-independent by relying on structural and contextual relationships between API calls and libraries rather than static properties or hard-coded signatures.

The proposed framework integrates graph-based embedding (Node2Vec), attention mechanisms (GAT), and sequential DL (CNN-GRU-3), which inevitably leads to a higher computational cost compared to traditional ML models. This increase is particularly notable in components such as GAT, due to multi-head attention over large node graphs, and GRU, due to sequential processing. Regarding computational overhead, traditional detection methods are highly efficient because they are rule-based, requiring minimal memory and processing power. ML models introduce moderate overhead, especially during training, while DL approaches incur the highest computational cost due to their layered architecture, high parameter counts, and GPU dependency. However, the overhead is often justified by the superior performance and generalization capabilities of DL-based frameworks. As demonstrated in Section 4.7, the model achieves 99.61% accuracy and extremely low error rates, significantly outperforming conventional baselines.

Like most DL-based malware detection systems, the proposed framework may face challenges when encountering novel or evolving malware that exhibits behavior patterns not present in the training data. Although the integration of graph-based and sequential learning components (Node2Vec, GAT, and CNN-GRU) enhances the model’s ability to generalize, it remains inherently limited by the scope and diversity of the training set. To address this, periodic retraining with newly collected and labeled malware samples can improve the framework’s adaptability. Additionally, more advanced solutions such as automated model updating via AutoML techniques may enhance responsiveness to evolving threats [20]. However, such techniques could not be used in this study due to their high computational costs and intensive resource requirements, which are beyond the scope of the current study. It is planned to focus on this issue in the future work.

6. Conclusions

This study proposes a new framework for malware detection by combining API calls and DLL information. The proposed Node2Vec + GAT + CNN-GRU-3 model effectively uses structural and contextual information by combining graph-based embedding techniques and DL methods. This model has outperformed many approaches in the literature by demonstrating a high performance of 0.9961 in understanding malware behaviors and classification accuracy. The experiments have shown that improving embedding techniques such as Node2Vec and GraphSAGE with GAT makes graph-based features richer and contextually meaningful. The effective use of GAT as both a classifier and feature extractor has contributed significantly to the success of the study. In addition, comparing multiple models (CNN-GRU-3, ensemble learning) during the study process allowed for an in-depth analysis of the effects of various approaches. In conclusion, this study has provided a new solution for malware detection with API calls and DLL-based heterogeneous graph representations and has become an important reference for future studies. In the future, it is suggested that the model be further developed by integrating multiple data sources for various malware types.