SV-GEN: Synergizing LLM-Empowered Variable Semantics and Graph Transformers for Vulnerability Detection

Abstract

Deep-learning-based vulnerability detection has made substantial progress, but two limitations remain prominent. Sequence-based methods linearize source code and thus weaken the explicit modeling of control-flow and data-flow dependencies. Graph-based methods preserve program structure, yet conventional graph neural networks still have difficulty capturing long-range interactions in large code property graphs (CPGs). In addition, standard CPGs usually lack explicit variable semantics and security-critical node roles, which limits their ability to represent vulnerability-relevant program behavior. To address these issues, we propose SV-GEN, a vulnerability detection framework that combines large-language-model-driven semantic enhancement with hybrid sequence-graph learning. The novelty of SV-GEN lies in introducing a semantically enriched code property graph, termed Sem-CPG, which augments conventional CPGs with variable semantic roles and security-oriented node labels, and in coupling this representation with an adaptive fusion mechanism over structural and sequential views. Specifically, we use a large language model as an external semantic annotator to assign variable roles and identify source, sink, and sanitizer nodes, and then encode the resulting Sem-CPG with a Graph Transformer while modeling the code sequence with GraphCodeBERT. A learnable gating module is further used to adaptively fuse the graph-level and sequence-level representations for final prediction. Experiments on Devign, ReVeal, and DiverseVul show that SV-GEN achieves competitive or superior overall performance across benchmarks, with particularly strong improvements on the large and highly imbalanced DiverseVul dataset.

1. Introduction

Software systems now support nearly every critical domain, from financial transactions and healthcare to industrial control and public infrastructure. As a result, software security directly affects economic stability and public safety [1]. Vulnerabilities continue to accumulate at a high rate, and public security reports show that the annual number of disclosed software vulnerabilities keeps increasing [2]. In 2023 alone, more than 25,000 vulnerabilities were reported. Once exploited, such vulnerabilities may lead to sensitive data disclosure, service disruption, and severe financial loss. The WannaCry ransomware outbreak in 2017, for example, exploited a vulnerability in the Windows SMB protocol, infected more than 200,000 computers worldwide, and caused losses estimated at billions of dollars. Early vulnerability detection during software development therefore remains a fundamental requirement for reliable and secure systems.

Traditional vulnerability detection mainly relies on static analysis, dynamic analysis, and manual expert auditing [3]. Static analysis detects suspicious patterns through rule matching and data-flow reasoning, but it often produces many false positives. Dynamic analysis executes programs and can detect runtime flaws, yet it is limited by code coverage and computational cost. Manual auditing can be accurate, but it does not scale to modern codebases. With the rise in deep learning, vulnerability detection has entered a new stage. Deep models can learn complex vulnerability patterns directly from labeled data without manually designed rules, and they often show strong performance in both efficiency and generalization [4].

Existing deep learning methods for vulnerability detection can be broadly divided into sequence-based methods and graph-based methods. Sequence-based methods treat source code as text, tokenize it, and use recurrent networks [5], LSTMs [6], or Transformers to learn semantic features from the resulting token sequence [7]. This representation, however, destroys the original program structure and makes it difficult to model control-flow and data-flow relations explicitly. Graph-based methods instead transform programs into structured graphs such as abstract syntax trees, control-flow graphs, data-flow graphs, or code property graphs (CPGs) [8], and then use graph neural networks to learn structural features. A representative example is Devign [9], which merges abstract syntax tree (AST) [10], control-flow graph (CFG), and data-flow graph (DFG) into a composite program graph and uses a gated graph neural network (GGNN) to capture vulnerability-related substructures.

Despite recent progress, existing deep-learning-based vulnerability detection methods still suffer from three limitations: insufficient modeling of dispersed control/data dependencies in sequence-based encoders, limited long-range propagation in message-passing graph models, and the lack of explicit variable/security semantics in conventional CPGs. Sequence-based methods still struggle to represent complex control propagation and data propagation patterns. Real vulnerabilities often depend on variable assignments across multiple branches, or on complete taint propagation paths from external input to dangerous operations. Once code is linearized, these nonlinear execution semantics become hard to express. When vulnerability conditions are scattered across distant code regions, the model must rely on attention alone to connect them, which is often unreliable in practice. Graph-based methods preserve program structure, but traditional graph neural networks propagate information through local message passing, and their effective receptive field is limited by network depth. Deeper networks also suffer from over-smoothing, which weakens informative local differences [11,12]. In realistic software projects, function sizes vary widely, and the resulting code property graphs are often large and sparse. Standard graph neural networks therefore struggle to propagate structural semantics across many hops, especially in vulnerability scenarios that depend on long-distance interactions. At the same time, conventional code property graphs do not model variable semantics and security-critical nodes sufficiently well. Variable nodes are usually reduced to generic placeholders such as VAR1 and VAR2, which removes their functional roles, while security-critical nodes such as taint sources, sinks, and sanitizers are not explicitly labeled. As a result, the model cannot focus on program elements that are directly involved in vulnerability triggering.

To address these limitations, we propose SV-GEN, short for Semantic-aware Variable-guided Graph Evolution Network, a vulnerability detection framework centered on representation enhancement rather than direct LLM-based end-to-end prediction. The key idea is to use a large language model (LLM) as a semantic annotator for program representation construction, instead of using it as the final detector. Starting from a conventional code property graph, we introduce a Semantically Enriched Code Property Graph (Sem-CPG) that explicitly augments the graph with two forms of vulnerability-relevant semantics: (1) variable-level semantic roles that characterize the functional role of variables in program logic, and (2) security-critical node roles that identify potential sources, sinks, and sanitizers. In this way, the resulting graph representation preserves the structural completeness of conventional CPGs while exposing higher-level semantic priors that are directly relevant to vulnerability formation.

On top of this enriched representation, we design a hybrid detection backend that combines a pre-trained code language model (CodeLM) with a Graph Transformer. The sequential encoder captures token-level and contextual code semantics, while the Graph Transformer models long-range structural dependencies over Sem-CPG through global attention. A learnable gating module then adaptively fuses the two views, allowing the model to balance sequential semantics and graph structure according to the characteristics of the input sample. Therefore, the novelty of SV-GEN does not lie in the isolated use of LLM annotation, Graph Transformers, or multimodal fusion as standalone techniques, but in their task-oriented integration around a vulnerability-oriented semantic graph representation.

Accordingly, the main contributions of this paper are as follows:

• We propose Sem-CPG, a vulnerability-oriented semantic program representation that enriches conventional code property graphs with explicit variable role semantics and security-critical node annotations. Unlike standard CPGs that mainly preserve structural relations, Sem-CPG injects semantic priors that help the model focus on program elements and propagation patterns closely related to vulnerability triggering.
• We develop SV-GEN, a hybrid sequence–graph vulnerability detection framework tailored to Sem-CPG. By combining a pre-trained code language model, a Graph Transformer, and an adaptive gating mechanism, the framework jointly exploits sequential code semantics and long-range structural dependencies in semantically enriched program graphs.
• We conduct a systematic empirical study on three public vulnerability detection benchmarks to verify the effectiveness of the proposed representation and framework. The results, together with ablation analysis, show that the gains of SV-GEN are consistently associated with semantic graph enrichment and hybrid sequence–graph modeling.

2. Related Work

2.1. Deep Learning with Sequential Representations

Source code is expressed in a language-like form, which makes it natural to borrow methods from natural language processing. In vulnerability detection, code is first divided into fragments and then converted into token sequences that are fed into deep models.

Li et al. first proposed VulDeePecker [13], which observes that vulnerabilities in C programs often occur around library calls. It therefore extracts library-call-related code fragments, names them Code Gadgets, and maps them into token sequences with Word2Vec. Although this design incorporates domain knowledge at preprocessing time, focusing only on library calls is far from sufficient for complex real-world projects. To obtain code slices that align better with vulnerability characteristics, Li et al. [14] later proposed SySeVR, which introduces syntax-based vulnerability candidates and semantics-based vulnerability candidates. Inspired by image recognition, SySeVR partitions programs into candidate fragments according to syntax and semantics and then converts them into token sequences with Word2Vec. To further improve the representation, Li et al. [15] proposed VulDeeLocator, which uses compiler techniques to optimize program representation. It first extracts syntax-based vulnerability candidates from source code, converts them into SSA-form intermediate representations with LLVM, derives semantics-based candidates from the intermediate form, and finally maps them into vectors. Beyond syntax and semantics, control information can also be exploited. Zou et al. proposed $\mu$VulDeePecker [16], which first transforms source code into a system dependence graph and then generates Code Gadgets based on the call relations of library functions in that graph.

After the introduction of the Transformer architecture, attention-based models quickly became dominant in natural language processing. Benefiting from the expressive power of Transformers, Fu et al. proposed LineVul [17], which models functions as token sequences and converts them into subword tokens through byte pair encoding [18]. Byte pair encoding improves vocabulary coverage and compression by repeatedly merging frequent byte pairs. The LineVul study shows that this tokenization strategy is one of the most important factors behind its performance. To push slicing granularity beyond line-level candidates, Wu et al. proposed UltraVCS [19]. Instead of taking an entire line as the basic slicing unit, UltraVCS identifies insecure key variables by jointly considering key operations and insecure operations such as sensitive API calls, variable assignments, and pointer offsets. However, the method still depends heavily on manually designed slicing rules and PDG-based dependency analysis, which means that its vulnerability representation is largely constrained by predefined heuristics rather than explicit learned semantic understanding.

2.2. Deep Learning with Graph Representations

In programming language research, graph-based program representations explicitly model syntax, control dependencies, and data dependencies. Abstract syntax trees, control-flow graphs, and data-flow graphs form the basis of structural code analysis. When combined into a code property graph, they provide a unified structural view for downstream vulnerability detection. Compared with sequence-based models, graph representations retain control constraints, variable propagation paths, and cross-statement semantic relations, and they have therefore become a major research direction in deep-learning-based vulnerability detection.

An early representative work is Devign [9], which merges AST, CFG, and DFG into a composite program graph and uses graph neural networks to capture key substructures that trigger vulnerabilities. ReGVD [20] revisits graph neural architectures systematically, points out that existing code property graph models are easily distracted by noisy nodes, and proposes more stable aggregation operators with multiscale skip connections. IVDetect [21] combines graph representations with interpretable views so that the model can explain its predictions from control and data perspectives. More recently, ANGEL [22] targets the challenge of vulnerability detection on large code graphs. It introduces a hierarchical graph refinement strategy to filter redundant nodes and reduce graph size, and then combines GNNs with a Graph Transformer for context-aware graph encoding. In this way, ANGEL captures both local structural information and long-range dependencies, making it a strong recent graph-based baseline. Liu et al. further proposed Vul-LMGNNs [23], which combines pre-trained code language models with gated graph neural networks on CPGs. To alleviate the locality of standard message passing, it introduces online knowledge distillation between parallel student graph neural networks (GNNs) and fuses sequence and graph views through an implicit–explicit joint training scheme. Although this design improves semantic–structural fusion, it still builds on conventional CPG semantics and GNN-style propagation, leaving explicit vulnerability-oriented node semantics insufficiently modeled.

2.3. Large Language Models for Vulnerability Detection

Large language models have recently achieved major progress in natural language processing, and their strong contextual understanding and reasoning capabilities have attracted growing interest in software vulnerability detection. Compared with traditional deep learning models, large language models provide stronger semantic understanding and better zero-shot or few-shot generalization, which opens a new direction for vulnerability analysis [24].

With the emergence of general-purpose models such as GPT [25], LLaMA [26], and Claude [27], researchers have begun to explore whether large language models can detect vulnerabilities directly. Existing work mainly follows three directions. The first direction uses pre-trained or fine-tuned large language models directly for vulnerability detection. These methods usually cast vulnerability detection as classification or sequence labeling and rely on prompting, few-shot learning, or fine-tuning so that the model can understand program semantics and predict vulnerable locations or types. Empirical studies show that large language models can achieve reasonable function-level performance and may detect security patterns that escape traditional static analysis tools, but they still face limitations in detection granularity, representation robustness, and the timeliness of vulnerability knowledge [24]. The second direction focuses on specialized tuning. SecureFalcon [28], for instance, fine-tunes Falcon for security tasks and improves recognition of common vulnerability categories such as Common Weakness Enumeration (CWE) classes. Prompt engineering has also been explored. GPTScan [29] uses carefully designed prompts to guide ChatGPT-like models toward zero-shot or few-shot vulnerability identification and reports promising results in selected scenarios. The third direction concerns system design. Many studies adopt retrieval-augmented generation to supply the model with relevant external context, while some recent systems employ multiple agents that cooperate during vulnerability analysis and iterative refinement [24].

3. Motivating Example

This section analyzes a real vulnerability from the open-source multimedia framework FFmpeg, namely CVE-2023-47470, to illustrate the core challenges faced by existing deep-learning-based vulnerability detection methods and to motivate the design of our approach.

Figure 1 shows the vulnerable code fragment from the function ref_pic_list_struct. It is a typical out-of-bounds write vulnerability. The root cause is the absence of proper boundary checking for external input. The variable ref_pic_num is read directly from an external bitstream through get_ue_golomb_long(gb), so it is untrusted input. The target array rpl->ref_pics[] is declared as a statically sized array inside the header file structure. Before the code uses ref_pic_num to control repeated writes to rpl->ref_pics[i], it never checks whether the value exceeds the array capacity. An attacker can therefore craft an input whose value is larger than the array bound, causing the loop to write beyond the end of rpl->ref_pics[], corrupt heap memory, and potentially trigger arbitrary code execution or a crash.

Figure 1. A vulnerability example from FFmpeg, CVE-2023-47470. The orange and red regions denote the taint source and taint sink.

From a semantic perspective, the triggering path of this vulnerability has a clear long-range nature. The source, where ref_pic_num receives user-controlled data, and the sink, where ref_pics[i] is written out of bounds, are far apart in the code graph. The path crosses multiple loops and branches. More importantly, it is mixed with many nodes that are irrelevant to the vulnerability itself, such as bit-shift operations on strp_entry_sign_flag and sign-dependent conditions on delta_poc_st. These irrelevant nodes introduce heavy semantic noise into the graph and can easily distract the model from the true vulnerability propagation path.

This observation motivates two design choices. First, variable roles should be identified explicitly so that the code language model can understand how different variables contribute to vulnerability triggering. Second, source and sink information should be labeled explicitly in the program graph so that the graph encoder receives clear security priors and can focus on nodes and paths that are directly relevant to vulnerability formation.

4. Methodology

This section presents the SV-GEN framework in detail, including the task formulation, the construction of Sem-CPG, and the hybrid backend for feature aggregation and prediction.

4.1. Overview

SV-GEN is a vulnerability detection framework that improves performance and generalization by constructing a semantically enriched program graph and combining a pre-trained code language model with a Graph Transformer. As shown in Figure 2, the pipeline contains four stages: input preprocessing, LLM-driven Sem-CPG construction, hybrid feature aggregation, and final classification. We first formalize the task and then describe each stage.

Figure 2. Overview of SV-GEN. The framework contains an LLM-driven semantic enhancement stage for CPG construction and a hybrid sequence-graph backbone for vulnerability detection.

4.1.1. Problem Formulation

We study function-level vulnerability detection. Given a function, the model predicts whether the function contains a security vulnerability. The dataset is defined as $\mathcal{D}={\left\{(c_i,y_i)\right\}}_{i=1}^n$, where $c_i\in \mathcal{C}$ is a code sample and $y_i\in \mathcal{Y}=\{0,1\}$ is its label, with 1 for vulnerable and 0 for non-vulnerable samples.

To exploit structured program semantics, we cast vulnerability detection as a graph classification problem. For each code sample $c_i$, we construct a Sem-CPG $G_i=(V_i,E_i,X_i)\in \mathbb{G}$, where $V_i=\{v_{i,1},\dots ,v_{i,m_i}\}$ is the node set and $m_i=\left|V_i\right|$. The node feature matrix is $X_i\in {\mathbb{R}}^{m_i\times d}$, and node $v_{i,j}\in V_i$ is represented by a feature vector $x_{i,j}\in {\mathbb{R}}^d$. The edge set is $E_i=\{e_{i,1},\dots ,e_{i,k_i}\}$, where each edge $e_{i,r}=(v_{i,s},v_{i,t},{\tau }_{i,r})$ denotes a directed dependency of type ${\tau }_{i,r}\in \mathcal{T}$ from source node $v_{i,s}$ to target node $v_{i,t}$. The learning objective is to learn a classifier $f:\mathbb{G}\to \mathcal{Y}$ that predicts the vulnerability label ${\widehat{y}}_i=f\left(G_i\right)$.

4.1.2. SV-GEN Workflow

As illustrated in Figure 2, SV-GEN follows a four-stage pipeline. The input function first undergoes preprocessing, including comment removal, identifier normalization, and static-analysis-based simplification. We then generate an initial code property graph with Joern and use a large language model to annotate variable semantic roles and identify security-critical nodes such as sources, sinks, and sanitizers, which yields Sem-CPG. Next, a Graph Transformer encodes Sem-CPG while a code language model encodes the natural code sequence, and a gating module fuses the two views into a unified representation. Finally, a classifier outputs the vulnerability probability.

4.2. Construction of the Semantically Enriched Program Graph

Stage 2 in Figure 2 is implemented through four substeps: (1) preprocessing and variable-level semantic labeling (Section 4.2.1), (2) initial CPG construction and Code Natural Sequence (CNS) edge insertion (Section 4.2.2), (3) security-critical node identification (Section 4.2.3), and (4) semantic feature embedding to obtain Sem-CPG node representations (Section 4.2.4). Algorithm 1 summarizes the full construction procedure.

Conventional CPGs unify syntax, control flow, and data dependencies, but they still lack the semantic information needed for vulnerability detection. Variable nodes are often represented only by generic identifiers such as VAR1 and VAR2, which removes their roles in program logic. Security-critical nodes such as taint sources, sinks, and sanitizers are also unlabeled, so the model cannot easily focus on elements that directly trigger vulnerabilities. To address this issue, we introduce the Semantically Enriched Code Property Graph, abbreviated as Sem-CPG. It uses the semantic reasoning ability of large language models to annotate variable roles and security-critical nodes and therefore provides a richer vulnerability-oriented program representation. As shown in Figure 3, after converting a certain data point in the DiverseVul dataset into Sem-CPG, its variable name is modified to include the name of the semantic role, and semantic labels are added to the graph structure.

Algorithm 1 Construction of the Semantically Enriched Program Graph

Require: Source code function $c_i$, large language model ${\mathcal{M}}_{LLM}$, pre-trained code language model ${\mathcal{M}}_{CodeLM}$, and the static analysis tool Joern Ensure: Semantically enriched program graph $G_i^{sem}=(V_i,E_i,{\mathcal{A}}_{sem,i},{\mathcal{A}}_{sec,i},{\Phi }_i)$  1: $c_{pre}\leftarrow \mathrm{PREPROCESS}\left(c_i\right)$  2: ${\mathcal{P}}_{i,var}\leftarrow \mathrm{BUILDVARPROMPT}\left(c_{pre}\right)$  3: ${\mathcal{A}}_{sem,i}\leftarrow {\mathcal{M}}_{LLM}\left({\mathcal{P}}_{i,var}\right)$  4: $c_{labeled}\leftarrow \mathrm{APPLYVARLABELS}(c_{pre},{\mathcal{A}}_{sem,i})$  5: $(V_i,E_{i,ast},E_{i,cfg},E_{i,dfg},E_{i,cdg})\leftarrow \mathrm{JOERN}\left(c_{labeled}\right)$  6: $E_{i,cns}\leftarrow \mathrm{BUILDCNSEDGES}(V_i,c_{labeled})$  7: $E_i\leftarrow E_{i,ast}\cup E_{i,cfg}\cup E_{i,dfg}\cup E_{i,cdg}\cup E_{i,cns}$  8: ${\mathcal{B}}_i\leftarrow \{v\in V_i\mid \mathrm{TYPE}\left(v\right)=\mathrm{BasicBlock}\}$  9: ${\mathcal{A}}_{sec,i}\leftarrow \mathfrak{\varnothing }$ 10: for each $b\in {\mathcal{B}}_i$ do 11:      ${\mathcal{P}}_{i,taint}\leftarrow \mathrm{BUILDTAINTPROMPT}\left(b\right)$ 12:      $(role,conf)\leftarrow {\mathcal{M}}_{LLM}\left({\mathcal{P}}_{i,taint}\right)$ 13:      if $role\in \{\mathrm{Source},\mathrm{Sin}\mathrm{k},\mathrm{Sanitizer}\}$ then 14:            ${\mathcal{A}}_{sec,i}\left(b\right)\leftarrow (role,conf)$ 15:      end if 16: end for 17: for each $v\in V_i$ do 18:      ${\mathbf{x}}_{code}\leftarrow {\mathcal{M}}_{CodeLM}\left(\mathrm{CODE}\left(v\right)\right)$ 19:      ${\mathbf{x}}_{sem}\leftarrow \mathrm{LABELENCODE}\left({\mathcal{A}}_{sem,i}\left(v\right)\right)$ 20:      if $v\in \mathrm{dom}\left({\mathcal{A}}_{sec,i}\right)$ then 21:            ${\mathbf{x}}_{sec}\leftarrow \mathrm{ENCODE}\left({\mathcal{A}}_{sec,i}\left(v\right)\right)$ 22:      else 23:            ${\mathbf{x}}_{sec}\leftarrow \mathbf{0}$ 24:      end if 25:      ${\Phi }_i\left(v\right)\leftarrow W_{proj}·[{\mathbf{x}}_{code};{\mathbf{x}}_{sem};{\mathbf{x}}_{sec}]+{\mathbf{b}}_{proj}$ 26: end for 27: return $G_i^{sem}=(V_i,E_i,{\mathcal{A}}_{sem,i},{\mathcal{A}}_{sec,i},{\Phi }_i)$

Figure 3. An example of Sem-CPG construction.

4.2.1. Preprocessing and Variable-Level Semantic Role Annotation

In many preprocessing pipelines, variable names and function names are normalized into placeholders such as VAR1 and FUNC1. This design is intended to reduce project-specific lexical bias so that the model can focus on structural vulnerability patterns. However, aggressive normalization also removes meaningful semantic cues from code tokens and weakens the semantic completeness of the representation.

We therefore introduce large-language-model-based variable semantic annotation. The idea is to replace raw variable names with semantically meaningful role names so that the program keeps its original structure while recovering the functional meaning of variables in context. The process contains two stages. We first simplify the source code through static analysis. We then ask the large language model to infer semantic roles for variables.

In the preprocessing stage, we preserve program information that is relevant to behavioral logic as much as possible. We simplify the source code through constant folding, dead code elimination, and related techniques so that analyzable and informative statements are concentrated into a cleaner representation. This step reduces redundant structure while preserving key computations and data-flow information, which gives the language model a clearer input for semantic analysis.

To keep the labeling process stable and interpretable, we use a fixed vulnerability-oriented vocabulary of variable roles, including categories such as user_input, buffer_ptr, buffer_len, size_counter, loop_bound, return_value, status_flag, and temp_variable. These labels are used only to describe the functional role of variables in context and do not change the original program logic.

Once preprocessing is complete, the resulting code fragment is fed into a large language model for semantic role annotation. As shown in Figure 4, the prompt template guides the model to infer the role of each variable from its surrounding context. The model then rewrites variable identifiers with role-aware names while preserving the original control flow, data flow, and statement structure. After this step, variable nodes are no longer meaningless placeholders but identifiers that convey program behavior, which enriches the sequential representation used in later stages.

Figure 4. Prompt template for LLM-based variable semantic labeling.

The complete semantic labeling pipeline therefore starts from a raw code fragment, simplifies it through preprocessing, and then invokes the large language model through the prompt template to assign variable semantic roles while preserving the original structural relations.

4.2.2. Program Graph Construction

We use the open-source static analysis tool Joern [30] to parse source code and generate an initial code property graph. The code property graph integrates several program representations within a unified graph, including the abstract syntax tree, control-flow graph, data-flow graph, and control dependence graph. It therefore describes program behavior from multiple perspectives, namely syntax, control dependencies, and data dependencies. The resulting graph contains multiple types of nodes, such as function calls, variables, and constants, as well as heterogeneous edges, such as AST edges, CFG edges, and DFG edges. This structured graph provides the foundation for later vulnerability-oriented semantic enhancement.

On top of the initial code property graph, we add Code Natural Sequence edges according to the natural writing order of the source code. These edges encode the linear execution order of statements and complement the non-structural semantics that pure dependency graphs miss. The final graph therefore contains both structured dependency semantics, carried by AST, CFG, DFG, and control dependence graph (CDG) edges, and linear sequence semantics, carried by CNS edges. A function $c_i$ is thus represented as a heterogeneous graph $G_i=(V_i,E_i)$ in which all edge types share the same node set $V_i$, and multiple semantic views are expressed jointly through heterogeneous relations.

4.2.3. LLM-Based Identification of Security-Critical Nodes

In most existing deep-learning-based vulnerability detection methods, the program graph is directly fed into a graph neural network for classification. These methods mainly rely on the model to learn vulnerability patterns implicitly from training data, but they do not represent security semantics explicitly. Because the graph itself does not encode security roles, the model often cannot distinguish potential data sources, dangerous operations, and sanitization logic. This limits how well vulnerability semantics can be expressed.

Inspired by the way human analysts first identify key data-flow roles and then examine propagation paths, we enrich the program graph with explicit security semantics. We use a large language model to annotate security-critical code blocks at the BasicBlock level. This design choice is motivated by the fact that source, sink, and sanitizer semantics are usually expressed by short statement groups rather than by isolated tokens.

Specifically, for each BasicBlock in the graph, the large language model analyzes the statements inside the block and determines whether the block acts as a source, a sink, or a sanitizer. When a block is recognized as one of these roles, we attach both the predicted label and its confidence score to the node. If the block is not security-critical, no additional security label is assigned.

The prompt template is shown in Figure 5. Through this process, a graph that originally contains only structural relations is extended with explicit security semantics. Because this information is attached directly to graph nodes, the downstream graph encoder can use structural relations and security semantics jointly during representation learning.

Figure 5. Prompt template for LLM-based taint semantic labeling.

The complete labeling procedure takes the original code property graph as input and examines all BasicBlock nodes in sequence. Each node is passed through semantic role classification and label attachment. The output is a security-enhanced program graph that is then used for vulnerability detection. By explicitly introducing source, sink, and sanitizer information into the graph, the model can focus more accurately on risky data-flow patterns during graph representation learning.

4.2.4. Sem-CPG and Feature Embedding

After the previous stages, the program graph contains two types of semantic enhancement. The first is variable role information, which describes the functional roles of variables in program logic. The second is security role information, which identifies security-critical BasicBlocks as sources, sinks, or sanitizers together with confidence scores. We combine these annotations with the original code property graph to form the Semantically Enriched Code Property Graph, or Sem-CPG.

The node set and edge set of Sem-CPG are inherited from the original code property graph, while the semantic labels are attached as additional node attributes. In this way, the graph preserves structural completeness while explicitly encoding vulnerability-relevant semantics.

After semantic augmentation, every node is converted into a dense vector for the downstream graph encoder. For each node, the final representation is built from three parts: a code-text embedding, a variable role embedding, and a security role embedding. The code-text embedding is obtained by encoding the node-related code fragment with a pre-trained code language model. The variable role embedding is added when the node corresponds to an annotated variable. The security role embedding is added when the node belongs to a security-critical BasicBlock. For nodes without the corresponding label, we use a default embedding or a zero vector so that all node representations remain in the same feature space.

The three components are then concatenated and projected into a unified feature space:

$$\Phi \left(v\right)=W_{proj}[x^{code}\left(v\right);x^{sem}\left(v\right);x^{sec}\left(v\right)]+b_{proj},$$

(1)

where $x^{code}\left(v\right)$ denotes the code-text embedding, $x^{sem}\left(v\right)$ denotes the variable role embedding, and $x^{sec}\left(v\right)$ denotes the security role embedding. In this way, each node in Sem-CPG contains structural context, program semantics, and security-related cues at the same time, where $W_{proj}\in {\mathbb{R}}^{d\times (d_{code}+d_{sem}+d_{sec})}$ is the projection matrix and $b_{proj}\in {\mathbb{R}}^d$ is the bias vector.

4.3. Hybrid Feature Aggregation and Classification

The inference stage of SV-GEN adopts a hybrid backend that combines the strengths of a Graph Transformer and a pre-trained code language model.

4.3.1. Graph Transformer

The graph branch of SV-GEN uses a Graph Transformer to capture long-range structural dependencies in Sem-CPG. Given the node feature matrix of the input graph, the Graph Transformer applies multiple layers of self-attention so that each node can interact with other structurally related nodes beyond local message passing.

The Graph Transformer is a graph encoder based on self-attention. It addresses the long-range dependency limitations of conventional graph neural networks. Given a program graph $G_i^{sem}$ and its node feature matrix $X_i$ (initialized by ${\Phi }_i$), the Graph Transformer applies multiple layers of self-attention aggregation. Let $H_i^{\left(0\right)}=X_i$. At layer l, the attention computation is defined as

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}+A_{\mathrm{mask}}\right)V$$

(2)

where Q, K, and V are the query, key, and value matrices, $d_k$ is the key dimension, and $A_{\mathrm{mask}}$ is a mask matrix constructed from graph adjacency. To handle multiple edge types, we use multi-head attention and allow different heads to focus on different relation types:

$$\mathrm{MultiHead}(H_i^{\left(l\right)})=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h)W^O$$

(3)

where each attention head can use a customized mask $A_{\mathrm{mask}}^{\left(i\right)}$ for a specific subset of edge types, and $W^O\in {\mathbb{R}}^{h{d}_h\times d}$ is the output projection matrix. After multiple Graph Transformer layers, we obtain a graph-level embedding through global average pooling:

$$h_i^g={\displaystyle \frac{1}{|V_i|}}\sum _{v\in V_i}{h}_{i,v}^{\left(L\right)}$$

(4)

where $h_{i,v}^{\left(L\right)}$ denotes the representation of node v at the last layer.

4.3.2. CodeLM for Sequential Semantic Features

The natural code sequence ${\mathcal{S}}_i$ is encoded by a pre-trained code language model such as CodeBERT [31] or GraphCodeBERT [32]. We have designed a flexible and replaceable architecture that allows for the selection of appropriate language models based on different types of code. Since such models are pre-trained on large code corpora, they provide strong semantic representations of source code.

Given the token sequence ${\mathcal{S}}_i=[t_{i,1},\dots ,t_{i,l_i}]$, the code language model produces

$$H_i^s=\mathrm{CodeLM}\left({\mathcal{S}}_i\right)=[s_{i,1},s_{i,2},\dots ,s_{i,l_i}]$$

(5)

where $H_i^s\in {\mathbb{R}}^{l_i\times d}$ is the sequence encoding matrix and $s_{i,j}$ is the embedding of the j-th token. We then derive the sequence-level embedding $h_i^s$ through weighted pooling:

$$h_i^s=\sum _{j=1}^{l_i}{\beta }_{i,j}{s}_{i,j}$$

(6)

where the weights ${\beta }_{i,j}$ can be obtained from the last-layer self-attention scores of the code language model or learned directly as pooling parameters.

4.3.3. Gating Fusion

The graph embedding $h_i^g$ emphasizes structural information and long-range dependencies, while the sequence embedding $h_i^s$ captures semantic details from the source code. Different vulnerability types may rely on these two views to different degrees. Vulnerabilities driven by complex data propagation may depend more on graph structure, whereas vulnerabilities dominated by semantic misuse may depend more on sequential code semantics.

We therefore introduce a learnable gating network that adaptively balances the two representations:

$${\alpha }_i=\sigma (W_g[h_i^g;h_i^s]+b_g)$$

(7)

where $[h_i^g;h_i^s]$ denotes the concatenation of the two branch embeddings, $W_g\in {\mathbb{R}}^{1\times 2d}$ and $b_g\in \mathbb{R}$ are learnable parameters, and $\sigma$ is the sigmoid function. The scalar ${\alpha }_i\in [0,1]$ gives the weight assigned to the graph representation.

The final fused representation is defined as

$$h_i={\alpha }_i·h_i^g+(1-{\alpha }_i)·h_i^s$$

(8)

This design allows the model to adjust the contribution of graph structure and sequence semantics according to the input. The fused representation is then fed into a fully connected classifier to predict the vulnerability probability:

$${\widehat{y}}_i=\mathrm{sigmoid}(W_c{h}_i+b_c)$$

(9)

where $W_c\in {\mathbb{R}}^{1\times d}$ and $b_c\in \mathbb{R}$ are trainable parameters. The output ${\widehat{y}}_i\in [0,1]$ denotes the predicted probability that the function is vulnerable.

4.3.4. Joint Optimization

During training, we optimize the Graph Transformer, the adaptation layers of the code language model if fine-tuning is enabled, and the gating network jointly. The total loss is defined as

$${\mathcal{L}}_{total}={\mathcal{L}}_{cls}+{\lambda }_1{\mathcal{L}}_{reg}$$

(10)

where

$${\mathcal{L}}_{cls}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n\left[y_{i}log\left({\widehat{y}}_i\right)+(1-y_i)log(1-{\widehat{y}}_i)\right]$$

(11)

is the binary cross-entropy classification loss, ${\mathcal{L}}_{reg}$ is a regularization term such as weight decay, and ${\lambda }_1$ is a hyperparameter. This adaptive fusion strategy gives SV-GEN more flexibility and robustness than single-modal or fixed-weight fusion methods.

5. Experimental Evaluation

To evaluate the effectiveness of SV-GEN comprehensively, we conduct systematic experiments on three widely used public vulnerability detection benchmarks. We first describe the research questions, datasets, baselines, evaluation metrics, and implementation details, and then analyze the experimental results.

5.1. Research Questions

Our study addresses four research questions. RQ1 asks whether SV-GEN outperforms existing baselines in vulnerability detection, which evaluates the overall effectiveness of the framework. RQ2 examines whether Sem-CPG is more effective than a standard code property graph, which reveals the contribution of variable semantic role annotation and security-critical node labeling. RQ3 studies the effect of major components in SV-GEN, including the Graph Transformer, the code language model, and the gating fusion module. RQ4 evaluates how sensitive SV-GEN is to different vulnerability categories and whether its gains remain stable across diverse CWE types.

5.2. Datasets

We evaluate SV-GEN on three widely used real-world benchmark datasets that cover multiple vulnerability types and different dataset scales.

Devign is a classic function-level vulnerability detection benchmark introduced by Zhou et al. [9]. It remains one of the most widely cited datasets in this area. The dataset is collected from four representative large open-source C projects, namely the Linux Kernel, QEMU, Wireshark, and FFmpeg. The authors retrieve security-related commits through automated scripts and then combine automatic processing with manual verification.

ReVeal [33] is another large-scale function-level vulnerability dataset that contains C and C++ code from multiple open-source projects. It is built through automated collection from GitHub together with static analysis and manual validation. The dataset contains multiple vulnerability categories, including buffer overflows and format string vulnerabilities, and therefore offers strong diversity and practical relevance.

DiverseVul [34] is a more recent dataset designed to address the imbalance of vulnerability categories in earlier benchmarks. It contains more diverse vulnerability patterns and coding styles collected from multiple large open-source projects, which makes it a challenging benchmark for evaluating model robustness.

5.3. Baselines

To evaluate the performance of SV-GEN comprehensively, we compare it with eight representative baselines that cover mainstream vulnerability detection techniques, including static analysis tools, sequence-based methods, and graph-based methods.

(1) 

Flawfinder [35] is a classic open-source static analysis tool. It scans C and C++ source code for potential security flaws through a built-in database of vulnerable patterns. The tool identifies risky function calls, such as strcpy and sprintf, through lexical matching rules and ranks the matches by risk level. Flawfinder requires no training and is suitable for rapid scanning, but its detection ability is constrained by the coverage of predefined rules and it often produces both false positives and false negatives.

(2) 

Devign [9] is an early graph-neural-network-based vulnerability detection method. It is the first to fuse AST, CFG, and DFG into a composite program graph and then learn vulnerability patterns with a gated graph neural network. Devign verifies the importance of structural graph information in vulnerability detection and remains a classic baseline in this area.

(3) 

GraphCodeBERT [32] is a pre-trained code language model proposed by Microsoft. It extends CodeBERT with data-flow information. Through two pre-training tasks, edge prediction and node alignment, the model learns semantic representations of code and data dependencies among variables. In vulnerability detection, GraphCodeBERT encodes the code sequence and performs binary classification through the output representation of the [CLS] token.

(4) 

LineVul [17] is a Transformer-based function-level vulnerability detection method. It uses a pre-trained RoBERTa model to encode code sequences. The method adopts byte pair encoding to process code tokens and uses attention to support line-level vulnerability localization. LineVul shows strong detection and localization performance on large-scale datasets.

(5) 

ReGVD [20] is a recent graph-neural-network-based vulnerability detection method that revisits graph representation and message passing systematically. It introduces more stable residual aggregation operators and multiscale skip connections, which effectively alleviate the over-smoothing problem in graph neural networks. ReGVD reports competitive performance on multiple benchmark datasets.

(6) 

CFExplainer [36] is a graph-neural-network-based vulnerability detection method with counterfactual explanation. On top of graph-based vulnerability classification, it generates counterfactual graphs to explain model predictions and identify the subgraph structures that most strongly influence the final decision. CFExplainer improves interpretability and also enhances detection performance through explanation-guided feature selection.

(7) 

DeepWuKong [37] is a vulnerability detection method based on program slicing and deep learning. It first extracts security-relevant code slices through a program dependence graph and then uses graph neural networks to learn structural features for classification. By focusing on security-critical program fragments, DeepWuKong reduces interference from irrelevant code and improves detection specificity.

(8) 

ANGEL [22] is a recent graph-based vulnerability detection method designed for large code graphs. It addresses this setting through two key components: a hierarchical graph refinement module that progressively removes less important nodes to simplify the input graph, and a context-aware graph encoding module that combines GNNs with a Graph Transformer to capture both local structural interactions and long-range dependencies. By reducing graph redundancy while preserving vulnerability-relevant context, ANGEL achieves strong performance on public vulnerability detection benchmarks and therefore serves as a competitive recent baseline.

5.4. Evaluation Metrics

To evaluate detection performance comprehensively, we use the following four widely adopted metrics.

(1) 

Accuracy (Acc). [38] Accuracy measures the proportion of correctly classified samples among all samples. It is defined as

$$\mathrm{Acc}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(12)

where $TP$, $TN$, $FP$, and $FN$ denote true positives, true negatives, false positives, and false negatives, respectively. Accuracy is the most intuitive metric, but it can be misleading when the class distribution is imbalanced.

(2) 

Precision. [38] Precision measures the fraction of predicted vulnerable samples that are truly vulnerable. It is defined as

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(13)

In vulnerability detection, precision reflects the false alarm rate. Higher precision means the model is better at avoiding the misclassification of benign code as vulnerable.

(3) 

Recall. [38] Recall measures the fraction of truly vulnerable samples that the model detects. It is defined as

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(14)

Recall is especially important in vulnerability detection because missed vulnerabilities may lead to severe security risks. Higher recall means the model discovers more potential vulnerabilities.

(4) 

F1 score. [38] F1 score is the harmonic mean of precision and recall. It balances precision and recall jointly and is defined as

$$\mathrm{F}1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(15)

Under class imbalance, F1 score provides a more objective summary of overall model performance and is therefore one of the most widely used metrics in vulnerability detection.

5.5. Implementation of SV-GEN

5.5.1. Experimental Environment and Settings

We implement SV-GEN in Python 3.11.14 and PyTorch 2.4.0. All experiments run on a server equipped with NVIDIA Tesla V100 32GB GPUs, NVIDIA driver 525.105.17, and CUDA 11.8. During Sem-CPG construction, variable role annotation and security-critical node labeling are performed by DeepSeek-V3 [39], which provides a practical balance between cost and quality. In the hybrid backend, the Graph Transformer encodes Sem-CPG, while GraphCodeBERT serves as the pre-trained sequential encoder. Their outputs are fused by a learnable gating network and then passed to the classifier. We use AdamW as the optimizer with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ$ = $1\times {10}^{-8}$. The learning rate is set to $1\times {10}^{-3}$, together with linear warmup at the beginning of training to stabilize optimization. To avoid overfitting, we use early stopping based on the validation F1 score. Training stops when the validation F1 score does not improve for ten consecutive epochs, and the model is restored to the best checkpoint. For each baseline, we reproduce the publicly available implementation and follow the hyperparameter settings described in the original paper as closely as possible.

5.5.2. Data Split

For each dataset, we randomly split the samples into training, validation, and test sets with a ratio of 8:1:1. We fix the random seed to 42 for reproducibility. To reduce data leakage, we ensure that code samples from the same project do not appear in both training and test sets.

Rather than being derived from the full dataset-level statistics in Table 1, all metrics reported in Table 2 are computed on the held-out test split after the 8:1:1 partition.In addition, for Flawfinder baselines, we are using command-line version 2.0.19. We observed occasional sample-level parsing or reporting instability on a small subset of benchmark instances. To ensure fairness, we applied a consistent metric aggregation protocol for all runs under the same split setting.

Table 1. Dataset statistics.

Dataset	Total Samples	Vulnerable	Non-Vulnerable	Vulnerability Ratio
Devign	22,361	10,348	12,013	46.28%
ReVeal	18,169	1664	16,505	9.16%
DiverseVul	330,492	18,945	311,547	5.73%

Table 2. Performance comparison between SV-GEN and baseline methods on three datasets (evaluated on held-out test splits).

Method	Devign				ReVeal				DiverseVul
	Acc	Pre	Recall	F1	Acc	Pre	Recall	F1	Acc	Pre	Recall	F1
Flawfinder	51.57	45.86	35.34	39.92	80.21	27.83	24.64	26.14	72.62	12.53	12.84	12.68
Devign	57.34	55.68	57.41	56.53	82.43	36.72	32.58	34.53	80.74	17.86	14.32	15.89
GraphCodeBERT	62.12	58.23	60.41	59.30	88.92	41.36	36.87	38.99	92.87	29.43	16.08	20.78
LineVul	60.34	57.47	62.71	59.98	89.17	49.54	38.23	43.16	93.14	30.27	17.86	22.47
ReGVD	59.53	57.14	59.86	58.47	88.54	56.93	35.46	43.70	86.58	28.14	19.53	23.06
CFExplainer	57.46	56.31	58.74	57.50	86.13	45.27	34.18	38.95	88.23	19.87	15.64	17.50
DeepWuKong	59.54	55.93	59.47	57.65	87.76	43.86	32.74	37.49	89.96	25.34	14.87	18.74
ANGEL	60.71	59.84	63.12	61.44	89.43	50.87	39.42	44.42	93.26	24.42	18.64	21.14
SV-GEN	61.59	57.21	65.54	61.09	91.81	52.68	41.50	46.43	93.53	31.59	23.83	27.17

5.6. Results and Analysis

5.6.1. Overall Performance (RQ1)

To evaluate the overall performance of SV-GEN, we compare it with eight baselines on the three benchmark datasets. Table 2 reports Accuracy, Precision, Recall, and F1 score on the held-out test splits.

Table 2 reveals several important findings. SV-GEN shows stable competitive performance across all three benchmarks and achieves especially strong results on DiverseVul, where it obtains the best score on all four metrics. This result is important because DiverseVul is the largest benchmark and contains the most severe class imbalance, which makes it closer to practical deployment settings.

The contrast with Flawfinder highlights the general advantage of data-driven models. Flawfinder relies on a predefined set of patterns and therefore performs poorly across all datasets. On DiverseVul, for example, its precision and recall are only 12.53% and 12.84%, which means that it misses many true vulnerabilities while still producing many false alarms. In contrast, SV-GEN learns vulnerability semantics directly from labeled data and therefore achieves much stronger coverage and reliability.

Compared with sequence-based baselines, SV-GEN also shows clear advantages. GraphCodeBERT and LineVul are two strong representatives of token-sequence modeling, yet they still compress nonlinear program behavior into linear token streams. On DiverseVul, SV-GEN improves the F1 score over LineVul by 4.70 percentage points and over GraphCodeBERT by 6.39 percentage points. On ReVeal, SV-GEN reaches a recall of 41.50%, which exceeds GraphCodeBERT by 4.63 percentage points and LineVul by 3.27 percentage points. These gains indicate that explicit structural modeling remains important when vulnerability conditions depend on dispersed control and data dependencies.

The comparison with graph-based baselines is equally informative. Devign, which directly applies graph neural networks to composite code graphs, trails SV-GEN on all three datasets. ReGVD improves graph learning through better aggregation and skip connections, but its recall on ReVeal remains limited at 35.46%, which suggests a more conservative prediction behavior. CFExplainer and DeepWuKong improve interpretability and slicing quality, respectively, but they still do not match the overall performance of SV-GEN. ANGEL is the strongest graph-based baseline in our comparison and achieves the best F1 score on Devign. However, SV-GEN surpasses ANGEL on ReVeal and DiverseVul, and its margin on DiverseVul is substantial. This pattern suggests that the global self-attention of the Graph Transformer and the explicit security priors in Sem-CPG become increasingly useful as the graph grows larger and the vulnerability distribution becomes more diverse.

Across the twelve evaluation settings, that is four metrics on three datasets, SV-GEN achieves the best result in eight settings. The overall pattern shows that the framework is not only competitive on average but also robust across datasets and metrics.

5.6.2. Effectiveness of the Graph Representation (RQ2)

This experiment evaluates whether Sem-CPG improves vulnerability detection compared with simpler graph representations. To isolate the effect of the graph representation, we keep the model architecture and hyperparameters fixed and change only the input graph. Because DiverseVul is the largest dataset and has the lowest vulnerability ratio at 5.73%, it is the most sensitive benchmark for measuring representation quality. We, therefore, conduct this experiment on DiverseVul. We compare three graph variants: a standard code property graph that combines AST, CFG, DFG, and CDG; an enhanced graph that additionally includes Code Natural Sequence edges; and the full Sem-CPG, which further adds variable semantic role annotations and taint labels. Figure 6 summarizes the results on accuracy, precision, recall, and F1 score.

Figure 6. Performance comparison of CPG, CPG+CNS, and Sem-CPG on DiverseVul.

Figure 6 shows that Sem-CPG achieves the best result on all four metrics. The difference in accuracy is relatively small, with 90.58%, 90.94%, and 92.53% for CPG, CPG+CNS, and Sem-CPG, respectively. This is expected because non-vulnerable samples account for 94.27% of DiverseVul, and accuracy is therefore less informative under such an imbalance. Precision, recall, and F1 score provide a clearer picture. Sem-CPG reaches a precision of 31.59%, which improves over CPG by 3.85 percentage points. Its recall reaches 23.83%, which is 6.65 percentage points higher than that of CPG. The F1 score rises from 21.22% with CPG to 22.92% with CPG+CNS and then to 27.17% with Sem-CPG, which yields a total gain of 5.95 percentage points over the standard code property graph.

The gains from the two enhancement stages follow different patterns. Adding CNS edges improves the F1 score from 21.22% to 22.92%, which indicates that natural sequence order complements structural dependency information. This is particularly helpful when vulnerability triggering depends on consecutive statements that are close in source order but not strongly connected in the dependency graph. The larger gain comes from adding semantic roles and security labels. Variable role annotation turns generic identifiers into functional entities such as input variables, buffers, and loop counters, which helps the model understand program behavior at the node level. Security labels provide explicit priors about sources, sinks, and sanitizers, which guide the Graph Transformer toward risky regions and propagation paths. The two signals are complementary, because one increases local semantic discriminability and the other identifies graph regions that are security-sensitive.

Overall, the results show that Sem-CPG provides a stronger foundation for vulnerability detection than standard graph representations. By enriching the graph first with natural sequence semantics and then with LLM-driven security semantics, it yields more discriminative program representations for large and sparse real-world benchmarks.

5.6.3. Ablation Study (RQ3)

To examine the contribution of the major components in SV-GEN, we conduct ablation studies from three perspectives: the effect of different large language models on Sem-CPG labeling quality, the effect of the graph encoder architecture, and the effect of the pre-trained code language model. All experiments are conducted on DiverseVul, whose scale and imbalance make component differences easier to observe. Unless otherwise stated, all remaining modules and hyperparameters are kept fixed.

We first study how the choice of large language model affects the quality of semantic labeling. Since Sem-CPG relies on the large language model to annotate variable roles and security-critical nodes, the quality of this annotation directly affects downstream detection. We compare Qwen2.5-Coder-7B [40], Qwen3-Coder-30B-A3B, and DeepSeek-V3 [39]. The resulting Sem-CPGs are then fed into the same backend. Table 3 reports the results.

Table 3. Effect of different large language models on Sem-CPG labeling quality on DiverseVul.

LLM	Acc (%)	Pre (%)	Recall (%)	F1 (%)
Qwen2.5-Coder-7B	92.87	29.42	17.56	21.99
Qwen3-Coder-30B-A3B	93.24	31.08	18.94	23.54
DeepSeek-V3	93.53	31.59	23.83	27.17

DeepSeek-V3 performs best, reaching 93.53% accuracy and 27.17% F1 score. This result is consistent with the expectation that a stronger code model provides better semantic reasoning. Qwen3-Coder-30B-A3B improves the F1 score over Qwen2.5-Coder-7B by 1.55 percentage points, while DeepSeek-V3 further improves over Qwen3-Coder-30B-A3B by 3.63 percentage points. This pattern suggests that higher-quality semantic labeling produces a clear downstream benefit. For straightforward cases such as clear user inputs or obvious buffer operations, even a smaller model can often produce correct labels. The stronger model mainly helps with ambiguous cases, for instance, when a variable can plausibly be interpreted as either a loop counter or an array index, or when a function call may or may not act as a sanitizer. The recall difference is especially notable, which suggests that better semantic labeling helps reduce missed vulnerabilities.

We next study the graph encoder. In the default SV-GEN setting, the graph branch uses a Graph Transformer. We replace it with a gated graph neural network and a graph attention network (GAT), and we also consider a setting without any graph encoder, denoted as w/o GNN, where the model relies only on sequential features. All graph encoders use the same three-layer depth, and the remaining settings stay unchanged. Table 4 reports the results.

Table 4. Effect of different graph encoders on detection performance on DiverseVul.

GNN Variant	Acc (%)	Pre (%)	Recall (%)	F1 (%)
w/o GNN	92.47	28.14	16.72	20.98
GAT	92.78	29.53	17.48	21.96
GGNN	93.06	30.86	18.37	23.03
Graph Transformer	93.53	31.59	23.83	27.17

The Graph Transformer performs best on all four metrics and reaches an F1 score of 27.17%. Removing the graph branch lowers the F1 score to 20.98%, which confirms that structural information is essential for vulnerability detection. GAT improves over the no-graph setting, but it still trails the Graph Transformer by 5.21 percentage points in F1 score. Although GAT uses attention, it only attends over local neighborhoods and does not exploit the full heterogeneity of Sem-CPG effectively. GGNN performs better than GAT because its gated update mechanism accumulates information over several propagation steps, yet its effective receptive field is still limited by network depth. The Graph Transformer avoids this bottleneck by allowing every node to attend globally, which is particularly useful when vulnerabilities depend on long propagation paths and cross-branch relations.

Finally, we study the sequential encoder. We replace the default GraphCodeBERT with CodeBERT and also consider a setting without any code language model, denoted as w/o CodeLM, where the model depends only on the graph branch. Both code language models are fine-tuned under the same learning schedule. Table 5 reports the results.

Table 5. Effect of different pre-trained code language models on DiverseVul.

CodeLM Variant	Acc (%)	Pre (%)	Recall (%)	F1 (%)
w/o CodeLM	92.68	29.27	17.63	22.01
CodeBERT	93.18	31.24	18.72	23.41
GraphCodeBERT	93.53	31.59	23.83	27.17

Removing the sequential encoder lowers the F1 score from 27.17% to 22.01%, which shows that source-level semantic features remain important even when the graph branch is present. GraphCodeBERT outperforms CodeBERT by 3.76 percentage points in F1 score. The likely reason is that GraphCodeBERT is pre-trained with data-flow-aware objectives such as edge prediction and node alignment, while CodeBERT focuses only on masked language modeling and replaced token detection. The structural priors learned during GraphCodeBERT pre-training therefore complement Sem-CPG more effectively.

Taken together, the ablation studies show that the major components of SV-GEN contribute complementary gains. The Graph Transformer provides the strongest graph representation, GraphCodeBERT supplies richer sequential semantics than a generic code language model, and stronger large language models yield better semantic annotation for Sem-CPG. Removing any of these components leads to a noticeable drop in performance.

5.6.4. Sensitivity to Vulnerability Types (RQ4)

To further examine how SV-GEN behaves across different vulnerability patterns, we compare SV-GEN with GraphCodeBERT on the ten most frequent CWE categories in the DiverseVul test set. These categories cover several security dimensions, including memory boundary violations, resource lifetime errors, input validation problems, and concurrency defects. GraphCodeBERT is used as the reference because it is the strongest sequence-based baseline on DiverseVul and already includes data-flow-aware pre-training. The comparison therefore highlights the additional value of semantic graph enhancement and hybrid modeling.

Figure 7 shows that SV-GEN outperforms GraphCodeBERT on all ten vulnerability categories, but the margin varies considerably, ranging from 11.83 percentage points on CWE-416 to 3.74 percentage points on CWE-200. This variation is closely related to the semantic characteristics of each vulnerability type and to the design of our method.

Figure 7. Per-CWE comparison between GraphCodeBERT and SV-GEN on DiverseVul.

SV-GEN shows the largest gains on vulnerabilities with explicit data propagation paths. The strongest improvement appears on CWE-416, use-after-free, where the F1 score increases by 11.83 percentage points. This category depends on a source, namely a memory release operation, and a sink, namely a later use of the dangling pointer, which are often separated by multiple function calls and branches. In Sem-CPG, these critical points can be marked explicitly, and the Graph Transformer can connect them even when they are far apart in the graph. CWE-787, out-of-bounds write, and CWE-125, out-of-bounds read, follow a similar pattern, with gains of 10.63 and 9.25 percentage points. These vulnerabilities depend on propagation from external input to buffer access, and explicit source-sink labels are particularly effective in such settings. CWE-362, race condition, also shows a large gain of 10.82 percentage points. Although the absolute F1 score of this category is not the highest, the improvement is substantial because race conditions often involve dispersed shared-state accesses. Variable role annotation helps the model identify shared resources more accurately, while the Graph Transformer captures relations across distant control paths.

SV-GEN also improves steadily on categories where variable semantics matter. CWE-119, generic buffer boundary errors, and CWE-476, null pointer dereference, gain 7.86 and 7.46 percentage points, respectively. Detecting these vulnerabilities requires the model to distinguish buffers, pointers, and control variables and to understand whether relevant checks are present along feasible paths. Sem-CPG helps because it converts anonymous identifiers into semantically meaningful program entities. CWE-20, improper input validation, improves by 6.93 percentage points, which further confirms that explicit source labeling helps the model follow the propagation of unchecked input toward dangerous operations.

The gains are smaller on CWE-200, information exposure, and CWE-401, memory leak, with improvements of 3.74 and 4.45 percentage points. This pattern is also understandable. Information exposure often depends less on a clear taint path and more on broader program intent, for example, whether an error message leaks internal state or whether a log statement outputs sensitive fields. Such cases require a higher-level understanding of design intent and policy, which is not fully captured by source-sink-style graph enhancement. Memory leak detection, in contrast, depends on path completeness. The model must infer whether allocated memory is released on all relevant execution paths, which is harder to express through local semantic labels alone.

Overall, SV-GEN is especially effective on data-flow-intensive vulnerability categories, where explicit semantic labeling and global graph attention align well with the structure of the problem. Its gains are smaller but still consistent on vulnerability types that require broader intent understanding or path-complete reasoning, which points to promising directions for future work.

5.7. Runtime Efficiency

To further evaluate the practical efficiency of SV-GEN, we conducted an additional runtime analysis on randomly sampled test functions from the three benchmark datasets. Specifically, we randomly selected 300 functions from each dataset, resulting in a total of 900 functions for runtime measurement. All experiments were conducted under the same hardware and software environment described in Section 5.5.1. Since the semantic enhancement stage in SV-GEN relies on calls to DeepSeek-V3, we report the average per-function runtime of each major component separately.

Table 6 presents the runtime breakdown. The average preprocessing time is only 1 ms per function. Joern-based graph construction together with CNS edge construction requires 300 ms on average. The hybrid backend forward pass, which includes the Graph Transformer, GraphCodeBERT, and gating-based fusion, takes only 4 ms per function. In contrast, the dominant overhead comes from the two LLM-driven semantic enhancement stages. Variable semantic role annotation requires 5.62 s on average, while security-critical node labeling (i.e., source/sink/sanitizer identification) requires 7.86 s. Summing all measured components, the average end-to-end processing time is approximately 13.79 s per function.

Table 6. Runtime breakdown of SV-GEN on 900 sampled functions.

Component	Average Runtime per Function
$T_{\mathrm{prep}}$ (preprocessing)	1 ms
$T_{\mathrm{var}-\mathrm{llm}}$ (variable semantic role annotation)	5.62 s
$T_{\mathrm{joern}}$ (Joern graph construction + CNS edges)	300 ms
$T_{\mathrm{taint}-\mathrm{llm}}$ (source/sink/sanitizer labeling)	7.86 s
$T_{\mathrm{infer}}$ (hybrid backend forward inference)	4 ms
Total end-to-end runtime	13.79 s

These results reveal a clear efficiency pattern. The computational bottleneck of SV-GEN does not lie in the neural classifier itself, but in the API-based semantic graph construction stage. In particular, the two LLM-driven annotation stages account for more than 98% of the total runtime, whereas graph construction and model inference together contribute only a very small fraction. This observation is important because it shows that the main cost of SV-GEN is a front-loaded semantic enrichment cost rather than a heavy downstream prediction cost.

From a deployment perspective, this runtime profile suggests that SV-GEN is best suited to a two-stage usage mode. In practical vulnerability analysis pipelines, the semantic annotation and Sem-CPG construction stages can be executed offline and cached in advance, while the trained hybrid model can be used for lightweight online prediction. Since the forward inference time of the hybrid backend is only 4 ms per function, the prediction module itself introduces negligible latency once the semantic graph has been prepared.

In addition, we adopt a concurrency level of 20 in the practical implementation, which substantially improves throughput in batch processing scenarios by overlapping the latency of multiple LLM requests. Therefore, although the cold-start end-to-end latency of SV-GEN is dominated by semantic annotation, the framework remains practically usable for real-world vulnerability screening tasks, especially in offline auditing, batch analysis, and continuous integration settings where semantic preprocessing can be amortized across many functions.

Overall, the runtime analysis indicates that SV-GEN achieves a favorable trade-off between effectiveness and efficiency. Its main overhead is introduced by semantic enrichment, which is also the key factor behind the performance gains observed in previous sections. At the same time, the downstream prediction stage remains lightweight, making the framework compatible with realistic large-scale security analysis workflows.

5.8. Direct Validation of LLM-Based Semantic Labeling Quality

To directly evaluate the reliability of the LLM-based semantic augmentation used in Sem-CPG, we conduct a focused manual verification study on a sampled subset from the test partitions of the three benchmark datasets. Specifically, we sample 30 functions in total, including 10 functions from Devign, 10 from ReVeal, and 10 from DiverseVul. This experiment is designed to provide direct evidence for the quality of the semantic labels produced by the LLM, complementing the downstream detection results reported in the previous experiments.

We evaluate two labeling tasks that are used during Sem-CPG construction. The first task is variable semantic role labeling, in which the LLM assigns semantic role names to variables according to their contextual usage in the code. The second task is security-critical node labeling, in which the LLM classifies each BasicBlock as one of four categories: source, sink, sanitizer, or none. In this experiment, DeepSeek-V3 is used as the labeling model, consistent with the main implementation of SV-GEN.

To obtain the reference annotations, two annotators independently reviewed the sampled functions and assigned labels according to the same semantic criteria used in the main method. Disagreements were then resolved by a third adjudicator, and the adjudicated results were treated as the gold standard. We compare the labels generated by DeepSeek-V3 against this gold standard. Because both tasks may involve class imbalance, we use Macro-F1 as the evaluation metric.

Table 7 summarizes the manual verification results. DeepSeek-V3 achieves a Macro-F1 score of 76.21 on variable semantic role labeling and 82.94 on security-critical node labeling. These results provide direct evidence that the LLM-generated semantic annotations are reasonably consistent with human judgment. Therefore, the semantic augmentation strategy used in Sem-CPG is supported not only by downstream vulnerability detection performance, but also by direct manual verification on sampled instances.

Table 7. Manual verification results of DeepSeek-V3 semantic labeling on a sampled subset.

Task	Annotation Unit	Macro-F1 (%)
Variable semantic role labeling	Variable	76.21
Security-critical node labeling	BasicBlock	82.94

6. Conclusions

This paper presents SV-GEN, a new framework for function-level source-code vulnerability detection. The core idea is to use large-language-model-driven semantic enhancement to compensate for the limited vulnerability semantics of standard CPGs and then combine global structural modeling with sequential semantic modeling in a hybrid backend. We design Sem-CPG as the unified program representation. Starting from a standard code property graph, we use a large language model to annotate variable nodes with semantic roles so that the functional meaning removed by identifier normalization can be recovered. We also identify security-critical nodes and attach source, sink, and sanitizer labels so that program elements directly involved in vulnerability triggering are marked explicitly. On top of Sem-CPG, we use a Graph Transformer to encode global structural dependencies and GraphCodeBERT to encode sequential code semantics, and we combine the two views through a learnable gating mechanism to predict the vulnerability probability.

Extensive experiments on Devign, ReVeal, and DiverseVul show that SV-GEN outperforms strong baselines in most settings, which confirms the effectiveness of the framework. Additional analysis shows that each major component contributes positively to the final performance. Sem-CPG consistently improves over standard CPGs; the Graph Transformer models long-range dependencies more effectively than conventional graph neural architectures; and stronger large language models yield higher-quality semantic annotations, although the marginal gain gradually shrinks. The per-CWE analysis further shows that SV-GEN is particularly effective on data-flow-intensive vulnerabilities such as use-after-free and out-of-bounds writes, while categories that require higher-level intent understanding or path-complete reasoning remain more challenging.

There are several important directions for future work. First, the current Sem-CPG mainly augments standard CPGs with variable semantic roles and source/sink/sanitizer labels. A natural next step is to incorporate richer program logic information, such as inter-procedural dependence, API usage constraints, control-dominance patterns, and even lightweight symbolic or dynamic evidence, so that the representation can better capture vulnerabilities whose manifestation depends on long execution contexts or implicit semantic conditions. Second, although SV-GEN currently performs function-level prediction, practical security auditing often requires finer-grained outputs. We therefore plan to extend the framework from coarse-grained detection to statement-level or line-level vulnerability localization, and to further improve interpretability by identifying the critical nodes, edges, and semantic annotations that most strongly support each prediction. Third, the current pipeline still incurs non-negligible overhead during the LLM-based annotation stage. Future work will investigate more efficient semantic enrichment strategies, such as cached annotation reuse, selective annotation for suspicious regions only, and distilled or local lightweight annotators, to improve scalability in large real-world codebases. Finally, we will evaluate the generalization ability of SV-GEN under more realistic settings, including cross-project and cross-dataset transfer, mixed-language repositories, and industrial continuous-analysis scenarios, to further assess its robustness and practical applicability.