A Hybrid Semantic and Multi-Attention Mechanism Approach for Detecting Vulnerabilities in Smart Contract Code

Abstract

Driven by blockchain technology, numerous industries are increasingly adopting smart contracts to enhance efficiency, reduce costs, and improve transparency. As a result, ensuring the security of smart contracts has become critical. Traditional detection methods often suffer from low efficiency, are prone to missing complex vulnerabilities, and have limited accuracy. Although deep learning approaches address some of these challenges, issues with both accuracy and efficiency remain in current solutions. To overcome these limitations, this paper proposes a symmetry-inspired solution that harmonizes bidirectional and generative semantic patterns. First, we generate distinct feature extraction segments for different vulnerabilities. We then use the Bidirectional Encoder Representations from Transformers (BERT) module to extract original semantic features from these segments and the Generative Pre-trained Transformer (GPT) module to extract generative semantic features. Finally, the two sets of semantic features are fused using a multi-attention mechanism and input into a classifier for result prediction. Our method was tested on three datasets, achieving F1 scores of 93.33%, 93.65%, and 92.31%, respectively. The results demonstrate that our approach outperforms most existing methods in smart contract detection.

1. Introduction

Currently, blockchain technology [1,2] and smart contracts [3,4] have become increasingly important in the digital landscape, fostering the development of numerous decentralized applications. In particular, smart contracts are widely used in China’s supply chain management, especially within industries such as food and pharmaceuticals, where blockchain technology ensures product traceability and authenticity. With the rapid growth of the virtual currency market, smart contracts have become an attractive target for cyber attackers. Despite intensified efforts to strengthen security audits for smart contracts, many cybercriminals continue to exploit undiscovered vulnerabilities, which, when exploited, can lead to the leakage of assets and user information, causing significant issues for users. According to statistics, the cumulative losses due to hacker attacks in the DeFi [5] sector have surpassed $9.11 billion (approximately RMB 6 billion), with more than 150 attacks related to smart contracts occurring in 2024 alone, resulting in losses exceeding $328 million (around RMB 230 million). As a result, the security of smart contracts has become a top priority for researchers across various fields.

Over the years, significant efforts have been made to design many representative traditional detection tools. However, with the expansion of blockchain transaction scales, these traditional tools have become insufficient to handle emerging vulnerabilities. Consequently, researchers have turned to a new approach—deep learning techniques. Among the related methods, Recurrent Neural Network (RNN) [6] and Long Short-Term Memory (LSTM) [7] face limitations related to the length of the input text, with their performance deteriorating as text length increases. Bidirectional Encoder Representations from Transformers (BERT), a powerful natural language processing model, mitigates the issue of memory degradation by using an attention mechanism, ensuring that performance remains stable even as the length of the code increases. This makes BERT effective for detecting potential vulnerabilities by extracting semantic features from the source code.

To further address the current security issues of smart contracts, this paper proposes a more effective model for extracting semantic information from the code: the BERT-GPT Multihead Fusion Model (BGMF). This model not only extracts the original semantic features from the source code but also extracts generative semantic features. Additionally, our model incorporates a multi-head attention mechanism to improve the ability to capture hybrid semantic features from different dimensions. We tested this approach on reentrancy, timestamp, and integer overflow vulnerabilities, and the results were promising. The contributions of this paper are as follows:

• Based on a comprehensive analysis of existing literature on reentrancy vulnerabilities, this paper further investigates other dangerous operations in contracts beyond call.value. We identify that call and delegatecall can also trigger reentrancy vulnerabilities. By expanding the scope of previous research, we enhance the extraction of vulnerable contract fragments, thus improving the accuracy of reentrancy vulnerability detection.
• Through an in-depth analysis of integer overflow vulnerabilities, this paper proposes a new method for extracting vulnerability contract fragments. This method overcomes some of the limitations of existing integer overflow detection techniques and provides effective support for improving the comprehensiveness and accuracy of vulnerability detection.
• This paper introduces the Generative Pre-trained Transformer (GPT) module for generative semantic feature extraction. Through this, input text is encoded to generate contextually relevant feature representations, enhancing the model’s ability to understand complex semantics and long-distance dependencies.
• We integrate the generative semantic features from GPT with the original semantic features from BERT via a multi-head attention mechanism. This fusion significantly improves the accuracy of vulnerability detection.

The organization of this paper is as follows: Section 2 reviews related work on smart contract vulnerability detection; Section 3 discusses the experimental preparations; Section 4 introduces the experimental methodology; Section 5 analyzes the experimental results; and Section 6 concludes the paper.

2. Related Works

To effectively prevent vulnerabilities in smart contracts, researchers have conducted extensive studies. This section introduces six traditional detection tools and four deep learning-based detection methods.

2.1. Traditional Detection Tools

Researchers have made significant efforts to detect vulnerabilities in smart contracts, and traditional detection tools [8] have played a crucial role in this process. These tools help developers identify potential security risks by analyzing the source code of smart contracts or examining control and data flows using various technical methods. Static detection tools, such as Slither [9], perform early-stage code auditing by converting Solidity [10] code into a simplified instruction set and applying techniques like data flow analysis and taint tracing to detect vulnerabilities early in the development cycle. Dynamic analysis tools, including Mythril [11] and Manticore [12], simulate contract execution under different conditions to identify security vulnerabilities. Mythril uses symbolic execution to explore all possible states of a smart contract, detecting 16 types of potential vulnerabilities, while Manticore offers a flexible framework for analyzing Ethereum smart contracts and binaries, providing significant advantages in contract analysis. Security [13] is another fully automated tool that uses symbolic analysis to extract the dependency graph of a contract, reducing the workload of manual auditing and verifying compliance with security properties. Fuzz testing tools [14] like Echidna [15] help detect anomalous behaviors by generating a large number of random or variant inputs to assess contract security. Finally, formal verification tools, such as K Ethereum Virtual Machine (KEVM) [16], ensure contract security through mathematical proofs by using the K-framework to provide clear, executable specifications for Ethereum Virtual Machines (EVMs). Despite the usefulness of these traditional tools, they all share a common issue: they often fail to sufficiently validate the complex interactions within smart contracts, which may result in the oversight of critical security issues.

2.2. Deep Learning Approaches

Currently, deep learning models [17,18] have reached a high level of maturity and are widely used in various fields, including applications in smart contract vulnerability detection. Compared with traditional detection tools, deep learning methods offer greater flexibility, as they do not rely on expert-defined rules and can effectively handle large-scale contracts and complex vulnerability patterns. Below are four representative categories of deep learning methods used for smart contract vulnerability detection.

The first category involves feature extraction based on bytecode images [19,20]. In this approach, the bytecode of a smart contract is first converted into an image format using the principles of RGB images and then used as input to a Convolutional Neural Network (CNN). Through this method, different parts of the bytecode are treated as image pixels, which enables feature extraction. The second category is based on Graph Neural Networks (GNNs) [21], where the source code of a smart contract is transformed into a graph structure, with nodes representing key function calls and edges denoting dependencies between program elements. This approach allows GNNs to capture complex features in the code more effectively, thereby improving vulnerability detection. The third category involves direct feature extraction from the source code or bytecode [22]. In this method, the source code or bytecode is pre-processed and converted into vector representations using Word2Vec [23], which are then used as input to an LSTM model. This enables the processing of sequence data to capture temporal dependencies and syntactic-semantic information within the contract code.

Although these three traditional deep learning methods address certain limitations of conventional detection tools, particularly in handling contracts with complex interactions, they still face some challenges. With advancements in computational power and the availability of large-scale datasets, a fourth, novel class of deep learning methods has emerged, leveraging large models [24] for improved smart contract vulnerability detection. For instance, Chen et al. [25] use ChatGPT (Website link: https://chat.openai.com, accessed on 8 July 2025) to detect smart contract vulnerabilities, primarily relying on the capabilities of its large language model and combining it with pre-designed prompts to analyze the contract code and identify vulnerabilities. Bu et al. [26] integrate BERT with Control Flow Graph (CFG) analysis to detect vulnerabilities directly from bytecode. Furthermore, research in smart contract vulnerability detection has introduced a contrastive learning approach [27], which learns the similarities and differences between pairs of contracts by comparing their loss functions. This approach captures the relationships between contracts and enhances the accuracy of vulnerability detection.

2.3. Motivation

Although using large models to detect smart contract vulnerabilities addresses some of the limitations of traditional deep learning methods, such as the difficulty that traditional RNNs and LSTMs face in capturing long-distance dependencies as sequence lengths increase. We found that there is still room for improvement and innovation in applying the BERT [28] to smart contract vulnerability detection. Previous studies have primarily focused on extracting high-level semantics, often neglecting the deeper internal semantics of the code. To address this gap, we innovated the decoding component of the transformer model by incorporating GPT to extract generative semantics from the code. Additionally, we introduced a multi-attention mechanism to fuse both the original and generative semantics, allowing for feature extraction from multiple perspectives.

3. Background and Technology of Vulnerability Detection

3.1. Ethereum

Understanding Ethereum [29] is essential for detecting smart contract vulnerabilities, as its design and operational principles directly influence contract security. Ethereum’s core component, the EVM [30], executes smart contracts to carry out predefined logic. Gas [31] measures computational and storage resources, with each transaction and contract execution consuming a specific amount. State variables of a smart contract are stored in Ethereum’s global state tree, and modifications to these variables require gas. In Solidity, state variables are stored in slots in the order of declaration. Ethereum features two types of accounts: external accounts, controlled by private keys for transactions, and contract accounts, governed by contract code for executing smart contracts. Ethereum’s consensus mechanism is vital for network security. Initially, it used Proof-of-Work (PoW) to secure the network through computational power, but due to PoW’s inefficiency and high energy consumption, Ethereum transitioned to Proof-of-Stake (PoS) to improve security and scalability.

3.2. Smart Contract Security Vulnerabilities

Smart contract security vulnerabilities [32] refer to design flaws or coding errors in smart contracts deployed on the blockchain. These vulnerabilities can be exploited by malicious attackers, resulting in loss of user funds, data breaches, or other severe consequences. Given that smart contracts are immutable once deployed, ensuring their security becomes particularly critical. Common vulnerabilities in smart contracts include reentrancy attacks, timestamp dependencies, and integer overflows, among others. These vulnerabilities not only jeopardize user assets but can also trigger a chain reaction within the blockchain ecosystem, destabilizing the entire network. As smart contract applications become increasingly widespread, research into vulnerability detection technologies has become a key focus in blockchain security. Numerous tools and methods have emerged to address these concerns, providing essential safeguards for the security of smart contracts. This paper specifically focuses on detecting and analyzing reentrancy vulnerabilities, timestamp vulnerabilities, and integer overflow vulnerabilities, which are briefly described below.

3.2.1. Reentrancy Vulnerability

Reentrancy vulnerability [33] is a classic issue in smart contracts, with severe consequences if exploited by malicious attackers. One of the features of Ethereum smart contracts is the ability to make external calls between contracts. If the target of an external call is a malicious contract controlled by an attacker, the attacker can execute a malicious function during the contract call, re-enter the attacked contract, and alter its normal logic, ultimately leading to the loss of funds. The 2015 Decentralized Autonomous Organization (DAO) attack was a historic event that exposed the potential risks of reentrancy vulnerabilities. Malicious attackers managed to steal a substantial amount of Ether (approximately $60 million) by exploiting a reentrancy vulnerability in the DAO contract. This attack caused significant damage to the DAO and severely impacted the credibility of Ethereum.

3.2.2. Timestamp Vulnerability

In Ethereum, certain aspects of smart contract execution rely on block timestamps [34], and variations in these timestamps can lead to discrepancies in contract execution outcomes. The timestamp is determined by the local time of the node (miner) that processes the contract, and its value is constrained to a range of $\pm 15$ s relative to the timestamp of the preceding block. While this restriction limits the potential for arbitrary manipulation of timestamps to some extent, an attacker can still influence the contract’s execution logic by finely adjusting the timestamps, which presents a security vulnerability. Although timestamp-related vulnerabilities are not as widely discussed as reentrancy vulnerabilities (such as the one exploited in the DAO hack), their potential risks have been highlighted in real-world incidents. A notable example is the Fomo3D gambling game, in which a malicious attacker was able to extend or prematurely end the game by manipulating the timestamps through miner node coordination, thereby increasing the likelihood of winning.

3.2.3. Integer Overflow Vulnerability

Integer overflow vulnerabilities [35] are a prevalent security issue in smart contracts. The Solidity programming language imposes restrictions on integer variables, such that when an arithmetic operation exceeds the maximum or minimum allowable value, an overflow occurs. This vulnerability can be exploited by malicious actors to illicitly acquire funds or disrupt the contract’s functionality. One of the most notable cases involved the BEC contract, which failed to implement security mechanisms such as the SafeMath library to validate the boundaries of arithmetic operations. This oversight allowed for the exploitation of the integer overflow vulnerability, ultimately leading to a sharp decline in the value of the tokens, the suspension of trading on exchanges, and the destruction of tokens in the compromised contract by the project team. This incident resulted in significant economic losses and sparked widespread concern regarding the security of smart contracts.

3.3. BERT

In recent years, driven by advancements in deep learning, Google introduced a novel pre-trained language model, BERT [36], which has had a significant impact, particularly in the field of natural language processing (NLP). BERT addresses the limitations of traditional NLP methods that rely on task-specific data by pre-training on large amounts of unlabeled textual data, thereby enabling it to generalize across various tasks. Vulnerability detection in smart contracts is akin to NLP tasks that involve understanding the code. When applying BERT to smart contract vulnerability detection, the model processes not ordinary natural language but the lexemes, statements, and structures inherent in the code. The input vector for BERT is computed as follows:

$$\begin{array}{cc}\hfill \mathrm{Input}\mathrm{Embedding}& =\mathrm{Token}\mathrm{Embedding}+\mathrm{Segment}\mathrm{Embedding}\hfill \\ \hfill & +\mathrm{Position}\mathrm{Embedding}\hfill \end{array}$$

(1)

where the key code fragments of the detected smart contract are represented through a series of embeddings, Token Embedding refers to the vector representation of each lexical unit derived from the segmentation of the key code fragments. Segment Embedding is a vector representation used to differentiate distinct code fragments, capturing information about the relationships between these blocks and aiding the model in understanding the overall structure of the code. Position Embedding provides a vector representation that encodes the absolute position of each lexical unit within its sequence. By combining these three embeddings, Input Embedding is generated, which offers BERT a comprehensive representation of the smart contract.

Subsequently, the Input Embedding is processed sequentially through 12 layers of multi-head attention computations. The output from the final computation is used as the feature vector representing the smart contract.

3.4. GPT

The fundamental principle behind GPT [37] lies in generating text through autoregressive language modeling, which is integrated with the decoder component of the Transformer architecture, thereby enhancing its ability to interpret the input text. The input to GPT is represented as the sum of word embedding vectors and positional encoding vectors. Equation (2) illustrates the computation of GPT’s input, where $W_t$ denotes the lexical unit derived from word segmentation, and each lexical unit is mapped to a vector representation through word embedding. Specifically, $\mathrm{Embedding}\left(W_t\right)$ refers to the mapping of the discrete lexical unit $W_t$ into a continuous word vector space. $P{E}_{(\mathrm{pos},\mathrm{t})}$ encodes information regarding the position of the lexical unit within the sequence, and $Z_t$ represents the final input, which is the summation of the word embedding and the positional encoding.

$$\begin{array}{cc}\hfill Z_t& =\mathrm{Embedding}\left(W_t\right)+P{E}_{\left(\mathrm{pos},t\right)}\hfill \end{array}$$

(2)

The core mechanism of GPT is based on the self-attention structure, which efficiently processes sequence data. The high-dimensional matrices captured by the multi-head self-attention mechanism are subsequently passed through a feed-forward neural network to produce the final output. Equations (3)–(6) illustrate the process of calculating the core component:

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(3)

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}\left({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h\right)W^o$$

(4)

$$\mathrm{FFN}\left(x\right)=\max (0,x{W}_1+b_1)W_2+b_2$$

(5)

$$P\left(W_t\right|W_1,W_2,\dots ,W_{t-1})=\mathrm{Softmax}\left(z_t^{\prime }{W}^y+b^y\right)$$

(6)

In the formulation above, the model captures the relationships between words in the sequence through the interaction of the query Q, key K, and value V. This interaction is subsequently enhanced by multiple independent attention heads, denoted as ${\mathrm{head}}_1,\dots ,{\mathrm{head}}_h$. The outputs from all attention heads are concatenated and then mapped back to the original dimensionality via a linear transformation matrix $W^o$, as illustrated in Equation (5). Specifically, the input x is processed through two linear transformation matrices, $W_1$ and $W_2$, with the activation function $\max \left(\right)$ applied to introduce nonlinearity. This nonlinearity enables the model to capture more complex patterns within the data. Ultimately, the model predicts the next word $W_t$ based on the previously generated word sequence $W_1,W_2,\dots ,W_{t-1}$. The output is then mapped to the glossary dimension through the linear transformation matrix $W^y$, with the bias term $b^y$ incorporated into the prediction.

3.5. Attention Mechanism

The basic idea behind the attention mechanism [38] is inspired by how humans focus on specific parts of a task to process information more efficiently. The multi-head attention mechanism was introduced to address the limitations of the single-head attention mechanism, which focuses solely on information from a single perspective. In contrast, the multi-head attention mechanism enhances the model’s representational capacity by capturing a wider range of features. It achieves this by introducing multiple “heads” that process the information in parallel, with each head independently learning from different perspectives, and the results are eventually combined. The working principle is illustrated in Figure 1, where “Linear” represents the linear transformation layer, and “Scaled Dot-Product Attention” refers to the scaled dot-product attention mechanism, which constitutes a fundamental component of the self-attention mechanism. “Concat” denotes the concatenation of the outputs from each attention head that has undergone the scaled dot-product attention operation, and h represents the number of attention heads.

4. Our Approach

This paper focuses on the detection of reentrancy, timestamp, and integer overflow vulnerabilities. During the vulnerability detection process, we adopt specific data preprocessing strategies for each type of vulnerability to ensure that the input data for subsequent model training is both accurate and effective [39]. For feature extraction, we utilize BERT and GPT to extract semantic features of varying natures from the input data. To further enhance feature representation, we integrate a multi-head attention mechanism to fuse the features extracted by BERT and GPT, thereby obtaining a richer, multi-dimensional feature representation. This approach aims to improve the accuracy and robustness of vulnerability detection. The overall workflow is shown in Figure 2.

4.1. Data Processing

In the data processing phase, the first step is to remove redundant data, as vulnerabilities are not directly related to comments or whitespace characters. To ensure that the feature extraction process focuses on the code logic itself, we eliminate code comments and compress the spaces. Next, we use specific keywords associated with different vulnerabilities to extract valid data fragments. Since each vulnerability is linked to specific keywords, we have supplemented and innovated existing methods for obtaining valid data fragments related to smart contract vulnerabilities. Finally, the intercepted snippets are matched with the original text and labeled according to the corresponding categories: 0 for vulnerabilities and 1 for non-vulnerabilities.

• Reentrancy vulnerability: The fundamental operation for detecting reentrancy vulnerabilities involves checking the call.value of the contract to determine if the contract is susceptible to such vulnerabilities. An attacker can exploit the callback function of an external contract to repeat an operation before the contract’s state is updated, which can lead to unauthorized fund withdrawals. In this paper, we extend the basic reentrancy vulnerability checks by adding examinations of two other functions: call and delegatecall. While call is similar to call.value, delegatecall allows the target contract to modify the storage of the calling contract, presenting heightened security concerns. Figure 3 illustrates the code where the reentrancy vulnerability occurred. The attacker triggered the fallback function of the external contract executed by delegatecall, which caused the contract’s withdrawal operation to be executed repeatedly, resulting in the theft of approximately $60 million in Ethereum.
• Timestamp vulnerability: The fundamental operation for detecting timestamp vulnerabilities involves checking the block.timestamp of the contract to determine if the contract is susceptible to such vulnerabilities. block.timestamp is a commonly used method for obtaining timestamps, and an attacker can manipulate the timestamps of previous blocks to influence the contract’s behavior, leading to execution results that deviate from the intended outcome. In this paper, we extend the basic operation of checking for timestamp vulnerabilities by examining other time-related functions and variables in the contract, such as now and block.number. Now was the keyword used to represent timestamps in earlier versions of Solidity, and there is a potential risk if now is still used in the contract’s code. Regarding block.number, an attacker could potentially influence conditional judgments and time limits based on the block height by controlling the block’s serial number. Figure 4 presents a partial example of a timestamp vulnerability. block.number reflects the number of blocks, and by controlling the block generation speed and affecting the progress of block.number, a miner could make the duration of bidding appear shorter than it actually is, resulting in unfair competition. Similarly, with block.timestamp, if the bidding time is nearing its end, the miner could delay the bidding by adjusting the timestamp to provide themselves with more time to bid or end the bidding early to prevent others from participating.
• Integer overflow vulnerability: After reviewing a substantial body of existing literature, it is evident that research on detecting integer overflow vulnerabilities is relatively limited. Although overflow protection mechanisms have been incorporated into Solidity versions 0.8 and higher, many contracts continue to use older versions or integrate older libraries, making the detection of integer overflow vulnerabilities still relevant. Unlike reentrancy and timestamp vulnerabilities, which have obvious keyword characteristics, integer overflow vulnerabilities do not exhibit such distinctive markers. Therefore, any function in a contract that involves an arithmetic operation should be considered a key component when determining whether an integer overflow vulnerability exists. In this paper, we include all functions involving the following operations in the valid fragments being tested:
  • Focus on all instances in the contract where addition, subtraction, multiplication, division, and remainder operations are performed.
  • Focus on operations involving uint 256, where an overflow or underflow occurs if a uint 256 variable performs an operation that exceeds its maximum or minimum value.
  • Consider the use of SafeMath, a common library used to prevent integer overflow. In the absence of SafeMath, the contract is more prone to integer overflow vulnerabilities.

Figure 5 shows an example of an integer overflow vulnerability. In this contract, balance is a uint 8 variable with a maximum value of 255. In the example, when the add function is triggered, the value of balance becomes 0 instead of the expected value of 256, resulting in an integer overflow.

By implementing the appropriate data preprocessing strategies for the three vulnerabilities, we extracted the key code fragments to be detected for each smart contract. Figure 6 illustrates the smart contract source code along with its processed key code fragments. To provide a clearer understanding of the terms used in the data processing in this section, we have included them in

Table 1. Glossary.

Term	Remarks
call	Used to call functions of other contracts
call.value	Used to perform a low-level contract call with ETH
delegatecall	Used to call the code of another contract
block.number	Represents the current block number
block.timestamp	Represents the timestamp of the current block
SafeMath	Used to perform safe mathematical operations

for reference.

4.2. Models

4.2.1. BERT-Based Semantic Feature Extraction

For semantic feature extraction in the contract, this paper uses BERT as the base model. BERT is a pre-trained, deep, bidirectional transformer model with strong contextual understanding capabilities. Unlike traditional language models, which are either unidirectional or bidirectional, BERT is trained to fully capture the features of words from multiple dimensions within the context, making it highly effective for smart contract vulnerability detection. Specifically, we first tokenize the internal information of each word in the contract using the embedding layer and transform the contract information into three sets of embedding vectors, which represent the word, segment, and position features of each word in the contract. These three sets of embedding vectors are then passed through a multi-layer transformer encoder, and the output from the final hidden layer is obtained. Since each token in the BERT output has a corresponding vector, we extract the vector corresponding to the classification ([CLS]) token as the semantic representation of the entire input contract. Furthermore, the model includes several normalization layers to ensure a more stable training process at each layer, improving both the speed and performance by normalizing the output. The Dropout mechanism is applied in the normalization layer, reducing the model’s over-reliance on specific features by randomly dropping neurons in the neural network, which enhances the generalization ability of the model. In summary, the BERT model is well-suited for extracting semantic features in smart contracts. Figure 7 illustrates the basic steps of the model for semantic feature processing.

4.2.2. Generative Semantic Feature Extraction Based on GPT

In previous experiments, the implicit structure in the data was often overlooked when extracting the semantic features of smart contracts. In this paper, we use the GPT to develop a method specifically for extracting the implicit structure of the data in smart contracts. This approach transforms the original semantic features into higher-level, more abstract representations, thereby uncovering more useful features from the original data to provide a more accurate semantic understanding for the vulnerability detection task. Specifically, the text content is first encoded through the embedding layer, where each token is represented by an embedding vector that encodes the “DNA” of the number. These vectors are then passed into the decoding layer of the GPT model, which consists of two sub-layers: the multi-attention layer and the feed-forward network layer. In the multi-attention layer, the primary goal is to learn and capture the dependency relationships between various positions in the input sequence, enabling the model to better understand and generate more accurate representations of the original language. In the feed-forward network layer, we apply the ReLU activation function between the two fully connected layers to enable the model to learn more complex feature representations. Additionally, a layer normalization step is applied to each residual block input within the decoding layer to reduce variance during the pre-training process, which helps stabilize the gradient and prevents instability. To further mitigate overfitting, a dropout layer is added after the fully connected layer. Finally, we extract the generative semantic features of the entire contract by mapping the final hidden state to the dimension size of the vocabulary through the output layer. In summary, this model provides a more accurate semantic representation of the entire contract. Figure 8 illustrates the basic steps of the model’s process for generative semantic feature extraction.

4.2.3. Feature Fusion Based on Multiple Attention Mechanisms

To better combine the semantic features provided by the two models, this paper fuses these two sets of semantic features using a multi-head attention mechanism. The purpose of applying the multi-head attention mechanism is to ensure that the newly introduced generative semantic features enhance the original semantic features while also facilitating a more effective combination of the output vectors from BERT and GPT. This fusion capitalizes on the structural symmetry inherent in BERT’s bidirectional and GPT’s autoregressive approaches, aiming to refine the extraction of more accurate information from the contract. The Multihead Attention Mechanism layer contains eight independent attention heads, each consisting of three fully connected layers that compute the query (Q), key (K), and value (V) vectors in each head. Through this process, the attention mechanism automatically selects the most relevant information for an enhanced representation, which is particularly effective in addressing long-distance dependencies. The BGMF effectively merges features from different models, thereby improving the overall performance of the model. Specifically, the first step involves adding and concatenating the vectors of different semantic features and regularizing them. This not only helps control dimensionality growth but also smoothens the fusion of different vectors, making the model more robust during training. The concatenated vectors are then passed into each attention head, and the attention weights for each head are derived from the softmax function within its independent subspace. The attention weight matrix for each head is obtained. Finally, the concatenated high-dimensional vectors are fed into a fully connected layer for classification, where the final classification prediction score is produced. This model allows generative semantic features to better support the original semantic features, thus improving the ability to detect vulnerabilities in smart contracts that would be difficult to identify relying solely on the original semantic features. When the detection probabilities for multiple vulnerabilities are similar, the multi-head attention mechanism can differentiate these vulnerabilities by assigning different attention weights, ensuring that the model makes accurate predictions even when the probabilities are close. This model demonstrates the potential for innovation in semantic feature-based detection of smart contract vulnerabilities. Figure 9 illustrates the processing flow of the BGMF model.

4.3. Migration Learning

Migration learning [40] is a strategy in deep learning that allows a model to first learn from a wealth of experience gained in one task and then apply this accumulated knowledge to a related task, thereby enhancing the model’s performance. Upon reviewing the literature, we identified the scarcity of detection samples in smart contract vulnerability detection, particularly those containing vulnerabilities. The limited availability of such samples results in insufficient model training and poor performance. To address this issue, we introduce migration learning in this approach. Specifically, before training the BGMF, we pre-train foundational models such as BERT and GPT on large corpora, including BooksCorpus (containing approximately 800 million words) and English Wikipedia (containing 2.5 billion words). This pre-training enables BERT and GPT to acquire rich linguistic representations, allowing them to better understand and learn the vulnerability characteristics of smart contracts.

5. Experimental Procedure

5.1. Datasets

Smart contract vulnerability detection is a relatively niche and emerging research area, and currently, there are no official or widely recognized third-party datasets available. Therefore, for the experiments in this paper, we synthesized data from various sources. A total of 2307 contracts containing 6744 functions were collected. Among these, 1180 contracts involving vulnerabilities contain 3017 unique functions, while 1127 contracts not involving vulnerabilities contain 3837 functions. Part of the dataset is pre-labeled and includes three types of vulnerabilities: reentrancy, timestamp, and integer overflow, with a total of 1093 contracts and 1173 functions. The remaining data were selected from a larger dataset of over 12,000 Ethereum contracts, which cover eight types of vulnerabilities. From this larger dataset, we extracted the three types of vulnerabilities (reentrancy, timestamp, and integer overflow), de-weighted the data, cleaned it, and labeled it accordingly. Ultimately, 1214 contracts containing 5681 functions were selected. Before conducting the experiments, each subset of the dataset was partitioned into a training set and a validation set, with 80% of the data allocated for training and 20% reserved for validation. All data were derived from real contracts deployed on the Ethereum network, ensuring both the diversity and real-world relevance of the data. This, in turn, supports the generalization ability of the subsequent training models.

Table 2. Dataset.

Dataset Property	Description
Total Contracts	2307 contracts
Total Functions	6854 functions
Contracts with Vulnerabilities	1180 contracts containing 3017 functions
Contracts without Vulnerabilities	1127 contracts containing 3837 functions
Pre-labeled Data	1093 contracts containing 1173 functions
Unlabeled Data	1214 contracts containing 5681 functions
Data Partitioning	80% for training, 20% for validation

clearly presents the composition of the dataset used in this paper.

5.2. Experimental Environment

The experimental parameters are shown in

Table 3. Experimental parameters settings.

Parameter	Value	Remarks
Learning_rate	1 × 10−5	Initial learning rate
Epoch_num	100	Training rounds
Batch_size	4	Sample training volume for each batch
Droppout	0.1	Random inactivation
Optimizer	AdamW	Learning rate optimization algorithm

.

The experimental environment is shown in

Table 4. Experimental environment.

Hardware/Framework	Remarks
GPU	NVIDIA RTX 2080 Ti
CPU	Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50 GHz
RAM	48 G
Framework	Pytorch 2.1.2

.

5.3. Evaluation Metrics

In this experiment, we use several common evaluation metrics to assess the performance of the model in the smart contract vulnerability detection task. These metrics include accuracy, precision, recall, and F1 score.

Accuracy: This metric indicates the percentage of samples for which the model predicts the correct result.

$$\mathrm{ACC}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$$

(7)

Precision: This metric indicates the percentage of true positive categories (i.e., vulnerabilities) that are correctly predicted as positive by the model. The magnitude of the precision value reflects the false alarm rate of the detection task.

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(8)

Recall: This metric indicates the ratio of samples correctly predicted as positive among all samples that are truly positive (i.e., those containing vulnerabilities). Increasing the recall rate ensures that more vulnerabilities are detected.

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(9)

Precision and recall both influence the value of the F1 score, which serves as a comprehensive measure of the model’s performance, particularly in addressing the class imbalance problem present in the dataset of this paper.

$$F1={\displaystyle \frac{2·(\mathrm{Precision}·\mathrm{Recall})}{\mathrm{Precision}+\mathrm{Recall}}}$$

(10)

5.4. Results and Analysis

In this subsection, we compare the BGMF detection method with nine other detection methods across three types of vulnerabilities: reentrancy, timestamp, and integer overflow, and carefully analyze the experimental results. Among these ten detection methods, there are three categories of detection methods:

• Traditional detection methods: Tools such as Mythril, Manticore, Oyente [41], and Slither primarily rely on static analysis and symbolic execution. These methods are effective for detecting well-established vulnerabilities and offer high computational efficiency. However, their reliance on predefined rules and models limits their ability to detect new or intricate vulnerabilities, particularly those that deviate from known patterns.
• Deep learning approaches: Methods such as CNN, LSTM, Degree-Free Graph Convolutional Networks (DR_GCN), and Message Propagation Networks (TMP) leverage neural network models trained on extensive datasets. These approaches are adept at capturing complex patterns of vulnerabilities that might not be readily identifiable by traditional methods. They excel particularly in scenarios where the analysis requires the understanding of structured information within smart contracts or the temporal dependencies inherent in contract execution.
• Large model approaches: Recent advancements in deep learning have introduced large model-based approaches, such as BERT and SmartBugBert, which utilize pre-trained models to deeply understand the logic of contract code in context. These models are capable of identifying potential vulnerabilities by leveraging their ability to process complex contract structures and semantic relationships. The strength of large models lies in their superior generalization capabilities, allowing them to recognize previously unseen vulnerabilities and adapt to a variety of contract structures.

In this paper, the implementation of all comparative experiments is divided into three scenarios:

• When the source code is provided: If the article provides the source code, we will replicate the original code, utilize our own dataset, and preprocess the data according to the methods described in the paper. Experimental results will then be recorded based on the evaluation metrics.
• When the source code is not provided: If the article does not publicly release the source code, we will contact the authors via email to inquire whether they can provide the source code for the purpose of conducting comparative experiments.
• When no response is received: If no response is received after emailing the authors, we will independently implement the algorithm based on the descriptions in the paper and perform the comparative experiments.

Additionally, we conducted ablation experiments on the two modules involved in the BGMF model and carefully analyzed the results. Due to the data imbalance in this experiment and the need to balance between minimizing misses and reducing false positives, we focus on the F1 score and recall values.

Table 5. Detection results of reentrancy vulnerability by different methods.

Methods	Reentrancy Vulnerability
	ACC (%)	Precision (%)	Recall (%)	F1 (%)
Mythril	62.63	38.21	64.38	47.95
Manticore	66.56	49.70	49.99	49.85
Oyente	78.62	54.17	65.63	56.44
Slither	81.07	65.00	83.00	72.91
CNN	86.95	77.50	84.46	80.83
LSTM	85.53	85.12	79.21	82.04
DR_GCN	90.75	88.63	82.30	85.16
TMP	88.51	85.01	91.02	88.13
BERT	89.10	78.34	88.27	83.01
SmartBugBert	90.45	90.62	91.76	91.19
Ours	96.40	93.33	93.13	93.23

presents the comparison results between the nine smart contract vulnerability detection methods and our method for detecting reentrancy vulnerabilities. From the table, it is evident that the performance of traditional detection tools is generally low, especially with Mythril and Manticore, which have low F1 values and accuracy rates. This indicates a high false-positive rate, as both methods predict potential vulnerabilities from the source code, while some reentrancy vulnerabilities occur during external calls, state changes, or contract interactions. This dynamic behavior results in some vulnerabilities not being correctly identified. However, Slither performs better and significantly outperforms other traditional detection methods in all metrics.In comparison with traditional detection methods, the overall performance of deep learning detection methods shows significant improvement. The difference in F1 scores between the worst-performing deep learning method and the best-performing traditional method is nearly 8 percentage points, indicating that deep learning methods have clear advantages in detecting reentrancy vulnerabilities. The BERT, a detection method derived from traditional deep learning, does not perform as well, with an F1 score of only 83.01%. This is because BERT is primarily designed for language modeling and text-based tasks, requiring more pre-training and fine-tuning for smart contract detection. The BGMF model presented in this paper overcomes this limitation by pre-training on a specific dataset, achieving superior performance across all metrics. It achieves an accuracy of 96.40%, an F1 score of 93.23%, and the best balance, demonstrating the robustness and practicality of the model.

Table 6. Detection results of timestamp vulnerability by different methods.

Methods	Timestamp Vulnerability
	ACC (%)	Precision (%)	Recall (%)	F1 (%)
Mythril	52.34	53.45	33.56	40.78
Manticore	67.50	63.16	45.00	52.55
Oyente	74.29	65.47	46.08	54.18
Slither	77.45	63.38	50.20	56.78
CNN	76.06	76.20	81.00	78.53
LSTM	77.66	88.74	75.05	81.32
DR_GCN	87.39	81.50	85.37	83.39
TMP	90.35	78.68	96.09	86.52
BERT	87.10	88.89	86.48	87.67
SmartBugBert	91.05	87.84	89.04	88.43
Ours	92.90	95.48	91.89	93.65

presents the detailed detection results for timestamp vulnerability detection across different methods. Among the traditional detection tools, Mythril performs relatively poorly, with an accuracy of 52.34%, a precision of 53.45%, and a recall as low as 33.56%, resulting in an F1 score of only 40.78%. This ineffective detection is primarily due to the static analysis methods, which are unable to effectively capture the dynamic behavior of timestamp vulnerabilities. Manticore and Oyente show improved performance, especially in terms of recall. Slither performs relatively well, achieving an F1 score of 56.78%, although it still falls short of the performance of deep learning methods. Among the deep learning methods, CNN and LSTM outperform traditional detection methods, with F1 scores of 78.53% and 81.32%, respectively, indicating a better balance between precision and recall. The BERT model further enhances the overall performance compared with traditional deep learning methods, demonstrating the significant potential of large models for smart contract vulnerability detection. The method presented in this paper more effectively mines the deep semantic features of the code, achieving an F1 score of 93.56% in the timestamp vulnerability detection task. This represents a 5.22% improvement over the best existing method and far surpasses the performance of other smart contract detection methods.

Table 7. Detection results of integer overflow vulnerability by different methods.

Methods	Integer Overflow Vulnerability
	ACC (%)	Precision (%)	Recall (%)	F1 (%)
Mythril	61.17	74.64	77.64	75.82
Manticore	72.45	66.35	31.22	42.46
Oyente	73.12	72.30	65.32	69.57
Slither	75.12	76.01	80.01	77.97
CNN	80.53	61.52	61.43	61.47
LSTM	85.43	86.15	84.42	85.28
DR_GCN	86.54	87.23	85.66	86.43
TMP	87.66	88.12	89.98	89.01
BERT	88.12	83.00	81.13	82.05
SmartBugBert	90.12	89.50	90.10	89.80
Ours	94.50	98.50	85.70	91.66

presents the detection results for integer overflow vulnerability across ten different models. From the data, it is evident that deep learning methods outperform traditional detection tools, with the method proposed in this paper achieving the most outstanding results. It demonstrates an accuracy rate close to 100%, efficiently detecting vulnerabilities with almost no false positives. Furthermore, previous experiments on integer overflow vulnerability detection have been limited, and our work addresses these gaps, significantly advancing the field of integer overflow vulnerability detection.

The comparative experimental data on reentrancy, timestamp, and integer overflow vulnerabilities presented above indicate that most traditional tools still exhibit certain limitations compared with deep learning approaches. Our method combines the advantages of previous research, addresses its shortcomings, and introduces innovations that improve the original model. As a result, it outperforms the best-performing previous methods for each of the vulnerabilities discussed in this paper. The three graphs in Figure 10 further visualize the detection results of the ten detection methods for reentrancy, timestamp, and integer overflow vulnerabilities. It is clearly evident that our method outperforms all other methods. (In the figures, (a–j) represent Mythril, Manticore, Oyente, Slither, CNN, LSTM, DR_GCN, TMP, BERT, and our method, respectively).

To demonstrate the effectiveness of the improved method proposed in this paper, we conducted ablation experiments. This paper conducts a comparative analysis of the following models:

• Old Model: This model represents the original detection method, serving as a baseline for performance comparison. Its performance reflects the detection capability without the inclusion of the innovative modules proposed in this paper.
• NoGPT: This model removes the generative semantic feature extraction module introduced in this paper while retaining the multi-head attention mechanism module. It is designed to evaluate the performance of smart contract vulnerability detection using a single semantic feature, where only the original semantics are retained.
• NoMHA: This model removes the multi-head attention mechanism module proposed in this paper while keeping the generative semantic feature extraction module. It aims to assess the performance of smart contract vulnerability detection without the feature fusion step.
• Ours: This model represents the fully improved approach proposed in this paper, incorporating both the generative semantic feature extraction module and the multi-head attention mechanism module. Its performance demonstrates the optimal detection capability resulting from the synergy of all the proposed enhancements.

Table 8. Improved detection results of our model for reentrancy vulnerability.

Model	Reentrancy Vulnerability
	ACC (%)	Precision (%)	Recall (%)	F1 (%)
Old Model	89.10	78.34	88.27	83.01
NoGPT	90.90	77.80	92.87	84.80
NoMHA	94.51	87.50	93.33	90.32
Ours	96.40	93.13	93.33	93.23

and

Table 9. Improved detection results of our model for timestamp vulnerability.

Model	Timestamp Vulnerability
	ACC (%)	Precision (%)	Recall (%)	F1 (%)
Old Model	87.10	86.48	88.88	87.67
NoGPT	90.00	89.18	91.66	90.41
NoMHA	90.80	87.17	94.44	90.66
Ours	92.90	91.89	95.48	93.65

present the performance of each module of the improved method compared with the original detection method. In the detection of reentrancy vulnerabilities, our improved method shows significant improvements across all metrics, with the F1 score being 10.22% higher than that of the original method. As shown in the data in Table 9, both the generative semantic extraction module and the multi-head attention mechanism module enhance the original model, with the generative semantic module providing the most noticeable improvement. This clearly demonstrates the effectiveness of the proposed improvements. For timestamp vulnerabilities, our improved method achieves a 5.89% increase in F1 score compared with the original method, indicating that the added modules are more effective in detecting timestamp vulnerabilities. Figure 11 visualizes the effectiveness of the improved modules.

6. Conclusions

In this paper, we proposed a smart contract vulnerability detection method based on hybrid semantic features and a multi-head attention mechanism. Our method first enhances the effectiveness of smart contract detection fragments during the data preprocessing stage. Secondly, we introduce generative semantic feature extraction in addition to the original semantic feature extraction, which makes the extracted semantic features more comprehensive. Finally, the inclusion of the multi-head attention mechanism enables the capture of hybrid semantic features from different perspectives, thereby improving the accuracy of vulnerability detection. The experimental results presented in the previous section demonstrate that the proposed BGMF model is effective for smart contract vulnerability detection, providing a new perspective and reference approach for future research in this field. Although the method designed in this paper outperforms the current best solution, there are still limitations in the experiments. While migration learning alleviates the issue of limited labeled samples, model performance is still constrained by this limitation. Therefore, future work should focus on addressing the problem of limited labeled samples and constructing a more comprehensive dataset to improve the generality of the experimental results and enhance the model’s generalization ability. While the use of large models for smart contract vulnerability detection is promising, training such models requires significant computational resources. Future efforts should also aim to optimize the model to reduce computational resource demands without compromising performance. Finally, the proposed method demonstrates strong potential for practical deployment. It can be integrated into existing smart contract automated auditing platforms or blockchain security frameworks to assist developers and security analysts in performing more efficient, accurate, and automated vulnerability detection, thereby enhancing the overall security and reliability of blockchain systems.