RTMS: A Smart Contract Vulnerability Detection Method Based on Feature Fusion and Vulnerability Correlations

Abstract

Smart contracts are at the core of blockchain technology, but the cost of fixing their security vulnerabilities is high, making pre-deployment vulnerability detection crucial. Existing methods rely on fixed rules, which have limitations in accuracy and scalability, and their efficiency decreases with the complexity of the rules. Neural-network-based methods can identify some vulnerabilities but are inefficient in multi-vulnerability scenarios and depend on source code. To address these issues, we propose a multi-vulnerability-based smart contract detection method called RTMS. RTMS takes bytecode as input, disassembles it into opcodes, uses the gas consumed by the contract for data slicing, and extends the length of input opcodes through a layered structure. It employs a weighted binary cross-entropy (BCE) function to handle data imbalance and combines channel-sequence attention mechanisms to extract vulnerability correlation features. By using transfer learning, it reduces training parameters and computational costs. Our RTMS model can detect multiple vulnerabilities simultaneously, enhancing detection accuracy and efficiency. In experiments with 100,000 real contract samples, the model achieved a Jaccard coefficient of 0.9312, a Hamming loss of 0.0211, and an F1 score that improved by about 11 percentage points compared to existing models, demonstrating its superiority and stability.

1. Introduction

Blockchain technology [1] is essentially a distributed shared transaction ledger, and all nodes in the blockchain network maintain the network with the help of consensus protocols [2]. Blockchain technology has characteristics such as decentralization, tamper-proofing, irreversibility, and traceability. This technology is being researched across multiple industries and has a wide range of applications, such as digital identity [3], cross-border banking [4], and insurance [5].

Ethereum [6] is a well-known programmable blockchain platform that allows us to use the Ethereum Virtual Machine to execute code included in smart contracts. The concept of smart contracts was proposed by Szabo [7] as early as 1994, and it was not until the advent of blockchain technology that a reliable execution environment and application foundation were provided. Smart contracts are essentially computer programs running on the blockchain, written in Turing-complete languages, and can be automatically executed within the blockchain network.

In the past, many traditional vulnerability detection techniques have been used to identify vulnerabilities in smart contracts, including formal verification methods [8], symbolic execution methods [9], fuzzing detection methods [10], and intermediate representation methods [11], etc. For more related research, refer to Section 2.2. However, these methods usually rely on rules set by security experts and have a large memory overhead and are time-consuming during detection.

With the rapid development of deep learning and machine learning technologies, they have shown great potential in the field of smart contract vulnerability detection. These advanced algorithms can automatically extract key features from massive data and perform complex detection tasks with high efficiency and accuracy [12]. Li et al. [13] proposed a method that combines opcodes and LSTM networks, which shows certain performance improvements compared to traditional detection methods. Liu et al. [14] proposed a detection method called DL4SC, which combines transformer–encoder and CNN, with an accuracy rate and F1 score of 95.29% and 95.68%, respectively. For more related research, refer to Section 2.3. However, current machine-learning-based smart contract vulnerability detection methods still face some challenges:

• Training and detection processes typically rely on source code or its decompiled form. However, in reality, the source code of most smart contracts is not publicly available, and what we can mainly obtain is their bytecode. This situation limits the application scope and effectiveness of traditional methods [15].
• Many existing detection tools can only detect a single type of vulnerability [16]. However, in actual situations, smart contracts often contain multiple vulnerabilities, and detecting these vulnerabilities one by one can significantly increase the time and computational costs of the detection process, thereby affecting detection efficiency.
• Many existing text-based detection methods lack effective and independent slicing strategies and cannot extract features from multiple scales.

To address the aforementioned issues, we have introduced an innovative detection method called RTMS for identifying multiple smart contract vulnerabilities. Built upon a multi-layered network framework, RTMS is capable of accurately detecting vulnerabilities such as re-entrancy, permission control, integer overflow, and insecure calling by integrating the associated features between vulnerabilities, thereby significantly enhancing detection accuracy. The system excels at managing complex vulnerability scenarios within a unified model framework and simplifies the detection process by optimizing the use of time and computational resources.

The main contributions of this paper can be summarized as follows:

• Utilizing opcode as input: We have chosen opcode as the input format. Compared to bytecode, opcode offers stronger expressive power and incurs less information loss during the decompilation process. Moreover, compared to Solidity code, opcode is more accessible and facilitates simpler computation of gas values.
• Proposing a gas-consumption-based slicing method: We have designed a slicing method based on gas consumption. By prioritizing the analysis of the most logically complex opcode segments, this method enhances the efficiency and accuracy of feature extraction.
• Employing a hierarchical network structure: We have adopted a hierarchical network structure to expand the range of acceptable input lengths and to extract features from multiple dimensions. This significantly improves the efficiency of feature extraction and the generalization capability of the model.
• Introducing channel–sequence attention mechanisms: By incorporating channel-sequence attention mechanisms, we enable the model to learn correlations between vulnerabilities and refine initial classification results. This approach markedly enhances the performance and accuracy of multi-vulnerability detection.

The rest of this paper is organized as follows: Section 2 introduces background and related work. Section 3 introduces the overall framework, data preprocessing methods, and the issue of data imbalance. Section 4 provides a detailed exposition of the structural principles of each sub-module. Section 5 presents the experimental results. Section 6 provides conclusion and future outlook.

2. Background

2.1. Classification of Smart Contract Vulnerabilities

RTMS is dedicated to the detection of contracts crafted in Solidity, the premier Turing-complete programming language for smart contracts on the Ethereum platform. In an effort to thoroughly uncover latent security threats, a meticulous selection of four prevalent smart contract vulnerabilities has been subjected to rigorous analysis and scrutiny: re-entrancy attacks [17], deficiencies in permission control [18], integer overflow issues [19], and the vulnerabilities of insecure calling protocols [20].

Re-entrancy attacks leverage the contract’s callback functions, enabling repeated function invocations during execution, which can lead to illicit gains for malefactors.

Permission control deficiencies stem from poor management of access privileges, potentially enabling unauthorized interference, manipulation of contract assets, or even theft of funds.

Integer overflow issues, a consequence of improper integer arithmetic within smart contracts, can result in erroneous computations, miscalculation of assets, and exploitation of logical flaws, posing a significant risk to the integrity of blockchain networks.

Vulnerabilities in secure calling protocols emerge during inter-contract function invocations; mismatches in function and parameter types may trigger the default invocation of a contract’s fallback function, presenting a fresh vector for re-entrancy attacks and potentially facilitating the execution of malicious code or the loss of funds.

These four categories of vulnerabilities are not only exceedingly perilous but also emblematic of the broader security challenges within the smart contract ecosystem. Ample samples of these vulnerabilities exist, providing a rich dataset for training purposes and thereby enhancing the precision of detection mechanisms.

2.2. Method of Smart Contract Vulnerability Detection

Smart contracts, as a core component of blockchain technology, are not only crucial for user experience but also profoundly impact the credit system of blockchain smart contracts. With the increasing popularity of smart contract applications, the issue of vulnerability detection has become a focus in the industry.

In the field of traditional software security detection, KEVM [21] serves as a formal analysis framework that constructs an executable formal specification based on EVM bytecode using the K framework. Jiang et al. [22] proposed the first cross-platform smart contract vulnerability detection framework based on Wasm bytecode using symbolic execution techniques, capable of detecting smart contracts in EOSIO and Ethereum. CrossFuzz [23] leverages fuzz testing methods, optimizing the mutation strategy of transaction sequences based on the data flow information between contracts, thereby enhancing the performance of fuzz testing. Contractsentry [24], based on intermediate representation, extracts key information by constructing contract information and intermediate representation, and builds comprehensive vulnerability detection rules, improving the accuracy of detection. However, these methods typically rely on rules set by security experts and have issues such as large memory overhead and long detection times. With the continuous advancement of deep learning technology, its potential in the field of smart contract vulnerability detection is also increasingly being demonstrated.

2.3. Related Work

In the research field where deep learning is applied to vulnerability detection, Qian [25] et al. proposed a method that combines Bi-LSTM with attention mechanisms for executing vulnerability detection tasks, significantly enhancing performance compared to unidirectional LSTM networks. Wu [26] et al. introduced a method that integrates hybrid attention mechanisms with Bi-GRU networks, which uses fewer parameters, simpler structures, and faster training speeds, matching the performance of Bi-LSTM. Both methods highlight the crucial role of bidirectional networks and attention mechanisms in vulnerability detection tasks.

Compared to numerous RNNs and their variants, the transformer–encoder has a distinct advantage in handling long-term dependencies and complex sequential data [27]. For instance, DL4SC [14] utilizes a combination of transformer–encoder and CNN networks to detect security vulnerabilities in smart contracts, further leveraging the potential of multi-attention mechanisms and improving detection performance. However, this method only extracts features from a single level. Guo [28] et al. proposed the multi-scale semantic encoder detection method MEVD, which comprehensively extracts features through local and global dimensions, thereby enhancing detection performance. Nonetheless, this method’s slicing approach to source code input, relying on vulnerability-related functions, is still limited by rules set by humans.

Tong [29] et al. proposed a multi-vulnerability detection method for smart contracts that combines the SecBERT model with WisdomNet, improving model classification accuracy by excluding difficult-to-classify samples. The SecBERT model employs a bidirectional multi-layer transformer–encoder structure, surpassing a single transformer–encoder in semantic understanding capabilities. However, this method only uses a parallel approach when dealing with multiple classification results, not fully considering the interrelationships between vulnerabilities, and feature extraction is conducted through a single channel. The EDSCVD [30] method combines BERT models, Bi-LSTM, and dual-channel CNN modules for feature extraction to obtain more comprehensive information. Despite this, the method does not expand the input scope of the BERT model, leading to some information loss, and the slicing method still relies on predefined vulnerability-related functions.

To address the aforementioned issues, RTMS employs a hierarchical network structure that allows for input expansion while effectively extracting information, using the CodeBERT model to extract sliced semantic features and bidirectional AGM (attention-based gated memory network) to extract global features. The CodeBERT model, compared to the SecBERT model, has a stronger code understanding capability as it was pre-trained with more code data. By utilizing the gas values consumed by instruction execution, the method selects the most logic-rich opcode segments. This method takes opcode as input, effectively solving the problem of insufficient data sources, and opcode, compared to bytecode, has stronger expressiveness. Compared to Solidity code, bytecode retains more information when disassembled into opcode, reducing information loss. RTMS can detect multiple vulnerabilities simultaneously and fully consider the interrelationships between vulnerabilities, significantly improving the overall Jaccard score by correcting initial classification results.

3. Method

3.1. Overview

As shown in Figure 1, RTMS employs a layered architecture, utilizing multiple CodeBERT models in parallel for local feature extraction, which can expand the range of scalable input and then rely on a bidirectional AGM network to achieve feature extraction at the global level. RTMS first standardizes the collected smart contracts, whether in the form of Solidity or bytecode, into opcode form. Compared to decompiling bytecode back into Solidity code, this method not only fully expresses the code logic with a more concise amount of text but also retains the original information of the bytecode to the greatest extent possible. Subsequently, the sequence of opcodes undergoes simplification processing. Considering that the EVM has over 140 instructions, some of which are logically weak and infrequently used, leading to scattered features. By merging these instructions, feature recognition can be enhanced, allowing the CodeBERT model to more effectively understand the semantics of the code. Next, the data are sliced and divided into several pieces according to a length of 512 tokens, which are then placed into containers. The data in the containers are input into the CodeBERT model for feature extraction, and by accumulating the outputs of the last four hidden layers, local feature vectors are obtained. These feature vectors are then input into the DFE module to map to the hidden layers, and finally, a bidirectional AGM network is used to integrate local features, extract global features, and complete preliminary classification. The preliminary classification results are input into the RTBlock module to extract correlations and refine the classification results. By integrating vulnerability correlations, the accuracy and stability of the classification results can be improved.

3.2. Data Preprocessing

The Ethereum smart contract platform has over 140 opcodes, many of which are functionally similar. To enhance the efficiency of feature extraction, these similar opcodes are merged to increase the concentration of features in the opcode sequence. This merging operation helps simplify the model’s input, making it more efficient.

Considering the input limit of the CodeBERT model is 512 tokens, when the opcode sequence exceeds this limit, it is necessary to process the input sequence appropriately. Within the opcode sequence, we observe consecutively repeated opcodes and logically similar opcodes. By merging these into one, we can effectively reduce the length of the sequence to fit the model’s input limit.

However, even after merging, the length of most opcode sequences still exceeds the input range, and a shard selection strategy will be adopted subsequently. Specifically, we combine the gas consumption table from the Ethereum Yellow Paper [31], and by analyzing the gas consumption of each shard during execution, we assess its logical complexity and importance. This method allows us to identify and select the most valuable shards for further processing, ensuring that the model focuses on the parts of the smart contract that are most critical for security and functionality. In this way, while maintaining the model’s input limit, we maximize the capabilities of the CodeBERT model to achieve more accurate and efficient smart contract vulnerability detection.

3.2.1. Opcode Simplification

Use the pyevmasm library to disassemble the collected bytecode into opcodes within the EVM (Ethereum Virtual Machine) and then simplify the opcodes according to the following steps. The effect of the simplification is shown in Figure 2.

• Replace specific addresses in the opcodes with the symbol “adr”. Since the majority of vulnerabilities are related to code logic rather than specific address values, this transformation turns irrelevant information into relevant information.
• Merge similar instructions to increase their frequency, which helps concentrate features and mitigates the issue of overly long input data. For example, instructions like PUSH1–PUSH32, where the number following PUSH indicates the byte size of the PUSH, can be unified into a single PUSH instruction. Additionally, instructions that are infrequently used, have low logic, and have similar functionality, such as TIMESTAMP (to obtain the block timestamp), NUMBER (to obtain the block number), and GASLIMIT (to obtain the block’s gas limit), can be merged into a single BLOCK INFO instruction, which means obtaining block information. To facilitate subsequent operations, remove instructions that consume too much or too little gas within the same group and ultimately, select a group with the same gas consumption. The specific merged instructions are shown in Table 1, with all other instructions remaining unchanged.
• Remove consecutively appearing instructions. There are many instances of the same instruction appearing consecutively in the opcodes. For example, when the ADDRESS instruction (to get the current account address) appears consecutively, it represents only one logic. In such cases, the repeated instructions should be deleted to reduce the weight of that instruction.

Table 1. Opcode simplification table. Created by the authors.

Simplified	Original
PUSH	PUSH1-PUSH32
LOG	LOG0-LOG4
DUP	DUP1-DUP16
SWAP	SWAP1-SWAP16
BlOCK INFO	COINBASE, TIMESTAMP, NUMBER, GASLIMIT
CONTRACT INFO	GAS, ADDRESS, BALANCE, GASPRICE
SYSTEM INFO	ORIGIN, CODESIZE, EXTCODESIZE, RETURNDATASIZE
CALL	CALLER, CALLVALUE, CALLDATALOAD, CALLDATASIZE
LT	LT, SLT
GT	GT, SGT

Table 1. Opcode simplification table. Created by the authors.

Simplified	Original
PUSH	PUSH1-PUSH32
LOG	LOG0-LOG4
DUP	DUP1-DUP16
SWAP	SWAP1-SWAP16
BlOCK INFO	COINBASE, TIMESTAMP, NUMBER, GASLIMIT
CONTRACT INFO	GAS, ADDRESS, BALANCE, GASPRICE
SYSTEM INFO	ORIGIN, CODESIZE, EXTCODESIZE, RETURNDATASIZE
CALL	CALLER, CALLVALUE, CALLDATALOAD, CALLDATASIZE
LT	LT, SLT
GT	GT, SGT

3.2.2. Opcode Slicing

The CodeBERT model has an upper limit of 512 tokens when processing inputs. After the expansion of the local encoder, RTMS can handle input lengths of n × 512 tokens, where n represents the number of containers. Despite reducing the length through opcode sequence simplification, most opcode samples still exceed the processing limit of the CodeBERT model. To address this issue, RTMS adopts an innovative strategy that differs from simply truncating the opcode sequence. Instead, it implements a gas consumption-based block selection strategy according to the gas values required for each instruction specified in the Ethereum Yellow Paper.

Algorithm 1. Opcode merging and shard selection algorithm.

Algorithm 1: Algorithm for calculating the maximum gas consumption shard. Created by the authors.

Furthermore, using a gas-consumption-value-based slicing strategy can also reduce the interference of redundant information on vulnerability detection.

Specifically, RTMS first divides the opcode sequence into multiple blocks of length n × 512 tokens to ensure the coherence of code logic. Subsequently, it calculates the gas consumption of each block using the gas consumption table from the Yellow Paper. Among these blocks, the block with the largest gas consumption usually represents the most logically complex part of the contract and is therefore prioritized for further analysis. This gas- consumption-value-based block selection strategy not only improves the efficiency of feature extraction but also ensures that the model focuses on the most critical parts of smart contract security and functionality. For detailed algorithmic procedures, please refer to the pseudocode in Algorithm 1.

3.3. Data Imbalance Problem

In the field of machine learning, particularly in classification tasks, data imbalance is a common issue that refers to the significant difference in the number of data samples across different categories [32]. This disparity can cause models to favor predicting the majority class while neglecting the characteristics and patterns of the minority class, thereby affecting the model’s generalization ability and accuracy. The uneven distribution of sample sizes in the dataset is the main cause of this problem.

To address the data imbalance issue, RTMS employs a strategy that incorporates a weighted binary cross-entropy (BCE) function. The BCE function is one of the widely used loss functions for handling multi-class classification problems. By introducing a weighting mechanism, RTMS assigns different weights to each type of vulnerability based on the number of samples, giving more attention to those with fewer samples. This approach more effectively balances the model’s learning capabilities across different categories.

$$L\left(x,y\right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{j=1}^C\left[x_{ij}\mathrm{l}\mathrm{o}\mathrm{g}\left(y_{ij}\right)+\left(1-x_{ij}\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-y_{ij}\right)\right]$$

(1)

Equation (1) represents the binary cross-entropy (BCE) function, where $x_{ij}$ denotes the true value of the jth label for the ith sample, and $y_{ij}$ denotes the corresponding predicted probability value by the model. $N$ is the number of samples, and $C$ is the number of labels. To tackle the data imbalance problem, higher weights are assigned to the vulnerability types with fewer instances, leading to the proposal of a modified weighted BCE function.

$$L_{\omega }\left(x,y\right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{j=1}^C{\omega }_j\left[x_{ij}\mathrm{l}\mathrm{o}\mathrm{g}\left(y_{ij}\right)+\left(1-x_{ij}\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-y_{ij}\right)\right]$$

(2)

$${\eta }_j={\displaystyle \frac{1}{n_j}}$$

(3)

$$w_j={\displaystyle \frac{{\eta }_j}{{\sum }_{p=1}^C{\eta }_p}}$$

(4)

The improved weighted binary cross-entropy (BCE) formula is as follows, where ${\omega }_j$ is the weight for the j-th class, calculated based on the proportion of samples in the total dataset. ${\eta }_j$ represents the base weight for class j, and $n_j$ represents the number of samples for class j. Equation (3) can be adjusted based on the number of samples in each class to achieve an effect where the higher the number of samples, the lower the weight value. To ensure that the sum of the elements in the weight matrix is 1, the base weight matrix is normalized, and the result is $w_j$, which is the normalized weight.

3.4. Transfer Learning

In recent years, transfer learning has been widely applied in the field of software engineering, especially in learning semantic features of code. To enhance performance, many studies pre-train models using a large number of source code snippets. Yuan et al. [33] utilized transfer learning techniques to automatically learn features from the multimodality of Solidity source code, which can be extended across platforms. RTMS employs a layered network structure of local-global encoders, which not only extends the input range but also extracts higher-dimensional features. In the local encoder module, to reduce training overhead, RTMS adopts a pre-training approach. First, the CodeBERT model is trained on an opcode dataset for feature extraction, then it is frozen, and multiple instances are connected in parallel to form a local feature encoder. This approach ensures efficient feature extraction while reducing computational costs, allowing for more CodeBERT instances to be connected in parallel under the same hardware conditions, thereby increasing the size of input slices.

4. Structure

4.1. CodeBERT

RoBERTa model [34], as a variant of the BERT model, has been trained on a larger dataset and employs a dynamic masking mechanism that reduces the probability of repeated masking, allowing the model to adapt to different masking strategies and learn richer linguistic representations. In terms of model architecture, RoBERTa maintains the multi-layer bidirectional transformer–encoder structure of the BERT model, where each unit can absorb information from the current token and the tokens on both its left and right sides, rather than simply reversing the text and inputting it again.

CodeBERT [35] is a pre-trained model proposed by Microsoft, based on the RoBERTa model architecture, specifically designed for understanding and generating programming code, as shown in Figure 3. The CodeBERT model further optimizes this foundation, with pre-training specifically tailored to the characteristics of programming languages, making it more effective in handling code-related tasks such as smart contracts. The pre-training of the CodeBERT model includes not only open-source code repositories but also related documentation, enabling the model to capture rich semantic information of code, providing strong support for subsequent feature extraction and vulnerability detection. Compared to RoBERTa, CodeBERT has a significant advantage in capturing semantic information of code as it is pre-trained on a larger scale of code data. The core of the CodeBERT model lies in its ability to deeply understand the complexity of smart contract code, thereby improving the accuracy of classification in vulnerability detection tasks.

4.2. Detail Future Encoder

The DFE (detail future encoder) module, as a crucial component of the RTMS architecture, is depicted in Figure 4. It utilizes a deep separable convolution structure that integrates dilated convolutions, depthwise convolutions, and pointwise convolutions, along with gating mechanisms, to achieve bidirectional extraction and fusion of local semantic features. This module enhances feature representation by adjusting the features of the reverse input based on the forward input and summing them with the original forward input.

The gating mechanism plays a crucial role in the DFE, acting like a valve in neural networks, allowing the network to selectively pass or discard information at each time step, thereby controlling the flow of information and making the network more flexible and dynamic in processing data. In the forward path of the DFE, the GELU (Gaussian error linear unit) activation function is used to activate the output, which is then element-wise multiplied with the output from the reverse path. This adjusts the strength of the reverse path, achieving bidirectional feature fusion.

The introduction of dilated convolutions, by setting different dilation rates, allows a convolution kernel of the same size to have a larger receptive field, effectively capturing structural features of different scales in the code. The combination of depthwise convolutions and pointwise convolutions forms the deep separable convolution structure, where depthwise convolutions are responsible for capturing local features, and pointwise convolutions are responsible for mixing channel information. This structure maintains the effectiveness of feature extraction while reducing computational load. The mathematical expression of DFE is as follows:

$$\overrightarrow{X}=GELU\left(Pointwise\left(Depthwise\left(Dilated\left(X\right)\right)\right)\right)$$

(5)

$$\overleftarrow{X}=Pointwise\left(Depthwise\left(Dilated\left(Reverse\left(X\right)\right)\right)\right)$$

(6)

$$Gate\left(X\right)=\overrightarrow{X}\odot \overleftarrow{X}$$

(7)

$$\widehat{X}=LN\left(W_p\left(Gate\left(X\right)\right)\right)+X$$

(8)

In the equation, $W_p^{\left(·\right)}$ represents the pointwise convolution layer, $W_e^{\left(·\right)}$ represents the depthwise convolution layer, $W_i^{\left(·\right)}$ represents the dilated convolution layer, $X$ is the input vector, $\mathrm{R}\mathrm{e}verse$ is the reversed input vector, $\widehat{X}$ is the output vector, $\odot$ denotes element-wise multiplication, and $LN$ stands for the layer normalization layer. Through this structure, the DFE module not only improves the efficiency of feature extraction but also enhances the model’s understanding of code semantics, providing a solid foundation for the high-performance vulnerability detection of the RTMS model. The use of the DFE structure can integrate bidirectional semantic features, mapping the feature vectors extracted by CodeBERT to the hidden layer.

4.3. Attention-Based Gated Memory Network

The AGM module (attention-based gated memory network), as a key component in the RTMS architecture, is designed to capture long-range dependencies across containers and model global information, as shown in Figure 5. This module utilizes a self-attention mechanism to select important information and integrate it into memory, thereby achieving an in-depth understanding and representation of global features within smart contracts.

The self-attention mechanism endows the AGM module with flexibility in processing data. It learns the weights of the occupied elements to identify and emphasize key features. This mechanism not only enhances the model’s sensitivity to local features but also strengthens its grasp of the global context.

The computation process of the AGM module involves a memory vector m and a gating vector g. The memory vector is used to record important information from local representations, while the gating vector controls which information needs to be retained or forgotten during training. The recursive architecture of the AGM module allows it to update the memory content at each time step, reflecting the most recent learning state.

Taking the output of the DFE module as $\left[t_1,t_2,...,t_n\right]$, which represents the local representation of the ith container. With the aid of the self-attention mechanism, we learn the weights of the occupied elements.

$$e_i=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(W_e{t}_i+b_e\right)$$

(9)

$${\alpha }_i={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(e_i\right)}{{\sum }_{j=1}^{\left|D\right|}\mathrm{e}\mathrm{x}\mathrm{p}\left(e_j\right)}}$$

(10)

Where $W_e$ and $b_e$ represent the weight matrix and bias term of the attention layer, respectively. The final result ${\alpha }_i$ is the attention score for the ith local vector, which is used as a weight.

$$g_i=sigmoid\left(W_g{t}_i+U_g{m}_{i-1}\right)$$

(11)

$${\widehat{m}}_i=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(W_h{t}_i+g_i\odot \left(U_h{m}_{i-1}\right)\right)$$

(12)

$$m_i=\left(1-{\alpha }_i\right)\odot m_{i-1}+{\alpha }_i\odot {\widehat{m}}_i$$

(13)

$$\mathrm{o}=\left[{\overrightarrow{\mathrm{m}}}_{\mathrm{v}};{\overleftarrow{\mathrm{m}}}_1\right]$$

(14)

In which $W_g$, $U_g$, $W_h$, and $U_h$ are trainable weight matrices, ${\widehat{m}}_i$ is the candidate state of the current unit, and $m_{i-1}$ is the output result of the previous unit. The gate $g_i$ determines how much of the previous information from the upper layer needs to be activated, ${\alpha }_i$ determines how much of the previous information needs to be retained and how much information from the candidate state needs to be remembered. To fully extract bidirectional global information, two layers of AGM networks are used, one to extract global features in the forward direction and the other in the reverse direction, concatenating the last unit of the forward layer ${\overrightarrow{m}}_v$ with the first unit of the reverse layer ${\overleftarrow{m}}_1$ to form the final output $o$. The sigmoid function is used to generate independent preliminary classification results. This design enables the AGM module to effectively integrate information from different directions, providing the model with a more comprehensive global feature representation.

4.4. RTBlock

RTBlock is an innovative component in the RTMS architecture, as shown in Figure 6, specifically designed to refine initial classification results by integrating vulnerability correlations, thereby enhancing the overall precision and stability of detection. This module combines channel attention and sequence attention mechanisms to learn the complex correlations between vulnerabilities and effectively capture dependent features across channels and sequences.

The channel attention submodule utilizes a multi-layer perceptron (MLP) to amplify cross-dimensional channel-sequence dependencies, while the sequence attention mechanism integrates channel features and promotes efficient information flow through rearrangement between channels, which helps to capture inter-channel vulnerability correlation features. Together, they accomplish the learning of vulnerability correlations. The formula for RTBlock is as follows:

$$X_{channel}=Sigmoid\left(Linear\left(GELU\left(Linear\left(X\right)\right)\right)\right)$$

(15)

$$X_{sequence}=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(Layer\left(Conv\left(GELU\left(Layer\left(Conv\left(X\odot X_{channel}\right)\right)\right)\right)\right)\right)$$

(16)

$$\widehat{X}=X+X_{serial}$$

(17)

Let $X$ be the original classification result, $X_{channel}$ be the output of the channel attention module, and $X_{sequence}$ be the output of the sequence attention module. The sequence attention submodule uses $X\odot X_{channel}$ as input, allowing it to integrate features learned from the previous channel attention layer. The use of the tanh activation function ensures that the output $X_{sequence}$ is distributed within the range (−1, 1), which can uniformly adjust the initial classification results $X$ positively or negatively, yielding the final classification sequence $\widehat{X}$. The design of RTBlock enables the model not only to learn local features but also to optimize global features by considering the correlations between vulnerabilities, thereby achieving higher accuracy and robustness in multi-vulnerability detection tasks.

5. Experiments

5.1. Datasets

The dataset originates from the slither-audited-smart-contracts dataset [36] on Hugging Face, which contains 106,474 bytecodes of smart contracts as well as 32,481 bytecodes of smart contracts crawled from Ethereum. The pyevmasm library is used for disassembly into opcode sequences, and the Slither [37] tool is used for automatic labeling.

To avoid overfitting, the dataset is divided into training, testing, and validation sets in an 8:1:1 ratio. This tool has marked out 38 detailed types of vulnerabilities. To further reduce data imbalance, similar vulnerabilities among the 38 are merged. The final dataset includes vulnerability types such as access control, integer overflow, re-entrancy, and insecure calls. The number of samples in each category of the dataset is shown in Figure 7.

5.2. Experimental Settings

All experiments were conducted on a machine equipped with 48 GB of RAM and two RTX 3080 GPUs with 20 GB of memory each. The methods were implemented using PyTorch. For details on the parameters used, please refer to

Table 2. Parameters of the MEVD model. Created by the authors.

Parameter	Value
Optimizer	Adam
Learning rate	2 × 10−5
Dropout rate	0.3
Batch size	16
Epochs	30

.

5.3. Evaluation Metrics

Precision, recall, and F1 score were used as the evaluation criteria. Precision is the proportion of correctly predicted vulnerabilities out of the total number of predicted vulnerabilities. Recall is the proportion of correctly predicted vulnerabilities out of the total number of actual vulnerabilities. The F1 score strikes a balance between the two and is the harmonic mean of precision and recall. The specific calculations are as follows:

$$Precision={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{|X_i\cap Y_i|}{\left|X_i\right|}}$$

(18)

$$recall={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{|X_i\cap Y_i|}{\left|Y_i\right|}}$$

(19)

$$F1=2\cdot {\displaystyle \frac{Precision\cdot Recall}{Precision+Recall}}$$

(20)

Let $X_i$ represent the predicted vulnerability sequence for the ith sample and $Y_i$ represent the true vulnerability sequence. n is the total number of samples in the dataset. In multi-label classification problems, the focus is on the degree of match between the predicted label set and the true label set [38]. In addition to these, the Jaccard similarity coefficient and Hamming loss are used to evaluate the overall performance of the model’s predictions. The Jaccard similarity coefficient measures the similarity between two sets, defined as the size of the intersection of the two sets divided by the size of their union. Hamming loss measures the proportion of labels that are inconsistent between the predicted labels and the true labels out of the total number of labels. The specific calculation formulas are as follows, using the average of the samples to represent the overall metrics. Both are commonly used evaluation metrics in multi-label classification problems.

$$Jaccard\left(X,Y\right)={\displaystyle \frac{\left|X\cap Y\right|}{\left|X\cup Y\right|}}$$

(21)

$$J={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}Jaccard\left(X_i,Y_i\right)$$

(22)

$$h\left(x,y\right)=\sum _{j=1}^l{1}_{x_j\ne y_j}$$

(23)

$$H=\left.{\displaystyle \frac{1}{n\cdot l}}\sum _{i=1}^n{h}_i\left(x_i,y_i\right)\right|$$

(24)

In this context, $l$ represents the number of labels that each sample may have. $h$ denotes the Hamming distance, which is the number of discrepancies between the predicted labels and the actual labels. The Hamming loss is generally the lower, the better.

5.4. Results Analysis

The purpose of all experiments is to input the opcodes to be detected and output a probability sequence indicating the likelihood of the opcodes having four types of vulnerabilities: re-entrancy, permission control, integer overflow, and insecure calling. The threshold is set at 50%. For example, if a segment of opcodes has both re-entrancy and permission control vulnerabilities, the corresponding probabilities for these two vulnerabilities will be greater than 50%, while the probabilities for the other two vulnerabilities will be below 50%.

To highlight the superiority of RTMS in detecting these four types of vulnerabilities,

Table 3. Detection performance of five methods for four types of vulnerabilities. Created by the authors based on experimental results.

Vulnerability	Metrics	RTMS	Bi-LSTM-ATT	Bi-GRU-ATT	MEVD	EDSCVD
Re-entrancy	pre	0.9662	0.7857	0.8159	0.8918	0.8778
	rec	0.9519	0.8128	0.8372	0.9266	0.8706
	F1	0.9589	0.7991	0.8264	0.9089	0.8742
Permission Control	pre	0.9329	0.7109	0.7527	0.8593	0.8342
	rec	0.8982	0.7461	0.7451	0.8744	0.8032
	F1	0.9153	0.7281	0.7489	0.8668	0.8184
Integer Overflow	pre	0.9538	0.7611	0.7819	0.8972	0.8664
	rec	0.9381	0.7916	0.8281	0.9070	0.8342
	F1	0.9459	0.7761	0.8043	0.9021	0.8501
Insecure Calling	pre	0.9468	0.7186	0.7659	0.8186	0.8090
	rec	0.9216	0.7595	0.7951	0.8242	0.7789
	F1	0.9289	0.7385	0.7802	0.8213	0.7937

presents a performance comparison with other detection methods (Bi-LSTM-ATT [24], Bi-GRU-ATT [25], MEVD [27], EDSCVD [29]) for vulnerability detection tasks. The performance for detecting the four types of vulnerabilities is primarily assessed across three dimensions: precision, recall, and F1 score.

Bi-LSTM-ATT and Bi-GRU-ATT exhibit similar detection performance across various vulnerabilities, while RTMS can achieve an average improvement of about 15 percentage points in terms of precision, recall, and F1 score. Compared to Bi-LSTM-ATT and Bi-GRU-ATT, MEVD, and EDSCVD can achieve an average improvement of about 10 percentage points in precision, recall, and F1 score, indicating that the use of multi-scale feature fusion can effectively enhance the extraction of vulnerability features. Furthermore, RTMS shows a maximum difference of about three percentage points in these three metrics for different vulnerabilities, whereas Bi-LSTM-ATT and Bi-GRU-ATT show a maximum difference of nearly seven percentage points, and MEVD and EDSCVD show a maximum difference of about five percentage points. This suggests that RTMS provides higher accuracy and stability in detecting various vulnerabilities in smart contracts compared to other methods. This is because RTMS can better extract correlations between vulnerabilities and has a smaller performance variation when dealing with different types of vulnerabilities, possessing better generalization ability and stronger detection accuracy.

Table 4. The performance of RTMS using different data slicing techniques. Created by the authors based on experimental results.

Method	Precision	Recall	F1 Score	Jaccard	Hamming
RTMS-GSS	0.9498	0.9341	0.9419	0.9312	0.0211
RTMS-GASS	0.9337	0.9285	0.9311	0.9104	0.0285
RTMS-OT	0.8636	0.7204	0.7855	0.7541	0.1027

lists the performance differences resulting from different slicing strategies used during the data preprocessing phase, with a focus on overall evaluation metrics. The GSS strategy refers to selecting slices of length n × 512 tokens that have the highest gas consumption. The GASS strategy refers to sequentially selecting n slices of length 512 tokens, ordered from highest to lowest gas consumption. The OT strategy refers to selecting the first n × 512 tokens as slices, with any insufficient portions padded with zeros and any excess directly truncated.

Analyzing the results in Table 4, it can be observed that the use of slicing strategies significantly improves model performance. RTMS-GSS, which uses the GSS strategy, shows an overall improvement of about two percentage points in Jaccard similarity coefficient compared to RTMS-GASS. This indicates that selecting slices of length n × 512 tokens, rather than n slices of length 512 tokens, maintains semantic coherence and extracts more logical features, thereby enhancing the performance of vulnerability detection.

To demonstrate the ability of different deep learning methods to learn vulnerability correlations,

Table 5. Comparison of the ability to learn vulnerability correlations using different neural network structures. Created by the authors based on experimental results.

Method	Precision	Recall	F1 Score	Jaccard	Hamming
Dense	0.9026	0.8873	0.8949	0.8762	0.0687
1DCNN	0.9282	0.9052	0.9166	0.9069	0.0373
Bi-LSTM	0.9331	0.9176	0.9253	0.9104	0.0285
Self-Attention	0.9307	0.9154	0.9230	0.9078	0.0314
Channel-Sequence-Attention	0.9498	0.9341	0.9419	0.9312	0.0211

and Figure 8 present a performance comparison of RTNetBlock layer using fully connected layers, one-dimensional CNNs, bidirectional LSTMs, self-attention mechanisms, and channel-sequence attention. The performance of extracting correlations between vulnerabilities is measured across five dimensions: precision, recall, F1 score, Jaccard coefficient, and Hamming loss.

From the data presented, it is clear that the use of channel–sequence–attention performs the best in terms of network performance, slightly outperforming the Bi-LSTM structure. It not only achieves higher precision, recall, F1 scores, and Jaccard coefficients compared to the other four networks but also has the lowest Hamming loss, indicating the lowest level of inconsistency between predicted labels and true labels. Furthermore, the precision of the channel-sequence attention is slightly greater than its recall rate, suggesting that the model leans towards being conservative in classification, that is, it tends to reduce false positives.

Table 6. Ablation study. Created by the authors based on experimental results.

Method	Precision	Recall	F1 Score	Jaccard	Hamming
DL-Encoder-RTNet	0.9498	0.9341	0.9419	0.9312	0.0211
DL-Encoder-TNet	0.7524	0.6263	0.6836	0.6486	0.2784
DL-Encoder	0.9162	0.9017	0.9089	0.8812	0.0724
DL-Encoder-BiLSTM	0.9386	0.9231	0.9307	0.9172	0.0261
CodeBERT	0.8752	0.7314	0.7895	0.7267	0.1054

presents the results of ablation studies, which demonstrate the necessity of various components by gradually removing key parts of the model. The RTNet layer consists of three RTBlock structures. The TNet is a network model without the shortcut connection structure, composed of three cascaded channel-sequence attention structures. DL-Encoder represents the local-global encoder structure. DL-Encoder-BiLSTM indicates the use of BiLSTM instead of DFE to map features to the hidden layer. CodeBERT refers to the model that does not use the layered architecture and relies solely on the CodeBERT model for classification.

The comparison reveals that when the shortcut connection structure is removed, there is a significant performance drop, especially in recall and F1 score. This indicates that the shortcut connection structure plays a crucial role in enhancing model performance. Conversely, retaining the shortcut connection can effectively improve vulnerability detection, providing the model with better stability and accuracy in detecting smart contract vulnerabilities. With the shortcut connection retained, the RTNet network aims to refine the initial classification results by leveraging correlations. Without the shortcut connection, it implies a secondary classification based on the original classification results. Additionally, compared to using only the DL-encoder, the inclusion of the RTNet structure leads to a slight improvement in all metrics, demonstrating that the RTNet structure helps extract correlations between vulnerabilities, enhancing the model’s overall detection performance. Replacing DFE with BiLSTM results in a slight decrease in overall performance, suggesting that the dual-channel structure in DFE is more effective than bidirectional networks like BiLSTM in mapping features to the hidden layer.

6. Conclusions and Future Work

6.1. Conclusions

RTMS is a multi-vulnerability detection method for smart contracts based on vulnerability correlations. RTMS addresses the issue of input data selection by utilizing the gas values consumed by smart contracts for data preprocessing. We employ a weighted binary cross-entropy (BCE) function to tackle the problem of data imbalance and design a layered structure to expand the length of input slices while enhancing the performance of feature extraction. Furthermore, RTMS combines channel attention and sequence attention mechanisms to learn the correlations between vulnerabilities, thereby improving the precision and stability of classification results.

The experimental results demonstrate that, in the detection of individual vulnerabilities, RTMS achieves F1 scores of 0.9589 for re-entrancy, 0.9153 for permission control, 0.9459 for integer overflow, and 0.9289 for insecure calling. Compared to other detection methods, these scores represent improvements of 8.47%, 9.69%, 9.58%, and 13.16%, respectively. Additionally, RTMS’s overall Jaccard coefficient for multi-vulnerability detection reaches 0.9312, which can be enhanced by 17.71% through the gas-value-based slicing strategy.

6.2. Future Work

Although RTMS has demonstrated superior performance in experiments, there is still room for improvement. Currently, the adjustments made to the CodeBERT model during subsequent training are insufficient, and the dataset we have collected is relatively limited in terms of vulnerability types, encompassing only four specific categories: re-entrancy, permission control, integer overflow, and insecure calling. Future research will focus on optimizing the fine-tuning strategy for the CodeBERT model, collecting more diverse datasets, and combining syntactic features to optimize the information extraction algorithm to enhance the model’s information processing capabilities. We will continue to follow the latest developments in the field of software vulnerability detection and the field of large language models, and we look forward to deeper optimization of the RTMS model.

The application of smart contracts is developing in a more complicated direction. We propose the following recommendations for addressing smart contract security issues and the above shortcomings:

• Build authoritative datasets. Future efforts should be directed towards gathering datasets that encompass a broader spectrum of vulnerability types. Additionally, it is essential to standardize the format of these datasets, facilitating the data preprocessing work for subsequent researchers. By expanding existing datasets, we can provide richer resources for the training and evaluation of deep learning models, thereby enhancing their performance and reliability.
• Research vulnerability remediation strategies. Currently, relevant research is still mainly focused on the detection of contract vulnerabilities. In the future, security research can be conducted on vulnerability contracts to help relevant researchers understand the logic behind contract vulnerabilities.
• Develop integrated static and dynamic analysis models. Focus on addressing the limitations of our current deep learning approach, which relies heavily on static source code analysis and may miss certain execution paths, leading to false positives. Integrate detection tools across different levels to enhance the identification of vulnerabilities in contracts.