Comprehensive Review of Software Reverse Engineering: Methods, Applications, and Emerging Challenges ^†

Abstract

With the increasing prevalence of pirated software, reverse engineering is regarded as a key technology. Reverse engineering enables the reconstruction of compiled code into a high-level language, revealing its functions and characteristics. We analyzed existing research results on software reverse engineering with a focus on the main methods, application scope, and emerging trends of combining reverse engineering with ML. The practical application of reverse engineering in intellectual property rights protection was examined by comparing it with existing methods, such as code obfuscation and anti-analysis mechanisms. Finally, this paper proposes future research directions, emphasizing the ethical and developmental aspects of reverse engineering, as well as the importance of AI.

1. Introduction

With the continuous emergence of pirated software in society, existing reverse engineering methods and corresponding countermeasures have become significant issues that warrant discussion. Reverse engineering at the software level aims to maintain and enhance systems, and its primary purpose is to understand system structures [1], such as transforming low-level abstractions into high-level abstractions. This process is not limited to converting source code or compiled programs into high-level programming languages that humans can understand [2]. Consequently, it might lead to deficiencies in information security. According to the Global Software Survey released by Business Software Alliance in 2018 [3], the commercial value of pirated software worldwide reached as high as USD 46.3 billion, indicating the role of technologies such as reverse engineering in bypassing software licensing and protection mechanisms and highlighting their significant research implications for information security and intellectual property protection [3].

To address the issues of piracy and illegal analysis arising from the increasing maturity of reverse engineering technologies, anti-reverse engineering techniques commonly employ methods such as code obfuscation to increase the time and cost required for attackers to analyze program logic [4]. Among these, the hybrid obfuscation method proposed by Al-Hakimi et al. [5] and the Generalized Dynamic Opaque Predicates proposed by Xu et al. [6] have significantly increased detection costs while maintaining good performance.

In contrast, reverse engineering focuses on analyzing systems and understanding their structures and methods, typically dividing into static and dynamic analysis approaches to reveal potential security vulnerabilities. With technological advancements and the rise of AI and machine learning (ML), traditional reverse engineering approaches that rely heavily on extensive manpower and effort have become outdated. ML techniques have proven to be one of the most effective solutions to this problem [7].

Code obfuscation significantly increases the difficulty for attackers and anti-reverse engineering to analyze and understand software. However, obfuscation is not used solely for legitimate software protection; it is also widely employed in malware. Malicious code is often concealed within programs, further complicating the detection of malware [8]. Therefore, the introduction of AI for this purpose enables automated feature extraction to identify potential malicious characteristics from large amounts of obfuscated program behaviors and structures, effectively improving the efficiency and accuracy of malware reverse engineering analysis [7,8].

In this study, we reviewed and analyzed reverse engineering methods in the software domain, anti-reverse engineering technical strategies, and the applications of AI and ML within this context. First, this study will organize the main methods of reverse engineering. Second, it will focus on analyzing anti-reverse engineering techniques, such as code obfuscation and control-flow obfuscation, primarily exploring their design and implementation, as well as the operational principles and strategies of different techniques.

We also explore how AI technologies assist in reverse engineering and anti-reverse engineering, particularly in their potential applications for automated feature code identification, malicious behavior prediction, and enhanced analysis efficiency. Through these comprehensive analyses, research outcomes related to reverse engineering are collected and examined.

In this article, Section 2 explores the differences in reverse engineering across traditional, model-driven, and AI-based approaches. Section 3 introduces various anti-reverse engineering methods. Section 4 presents several studies on the application of AI in reverse engineering. Section 5 provides our recommendations and conclusions.

2. Evolution of Reverse Engineering

2.1. Traditional Reverse Engineering

Reverse engineering is the process of analyzing compiled programs or binary files in the absence of source code or design documentation, using manual and tool-assisted approaches [9] to understand their internal operational mechanisms and behavioral logic. Such methods are widely applied in the information security domain, including malware analysis, vulnerability discovery, and the study of software protection mechanisms [7].

Common traditional reverse engineering techniques can be divided into static analysis and dynamic analysis. Static analysis relies on disassembly and recompilation techniques. Disassembly converts binary programs into assembly language to facilitate observation of program logic and functionality [9], while recompilation attempts to reconstruct program structures, variable names, and comments to aid in analyzing function semantics and logic [10]. Dynamic analysis, on the other hand, involves executing programs in a controlled environment to observe their runtime behavior, system calls, and memory operations [11].

The advantage of traditional reverse engineering lies in the high precision of its analysis results, enabling detailed observation at the instruction level and making it particularly suitable for vulnerability inspection. However, its main limitation is the heavy reliance on expert knowledge and the significant time costs associated with it. When dealing with large-scale or highly obfuscated programs, both analysis efficiency and scalability are limited [7].

2.2. Model-Driven Reverse Engineering

Model-Driven Reverse Engineering (MDRE) introduces the concept of Model-Driven Engineering (MDE) into the reverse analysis process. Its objective is not to reconstruct complete source code but to abstract high-level structural and behavioral models from existing systems, source code, or binaries to support understanding, analysis, and reuse [12,13,14].

In the MDRE approach, the output of reverse engineering is no longer limited to decompiled code representations but shifts from an object-oriented to a model-oriented perspective [12]. MDE is a system development methodology that utilizes models as its core [12]. Reference [15] explores the conversion of Unified Modeling Language (UML) models into Java. The integration of reverse engineering and MDE forms MDRE, which uses models generated from existing systems to support further engineering tasks such as refactoring [13].

Compared with traditional reverse engineering, MDRE provides better visibility and generality [13]. Scenarios where system characteristics have been studied or where mature models already exist for reference. Its effectiveness largely depends on the availability of models that correspond to relevant system aspects, enabling the achievement of the expected analysis outcomes. Moreover, since models are abstract representations, manual adjustments are required to address differences across programming languages and platform implementations. In practice, relying solely on automated MDRE analysis is often insufficient, and it is typically combined with manual or traditional reverse engineering methods for support [14].

2.3. AI-Oriented Reverse Engineering

With the maturation of AI and ML technologies, recent studies have increasingly attempted to incorporate AI and ML methods to reduce manual analysis effort and improve efficiency. ML techniques have proven to be effective tools for addressing the limitations of traditional reverse engineering. AI-assisted reverse engineering primarily applies ML algorithms to infer features and semantics from binary code [7,16,17]. Such methods are applied to various reverse engineering tasks. In recompilation, issues such as incorrect function identification and variable type errors frequently occur. By training large language models, the DE compilation process can be optimized, helping analysts more quickly understand program structures and detect vulnerabilities [15].

Although AI technologies can enhance the automation and efficiency of reverse engineering, their application is still constrained by data scarcity, which affects the reliability of model training and evaluation. In addition, existing AI models generally lack strong generalization capabilities, resulting in unstable analysis performance across binaries produced by different tools, compilation environments, or platforms. Furthermore, the limited interpretability of deep learning models makes their decision-making processes difficult to verify, reducing analytical trustworthiness, particularly in security-critical systems. Finally, AI-based reverse engineering also faces adversarial attacks and ethical and legal risks, indicating that it must be applied with caution and in conjunction with manual analysis [7].

3. Analysis of Anti-Reverse Engineering

We adopted the Systematic Literature Review method to systematically organize and analyze recent research on the technological developments and applications of software reverse engineering. We collected articles listed in the Institute of Electrical and Electronics Engineers Xplore Digital Library to ensure comprehensive coverage of relevant fields, including software engineering, information security, and program analysis. Keyword combinations are used in the literature search, focusing on core topics including reverse engineering, MDRE, AI reverse engineering, and software obfuscation. The collected literature undergoes multiple screening stages, including title screening, abstract review, and full-text analysis, to exclude studies that are not aligned with the research topic or lack sufficient technical depth. Ultimately, representative and valuable studies are selected for synthesis and comparative analysis. To enhance systematic analysis, the literature is further classified by technical type, application scenario, and research objective, providing a foundation for subsequent comparisons between traditional reverse engineering methods and AI-assisted reverse engineering techniques.

As reverse engineering techniques have become increasingly mature in fields such as software analysis, vulnerability research, and malware analysis, software developers and malware authors have also adopted anti-reverse engineering techniques to make program analysis and understanding more difficult. The objective of anti-reverse engineering is to disrupt the analysis process, conceal program logic, or increase the time and cost required by attackers [5,18]. This section introduces several common and representative anti-reverse engineering techniques, explaining their basic principles and impacts.

3.1. Code Obfuscation

Obfuscation is one of the most common and effective techniques for anti-reverse engineering. Its primary purpose is to improve code readability without altering program functionality [5,18,19]. Code obfuscation techniques can be applied at both the source code and compiled binary levels, such as renaming variables and functions and inserting redundant code [5,20].

Source-level and binary-level obfuscation differ significantly in their technical characteristics. Source-level obfuscation can leverage semantic and structural information unique to high-level languages, making it suitable for scenarios where source code must be distributed; however, its obfuscation effects may be partially eliminated during compilation. Binary-level obfuscation must be designed for specific architectures and is more difficult to implement, but it provides more persistent and direct protection effects [20]. Although code obfuscation provides strong resistance to reverse engineering, its drawback is that it increases normal program computation and execution costs. Therefore, a balance must be achieved between obfuscation strength and execution efficiency [18,21].

3.2. Control Flow Obfuscation

Control flow obfuscation is an anti-reverse-engineering technique that specifically targets program execution flows. Its goal is to disrupt the intuitive structure of control flow, increasing the complexity of the control flow graph (CFG) and making it difficult to analyze [22,23]. Common methods include inserting unnecessary branches, using indirect jumps, or introducing opaque predicates, which prevent analysis tools from determining actual execution paths [23].

Methods proposed by BiAn [18], such as inserting opaque predicates and flattening the control flow graph, are common control flow obfuscation techniques. On this basis, a chaotic mapping is introduced to generate opaque predicates that differ from traditional constant-true or constant-false conditions. Chaotic mapping is used to generate Boolean decision conditions whose initial values exhibit high sensitivity and randomness, effectively preventing the cracking of encrypted information or increasing the difficulty of cracking.

Control Flow Obfuscation with Intel Software Guard Extension (CFHider) is used for a hardware-assisted control flow obfuscation method [24]. While traditional approaches increase analysis cost by complicating program structures, CFHider replaces each conditional expression with a function call named ColdFusion Markup Language and moves the actual conditional logic into a trusted execution environment called Control Flow Enclave. CF Enclave performs computation and returns results only after remote attestation using Intel SGX, thereby determining the program’s actual control flow.

Such techniques significantly limit the accuracy of static analysis methods in modeling program control flow, greatly increasing the difficulty of control flow recovery in reverse engineering and leading to exponential growth in the time and resources required for analysis [23].

3.3. Code Virtualization

The difference between code virtualization and code encryption is that, in code virtualization, the source code is not restored in memory. By transforming original programs into custom virtual instructions that are interpreted and executed by an embedded virtual machine, key program logic can be effectively hidden during dynamic analysis, thereby increasing the time required for reverse engineering. However, its weakness lies in the fixed structure of virtualization components. During static analysis, attackers can extract the virtual instruction set from the virtual structure, infer its functionality, and gradually reconstruct program logic [19].

To address this issue, the Variant Call Format (VCF) is used to strengthen virtualization obfuscation [19]. Its core concept is to reduce the number of virtual instructions observable at any given time. VCF divides the entire virtual instruction sequence into multiple virtual instruction blocks, each corresponding to a basic block. Subsequently, new instruction contents are dynamically generated through folding operations using the previously executed block and randomly generated keys, ensuring that only one executable virtual instruction block exists in memory at any time.

In frontend applications, Wang et al. propose JavaScript (JSPro) [25], which enhances resistance to reverse engineering by virtualizing JavaScript. JSPro utilizes WebAssembly as an intermediate execution environment, dividing JavaScript into computational and Document Object Model-related code. Computational JavaScript code is converted into a custom virtual instruction set and interpreted and executed by a virtual machine within WebAssembly. As a result, control flow and computation logic no longer exist in readable source code form, increasing the difficulty of both static and dynamic analysis. Compared with traditional JavaScript obfuscation techniques that only modify syntax and control structures, JSPro alters program behavior at the execution level. By combining the binary characteristics and near-native execution performance of WebAssembly, it provides stronger program protection while maintaining performance.

3.4. Analysis of Anti-Reverse Engineering Challenges

To address illegal reverse engineering that infringes on software intellectual property rights, numerous studies have proposed solutions such as obfuscation and encryption, which effectively increase the difficulty of analysis and understanding. However, these techniques are accompanied by challenges in their practical application, including increased execution time, computational overhead, and development costs [5,18,19,21]. As summarized in

Table 1. Anti-reverse engineering.

Reference	Type	Method	Contribution
Al-Hakimi et al. [5]	Hybrid obfuscation	Combining different methods to make the code more complex.	Demonstrated that the hybrid obfuscation approach is more resistant to attacks than the standard method.
Xu and Wu [6]	Control flow obfuscation	Automatically transform common program structures into dynamic opaque predicates.	Proposed a novel dynamic opaque predicate technique.
Zhang et al. [18]	Control flow obfuscation	Opaque predicates combined with chaotic mapping to enhance the strength of control flow obfuscation.	Developed the first smart contract obfuscation tool.
Wang et al. [24]	Control flow obfuscation	Protecting control flow through Intel SGX.	Applied hardware-assisted trusted execution environments to the design of control-flow obfuscation for the first time.
Suk and Lee [19]	Virtualization obfuscation	Virtual code folding within a virtualization structure.	Proposed a virtualization-based obfuscation method that reduces the observability of virtual instructions.
Wang et al. [25]	Virtualization obfuscation	Virtualizing numeric JavaScript into bytecode executed on a WebAssembly-based virtual machine	Combined code virtualization with WebAssembly and applied it to JavaScript obfuscation for the first time.

, recent studies on software reverse engineering have highlighted the ongoing trade-offs between protection effectiveness and system efficiency. Therefore, in practice, it is necessary to achieve an appropriate balance among security, performance, and cost when designing suitable protection models.

4. Applications and Challenges of AI in Reverse Engineering

We summarized and compared various common anti-reverse engineering techniques, presenting their methods and advantages in a tabular form. These techniques are designed to counter traditional reverse-engineering approaches, such as control-flow obfuscation and code virtualization, and can pose significant obstacles to both static and dynamic analysis. However, malware also applies obfuscation techniques to evade detection. With the maturation of obfuscation techniques, traditional reverse engineering methods that rely primarily on manual analysis have gradually revealed limitations in efficiency and scalability when dealing with large-scale or highly obfuscated malware.

Analysts need to spend substantial time reconstructing program logic but face difficulties in extracting representative behavioral features from obfuscated programs within a short period. With the rapid development of AI and ML technologies, these approaches have been introduced into the reverse engineering process as tools for assisted analysis and automated decision-making. Therefore, reverse engineering methods were investigated based on AI ML models, summarizing their application scenarios in malware analysis and the challenges they face.

4.1. AI Applications

Regarding the application of large language models (LLMs) in reverse engineering, Liao et al. [15] proposed the SmartHalo method for smart contracts written in the Solidity programming language. Its idea is to combine LLMs with static analysis techniques to address limitations in the recompilation process, such as inaccurate function identification, incorrect variable types, and the lack of contract attributes. In this method, state, control-flow, and type dependencies are extracted from decompiled code via static analysis to construct a Dependency Graph (DG) that captures key information. DG is then used to generate prompts, including optimization target contexts and optimization of the result candidates. Chain-of-thoughts are employed to guide LLM reasoning, thereby improving the completeness and accuracy of recompilation.

Mi et al. [22] proposed the Instruction Stream-augmented Control Flow Graph (CFG)-based Malware Variant Detection method, which utilizes an Instruction Stream–Augmented CFG as the model and applies Graph Neural Networks for automatic identification of malware variants. Traditional CFGs are constructed through disassembly and control flow analysis. To compensate for the semantic limitations of CFGs, instruction streams within basic blocks are incorporated as features, and opcodes and operands in instructions are converted into vectors. GNNs are used to learn the relationships between control flow structures and instruction semantics, ultimately generating vector representations of entire programs for malware family or variant classification and comparison.

In studies on reverse engineering and security detection for the Android platform, Urooj et al. [17] decompiled Android Package Kit files to extract features such as permissions and application programming interfaces, converted these features into binary vectors, and applied multiple ML algorithms, including Support Vector Machines (SVM), Random Forest (RF), and K-Nearest Neighbors (KNNs). When comparing the performance of these classical ML algorithms, RF performed best.

4.2. AI Challenges

Many studies on ML-based reverse engineering combined with AI have indicated that AI technologies enhance the automation level and analysis efficiency of reverse engineering. However, their application is still constrained by data scarcity, which affects the reliability of model training and evaluation. In addition, existing AI models generally lack strong generalization capabilities, resulting in unstable analysis performance across binaries generated by different tools, compilation environments, or platforms. Furthermore, in obfuscated code, the original structure and semantic relationships are disrupted, leading analysis methods that rely on structural features to suffer significant noise. Therefore, applying AI techniques to reverse engineering remains a significant challenge [7,15,17,22].

5. Conclusions

We compared the major technological development trajectories of software reverse engineering, including traditional static and dynamic analysis, MDRE, and recently emerging AI-assisted analysis methods. The results indicate that traditional reverse engineering remains crucial for understanding program behavior and functional recovery. However, its efficiency is limited when faced with anti-reverse-engineering techniques, such as obfuscation and virtualization. In contrast, the introduction of AI and ML improves automation and analysis efficiency and can handle most complex program structures, demonstrating strong potential for future reverse engineering research. Nevertheless, current AI approaches face challenges such as data scarcity, limited generalization capability, and insufficient interpretability, indicating that they cannot yet fully replace expert-driven analysis processes. Based on these findings, the contribution of this study lies in clarifying the roles of different reverse engineering techniques in practical applications and in highlighting the ongoing adversarial competition between reverse engineering and anti-reverse engineering. In further research, it is necessary to focus on achieving a balance among automation, cost, and analytical outcomes in software reverse engineering.