SoK: An In-Depth Analysis of Intrusion Detection Systems Based on System Calls

Abstract

The increase and professionalization of cyberattacks calls for the development of relevant defense-in-depth mechanisms of which intrusion detection systems (IDSs) are essential components. This paper provides an in-depth analysis of system call-based IDSs as intelligence for detecting malicious activities. A systematic analysis of 209 publications from the scientific literature between 1996 and early 2026 highlights trends in this field of research and defines a taxonomy presenting the different approaches proposed by researchers. Eighteen state-of-the-art methods, representative of the diversity of approaches proposed in the literature, were reproduced and evaluated on two public datasets, ADFA-LD and NGIDS-DS. The detection performance and overhead of each method are examined in great detail, opening discussions on the shortcomings of the state of the art, limitations of system call-based IDSs, and lines of research that would enable this type of detection system to meet the challenges of deployment in a real-world environment. Finally, recommendations for future work are derived from these findings.

1. Introduction

The cybersecurity landscape has been evolving rapidly in recent years. On one hand, companies are increasingly relying on external cloud technologies for their enterprise infrastructure, leading to a rise of around 75% in cyberattacks against these systems in 2024 [1]. On the other hand, the healthcare sector and hospitals are among the most targeted by ransomwares [2,3], which are deployed by cybercriminals to extort money from organizations. In 2023, 389 U.S. healthcare institutions were hit by such malwares, leading to sometimes critical disruptions in patient care [4]. Industry is not spared either and is also the target of cyberattacks, which are often perpetrated by government-backed hackers known as Advanced Persistent Threats (APTs) [1,2]. Infrastructures such as those governing water treatment, power generation and distribution, and transportation have historically been isolated from the Internet, and they were not designed to meet cybersecurity constraints. These critical infrastructures, which are responsible for the management of real-life critical processes, are a prime target for cybercriminals with geopolitical interests at stake [4]. Moreover, attackers are taking advantage of an ever more mature ecosystem to improve their attacks, particularly in terms of tools and evasiveness [4]. The increase and professionalization of cyberattacks calls for the development of countermeasures to enhance the defense capability of computerized systems, from Information Technologies (IT) to Operational Technologies (OT). A strategy for identifying and containing attacks is to detect them before their effects become critical. Intrusion detection then appears as a relevant defense-in-depth mechanisms for a first step toward resilience by monitoring the state of protected infrastructures against signs of malicious activities.

1.1. What Is an Intrusion?

The National Institute of Standards and Technology (NIST) defines an intrusion as “a security event, or a combination of multiple security events, that constitutes a security incident in which an intruder gains, or attempts to gain, access to a system or system resource without having authorization to do so” [5]. ANSSI, the French Cybersecurity Agency, defines it as “the act of a person or object entering a defined space (physical, logical, relational) where its presence is not desired.” [6]. According to the above definitions, an intrusion can be defined as an event that needs to be closely monitored to ensure that the considered system is not altered by malicious activities.

1.2. What Is Intrusion Detection?

Intrusion detection may be defined as the process of actively detecting the presence of intruders in the system being protected. Such intrusion detection is performed by an intrusion detection system (IDS) whose role is to look for suspicious activities, by a misuse approach, i.e., looking for known intrusion patterns, or an anomaly one, i.e., looking for deviations from the expected behavior of the system. Alerts created by these IDSs are then collected in Security Operations Centers (SOCs) where experts analyze them in detail and respond accordingly to protect potentially compromised systems. The automated detection of cyberattacks also enables automated responses to be implemented, which reduces attack surface and reinforces overall system security. Many research studies and existing solutions are based on the analysis of packet transiting the network to identify malicious actions. These methods, known as network-based intrusion detection systems (NIDSs), have the advantage of being non-intrusive, since they involve passive probes placed on an existing network [7]. Others propose IDSs based on the behavioral analysis of strategic targets for intrusion detection. Known as host-based intrusion detection systems (HIDSs), they base their detection on the analysis of signals internal to a device, such as logs, Hardware Performance Counter (HPC) registers, or sequences of system calls. The first scientific publication to conceptualize host-based IDSs was the article published by Dorothy E. Denning in 1987 [8], which used operating system audit logs as a source of information for detecting abnormal behavior. Further research has been carried out along this line—among it, the work of Theresa F. Lunt in 1993 [9].

1.3. What Are System Calls and How Can They Be Used for Intrusion Detection?

A system call is an interface between a program running in user space and the operating system (OS). Application programs use system calls to request privileged services and functionalities from the OS kernel, such as reading a file, writing on a socket or running a new process. When a computer system is running, all its applications generate sequences of system calls which represent host behavior, and they can then be analyzed for intrusion detection. The studies of Forrest and her colleagues launched this line of research in 1996 with the first publication to explicitly use system call sequences for intrusion detection. They presented a method named TIDE, for Time-Delay Embedding, based on a reference dictionary of system call sequences observed during a normal and benign execution [10,11]. At runtime, a predefined number of mismatches raises an alert. This method is often considered a gold standard to compare the performance of later research, which is mostly based on machine learning.

With the growth of research around system call-based HIDSs, some publications have made it possible to better synthesize advances using techniques derived from language processing [12] or having been carried out on the same dataset [13]. However, to the best of our knowledge, none of them provide a complete overview of state-of-the-art approaches for intrusion detection based on system calls, a rigorous comparison of detection performances, or an analysis of the applicability of this type of HIDS to real-life applications. To overcome this deficiency and provide a comprehensive overview of the research field, the most extensive systematic analysis of the literature on intrusion detection systems based on system calls is proposed, from the studies of Forrest in 1996 [10] to the beginning of 2026, which represents 209 publications. Furthermore, to enable a fair evaluation of the different approaches proposed in the state of the art, 18 of these methods were re-implemented to experimentally compare their performance under similar conditions of execution, i.e., with the same preprocessed datasets, and with the same evaluation criteria. All materials used for literature analysis are made publicly available. This in-depth analysis of HIDS solutions based on system calls enables us to assess the applicability of such solutions in real-life environments, such as in industry, cloud, enterprise servers and personal computers. Finally, recommendations are drawn up for future research.

To summarize, the main contributions of this paper are as follows:

• A systematic literature review of 209 publications on intrusion detection research using dynamic monitoring of system calls, from 1996 to early 2026, whose materials are publicly available for possible reanalysis;
• The re-implementation of 18 state-of-the-art methods representing the diversity of system call-based intrusion detection approaches in the literature, offering a real evaluation of the performance of these IDSs on two public datasets, encouraging research reproducibility.

Section 2 presents other work related to the review of existing contributions and highlights the need for this paper. The methodology adopted for the literature review and the experimental evaluation of detection methods are described in Section 3. The literature is reviewed in Section 4, which is followed by the results of evaluated state-of-the-art methods in Section 5. Then, discussions are opened in Section 6 based on each analysis and the maturity of system call-based IDSs for real-life applications. The limitations of this paper are also discussed in Section 7. Finally, Section 8 concludes with the main lessons to be learned from intrusion detection research based on the analysis of sequences of system calls.

2. Related Work

Several studies summarize research carried out on host-based intrusion detection.

Liu et al. [14] propose an overview of HIDSs based on the monitoring of system calls. The authors present an overview of data processing methods and decision engines used for intrusion detection in the literature as well as associated datasets. The latter and evaluation metrics are not deeply analyzed because of the scope of the paper, whose aim is not to be a systematic literature review, although it offers an excellent overview of the main trends in this field. The applicability of such solutions on embedded systems is discussed despite the potential applicability of HIDS to other use cases. Data processing through cloud and big data architectures are also discussed.

Khraisat et al. [7] covers the main features of both network- and host-based IDSs, presenting the particularities of each approach. As the subject is very broad, not all host-based detection methods explored in the literature are mentioned such as the use of HPCs. Nonetheless, this paper provides a valuable overview of general intrusion detection strategies.

Bridges et al. [15] and Martins et al. [16] provide surveys on HIDSs in general—not only the ones focusing on system call-based detection. Bridges et al. review the different types of information that can be used for host-based detection for both Unix and Windows environments, which is illustrated by comparative analyses of existing work. On the other hand, Martins et al. observe HIDSs through their applicability to IoT systems, highlighting the needed properties of such IDSs for real-time detection and its main related issues. However, other applications are not discussed. Finally, they also developed multiple attacks (28) based on known CVEs and compared the score given by open-source HIDSs when launched against a target. While they are not as precise as the proposed work aim to be on syscall-based HIDSs, both studies provide a great overview of the state of the art in HIDS research.

Khandelwal et al. [13] propose a comparative analysis of several (17) system call-based HIDSs that were evaluated on the ADFA-LD [17] dataset. They provide a description of the methods used and the performance obtained as given by the authors. They also resume the strengths and weaknesses for each presented solution. This approach is interesting because it aims to compare methods evaluated using the same dataset; thus, the results would be comparable between them. Nevertheless, this study points out that the metrics used in each paper are different, so they cannot all be compared as is. Moreover, even when these studies use the same dataset, it is not mentioned that it can be used differently in training, validation, and testing phases, which also hampers the strict comparability of results.

Sworna et al. [12] made a systematic literature review leading to an analysis of 65 studies using Natural Language Processing (NLP) methods for host-based intrusion detection based on system calls. Data extracted from these studies enable the authors to classify methods according to the intrusion detection approach used, the decision algorithm and its learning mode, and the data used with the attacks on which the evaluation of the model is made. Despite a thorough analysis of the most commonly used datasets, the context around the environments in which data collection took place is missing. This could have helped to identify the use cases envisaged by researchers for HIDSs, either for IoT applications or cloud computing, as examples.

Satilmiş et al. [18] analyzed 21 studies between 2020 and 2023 through a systematic review of the HIDS literature. While not focusing solely on system call-based HIDSs, this SLR summarizes the main contributions of recent key studies. The authors analyze the benefits and drawbacks of encountered evaluation methods, algorithms used for detection, and datasets. In particular, it underlines the fact that the additional performance costs generated by intrusion detection are hardly ever emphasized either for time or memory overheads. Yet the small number of studies included in the analysis, and the limited time period covered, make it difficult to conduct a complete analysis of the evolution of these IDSs.

Despite the existence of several publications summarizing the state of research in intrusion detection based on host information, few of them are systematic reviews of the literature. This type of analysis is needed to rigorously interpret all the available work and assess progress on a well-defined research question [19]. In addition, existing reviews focus on recent work, post-2011, or are based on a limited amount of studies according to

Table 1. Existing works analyzing host-based intrusion detection research.

	Year	Scope of the Study	SLR?	Considered Studies	Covered Period	Experimental Evaluation
[14]	2018	Syscall-based HIDSs	✗	70 *	1996–2018	✗
[7]	2019	HIDSs, NIDSs	✗	70 *	1998–2018	✗
[15]	2020	HIDSs	✗	81	1993–2018	✗
[16]	2022	HIDSs for IoT	✗	18	2016–2021	✗
[13]	2022	Syscall-based HIDSs evaluated on ADFA-LD	✗	17	2013–2021	✗
[12]	2023	Syscall-based HIDSs using NLP methods	✓	65	2011–2022	✗
[18]	2024	HIDSs	✓	21	2020–2023	✗
This work	2026	Syscall-based HIDSs	✓	209	1996–2026	✓

SLR?: Whether the study is a systematic literature review (✓) or not (✗). Considered studies: Number of studies included in the analysis of the research field. Counts marked with a * are those which do not clearly indicate surveyed studies, which can lead to inaccuracies in this reporting. The number of counted studies has been rounded up to the nearest ten. Covered period: Time period covered by the analysis of considered studies. Experimental evaluation: Whether the study re-implements state-of-the-art methods for empirical evaluation (✓) or not (✗).

, which provides only a partial view of the evolution of research on HIDSs. In contrast, this paper provides an in-depth analysis of 209 publications, ranging from the very first studies conducted in 1996 through to 2026, and covering the entire research field related to system call-based intrusion detection. Additionally, an effort has been made to re-implement various methods in order to properly compare their performance, which to our knowledge has not been published previously.

3. Materials and Methods

A systematic literature review requires a well-defined process for collecting the primary studies that will form the foundations of the proposed analysis. The research method for this paper follows the guidelines from [19] to present a proper assessment of the research topic by using a rigorous and verifiable methodology. The research methodology described in this section is illustrated in Figure 1 with the number of publications included at each stage indicated in parentheses.

3.1. Research Questions

This systematic literature review aims to answer the following research questions.

RQ-1: How have IDSs based on system calls research evolved since 1996?

The objective is to identify trends in approaches developed to detect intrusions from system calls. These may be in the targets on which IDSs aim to be deployed, in the methods used to classify system call sequences, or any other trend relevant to the analysis of the research area.

RQ-2: How are system call-based IDSs built?

We seek to identify and analyze the methods used to build such IDSs. This includes the overall intrusion detection approach, whether the authors are looking to recognize known attack patterns or deviations from normal behavior, the processing of system calls, and the mechanisms that classify sequences of system calls as benign or malicious.

RQ-3: On what data are IDSs based on system calls trained and tested?

The aim is to describe the data used to evaluate the performance of these IDSs: their origin, the application case studied, and their availability for reproducibility.

RQ-4: How are IDSs based on system calls evaluated?

We are interested in how the overall performance of these IDSs is measured. This includes the metrics used to evaluate detection performance but also the measurement of the impact of intrusion detection on device performance.

Then, the experimental evaluation of some key state-of-the-art methods aims to answer the following research question:

RQ-5: How do state-of-the-art IDSs based on system calls identifiers perform under the same execution conditions?

The goal is to observe the experimental performance of key methods proposed in the literature relying solely on the analysis of system calls identifiers for intrusion detection by setting predefined datasets and common evaluation methods for all of the studied methods.

3.2. Literature Review Methodology

3.2.1. Search Strategy for Studies to Be Considered

This paper is based on a review of the scientific literature available from Scopus, IEEE Xplore and The ACM Guide to Computing Literature digital libraries. The keywords aligned with the study topic are all expressions derived from intrusion, anomaly, misuse or malware detection, with system calls, which are also abbreviated to “syscalls”. The search logic used to collect relevant publications from these libraries is shown in Figure 2, and the specific search strings applied for each library are available in Appendix A. Searches are performed on titles, abstracts and keywords for Scopus, on all metadata for IEEE Xplore, and on abstracts in The ACM Guide to Computing Literature.

Although there are many duplicate publications in the search results from the IEEE Xplore and The ACM Guide to Computing Literature libraries with publications indexed in Scopus, including these libraries is not redundant because, as the collection results in Figure 1 show, not all documents are indexed in Scopus. This can be explained by a delay in indexing and databases built by abstracts instead of full-text databases, which leads to complementary results.

3.2.2. Inclusion and Exclusion Criteria for Primary Studies Selection

Once duplicate publications have been removed from the collection results, a number of inclusion and exclusion criteria are applied, as presented in

Table 2. Inclusion and exclusion criteria for primary studies selection.

Inclusion criteria	I1	Publications in which the presented intrusion detection system is based on dynamic analysis of system calls. This excludes static analysis of system calls from binaries.
	I2	Studies that have been published between 1996 [10] and early 2026.
Exclusion criteria	E1	Publications that are not accessible.
	E2	Publications that are not written in English.
	E3	Conference reviews.
	E4	PhD thesis manuscripts and posters.
	E5	Conference versions of journal papers and prior studies that have been further developed in a second publication.
	E6	Surveys that do not provide an intrusion detection method.

, to identify relevant publications to study. The aim of these criteria is to exclude from the set of publications those that do not correspond to the scope of the review. These criteria are first applied by reading the titles and abstracts of the publications. The selection is then refined with the more in-depth full-text reading for data extraction.

3.2.3. Data Extraction from Primary Studies

From the full-text reading of these primary studies, a substantial amount of information is extracted to address review questions.

To answer RQ-1, bibliographical information from each publication is collected and linked to the information extracted for the other research questions to study research trends:

• Year of publication;
• Digital Object Identifier (DOI), if any;
• Authors;
• Title.

The answer to RQ-2 requires gathering all the information relating to the design of the proposed methods:

• Intrusion detection approach, focusing on recognizing known intrusion patterns or deviations from a learned normal behavior;
• If relevant to the method used, which learning paradigm is used to train the IDS;
• Data usage, therefore defining the granularity of alerts, by entire execution trace, by predefined-length sequences of system calls, or by defined time interval;
• Features extracted from system calls;
• Features reduction method, if any;
• Classification mean from extracted features;
• Other information used in conjunction with system calls;
• Whether the proposed IDS is collaborative, i.e., if it is intended to be deployed on a single target or if its alerts are based on data collected from multiple hosts.

RQ-3 requests information on the data used for each study:

• Data type, rather collected from real devices, testbeds, or simplified environments;
• Data augmentation method, if any;
• Data availability at the time of publication;
• Operating system from which system calls were collected;
• Use case of the host under study, i.e., desktop computer, enterprise server, IoT device, or other application environment.

Finally, IDS evaluation methods need to be analyzed to answer RQ-4:

• Evaluation metrics used to assess detection performance;
• Overhead measurements type and scopes, if any.

When a publication presents several intrusion detection methods, only the method corresponding to the main contribution of the authors is retained for analysis. In the case of contributions comparing several detection models, such as several machine learning models, the model that presented the best results according to the authors is the one that is considered.

3.3. Methodology for Experimental Evaluation of State-of-the-Art System Call-Based IDSs

3.3.1. Inclusion, Exclusion, and Quality Assessment Criteria for Selecting Candidate Studies for Re-Implementation

More selective inclusion and exclusion criteria are being applied to the previously selected primary studies in order to examine a smaller set of methods for reproduction. Among these criteria, described in

Table 3. Inclusion, exclusion and quality assessment criteria for selecting candidate studies for re-implementation.

Inclusion criteria	I3	Publications in which the presented IDS is only based on dynamic analysis of system call sequences from one host without further information such as arguments or return code.
Exclusion criteria	E7	Methods that are too specific to their environment, such as the monitoring of well-defined applications.
	E8	Studies published strictly before 2020 with fewer than strictly 35 citations *.
	E9	Studies published between 2020 and 2022 with less than strictly 15 citations *.
Quality assessment criteria	AQ1	Is all the information needed to reproduce the proposed HIDS clearly defined?

* The number of citations for each publication was retrieved from Google Scholar on 11 December 2025.

, are citation conditions; these allow us to consider key works in the field.

The quality of publications is then assessed based on a single criterion: the presence of all the information required to reproduce the work. At last, the final selection of works to be reproduced is made in a less systematic manner, aiming to prioritize methods representative of the diversity of approaches proposed in the literature and those presenting relevant research questions.

3.3.2. Data Extraction from Candidate Studies for Re-Implementation

The answer to RQ-5 requires the study of all particularities linked to the IDSs proposed by authors with the aim of reproducing their methods. Thus, the information extracted from these publications is technical information such as the following:

• Training and testing data splitting method;
• Pseudo-code for any proposed new algorithm;
• Hyperparameters used by machine learning algorithms.

More generally, any information needed to reproduce considered methods is extracted.

4. Results: Study of Literature

In this section, we address the following:

• RQ-1: How have IDSs based on system calls research evolved since 1996?
• RQ-2: How are system call-based IDSs built?
• RQ-3: On what data are IDSs based on system calls trained and tested?
• RQ-4: How are IDSs based on system calls evaluated?

4.1. Considered Studies for Literature Review and Experimental Evaluation

The complete list of studies and their rationale for exclusion is publicly available as described in the Data Availability Statement at the end of this paper.

As can be seen in Figure 3a, more than one quarter of the publications retrieved from the search were excluded from the analysis because their contribution did not feature an intrusion detection method (I1). Around one fifth of publications were also discarded because they were not written in English, were not accessible, or because the presented work was more thoroughly analyzed in other publications within the study set (E1–6).

Then, there remain 209 relevant papers for analysis.

The second set of inclusion and exclusion criteria were applied to this selection in order to reduce the number of publications whose method is a candidate for reproduction and evaluation. According to Figure 3b, around one third of methods were not considered because they integrated other data than system calls identifiers for intrusion detection; they were therefore dismissed (I3). Another third of the publications were discarded because they were deemed too specific for integration into a particular application, which would make an evaluation alongside more versatile methods of little relevance (E7), or because they had little impact on further work in this field of research (E8-9). As can be seen in Figure 4, these conditions seem appropriate since the number of publications considered per year follows the trend of the total number of publications of the first selection. Thus, 86 publications remain candidates for the evaluation.

4.2. Complete Breakdown of Studies Proposing an Intrusion Detection Method Based on System Calls

Data extracted from the 209 reviewed publications allow us to categorize intrusion detection methods using system calls according to several characteristics. In addition to seeking to answer research questions RQ-1, RQ-2, RQ-3 and RQ-4, these new taxonomies present the most explored approaches while being broad enough to allow for the characterization of future work.

4.2.1. Comprehensive Taxonomy of System Call-Based IDSs

In response to RQ-1 and RQ-2, several elements have been identified to characterize the approaches used for system call-based intrusion detection. These elements are provided in Figure 5 and detailed below. A more detailed visualization showing the number of publications involved in each of these categories, in the form of a Sankey diagram, is available in Appendix B.

Detection Type

The detection type corresponds to the approach used for intrusion detection. Two types are considered: Anomaly, which characterizes methods looking for deviations from a behavior considered as normal, or Misuse, which characterizes methods aiming to recognize a behavior considered as malicious. The first approach has the advantage of detecting malicious behavior that was not present in the training dataset, whereas misuse detection is limited to identify only known malicious patterns. The second method is similar to the signature-based recognition used by traditional antivirus software, which relies on predefined malicious signatures [20]. No temporal evolution seems to be emerging regarding the adoption of one approach over the other with anomaly detection being the most widely used approach in the literature.

Learning Paradigm

The type of learning paradigm defines how the IDS is trained if applicable. This may involve training a data representation model, a feature reduction algorithm, or the classification model. This is the case for IDSs using machine, including deep, learning models that require a training phase. Supervised learning exploits data which, in the context of intrusion detection, are usually labeled as “normal” and/or “attack”. A machine learning model uses this information in its learning phase to adjust its weights or thresholds so that its predictions align with the ground truth labels associated to sequences of system calls. Sometimes, only sequences from one label (“normal” or “attack”) are provided or used for training; this paradigm is known as one-class supervised learning, and it should not be confused with semi-supervised learning, which corresponds to learning from both labeled and unlabeled data. Self-supervised learning, which is a type of supervised learning, is a method of training models in which the supervision signal (the “labels”) is automatically derived from the data itself rather than being manually provided in a dataset. Examples of self-supervised learning include language models that predict the next word in a sentence, where the target is derived directly as a shift in the input sequence. Unsupervised learning corresponds to learning from unlabeled data, which means that sequences of system calls provided to the model are neither identified as “normal” nor “attack” during the learning phase. The model therefore cannot use this information in its learning phase and instead seeks to discover underlying patterns within the data. This type of learning is widely used for clustering and dimensionality reduction. It usually produces a score based on likelihood or a similarity measure and can therefore be adapted to binary classification problems by applying a threshold. Reinforcement learning involves embedding an agent in an environment in which it learns a strategy by receiving rewards or penalties based on its interactions with that environment. This approach differs from traditional supervised or unsupervised learning paradigms as the aim is not to identify patterns in the data and make a prediction but rather to maximize a reward process through the predictions made. Reinforcement learning then corresponds to learning a (near-)optimal policy of actions to be taken by an agent in order to maximize a reward function. Federated learning designates a collaborative learning approach based on several local models trained from local data. In the context of intrusion detection, this can be achieved by first training a specific model for different yet similar devices, such as IoT devices [21,22,23], and then by sharing the trained models, and only the trained models, in order to make a decision. This is one way of sharing knowledge while preserving data privacy. Local models then fall into other learning paradigms described above. In our analysis, all IDSs implementing federated learning use supervised learning for local models. When multiple learning paradigms are involved in training an IDS, the method presented is categorized by the most restrictive paradigm. For example, an IDS composed of a model learning a representation of data in an unsupervised way followed by a classifier trained using supervised learning will be categorized as supervised. Similarly, the deployment of federated learning models will be classified as such regardless of whether the learning mode is supervised or unsupervised. In this case, in our literature review, all methods implementing federated learning deployed supervised models.

Looking at the number of publications per year presenting an IDS with at least one learning mechanism, as shown in Figure 6, learning was predominantly unsupervised until 2014, with one exception in 2018, in favor of supervised learning techniques. This can be explained by the increasing use of machine and deep learning algorithms, which grew during the same period.The arrival of self-supervised learning mechanisms, notably with the growing use of deep learning language processing algorithms for detecting abnormal system call sequences, can be seen from 2019 onwards as well as in recent research in federated learning with early work in 2023.

Trained on Attacks?

Some intrusion detection methods require sequences corresponding to attacks in their training phase. This has the advantage of enabling models to better identify normal behavior from malicious behavior and to effectively detect attacks similar to those in the training set. On the downside, some of these models struggle to detect attacks they have never seen before. Lots of methods proposed in the literature present an IDS trained on attack traces, especially in recent work, in proportion with the use of supervised learning, as depicted in Figure 6.

Data Granularity

IDSs use different inputs, such as a sequence of system calls corresponding to an entire execution of a process or just a part of it, in order to make a prediction regarding its legitimacy. A full sequence data granularity corresponds to the use by an IDS of entire traces of system calls of an execution, whether by program, i.e., from start to finish of program execution, or by data acquisition cycle, for example from startup to shutdown of the target under study. This necessarily leads to a delay in detection, as it requires waiting until the end of an execution to raise an alert. A fixed-length sequence granularity corresponds to the use of sequences containing a fixed number of system calls. Such sequences are thus shorter than full sequences so that a behavioral analysis can be performed before the end of the execution of the studied system, allowing a possible alert to be raised earlier in the course of an intrusion than an IDS that performs an analysis at the end of each program execution. A time interval granularity corresponds to the use of sets of system calls executed over a fixed time period. Sequences can therefore be of variable size, as the number of system calls during a given time window depends on the system activity at that moment. As with fixed-length sequences, analyses can be performed during runtime rather than afterwards. There are about the same number of studies proposing detection by full system call sequences as by fixed-length sequences, and there are also occasional studies analyzing system calls at fixed time intervals for intrusion detection.

Features

Several features can be extracted from sequences of system calls. We summarize the main types of features below:

• A first representation is the one based on the sequences themselves. sequence-based features can take the form of raw sequences of system call identifiers, i.e., a list of temporally ordered system calls executed on the host under study, with or without system call selection [24,25,26,27,28]. Other sequence-based features are n-grams built from sequences of system calls.
• A second common representation of sequences of system calls is frequency-based. In this type of approach, sequences are encoded as vectors whose components are the values of the frequency of system calls or n-grams. It includes occurrence count and Term Frequency-Inverse Document Frequency (TF-IDF) features [22,29,30,31,32,33,34,35]. The main drawback is the loss of information on the execution order of system calls in the observed sequence in exchange for adding weighting information.
• Relationships can be derived from sequences of system calls, forming graph-based features. Often requiring the use of information other than system calls alone, these relationships can come from links between processes [36,37,38,39], links between resources on the studied target such as files or sockets [40,41,42,43,44,45,46], or network elements such as destination IP addresses [47,48].
• The democratization of machine learning algorithms for intrusion detection has led to the embedding-based representation of system calls. An embedding is a vector representation of an object under study, in our case of each system call. Embeddings can be predefined, like One-Hot Encodings [49,50,51,52,53], but are most often learned as Word2Vec, GloVe, or Locally Linear Embedding [54,55,56,57,58,59,60,61,62,63].
• Information on a group membership of system calls within a sequence can also serve as a feature of a detection model. group-based features represent system calls by their belonging to a group, whether linked to their scope of action [35,46,57,64,65,66,67,68], or following the prior application of a clustering algorithm [69,70,71,72,73].
• Other methods use descriptive statistics of the composition of each sequence [74,75]. These statistical description-based features allow for a complete agnosticity of system call identifiers, meaning that certain behaviors in the sequence of system calls can be learned on one system and effectively applied to another system [75].

In addition to the above feature types, we consider in the remainder a category Other which gathers “exotic” representations of sequences of system calls [27,76,77,78,79,80,81,82,83,84], like the Soundex representation described in [85].

When research into intrusion detection using system calls first began, IDSs mainly focused on analyzing the sequences themselves or the n-grams that made up the observed sequences, as can be seen in Figure 7. Around 2015, however, research began to shift toward using information related to the frequency of system calls. This is consistent with the evolution of technologies such as multithreading, which allows multiple programs to run on a single processor core, and the increase in the number of cores per processor. Since task scheduling is performed by the operating system, even on the same benign behavior, system calls very rarely have exactly the same execution sequence across an entire system. It is also worth noting the interest in representing system calls using graphs since 2009, and the recent use of embeddings as features, especially since 2020, coinciding with the growing use of deep learning algorithms and ensemble methods for intrusion detection, as can be seen in Figure 8.

Feature Reduction

Feature reduction consists of constructing, from the extracted features, new features of reduced size or dimension. The main approaches adopted are Principal Component Analysis (PCA) and Singular Value Decomposition (SVD), which are closely related unsupervised statistical methods for creating linear combinations of non-correlated variables from observed and possibly correlated variables [21,22,23,24,33,35,86,87]. In our case, observed variables are the features extracted from system call sequences, and the newly created variables are those given as input to the classification model. Other feature reduction methods can be used [30,88,89,90,91], such as Linear Discriminant Analysis (LDA) [92], using the encoder layer of an autoencoder model [93], Recursive Feature Elimination (RFE) [94], and rough sets [95]. It does not appear that any trend is emerging regarding the use or non-use of feature reduction mechanisms nor regarding the type of mechanism used.

Classification Method

Because intrusion detection is a classification problem, an IDS needs at least one way of classifying observed behavior as benign or malicious based on the computed features of system call sequences and potentially other information. Classification methods allow one to raise an alert when the observed behavior is considered malicious. They therefore have two major objectives: to present a maximum attack detection rate in order to detect all the threats for which it was designed while minimizing the number of false alerts that are raised when no malicious action takes place. Several classification methods are presented in the state of the art, which we have tried to group into 15 main and subcategories.

• Heuristics-based classification methods rely on observed patterns within the extracted features and most often define decision thresholds empirically on a measure of similarity or sequence coverage [25,39,42,44,80,96,97,98,99,100,101,102,103]. For instance, TIDE defines a mismatch threshold on the observed sequences compared to the reference sequences to raise an alert [10,11].
• Rule-based classifiers classify system call traces against a set of predefined decision rules, producing static and deterministic classification results [47,62,68,104,105,106,107,108,109,110,111,112,113,114,115,116]. These methods require a detailed analysis of the protected system by experts to ensure that its behavior is fully characterized as well as to avoid false alarms. Rules can be boolean, which means that the classifier assesses whether the observed traces comply or not with the defined rules in order to mark them as benign or malicious. Classification can also be fuzzy rule-based [117], meaning that the set of rules assigns a degree of membership to several classes, from benign to malicious, in a multiple-label manner.
• The execution of system calls can also be seen as a stochastic process which can be modeled with Markov chains [70,77,118,119,120,121], Bayesian networks [78,122], Hidden Markov Models (HMMs) [95,123,124,125,126,127,128], Hidden semi-Markov models (HsMMs) [129], non-stationary Markov chains [130], and Dynamic Bayesian Networks (DBNs) [131] in order to assign a probability score to an observed sequence of system calls so as to determine its likelihood relative to a learned normal distribution. A threshold on this value, often learned during the training phase, triggers an alert when the observed behavior is drastically different from the learned behavior.
• Machine learning encompasses a large number of algorithms that can be used for classification purposes. Some methods, such as Linear Discriminant Analysis (LDA) [24,132], Logistic Regression (LR) [94,133], and Naive Bayes (NB) [86], are linear classifiers that seek to identify a separation between normal and abnormal data and hence require training on both types of data to correctly identify this boundary. Support Vector Machines (SVMs) [27,32,36,67,76,91,93,134,135,136,137] go further than linear classifiers by using kernel functions to project data into a high-dimension space in which a linear separation is applied. A One-Class SVM (OC-SVM) [138,139] is a variant of SVMs trained only on data from one class whose decision boundary corresponds to the distribution of the learned data. This algorithm identifies, at inference, data deviating from this learned distribution and characterizes them as anomalies. Methods based on decision trees (DTs) [140,141], such as Random Forest (RF) [23,34,74,90,142,143,144] or Isolation Forest (IF) [75], define a set of conditions that differentiate normal sequences from abnormal sequences, either by a majority vote of the DTs on the nature of the sequences, or by identifying the conditions that isolate rare data from the rest of the data to characterize them as outliers. The use of DTs allows for an interpretability of classification results because the conditions applied by the DTs can be retrieved. Boosting algorithms such as XGBoost [35,145] can also use DTs but, unlike RF, they do not all have the same weight in the model’s final classification decision. The k-means [79,83,87] and k-Nearest Neighbors (kNN) [29,58,81,146,147,148,149,150,151] classifiers evaluate a function based on distance from a centroid or from neighboring data, respectively, in order to identify groups in the observed data and identify isolated data marked as anomalous. The Gaussian Mixture Model (GMM) [84] is a partitioning clustering algorithm that can assign a probability of belonging to groups of data considered similar to a given data point, thereby identifying data that deviate from these groups with a richer representation of the data than k-means clustering.
  Deep learning is a subset of machine learning defined by the use of deep Artificial Neural Networks (ANNs) [152,153] with several hidden layers of neurons. A Multi-Layer Perceptron (MLP) [22,31,33,46,89,154,155,156] is a feedforward neural network, meaning that information from the inputs propagates through the network only in the direction of the output, like Convolutional Neural Networks (CNNs) [50,57,157,158,159,160] such as WaveNet [56]. The main component of CNNs is convolutional layers, which apply a convolution kernel to the sequential data in order to identify particular local patterns. Conversely, in Recurrent Neural Networks (RNNs) [161], the network treats the sequences step by step, and the output of the recurrent cell at a given step also depends on the output of the cell at the previous step, allowing information to flow along the sequence. Among these RNN-based algorithms are Long Short-Term Memory (LSTM) [51,52,53,55,162,163] and Gated Recurrent Unit (GRU) models [60,164], which are used as classifiers of system call sequences when preserving order is important for anomaly detection. More recent architectures such as Transformers [165] are designed to process input data simultaneously and with better contextualization between the beginning and end of the sequence, unlike RNNs, which require sequential processing. Graph Neural Networks (GNNs) such as GrapheSAGE [48] are networks specifically built to ingest data composed of nodes, for instance processes or files, linked by edges, like system calls. Other specific neural networks such as Extreme Learning Machine (ELM) [82,166,167,168] have been used in the literature for the classification of system call sequences. An AutoEncoder (AE) [21,43,45] is a neural network architecture designed around an encoder, whose role is to compress input data, and a decoder responsible for reconstructing the original data from the compressed representation. This type of model can be used in intrusion detection by using the reconstruction error between input and output as an anomaly score for the observed sequence of system calls. Other variants include Variational AutoEncoder (VAE) [66] and Graph AutoEncoder (GraphAE) [169].
  Reinforcement prediction algorithms with, for instance, Markov reward processes [170,171,172], enable machine learning agents to be trained to recognize benign behavior from malicious behavior based on observed system call sequences.
  Finally, neuro-fuzzy systems use fuzzy set theory to provide fuzzy logic to a neural network which is able to assign to new samples of data a degree of membership in several classes from least abnormal to most abnormal [85].
• Colored Petri Nets are used to detect patterns in system call sequences, which are then characterized as functionalities. Chaining these functionalities together in the context of execution creates a signature for each program, which is then considered benign or malicious by comparison with a database of reference signatures [37,173,174,175,176]. This approach has the advantage of being able to track malicious actions performed on a system in addition to raising alerts.
• Artificial Immune Systems (AISs) are a biology-inspired approach to intrusion detection that uses algorithms abstracting clonal selection through hypermutation processes, negative selection, immune memory, and danger theory to identify abnormal behaviors. Even though Forrest [10] is one of the first studies partly using this type of approach for anomaly detection, we rather consider it as a heuristic-based classification approach, which is in line with more recent similar methods that do not refer themselves as AISs. Typical AIS-based methods used for anomaly detection are described in [28,177,178,179,180].
• Combining several models may provide better predictions than using a single model. Ensemble methods enable the training of machine learning models with varying hyperparameters [35,145,181,182]. Other combinations [30,69,71,72,88,183,184,185,186,187,188,189,190,191,192,193,194,195], such as a combination in ROC space [196,197,198], have also been studied for the detection of malicious behavior using sequences of system call identifiers.

As can be seen in Figure 8, researchers initially used heuristics, rules, and stochastic processes to distinguish benign sequences from malicious ones before turning more unanimously to machine learning algorithms from 2014 and deep learning from 2020. In recent years, IDSs using classifiers that do not rely on artificial intelligence have become almost occasional.

Collaborative

Intrusion detection can be based on the behavioral analysis of several targets at the same time—for instance, to identify rogue devices [199]. In this case, we call an IDS collaborative when several hosts are required for intrusion detection. The topic of collaborative IDSs was addressed sporadically between 2006 and 2013 [41,126,178,179,199]; then, it resumed with the use of federated learning starting in 2023 [21,22,23,169].

Additional Information Used

System call identifiers are information that can be used for intrusion detection, but they can also be used together with other information that provides additional context for detection.

• Additional information may include data from the dynamic analysis of system calls, i.e., directly interceptable with system call identifiers. Examples include timestamps corresponding to the start or end of the call execution [122,132,159], arguments passed as parameters [43,55,70,77,107,112,127,200], information linked to the calling process such as the process identifier (PID) [39,148] or parent-process identifier (PPID) [47,80,174], information linked to the privileges with which the calling process wishes to access resources like EUID and EGID for Linux devices [104,201,202,203], the return value of the system call which indicates the success or failure of the call [74,120], as well as the value of the error raised in the case of a failed call [99,109].
• Resources usage is a good indicator of how much a machine is being used, making it a key piece of information for identifying less evasive malware such as ransomware, whose encryption of the file system results in an abnormal increase in CPU activity, or botnets, whose activation generates significant network activity on infected hosts. Then, a number of studies have incorporated host activity information into intrusion detection in conjunction with system calls. These include the CPU time devoted to each process [84,179,180,185]; the RAM used, allocated and released during execution [84,179,180,185]; input and output usage [185], such as peripherals and GPIO usage; network usage in terms of number of active connections, the number of packets transmitted and received, or bandwidth saturation [179,180,204]; the thread status of monitored processes—that is, whether they are running, sleeping or dead [84]; host power consumption [193], which is intrinsically linked to its activity; and Hardware Performance Counters (HPC) [193], which reflects the micro-architectural state of each microprocessor, i.e., its number of well-predicted conditional branches, its number of cache misses, etc.
• Since system calls are functions executed at the kernel level, higher-level OS Events can also be used for intrusion detection. As an example, [151] uses calls to the Android API in addition to system calls as a feature of a machine learning classifier for anomaly detection. Starting from the observation that the majority of Android applications make use of middleware libraries, based on the Android API, indices about their compromission can be seen through the monitoring of these functions. However, since an application can also directly call kernel functions, this is not a substitute to using system calls to detect malicious behavior. Ref. [28] presents an IDS using keyboard interrupt signals to detect the presence of user-space keyloggers on a virtual machine. In conjunction with system calls related to file system management and networks, the authors succeed in observing the behaviors induced by the presence of each keylogger studied through Virtual Machine Introspection (VMI).
• For intrusion detection involving the monitoring of a particular process, a prior analysis of binaries enables all system calls that may be called during execution to be retrieved. In this way, the consistency between system calls from static analysis of the binaries and those actually observed at runtime can be used to detect anomalies in observed execution [139].
• Other information, not falling into the previous categories, has also been used in the state of the art and designated Other in our taxonomy, including the memory address at which the assembler jump instruction to the system call is located, i.e., the address of the kernel branch instruction making the system call execution routine [51,79], or shell commands causing system call execution [204].

4.2.2. Data Used to Study System Call-Based Intrusion Detection Mechanisms

To answer RQ-1 and RQ-3, it is necessary to be able to describe the data used for the training and evaluation of intrusion detection systems. A number of features have been identified, as detailed below and illustrated in Figure 9.

Data Availability

An important property of the data used in training and evaluating IDSs is their availability. The availability of the data used by authors at the time of publication of their studies was analyzed. Public data encourage the reproducibility of the results presented by authors as well as a comparison with other studies from the literature if they were conducted using the same public data. The use of private data may be due to industrial partnerships whose artifacts cannot be made public or to the study of a use case whose specificity is not present in already public data. In this case, comparison with other intrusion detection methods becomes more difficult, even if such data may come from real installations and the results reported reflect IDS performance under operational conditions. The literature is divided almost equally between work carried out on public and private datasets. A temporal analysis does not reveal any link between the publication of a new system call dataset and an increase in the use of public data. It is also important to note that the UNM [10] and DARPA98 [205] datasets are still used in recent work even though they were made public almost three decades ago. The same applies to ADFA-LD [17], which is the most widely used public dataset according to our analysis, despite the fact that the data were collected more than 14 years ago and are therefore certainly not representative of modern computer systems anymore. Furthermore, as shown in Figure 9, datasets that were once public like UNM are no longer accessible, which makes it difficult to reproduce the findings in the future. Publications considered below for the analysis of the type of data used by authors, the operating system on which they were collected, and the host environment are those whose research uses private data. The reason for this choice is that given that half of the literature is based on a limited number of public datasets, considering them could bias the characteristics of the data we wish to highlight through Figure 9, Figure 10 and Figure 11.

Data Type

Data type refers to the nature of the data used for training and evaluating IDSs. It is considered real when it comes from a real environment, namely existing systems operating for the use case under study and with the usual users. Data are considered coming from a dedicated testbed when it comes from an environment that attempts to reproduce as faithfully as possible the systems used for the use case under study. Finally, it is considered emulated if it comes from an environment that is simplified compared to the systems used for the use case under study. Data from executions on virtual machines are considered emulated data when the use case does not involve a cloud environment. If the use case under study involves user actions, the data are considered emulated when their behavior is approximated using automation frameworks such as Selenium. Whenever researchers use private data, they are mostly collected on testbeds built specifically for generating such data, as shown in Figure 9. A number of emulated systems have been used, probably for the ease and security of deployment when running real malware, at the expense of data quality, which is less representative of real-world use cases.

Data Augmentation

Some studies use data augmentation methods such as Generative Adversarial Networks (GANs) [154,157] and the Synthetic Minority Over-sampling Technique (SMOTE) [145,181] to generate missing data samples for an even distribution of normal and abnormal sequences of system calls or to expand datasets for training large neural networks.

Operating System

System calls constituting datasets may originate from multiple operating systems. Although most work has focused on Unix-based operating systems like FreeBSD and Linux, other work has been based on data from Windows, Android, or OSX systems. Some publications do not describe the origin of the system calls collected for their experiments.

Private data specifically collected for research come mainly from Unix environments (60 publications in our analysis, Figure 9) and Windows (40). A few isolated studies have been conducted on Android since 2014, as can be seen in Figure 10, and one study focused on OSX in 2011.

Host Environment

Regardless of the underlying operating system, data describe the host environment from which system calls are extracted and embedded.

This environment may include any of the following:

• Desktop, such as a personal computer or workstation;
• Server, when the target hosts a few web applications for personal use or a significant number of services as part of an enterprise server;
• Cloud, when the use case focuses on virtualized services running in containers or virtual machines;
• Mobile, when the subject of study is a smartphone;
• IoT/I-IoT, when the target is embedded in an environment with a specialized role of data aggregation and processing, in a home automation or industrial environment.

Some studies do not focus on a specific environment for their IDS research.

The main use case covered by private datasets is office computing, as shown in Figure 11. Interest in the cloud has been growing since 2012, and interest in the IoT has been growing since 2019.

4.2.3. Evaluation Method of System Call-Based IDS

Responding to RQ-1 and RQ-4, evaluations conducted by the authors were analyzed whether they focused on computational efficiency or on measuring the detection performance of presented methods. Findings are presented in Figure 12 and reported below.

Overhead Measurement

The evaluation of intrusion detection systems involves measuring their impact on the resources of the system on which they run. Measuring this overhead involves temporal complexity such as additional CPU load, increased latency or longer execution times; it can also involve spatial complexity such as the RAM used by the program during execution or the persistent memory footprint. These metrics can be measured at different stages of the IDS lifecycle, such as its training phase or during inference. This makes it possible to identify resources required for each phase. For example, an IDS with negligible inference overhead can be deployed on a system with limited resources, but it may be worthwhile to defer the learning phase of this solution to a more powerful machine if it requires a larger amount of computational resources. Conversely, if the target has the necessary resources, local relearning of the IDS is possible following the aggregation of new data. Only a few studies measure the overhead for their IDS, yet when they do, the measurement often focuses on inference time, i.e., once the solution has been deployed.

Evaluation Metrics

The detection performance of IDSs is measured using metrics. Some authors do not provide metrics but supply the raw predictions of their detection model during the testing phase. Detection methods that directly classify sequences as normal or malicious can have their results summarized using confusion matrices showing the number of True Positives (TP), i.e., the number of system call sequences identified by the IDS as malicious and labeled as malicious in the dataset; the number of False Positives (FP), i.e., the number of sequences identified by the IDS as malicious but labeled as normal; the number of True Negatives (TN), i.e., the number of sequences identified by the IDS as normal and labeled as normal; and the number of False Negatives (FN), i.e., the number of sequences identified by the IDS as normal but labeled as malicious. A number of metrics can be calculated from these results:

• True Positive Rate (TPR): also known as recall or sensitivity, this metric corresponds to the number of sequences correctly identified as malicious by the IDS out of all the malicious sequences in the dataset.
• False Positive Rate (FPR): number of sequences falsely identified as malicious out of all sequences labeled as normal.
• True Negative Rate (TNR): also known as specificity, it corresponds to the number of sequences correctly identified as normal out of all sequences labeled as normal.
• False Negative Rate (FNR): the number of sequences falsely identified as normal out of all sequences labeled as malicious.
• Accuracy: number of sequences correctly identified as normal or malicious out of all sequences in the test dataset. It should be noted that this metric is highly dependent on the dataset used, as an IDS with no detection capability that labels all traces as normal could achieve high accuracy if the dataset contained very few malicious samples [206].
• Precision: number of sequences correctly identified as malicious by the IDS out of all sequences labeled as malicious by the IDS.
• F1-score: metric computed as the harmonic mean between TPR and precision, which is preferred for addressing the problem of imbalanced classes where accuracy measurement may not be representative of the IDS’s actual detection capabilities.
• Matthews Correlation Coefficient (MCC): a metric that, like the F1-score, aims to address the problem of unbalanced class distribution by proposing a score proportional to the results obtained on the four metrics of the confusion matrix.

For detection methods that assign an anomaly score to each sequence of system calls, results can be presented using a Receiver Operating Characteristic (ROC) curve showing the detection model’s performance in terms of TPR (y-axis) and FPR (x-axis) for each threshold used to distinguish normal sequences from malicious ones. The Area Under the ROC Curve (AUC) thus represents the classification performance across several decision thresholds. The TPR obtained for a fixed FPR can then be deduced and, conversely, an FPR for a fixed TPR. Other measures, such as generality [207] or the number of virtual machines protected by the proposed detection framework [208], are occasionally used by authors to address the specific features of their solution. No particular trends were observed in measuring the impact of proposed IDSs on the system under study nor the metrics employed to evaluate the performance of detection methods.

5. Results: Evaluation of State-of-the-Art System Call-Based IDSs

This section addresses the following:

• RQ-5: How do state-of-the-art IDSs based on system call identifiers perform under the same execution conditions?

5.1. Selected Studies

The quality of the 86 candidate publications for experimental evaluation was carefully assessed (AQ1), leading to the choice of 14 of them which represents 15 intrusion detection methods to be evaluated alongside the 3 methods proposed by Forrest and her colleagues, namely TIDE [11], STIDE and t-STIDE [209], which have been considered by the research community as gold standard methods. The selected 18 methods are described below, and their characteristics are summarized in

Table 4. Characteristics of selected system call-based intrusion detection methods for experimental evaluation.

Method	Learning Paradigm	Trained on Attacks?	Features	Features Reduction	Classification Method	Data Granularity
[M1]	N/A	no	sequence-based	no	heuristics-based	full sequence
[M2], [M3]	N/A	no	sequence-based	no	heuristics-based	fixed-length sequence
[M4]	unsupervised	no	sequence-based	no	stochastic process (HMM)	fixed-length sequence
[M5]	supervised	no	frequency-based	no	machine learning (kNN)	full sequence
[M6]	N/A	no	group-based (categorization), frequency-based	no	heuristics-based	full sequence
[M7]	supervised	yes	frequency-based	no	machine learning (ELM)	full sequence
[M8]	supervised	yes	sequence-based	no	rough sets	fixed-length sequence
[M9]	unsupervised	no	sequence-based (STIDE, HMM), frequency-based (OC-SVM)	no	combination	full sequence
[M10]	N/A	no	sequence-based	no	heuristics-based	full sequence
[M11]	unsupervised	no	statistical description-based	no	machine learning (IF)	full sequence
[M12]	supervised	yes	sequence-based	no	deep learning (LSTM)	fixed-length sequence
[M13]	supervised	yes	frequency-based	no	deep learning (MLP)	fixed-length sequence
[M14], [M15]	unsupervised	no	embedding-based	no	deep learning ([M14]), combination ([M15])	full sequence
[M16]	supervised	yes	frequency-based	PCA/SVD	deep learning (MLP)	full sequence
[M17]	supervised	yes	sequence-based, group-based (categorization), embedding-based (Word2Vec)	no	deep learning (Text-CNN)	full sequence
[M18]	unsupervised	no	sequence-based	no	stochastic process (Markov chain)	full sequence

.

5.1.1. [M1]

Hofmeyr et al. [11] proposed to collect observed sliding windows of system calls of a predefined length k and of step 1 during a normal and benign execution and to store them as a reference dictionary of sequences of system calls. An anomaly score is then constructed at the detection stage by comparing sliding windows of length k of the observed sequence against those in the reference dictionary; sequences that do not appear are considered as mismatches, and the number of mismatches divided by the total number of subsequences constitutes the anomaly score for the entire sequence. A threshold is then set to distinguish between a normal and an abnormal sequence based on its anomaly score.

5.1.2. [M2], [M3]

Warrender et al. [209] presented two methods derived from TIDE. STIDE ([M2]) differs from TIDE at the detection stage by taking into account the number of mismatches in a temporally close window; the authors called this the Locality Frame Count (LFC). This means that the anomaly score of the entire sequence is computed from the number of consecutive sliding windows that have not been observed during normal execution, with the number of considered consecutive windows being fixed, as for the window size k. A threshold on LFC identifies abnormal sequences from normal ones. t-STIDE ([M3]) adds the occurrences of each sequence of system calls during normal execution to the reference dictionary. Hence, rare sequences in the reference dictionary (defined as sequences appearing less that 0.001% in the normal execution by the authors) are also considered as mismatches and included in the LFC calculation. Again, a threshold on LFC identifies abnormal sequences from normal ones.

5.1.3. [M4]

The method proposed by Yeung and Ding [123] is based on the training of a fully connected HMM from the sequences of system calls observed through sliding windows of length k and step of 1 on a normal and benign execution. A threshold is then chosen to identify normal and abnormal sequences based on the log-likelihood of the score returned by the HMM for each observed sequence.

5.1.4. [M5]

Sharma et al. [147] represent system call sequences in the form of vectors containing the binary presence of each system call in a sequence, i.e., “1” if it is in the sequence, “0” otherwise. The authors present a new measure to quantify the similarity between two of these vectors, which is a derivative of the binary weighted cosine measure [210]. This similarity measure is used as the kernel of a k-nearest neighbors algorithm (kNN), which classifies new sequences of system calls by examining its k closest sequences observed from the normal execution. Then, the anomaly score is computed as the average distance to its neighbors. A threshold on this probability is used to assert the classification of the sequence tested as normal or abnormal.

5.1.5. [M6]

Murtaza et al. [64] do not consider raw system calls but the type of operation they are related to, as defined in the Linux kernel: architecture (10 system calls), file system (131), inter-process communication (7), kernel (127), memory management (21), networking (2), security (3), or unknown (37). Sequences of system calls are therefore replaced by sequences of corresponding operations. The proposed method, called Kernel State Modeling, measures the occurrence of each of three operations (file system, kernel, and memory management) out of the total number of operations in a sequence and compares it with the same frequencies calculated during benign execution of the considered program. If the frequency of file system operations exceeds the maximum frequency observed during training, the anomaly is confirmed. Otherwise, the difference between the frequency of the other two operations and their maximum frequency observed during training is compared to an $\alpha$ threshold, which is computed during the training phrase. If one of these measurements is greater than the threshold, then the anomaly is also confirmed. If none of these conditions is met, the sequence is considered normal.

5.1.6. [M7]

Creech and Hu [166] propose to build a dictionary of “words” composed of sequences of contiguous system calls observed in the training set and of variable size, i.e., 1-g to n-grams with a predefined n. Only words appearing at least 200 times are kept, and then they are combined in all possible ways into “phrases” of a predefined maximum word size. This constitutes discontiguous sequences of system calls from the training set, enriching the number of sequences considered in the reference frame of benign behavior. The count of reference sentences appearing in observed sequence of system calls for each sentence size becomes the feature on which an Extreme Learning Machine (ELM) is trained and tested. A threshold on the score returned by the ELM defines the boundary between sequences considered normal and abnormal.

5.1.7. [M8]

Rather than viewing the anomaly detection problem as a binary normal/abnormal problem, Nauman et al. [207] use rough sets theory to define an in-between region in which suspicious sequences are found, i.e., whose abnormality or normality is not as certain as other sequences. Sequences of system calls in the training set are first split into subsequences by a sliding window of step 1 and a predefined size similar to [M1], [M2] and [M3]. Equal windows are grouped into equivalence classes, and each class is assigned a probability based on the number of windows in the class out of the total number of windows in the sequence as well as a conditional probability based on the number of malicious windows in the class out of the total number of windows in the class. Two thresholds, $\alpha$ and $\beta$, are optimized during the training phase and separate classes into sets of abnormal, in-between, and normal classes, respectively, based on their probability and conditional probability. A sequence of system calls is then predicted as abnormal if at least one of its window is predicted as abnormal. A window is predicted to be abnormal if its equivalence class has a conditional probability exceeding the $\alpha$ threshold; it is predicted to be normal if its equivalence class has a conditional probability lower than the $\beta$ threshold. Finally, if this probability lies between these two thresholds, it is considered suspicious, but it is not classified as abnormal.

5.1.8. [M9]

Khreich et al. [198] start from the intuition that combining several anomaly detection methods allows taking advantage of the strengths of each of them. Multiple models—STIDE ([M2]), an HMM, a One-Class Support Vector Machine (OC-SVM)—using two different representations of system calls—raw for STIDE and HMM, frequency of system calls in the observed sequence for OC-SVM—are trained independently on the same training set of normal sequences of system calls. The authors propose a method called Iterative Boolean Combination, which combines the predictions of each detection method using different thresholds for each model involved, that give the best AUC in the ROC space during the validation phase. A reference FPR is then chosen, and the two points on the validation ROC curve that are adjacent to the chosen FPR are selected, knowing that the considered points each correspond to a model combination rule at predefined thresholds. At inference, the two predictions corresponding to the two selected combination rules are computed, and the final prediction is chosen randomly between those two with a higher probability of choosing the combination linked to the point that is closest to the chosen FPR.

5.1.9. [M10]

Marteau [103] presents a sequence covering algorithm called Sequence Covering for Intrusion Detection (SC4ID) which aims to find the optimal coverage of a sequence of system calls by a set of subsequences extracted from reference sequences observed during benign executions of the system. An anomaly score is constructed as a function of the number of subsequences required for optimal coverage of the sequence analyzed relative to the size of the sequence itself. A threshold then defines the limit above which the sequence is considered abnormal from the computed score.

5.1.10. [M11]

Liu [75] uses neither the raw composition of system call sequences nor their frequency within these sequences as is but rather a set of descriptive statistics on the frequency of n-grams in a sequence. Mutliple values are computed, such as quantiles, maximum value, standard deviation, mean, skewness, kurtosis, standard error, and Shannon’s entropy from all frequencies of 1–3-gram in the observed sequence. These values are then given as input features to an Isolation Forest (IF) model that associates an anomaly score on which a threshold is applied to differentiate sequences considered normal from those considered abnormal.

5.1.11. [M12]

Wunderlich et al. [52] compare different representations of sequences of system calls as input to a recurrent neural network Long Short-Term Memory (LSTM) model, among a one-hot encoding and two embedding representations, Word2Vec and GloVe. Sequences given to the model are subsequences corresponding to a sliding window of predefined length and a step of 1 over the observed sequence of system calls. If one of these subsequences is considered abnormal, then the entire sequence is also considered abnormal. Experiments conducted by the authors showed better results with the first representation, i.e., one-hot encoding, so this is the method that was chosen. The LSTM model combined with densely connected neural network layers classifies each observed sequence as normal or abnormal following the one-hot encoding input of sequences of system calls.

5.1.12. [M13]

Zhang et al. [31] raise the question of the need to analyze the entirety of an execution trace in order to detect malicious behavior. Thus, their method proposes a TF-IDF representation of 2–4-gram by considering only the first 200 system calls generated by each process as input to a Multi-Layer Perceptron (MLP) model which is responsible for the classification of the observed sequences.

5.1.13. [M14], [M15]

Ring et al. [56] describe the Trace-Level Anormaly Detection algorithm, involving a language model called WaveNet [211] trained on sequences of system calls from normal executions. An embedding layer trained at the same time as the neural network is applied to the model input to represent each system call. Given all previous system calls in a sequence, the model determines the probability distribution for the following system call in that sequence. The probability of each sequence is then calculated by multiplying the probabilities of system calls for these sequences, and a threshold on the negative log-likelihood of this probability classifies the observed sequence as normal or abnormal ([M14]). An ensembling model of three WaveNets with a filter number of, respectively, 128, 256 and 512 in each convolution layer is also proposed ([M15]). The score on which classification is based is the average of the differences between the three scores and the median of the training scores for each model, which are input into the ReLU function.

5.1.14. [M16]

Subba and Gupta [33] question the relevance of using the TF-IDF of all the 5-g of system calls in a sequence as the input feature of a classifier. Thus, they apply Singular Value Decomposition (SVD) as feature reduction to identify the most discriminating linear combinations of n-grams between normal and abnormal sequences, and then they use them as input to an MLP that is in charge of classifying observed sequences.

5.1.15. [M17]

The architecture of the intrusion detection solution proposed by Zhang et al. [57] integrates several processes. Each sequence of system calls is first completed by a generative LSTM until it reaches a predefined fixed size. The extended sequences are then used, on the one hand, to build abstraction sequences of system calls according to their scope of action, similar to [M6]; on the other hand, they are used to build differential encoding sequences, i.e., system calls prefixed with an encoding according to their previous system call. The three sequences thus constructed (extended, abstract and encoded) are represented in an embedding form by Word2Vec models learned for each of these representations. Finally, these three embeddings form the input channels for a Text-CNN model [212,213] in charge of predicting the abnormality of the observed trace.

5.1.16. [M18]

The idea of Shamim et al. [121] is to find patterns in the ordering of system calls in a sequence, corresponding to repeated program execution, as can be seen in IoT applications. A segmentation of system call sequences is performed using autocorrelation, the application of the Savitzky–Golay filter [214], and prominence to identify peaks delimiting “segments” of system calls within the observed sequence. The starting point of the longest common subsequence between obtained segments is considered the beginning of the execution of the pattern. A Markov chain is then constructed to determine the probability of each segment of system calls during normal execution. During anomaly detection, system calls are ingested sequentially to find the starting point of the pattern execution. Once this point is found, the probability of the current segment of system calls is calculated from the Markov chain transition matrix. If the identified segment is of a similar length to the one observed during training and its probability is below a certain threshold, the sequence is classified as abnormal.

5.2. Evaluation Workflow

The evaluation of reproduced methods is performed on two public datasets from the scientific literature. The ADFA-LD dataset [17] is the most widely used by studies published over the last decade, as highlighted in Section 4.2.2. ADFA-LD is a set of system call traces collected on an Ubuntu 11.04 operating system running a vulnerable Apache web server with PHP, a MySQL database, a web collaboration tool named TikiWiki, and other services including FTP and SSH [17]. According to the authors, the platform represents a common small server for file sharing, database and web services with remote access. A first considered threat model is that of a remote attacker seeking to compromise exposed services. Four attacks have been developed for this purpose, including two password brute force attacks on FTP and SSH services, an exploitation using a TikiWiki CVE [215], and a custom PHP remote file inclusion. A second considered threat model is that of an attacker using social engineering to have a poisoned executable file run on the server, probably by a technician. Both of these attacks can be used to obtain a reverse shell or to create a new superuser. As the dataset is designed for IDSs with an anomaly detection approach, it is built around three subsets: a training set and a validation set containing, respectively, 833 and 4372 traces of normal web server usage, and a dedicated test set, comprising 746 traces from multiple executions of each attack. NGIDS-DS [216] is a more recent dataset than ADFA-LD containing traces of system calls executed on a corporate server running Ubuntu 14.04 and operating storage, FTP file sharing, email, web application, NAT, DNS, and SSH services. This machine is subject to a significant set of attacks grouped by the authors into seven families: Exploits, Denial of Services, Worms, Reconnaissance, Shellcode, Backdoors, and Generic. The dataset is divided into four long system call traces corresponding to normal server executions and 16 additional traces following the execution of attacks on the same system. To obtain similar system call sequences between both ADFA-LD and NGIDS-DS datasets, the latter traces are divided into sequences of size 2600, which is the average size of those in ADFA-LD. This results in 34,628 sequences of 2600 system calls, which are divided into 29,783 normal sequences and 4845 attack sequences.

Some selected methods require the training set to contain malicious traces, while others do not. Datasets were thus first imported as a whole without taking into account the training, validation, and attack separation of initial datasets. They were then divided accordingly into training, validation, and test sets. Training and validation sets represent 70% of the entire dataset with 20% of this subset being dedicated to validation. The other 30% constitutes the test set. For methods that have to learn only on normal traces, attack traces were divided between the validation set (20%) and test set (80%). One method ([M13]) only considers the first 200 system calls of each trace; datasets were therefore reduced to consider only traces that have at least 200 system calls, while other traces were truncated.

Table 5. Composition of datasets for experimental evaluation of state-of-the-art methods.

Methods	Dataset Composition (Normal + Attack)
	–	Train	Validation	Test
[M1], [M2], [M3], [M4], [M5], [M6], [M9], [M10], [M11], [M14], [M15], [M18]	ADFA-LD	2914 + 0	728 + 149	1563 + 597
	NGIDS-DS	16,678 + 0	4169 + 969	8936 + 3876
[M7], [M8], [M12], [M16], [M17]	ADFA-LD	2914 + 417	728 + 104	1563 + 225
	NGIDS-DS	16,678 + 2713	4169 + 678	8936 + 1454
[M13]	ADFA-LD	1925 + 248	481 + 62	1033 + 134
	NGIDS-DS	16,678 + 2713	4169 + 678	8936 + 1454

lists the content of each set and methods involved.

Once datasets have been initialized, they were preprocessed according to the requirements of each method. Methods were implemented in Python 3.10 using the scikit-learn 1.2.1 package for machine learning algorithms and evaluation metrics, Keras/Tensorflow 2.11 for deep learning models, hmmlearn 2.8 for HMM models, and Gensim 4.3.1 for Word2Vec embeddings. The runtime environment was configured to accelerate the training and inference phases of deep learning models thanks to NVIDIA CUDA, which enables calculations to be parallelized on GPUs. As a first step, the results obtained under the conditions described by the authors in their publications were sought to be retrieved. If these results could not be obtained, authors were contacted for insights. This was the case for several methods:

• [M7]: We encountered difficulties in executing the proposed method in a suitable time. Strictly following the pseudo-codes proposed in the publication exploded our computation time, the method running over several weeks when the paper speaks of several days of execution. After comparing the explanations with those of the thesis manuscript of the main author [217], we realized that the algorithm for generating what they call “sentences” is only based on a selection of “words” that appear at least 200 times in the training set. Once this selection had been made and implementation optimized using suffix trees, the proposed method was able to be executed in a decent amount of time on the ADFA-LD dataset. However, we were unable to evaluate this method on the NGIDS-DS dataset due to an execution time exceeding two months, which made it unsuitable for this paper. In addition, the presented detection performance could not be found despite contacting the team who had worked on the subject, who informed us that they no longer had the source code and could not help us any further.
• [M14], [M15]: The main difficulty encountered was that we were unable to obtain the results presented for the ensembling method despite the availability of the used source code, which is claimed to achieve superior detection performance compared to using the WaveNet model alone. The authors were very responsive and helpful, directing us to the right branch of the versioning repository. Their source code was used in order to avoid any errors on our side. Yet the experiments showed no significant improvement in intrusion detection performance between the model alone and the ensemblist model.
• [M17]: The proposed method is rather complex and, although many hyperparameters are given in the publication, it is not always clear to us what they correspond to. For instance, the number of layers in the encoder and decoder of the sequence completer is not specified nor is the length of each window or whether padding is used. A number of 150,000 epochs is mentioned for training the Word2Vec model, which seems disproportionate. The pool size for MaxPooling of the Text-CNN model is not mentioned nor do the authors mention the use of a validation set. We therefore contacted them but received no response despite several enquiries. Then, without feedback as to the quality of our re-implementation, the reproduced method does not allow the results presented by the authors to be retrieved.

The seed used to initialize machine and deep learning model weights is set before runtime. In addition, hyperparameters that are not given by the authors are retrieved by a grid search using the validation set so that all models are evaluated in their most favorable configuration. Each method is launched 10 times on ADFA-LD and 5 times on NGIDS-DS to have results that are independent of the random data split between train, validation and test sets, and then they are evaluated on a set of metrics depending on its approach. We chose to consider only five executions for NGIDS-DS due to the number of sequences contained in the dataset, which significantly increases the training and testing time of certain methods, sometimes exceeding our defined limit of 60 days per execution. Score methods ([M1], [M2], [M3], [M4], [M5], [M7], [M9], [M10], [M11], [M14], [M15]), which compute an anomaly score for each sequence of system calls, are evaluated on AUC, TPR for fixed FPR, and FPR for fixed TPR. Raw results and ROC curves are given in Appendix C. Prediction methods ([M6], [M8], [M12], [M13], [M16], [M17], [M18]), which answer a binary classification problem between normal and attack traces, are evaluated on accuracy, balanced accuracy, precision, F1-score, TPR, and FPR, as defined in Section 4.2.3. Raw results and confusion matrices are given in Appendix D. However, these metrics do not allow for a comparison between the performance of scoring methods and prediction methods. Therefore, we use Normalized Discounted Cumulative Gain (NDCG), imported from research in information retrieval, as a metric evaluating the quality of the classification. The NDCG @n measures the relevance of the anomaly score associated with the n sequences that had the best predictions. The relevance score is calculated between the associated anomaly score and the true label of the sequence, which penalizes normal traces predicted as abnormal. In other words, the more a detection system associates high anomaly scores with truly abnormal behaviors, the higher its relevance score will be, and therefore the better its NDCG will be. Although this metric does not represent a measure of the performance of models alone, it allows methods to be ranked according to the quality of their detection on the same sequences. Finally, execution times for training and inference phases are logged. These measurements are performed one method at a time on the same machine with an Intel bi-Xeon Gold 6348 (2×28 cores at 2.6 GHz) as CPU, 512 Go of RAM and an NVIDIA A100 GPU with 80 Go of VRAM, running Scientific Linux 7. Training is carried out on the GPU for deep learning models, but inference is forced per trace on CPU instead of per batch on GPU in fairness to other methods that cannot benefit from hardware acceleration and to become closer to an on-line inference that receives one trace after another.

5.3. Detection Performances

Figure 13 shows the detection performance achieved by reproduced score methods.

A first point to note is the poor performance of [M4]. Being based on an HMM, it seems unable to model the normal behavior described in both datasets. Specifically, the authors designed this method to track system calls of a single program, whereas ADFA-LD and NGIDS-DS are built from the system calls of each program running on the studied machine. This leads us to believe that intrusion detection at the system level is too complex to be achieved through a stochastic process. On the other hand, despite being designed by its authors for anomaly detection on a single program, [M1] is surprisingly among the most effective methods. Based on sequences coverage, it inspired [M10], which offers almost the best True Positive Rates for a False Positive Rate set at 1% but also provides the worst False Positive Rates for a detection rate of 100% on both datasets. [M5] also delivers strong detection performance, achieving the third-best TPR for an FPR set at 1% and the best FPR for a 100% TPR on ADFA-LD. However, it performs significantly worse on NGIDS-DS. It can be explained by the complexity of the dataset which represents a constantly busy enterprise server and that therefore generates highly diverse system call sequences. This method, which uses a similarity measure as the kernel of a kNN algorithm, thus seems to encourage anomaly detection by applying clustering on the presence of system calls in observed sequences but only when the use case implies that the system is being protected from being overloaded as in NGIDS-DS. The results presented by [M7] are encouraging, but it never achieved the 20% FPR for the 100% TPR described in the original publication despite being evaluated on the same ADFA-LD dataset. Exploring other combinations of system calls from the dataset likely permits the model to be more generic in characterizing normal behavior than methods such as [M1], [M2], [M3] and [M10], whose reference system call patterns are created solely from observed training data. This over-generalization seems also responsible for the low number of False Positives for the 100% detection of true positives, yet it fails to detect traces of actual attacks when a low FPR is enforced. Meanwhile, this method is very complex and requires tremendous computing resources to be executed in an acceptable amount of time. In our experiments, one iteration of training, validation, and test exceeds 60 days on the NGIDS-DS dataset, making the method impractical for anomaly detection when the target behavior is too scattered. The results presented by the authors are reproduced for [M14]. As it does not outperform the other methods on any single metric, it appears to perform well on all of them on ADFA-LD: it achieves twice the detection rate obtained by [M7] for a TPR of 1% even though the model was trained only on benign system call sequences; it therefore manages to detect malicious sequences better than a supervised model that has been exposed to malicious patterns during the training process. The FPR for a 90% TPR is also similar between these two methods. In contrast, the model has a quite high False Positive Rate for 100% detection. Using embeddings to represent system call sequences appears to be suitable for use with neural networks, but improvements are needed. Furthermore, using multiple WaveNet models with different hyperparameters does not seem to significantly improve detection performance compared to a single model, which is seen by [M15] following the results of [M14] on both datasets. [M11] fails to compete with other methods on ADFA-LD and has the worst AUC on this dataset apart from [M4]. Since the method uses a set of descriptive statistics on the frequency of system call n-grams in a sequence as detection features, there is a significant loss of information, first through the n-grams, which result in the loss of information about the order of execution of system calls, and then through frequency analysis. In addition, it achieves the second-best AUC on NGIDS-DS, which is due in particular to its reduced FPR for 90% TPR. The “coarse-grained” analysis of system calls therefore appears to be useful when the behavior of the target under study is too complex to be tracked by a detection method based on the sequences themselves. Then, it has the advantage of being independent of system call identifiers, which is an interesting property that would allow a model to learn behaviors on one system that remain valid on another. The combination of heterogeneous classifiers in [M9] seems to produce positive results despite its complexity, which made it impossible to evaluate it on NGIDS-DS. Given the results of [M4] based on an HMM, the contribution of this same model in the combination proposed by [M9] is questionable. The AUC obtained by this method is higher than that of [M2], which is consistent with the way the combination of models is constructed to maximize this metric. The fixed TPR or FPR values are better than [M2] alone, which is most likely due to the OC-SVM. However, the False Positive Rate for a TPR of 90% and 100% is quite high. Taking advantage of the strengths of several models therefore seems to be a good idea as long as this combination makes sense and the benefits of each model have been proven.

The detection performance of the prediction methods is presented in Figure 14.

The precision of [M13] and [M16] is especially noticeable on ADFA-LD, surpassing other methods with an extremely low FPR and good detection capability. Using a TF-IDF vectors of n-grams as a feature of an MLP model, with or without SVD, thus seems more than appropriate on this dataset. However, these methods have no discernment capability on NGIDS-DS given the systematic characterization of sequences as normal. The cumbersome mechanism deployed for [M17] also fails to identify abnormal traces in NGIDS-DS. Its TPR is the second highest on ADFA-LD among the reproduced methods, but it also has the second highest FPR on that dataset. [M6] is also in this situation with unsatisfactory results on both datasets. One reason is probably the replacement of system calls with their operation types, which seems to overly simplify sequences. In a frequency analysis of the composition of each trace of system calls, such simplification makes it hard to differentiate between sequences resulting from legitimate behavior and those resulting from malicious behavior.

Rough sets for system call sequences classification are an interesting approach given the results of [M8], but they do not perform as well as [M16]. As shown in [M12], using recurrent neural networks like LSTM along with one-hot encoding for each system call gives a really good TPR on ADFA-LD but at the cost of a really high FPR. A noteworthy feature of this method is that the LSTM processes sub-windows of 20 system calls, and the entire system call trace is predicted as abnormal if any of these windows are predicted as abnormal. This explains the highest FPR of all the reproduced methods as well as the fact that [M12] predicts nearly every sequence as abnormal on NGIDS-DS. Finally, recognizing patterns in sequences using a Markov chain as proposed by [M18] achieves a higher TPR than [M6] on ADFA-LD and compared to most methods on NGIDS-DS, but it also has a high FPR, giving it the lowest F1-score on ADFA-LD but the highest on NGIDS-DS.

NDCG scores are presented in Figure 15 in a sorted list to help assess the quality of predictions and scores produced by each intrusion detection method. From @1 to @1000, the variance of NDCG scores increases while bias decreases. [M10], [M12], [M13], and [M16] assign an anomaly score or accurate prediction to malicious traces on ADFA-LD. However, they do not perform well on NGIDS-DS and are among the lowest in NDCG @1000 scores. Surprisingly, [M1] achieves the best results on this dataset. The derived methods [M2], [M3], and [M10] are not far behind, as is [M5]. The standard deviation of [M4], [M5], [M12], [M13], and [M17] is higher than the other methods at @1 on ADFA-LD. Still, on this dataset, [M4], [M5], [M7], [M11], [M17], and [M18] maintain large standard deviations on @50, @100, and @500, meaning that the anomaly score these methods associate to truly malicious traces is not always adequately high. On NGIDS-DS, [M5], [M10], and [M16] have a high standard deviation at NDCG@1, and [M12], [M16], and [M18] maintain it at @50, @100, and @500. As such, methods whose classification is based on heuristics obtain more relevant predictions or anomaly scores on NGIDS-DS attack traces than the deep learning-based models that perform best on ADFA-LD. This seems counterintuitive, since this type of algorithm would seem to be better suited for modeling the complexity of behavior under NGIDS-DS and thus more suitable for complex datasets. The method to be considered for a detection solution therefore depends heavily on the dataset and, by extension, on the use case in question.

All methods seem to be completely misguided on NGIDS-DS despite having the same evaluation pipeline as ADFA-LD. In particular, grid searches are redone at each evaluation iteration to find hyperparameters that allow methods to obtain the best predictions. The fact that reproduced methods yield more contrasting results on ADFA-LD than on NGIDS-DS seems to confirm that the latter is a more difficult dataset than ADFA-LD, which we associate with it containing the system calls of a heavily used corporate server. Normal behavior then appears to be too complex to be satisfactorily characterized by the methods evaluated in this paper. Given the results of score-based and prediction-based approaches that were reproduced, it can be inferred that no single approach stands out from the others with significantly higher performance on both datasets. The analysis of raw system call identifiers frequencies in a trace followed by an MLP classifier ([M13], [M16]) is a powerful method compared to reproduced ones, maximizing TPR while minimizing FPR, although the feature reduction based on SVD ([M16]) seems to undermine results. Representing sequences by embeddings, whether one-hot ([M12]) or learned by another model ([M14], [M15]), is relevant when used as input to a neural network. Lastly, sequence coverage methods ([M1], [M2], [M3], [M10]) are also effective and are consistent in the relevance of the given score. However, no unsupervised method performs as successfully as supervised methods for a given FPR or TPR. Furthermore, methods using multiple representations of system calls such as [M17], based on extended sequences and encoded system calls, as well as their functional abstraction, seem like a good idea even if the reimplemented method does not achieve superior detection performance over simpler methods such as [M13].

5.4. Overhead Evaluation

Observing the training and inference times for each method, as shown in Figure 16, clearly reveals that there are no consistent correlations between a method’s temporal complexity and its detection capabilities.

Method [M4] obtained the poorest detection results on the ADFA-LD dataset despite being one of the most time-consuming methods to train and infer. [M6] and [M8] are among the least costly to train and infer, as is [M1], but their prediction quality is insufficient on both datasets. Methods [M2] and [M3], derived from [M1], achieve slightly lower detection performance and training time, but they take longer to infer. Similarly, [M10] requires more time to train and infer than [M1], which is probably due to the complexity of computing the largest coverage sequence for each trace. Clustering method [M5] has low training resource requirements but very long inference time as a result of kNN operation. Methods [M7] and [M9] are among the most challenging methods for both phases. This is the reason why these methods could not be evaluated on the NGIDS-DS dataset, as this dataset contains many more system call sequences than ADFA-LD, which caused the execution time to increase exponentially. As a result of the number of metrics computed as features for [M11] and the Isolation Forest model, this method is also as time consuming as [M13]’s MLP model to train and [M12]’s LSTM model to infer. Compared to lighter methods, [M11] requires more resources for training and inference but provides weaker detection performances on ADFA-LD. [M18] offers reasonable resource requirements at the cost of limited relevance of anomaly scores and is one of the least effective methods on ADFA-LD. Unsurprisingly, the use of deep learning models ([M12], [M14], [M15], [M16], [M17]), except [M13], generates very long learning and inference phases that could possibly be further reduced using model pruning [218] and weights quantization [219] if applicable.

6. Discussion

6.1. Shortcomings of Existing Work

6.1.1. Difficulty in Reproducing Methods from the Literature

As feedback, we can share the difficulty of reproducing the work of other researchers. Indeed, the information needed to reproduce presented methods is often missing, such as hyperparameter values for machine and deep learning models. This requires additional work to find the optimal parameters, which runs the risk of deviating from those actually used by the authors. To address this issue, we strongly encourage authors to publish all the elements needed to reproduce their method, including source code where possible.

6.1.2. Terminological and Evaluation Inconsistencies

The growing use of algorithms from fields that are rapidly evolving, such as machine learning as demonstrated in Section 4.2.1, also leads to confusion in the terminology used by authors—for example, in the use of the term “semi-supervised learning” to refer to single-class supervised learning. Similarly, authors do not always remind readers of the formulas used to calculate their evaluation metrics. However, this would make it easier to identify the same metrics presented under different names, such as “detection rate,” which is often synonymous with “true positive rate” in several publications [81,166,195]. The authors also do not always justify their choice of metrics, as illustrated by the analysis in Section 4.2.3: why choose to compare the performance of the method presented by accuracy, which depends heavily on the underlying dataset [206], rather than the F1-score? We believe that the metrics used to measure IDS detection performance should be dependent on the needs of the considered use case. For example, monitoring a critical system will tend to favor detecting as many attacks as possible without any False Positives. In this case, measuring TPR for a fixed FPR may be relevant. If an IDS is designed to be used in conjunction with other mechanisms aimed at reducing the number of false alarms, then TPR is an interesting metric. The relevance of the anomaly score associated with an alert is also valuable information that enables incident response teams to take suitable action. In this case, an IDS that maximizes the NDCG score would be appropriate. We recommend that authors always make the raw scores obtained during their evaluations available, such as ROC curves and confusion matrices, so that other metrics than those presented can be recalculated a posteriori.

6.1.3. Usage of Outdated Datasets

The presented IDSs performances are based on tests performed on two datasets from the literature, namely ADFA-LD and NGIDS-DS. The first, ADFA-LD [17], is the most widely and recently used for evaluating system call-based IDSs, as analyzed in Section 4.2.2. It contains few attacks and comes from a now obsolete system: the Linux kernel from which the system calls were collected was discontinued in 2011 [220]. This directly limits the credibility of evaluations conducted by researchers, as such data do not provide a basis for comparing the performance of detection methods against modern threats and on current systems. The second dataset, NGIDS-DS [216], offers a much more comprehensive set of attacks, but the threat model is not explicitly described in its publication. It is therefore difficult to clearly identify the use case represented in these data and to comment on the very poor performance of literature methods on these data, as presented in Section 5.3. Furthermore, although more recent than ADFA-LD, the Linux kernel of the system on which the NGIDS-DS system traces were collected also saw its maintenance discontinued in 2015 [221]. Other public datasets exist [10,205,222], but they are not always up to date [223] and only partially cover the various intrusion detection application cases, such as IoT and industrial systems [12,18]. On the other hand, the lack of updated datasets limits the study of concept drift in system calls for the detection task, which we did not see addressed in any of the publications analyzed for this paper.

6.2. Limitations of System Call-Based IDSs

Although intrusion detection based on system calls offer promising detection performance, this type of detection is intrinsically limited by design.

6.2.1. Possible Weaknesses in the System Call Collection Mechanism

IDSs based on the dynamic analysis of system calls are, by design, unable to detect attacks that do not invoke system calls. In contrast, there are very few actions that an attacker can perform without invoking a system call during the entire attack chain [224]. Given this observation, the effectiveness of an IDS based on system calls is highly dependent on its implementation. For example, the retrieval of system calls generated at runtime by a user-space hook can be bypassed by an attacker making direct system calls to the kernel [225]. Thus, a malicious actor could carry out their actions without the IDS recording their behavior. Possible countermeasures include collecting system calls at a privileged level, for example through a kernel module or an eBPF program for Linux operating systems, coupled with a whitelist of programs authorized to make direct system calls [225].

6.2.2. Vulnerability to Mimicry and Adversarial Attacks

If intrusion detection is based on recognizing sequences of system call identifiers that have been learned as normal behavior, then a malicious program capable of artificially reproducing these sequences can hide its actions within sequences authorized by the IDS, for example by calling system calls that are bound to fail. Such a program could also scramble its execution signature so that it does not match any known malicious behavior. This type of attack is known as a mimicry attack [224]. Although this threat is well known to researchers, it does not appear in any of the publicly available datasets identified in this literature review. This could explain why, in the experimental evaluation conducted, methods that are particularly sensitive to basic mimicry attacks, such as Forrest et al. [10] according to Wagner and Soto [224], achieve detection performance similar to or even better than the more complex methods based on machine learning outlined in Section 5.3. The latter, however, is not immune to adversarial attacks, as reported in several surveys [226,227,228,229]. Thus, system call identifiers alone are not sufficient for the comprehensive monitoring of protected systems. The use of other information such as arguments and return values has been studied in this regard, making the imitation of legitimate sequences a much more complex task for an attacker. On the other hand, adversarial attacks aimed at disrupting machine learning models responsible for detection need to be clearly studied in order to build robust IDSs.

6.3. Towards Deployable System Call-Based IDSs

6.3.1. Learning from Attack Traces Is Difficult to Achieve on a Real Installation

The vast majority of IDSs require a stage of characterization of the normal behavior of the system on which they aim to be deployed, as we saw in Section 4.2.1. This stage requires data, in our case system calls, from the final system in operation. However, our analysis of the literature has also shown that almost half of the proposed IDSs are trained on data that include malicious behavior. Yet, it is not always possible to obtain these data due to the operational risks involved. A possible approach would be to extract features that are independent of system call identifiers or other data that are highly dependent on the underlying system, such as the method proposed by Liu et al. [14]. This would allow detection systems to be trained in controlled environments in which a known set of malicious actions can be executed without the risk of compromising or affecting the real system. This kind of approach also supports the development of generalized and scalable intrusion detection systems.

6.3.2. Detect Intrusions in Real-Time

Some detection models are designed to detect anomalies in system call sequences corresponding to an entire program execution. This limits the response capabilities following an intrusion, as the IDS will only be able to raise an alert once an attacker has already impacted the target. Furthermore, this is not compatible with certain applications, particularly IoT and industrial systems, where it is not possible to wait for a process to finish executing and then raise an alert, as most application processes never stop. It would therefore be preferable to turn to detection models capable of raising alerts while processes are running either by analyzing the execution of a specific number of system calls, referred to as fixed-length sequences data granularity in Section 4.2.1, or by regularly analyzing system calls executed over a period of time, which is referred to as time interval in our literature review. Real-time detection also requires that detection mechanisms be capable of processing data as it arrives. This means that inferences must be fast whether they are integrated into the monitored device [21,121] or offloaded to an infrastructure with more computing resources [32,89]. The process of collecting system calls can also be lightened by selecting only a few key system calls for anomaly detection [24,25,132,202] or by reducing the number of features extracted as seen in Section 4.2.1. Unfortunately, publications in the literature do not systematically provide information on the computational or memory complexity of their solutions, as can be seen in Section 5.4, which is nevertheless crucial for evaluating the applicability of proposed solutions to real-world use cases.

6.3.3. Deal with Frequent and Not Always Explicable Alerts

A difficulty encountered by IDSs based on system calls is related to the extensive use of machine learning in the detection process, as illustrated in Section 4.2.1. Explaining decisions made by linear models (e.g., linear discriminant analysis, SVM with linear kernel) and decision trees easily provides an explanation for the anomaly scores obtained, whereas non-linear algorithms and neural networks require the use of additional mechanisms to make their results partially explainable [230]. This leads to alerts that cannot always be explained as such, which is not recommended when the origin of the alerts needs to be investigated further to respond appropriately. Another difficulty inherent in the use of machine learning is the frequency of False Positive raised by these models. Indeed, consider an IDS that analyzes every second the system calls executed in the last second with a False Positive Rate of 1%, which is lower than the performance obtained by the methods evaluated in Section 5.3 on ADFA-LD; this means that this model will raise, on average, one false alarm every 100 s, making it unmanageable for deployment in a real environment. One way to reduce these False Positives is to combine them with detection models that use information other than system calls, which could then confirm or refute intrusions. However, monitoring additional information or running multiple detection models necessarily impacts the performance of the monitored system. Precise overhead measurements are then required before this type of detection system can be deployed on actual devices.

6.3.4. Interfacing with Response Mechanisms

Detecting intrusions provides knowledge of malicious actions being carried out on a system, but it does not in itself provide protection against them. A highly virulent attack can compromise a machine and spread to others very quickly before an incident response team has had time to intervene. An Intrusion Prevention System (IPS) is an IDS equipped with the ability to autonomously block malicious actions as soon as they are identified. However, just as the incident response team needs to know the source of the alert to take the appropriate measures to contain an intrusion, an IPS needs certain context from the detection mechanism to identify which actions to take. Intrusion detection is therefore not just a matter of binary classification: the relevance of the anomaly score, the explainability of alerts, and the identification of resources potentially affected by an incident are essential for cybersecurity teams and for interfacing with intrusion response mechanisms.

7. Threats to Validity

The methodology adopted for this paper, defined by Kitchenham et al. [19], identifies several biases in conducting a systematic review of the literature. Three widely used scientific libraries by the computer science community were used to carefully select primary studies with a common research logic to mitigate selection biais. In addition to the extensive coverage of published work, the inclusion and exclusion criteria for the selection of primary studies were clearly defined during the definition of the methodology before any analysis of existing work to avoid being guided in any way toward the definition of any specific criteria. The attrition bias occurs when there is a difference in processing several publications, leading to their exclusion from an analysis where they deserve to be included. Then, the mentioned advice [19] was applied by noting the rationale for the exclusion of concerned studies which were cross-read by each author, which is available as described in the Data Availability Statement at the end of this article. There is therefore little room for the personal interpretation of an author, which might have been reflected in this paper. We would also like to emphasize that no Large Language Model (LLM) was used in the analysis of the publications presented in this study.

A second selection bias may have occurred during the identification of methods to be reproduced for evaluation. We attempted to minimize this by strictly applying inclusion criteria, particularly the requirement that each publication contain all the information necessary to replicate the described methods. This significantly reduced the number of candidate publications. We then identified the methods that seemed to us to be the most innovative and diverse, based on the characteristics identified in the taxonomy, in order to obtain a sample of the various methods proposed in the scientific literature. Furthermore, it is possible that one or more methods have been badly re-implemented, leading to an unfair evaluation and a performance biais. This biais was mitigated by cross-checking the re-implementation of methods between authors, which have been made using the same Python packages and environment, in addition to cross-reading the associated publication. The implementation of reproduced methods was first validated by retrieving the results obtained by the authors in their publication, which was followed by making contact with them when we were unable to do so, which greatly limits the unfairness of assessments. Finally, the measurement bias was limited by defining a common evaluation strategy for reproduced methods before any experiment and after analyzing the literature in order to identify what we recognized as best practices.

8. Conclusions and Recommendations for Future Work

In light of the drastic increase in the number of cyberattacks perpetrated on digital infrastructures worldwide, it is becoming necessary to develop new ways of detecting such malicious actions. Researchers have focused in particular on intrusion detection based on the dynamic analysis of system calls at runtime.

In order to characterize the different proposed approaches and identify trends in this area of research over the last few years, a systematic review of the literature of 209 publications from 1996 to early 2026 was conducted, the methodology of which is rigorously presented in Section 3. From this, five research questions have been identified:

• RQ-1: How has research on IDSs based on system calls evolved since 1996?
• RQ-2: How are system call-based IDSs built?
• RQ-3: On what data are IDSs based on system calls trained and tested?
• RQ-4: How are IDSs based on system calls evaluated?
• RQ-5: How do state-of-the-art IDSs based on system calls identifiers perform under the same execution conditions?

The first four research questions, RQ-1, RQ-2, RQ-3 and RQ-4, were addressed in Section 4 by analyzing the literature and defining a taxonomy comprising 16 categories from the detection method to the data and evaluation metrics used. Methods based on heuristics, rules, and stochastic processes, although widely developed in the early days of intrusion detection research, are becoming less common in recent work in favor of more complex algorithms. In particular, this paper highlighted the emergence and subsequent dominance of machine learning algorithms in the task of intrusion detection, which can be assimilated to a classification problem. The most studied targets are Unix targets in desktop environments with growing interest in Cloud and IoT use cases. Finally, the evaluation metrics favored by researchers for assessing the detection performance of proposed methods are the TPR and FPR, which are followed by accuracy, F1-score, and precision measures. The majority of publications do not present an analysis of the resources required to run presented IDSs, and when they do, they focus on a measure of temporal complexity during the inference of detection methods. The study of computational resources required to operate a proposed detection system is nonetheless crucial to put into perspective the trade-off between achieved detection performance and required resources.

The last research question, RQ-5, is addressed in Section 5 by reproducing 18 methods representing the diversity of approaches proposed in the literature according to the aforementioned taxonomy. Methods were evaluated on two datasets from the literature, ADFA-LD and NGIDS-DS, and according to the metrics identified in response to RQ-4, to which we added the NDCG calculation to rank methods according to the relevance of the alerts they raise. These results showed that analyzing system calls for intrusion detection is highly relevant, as it yields promising results as a defense-in-depth mechanism, particularly when using an anomaly detection approach whose main advantage lies in the detection of zero-day, i.e., unseen, attacks. However, no unsupervised method performs better than supervised methods, and none of them achieve satisfactory results on a difficult dataset like NGIDS-DS. Then, measuring the training and inference time of each reproduced method also revealed that there is not always a link between the computational complexity of a method and the accuracy of its detection.

We have analyzed in more depth in Section 6.1 the shortcomings found in existing work, particularly the difficulty in reproducing methods from the literature due to the lack of information in publications. The use by researchers of public datasets that are obsolete, both in the system studied and in the considered attacker model, is also cited as a potential factor undermining the credibility of the results obtained, particularly when compared to modern systems and threats. The main inherent weaknesses of IDSs based on a dynamic analysis of system calls are summarized in Section 6.2, from the implementation of system call collection mechanisms to the vulnerability to mimicry and adversarial attacks. Finally, the paper concludes in Section 6.3 by identifying the main challenges associated with deploying methods developed by researchers in real-world environments, from the training to the deployment of an on-line solution.

In summary, this paper has helped identify critical issues in the research on intrusion detection from dynamic analysis of system calls. Researchers are therefore encouraged to pursue the following avenues in future work:

• Build test benches that enable the creation of public datasets representative of current systems and threats based on realistic knowledge databases such as MITRE ATT&CK and Cyber Kill Chain;
• Propose detection methods capable of learning solely from normal execution traces and able to raise alerts during program execution;
• Evaluate proposed IDSs on public datasets, including as much information as possible for reproducing their work and, if possible, publishing the source code.
• Justify the choice and provide definitions of the metrics used to evaluate the detection performance of the methods they present;
• Provide an analysis of the computational complexity and memory requirements for the execution of their method;
• Propose mechanisms explaining raised alerts.