A Survey on Self-Supervised Learning in Cybersecurity: Network Intrusion and Malware Detection

Abstract

Self-Supervised Learning (S-SL) is a recent line of research that could represent the next step to understanding human intuition. By blending the strengths of unsupervised and supervised learning paradigms, S-SL endows Deep Learning models with stronger generalization capabilities. Although better known for its applications in Computer Vision and Natural Language Processing, S-SL has also proved its value in other fields, such as cybersecurity. In this work, we review the current progress and future trends in S-SL for the two most relevant problems discussed in the cybersecurity literature: network intrusion and malware detection. The scope of this survey spans from 2019 to 2025. From an initial analysis of over 200 documents, we distill the 50 most relevant papers. We also highlight opportunity areas, such as attack detection over encrypted network traffic, RAM-based analysis of obfuscated malware, creating S-SL models for tabular data and resource-constrained devices, as well as the research on backdooring, encoder extraction, the transferability of vulnerabilities, and data memorization in S-SL. To the best of our knowledge, this is the first comprehensive survey regarding the application of Self-Supervised Learning in cybersecurity, benchmarking contrastive learning vs auxiliary pretext tasks and presenting the data requirements for implementing S-SL solutions in this field. We hope this paper provides a firm ground for further exploration.

1. Introduction

Supervised learning encompasses the development of predictive models through the utilization of a training dataset, which comprises a multitude of instances paired with the corresponding labels that represent the ground truth outputs [1]. However, the deployment of supervised learning techniques becomes increasingly challenging with the expansion of dataset volumes, primarily due to the necessity of human intervention in labeling each data record. This process not only incurs significant costs but also introduces the potential for annotation biases [2]. On the other hand, unsupervised learning methods emerge as an alternative to alleviate the dependency on extensive labeled data. Despite this advantage, models based on unsupervised learning have encountered issues in achieving effective generalization [3].

Self-Supervised Learning (S-SL) emerges as a possible solution to the two problems mentioned above, where its capacity to learn from unlabeled data and strong generalization can represent the next step to understanding human intuition [4]. S-SL can be seen as a fusion between unsupervised and supervised learning, where the model is pre-trained with unlabeled data to learn a strong representation of the knowledge domain, via carefully defined pretext tasks or via contrastive learning, and then the resulting model is fine-tuned with labeled data for specific target tasks (also known as downstream tasks) [5]. Although S-SL still considers labeled samples as part of the training process, it reduces the need for large-scale labeled data and exploits the advantages of Few-Shot Learning (FSL) [6]. Moreover, S-SL exploits the benefits of Deep Learning (DL) networks regarding representation learning [7], whilst saving time originally used for feature engineering [8].

Semi-supervised learning is a training paradigm used when there are large amounts of unlabeled data available and typically smaller sets of labeled data [9]. Although there have been few direct comparisons between S-SL and semi-supervised learning, both can work in tandem given that S-SL deals with model pre-training, and semi-supervised learning handles the fine-tuning process [10,11]. See Figure 1 for a distinction between the learning paradigms discussed thus far.

Figure 1. Distinction between Self-Supervised Learning and the main paradigms of unsupervised and supervised learning. Regarding semi-supervised learning, Self-Supervised Learning differs in the incorporation of an initial context extraction phase. X indicates the matrix of samples, whilst y is the corresponding vector of labels.

S-SL has been successfully applied in Computer Vision (CV) and Natural Language Processing (NLP), achieving better performance than supervised methods. For CV, S-SL has been used in tasks such as object segmentation, image colorization, and frame’s order verification [12]. Regarding NLP, S-SL has been widely adopted for masked prediction (i.e., predicting words and sentences) and language translation [10]. Concerning communication networks, recent research has shown its viability for content type detection in encrypted traffic [13,14], outperforming supervised approaches in deployed Software-Defined Networking [15] scenarios by about 2% in terms of accuracy. Indeed, in the context of cybersecurity, S-SL has been shown to be valuable in the detection of attacks for non-encrypted traffic [3,16,17].

Cybersecurity emerges as a critical domain, especially in the context of escalating threats such as malware, data breaches, and exploitation of system vulnerabilities. These challenges have significantly transformed defensive strategies, marking a transition from traditionally reactive measures to proactive approaches. Contemporary strategies heavily rely on predictive modeling to preemptively identify and thwart cyberattacks, thereby mitigating potential damage [18]. The repercussions of these cyber incidents are multifaceted, encompassing not just financial and temporal losses but also reputational harm, impacting diverse stakeholders within the system, including clients and software developers [19]. Consequently, enhancing the efficacy of such proactive responses is not merely a technical necessity but also a critical determinant in organizational resilience and strategic decision-making regarding investments in cybersecurity [20].

S-SL is a recent line of research. In the state of the art, we can find surveys from 2021 that explain the potential of this novel learning paradigm for videos, images, audio, text, and graphs [2,5,10,12,21,22]. Nevertheless, to the best of our knowledge, there is no literature review that studies the applicability of S-SL in the cybersecurity domain. Additionally, state-of-the-art surveys regarding the application of DL to cybersecurity [23,24,25,26,27,28,29,30,31,32] ignore S-SL as an Artificial Intelligence (AI) paradigm for model training. Only the supervised, unsupervised, and semi-supervised learning strategies are mentioned, as well as Deep Reinforcement Learning (DRL). Cybersecurity can benefit from S-SL due to the lack of labeled real-world datasets and the pressing need for generalization in detecting constantly evolving attack patterns.

Therefore, in this literature review, we explore the details of the current progress and future trends of S-SL for cybersecurity applications, paying particular attention to the most popular problems, such as network intrusion and malware detection. We focus our analysis on Q1 and Q2 papers from the Scopus and Web of Science repositories. In addition, we extended our search to arXiv, where we also looked for the final peer-reviewed version of any of the presented articles. See Table 1 for the queries used to choose the final papers and Figure 2 for the overview of this selection process. Although the queries used to select the final 10 surveys for Section 2 do not explicitly consider the search term “Self-Supervised Learning”, the scope of the parameters used for the title, abstract, and keywords fields is intended to analyze in a general way the many different AI training paradigms presented by the available literature reviews on cybersecurity.

Table 1. Queries used to write the different sections of this literature review. These queries were search parameters for the title, abstract, and keywords fields of the available documents in the Scopus, Web of Science, and arXiv databases.

Sections	Queries	Number of Papers Selected
Deep Learning for Cybersecurity Applications	[“Deep Learning” AND “Cybersecurity” AND “Survey”]	10
Self-Supervised Learning for Cybersecurity	[“Self-Supervised Learning” AND (“Network Intrusion” OR “Malware”)]	20
Other Applications and Future Research Directions in Cybersecurity for Self-Supervised Learning	[“Self-Supervised Learning” AND (“Misinformation” OR “Graph Anomaly Detection” OR “Financial Fraud” OR “Adversarial Attacks” OR “Cyber-Physical Systems” OR “Infrastructure Security” OR “Cybersecurity” OR “Backdoors” OR “Model Extraction” OR “Memorization”)]	20

Figure 2. Overview of the selection process of the 50 documents analyzed in detail in this survey. The queries are in Table 1. Regarding arXiv, most of the manuscripts available are originally a preprint, and served as a basis to look for the equivalent in the refereed databases of Scopus and Web of Science.

From the carefully curated list of 50 papers, 10 correspond to similar surveys in the field of AI for cybersecurity, 20 to S-SL applications for network intrusion (11) and malware detection (9), and the remaining 20 to other important topics, from S-SL in graph anomaly detection, to model extraction and memorization. Out of the thousands of available records in the refereed Scopus and Web of Science databases, the literature quality consisted of evaluating the title and abstract of mainly Q1/Q2 journals. Regarding arXiv, for any manuscript of interest according to our search queries, we prioritized its most recent published version in the other two databases, discarding preprints and any potential duplicates.

Thus, given the previous considerations, we present the following research questions:

• How do S-SL cybersecurity applications work?
• Is S-SL a suitable strategy in cybersecurity applications?
• What cybersecurity areas can benefit from S-SL solutions?

While traditional Machine Learning (ML) models, such as Decision Trees (DTs), have been proven to exhibit exceptional performance in terms of classification accuracy and processing speed in the context of Distributed Denial of Service (DDoS) attack detection [33,34], our review is primarily focused on DL models. This decision is underpinned by the distinctive capability of DL models to engage in nonlinear processing and their proficiency in constructing multi-level data representations from extensive datasets comprising raw data [35]. Nonetheless, it is pertinent to acknowledge that the scope of S-SL works extends to encompass traditional ML models as well [16].

We summarize our contributions as follows:

• An original survey considering S-SL as a strategy for model training to create applications in different problem domains of cybersecurity.
• A glimpse into opportunity areas in cybersecurity that can benefit from S-SL solutions, fostering ideas for future research on this topic.

The rest of this document is organized as follows: in Section 2, we provide a review of different surveys about DL for cybersecurity applications to observe the learning paradigms used in the state of the art; Section 3 explains what S-SL is and creates its taxonomy, as well as presents the different applications of this novel paradigm in network intrusion and malware detection; in Section 4, we show additional problem domains in cybersecurity that can benefit from S-SL for model training and research directions for S-SL in tabular data and resource-constrained devices; finally, Section 5 summarizes our key findings, and in Section 6, we present the conclusions of this work and research questions for future exploration.

2. Deep Learning for Cybersecurity Applications

In this section, we present a detailed study of previous reviews on DL models for cybersecurity. See Table 2 for the learning paradigms considered in these literature reviews and Table 3 for the cybersecurity problems that they discuss. Network intrusion and malware are the two most popular cybersecurity problems, along with adversarial attacks.

Table 2. Learning paradigms considered by the selected surveys of Deep Learning for cybersecurity applications.

References	Supervised Learning	Unsupervised Learning	Semi-Supervised Learning	Deep Reinforcement Learning	Self-Supervised Learning
Mahdavifar and Ghorbani [23], 2019	✔	✔	✔	✗	✗
Aldweesh et al. [24], 2020	✔	✔	✔	✗	✗
Gümüşbaş et al. [25], 2021	✗	✔	✗	✗	✗
Dixit and Silakari [26], 2021	✔	✔	✔	✔	✗
Sewak et al. [27], 2022	✗	✗	✗	✔	✗
Halbouni et al. [28], 2022	✔	✔	✔	✔	✗
Macas et al. [29], 2022	✔	✔	✔	✔	✗
Pawlicki et al. [30], 2022	✔	✔	✔	✗	✗
Kaur et al. [31], 2023	✔	✔	✗	✔	✗
Ozkan-Okay et al. [32], 2024	✔	✔	✔	✔	✗

Table 3. Top three of the cybersecurity problems discussed in the 10 surveys listed in Table 2.

Problem	Description	Number of Surveys
Network intrusion	Unauthorized access to a network for compromising an application and stealing data, as well as the launch of denial of service attacks to alter the quality of service.	10
Malware	Software with malicious intentions, including data encryption, espionage, and taking full control over the victim.	6
Adversarial attacks	Targeting the AI defense system with carefully manipulated input data to get incorrect predictions. Fooling a neural network to detect on purpose a threat as benign.	6

In [23], the authors provide a comprehensive review of various DL methodologies employed for the detection of an array of cybersecurity threats, including network intrusion, malware, phishing, spam, and website defacement. The authors analyze the reliance of these methodologies on supervised learning, underscoring the challenges associated with the acquisition of labeled data. Furthermore, they propose a classification schema for DL architectures pertinent to both generative and discriminative learning paradigms. This schema encompasses a range of architectures, notably Autoencoders (AEs), Recurrent Neural Networks (RNNs), Deep Belief Networks (DBNs), Generative Adversarial Networks (GANs), and Convolutional Neural Networks (CNNs).

Network intrusion has garnered special attention in recent literature [24,25,28,30], particularly in the context of the increasing Internet traffic and the emergence of new networking paradigms, such as the sixth generation (6G) mobile communication system [36]. Intrusion Detection Systems (IDSs) [37] play a pivotal role in safeguarding against prevalent cybersecurity threats. Among these, we find probing attacks like ping sweep and port scan [38,39], instrumental in identifying accessible hosts and active services and brute force attacks aimed at password theft, exemplified by those executed using tools like Patator [40].

Aldweesh et al. [24] advocate for the exploration of Ensemble Learning [41], highlighting its potential to combine varied architectures to bolster generalization. In a similar vein, Gümüşbaş et al. [25] catalog an array of Deep Neural Network (DNN) architectures, including AEs, CNNs, Long Short-Term Memory Networks (LSTMs), and GANs, which can be adeptly trained using unsupervised learning for anomaly detection. Concurrently, Halbouni et al. [28] delve into the nuances of Transfer Learning [42], an AI strategy centered on the transference of knowledge predominantly between akin source and target domains [43]; in contrast, whenever we have the same source and target tasks but different source and target domains, we are talking about Domain Adaptation [44]. Pawlicki et al. [30] then pivot to discuss the criticality of explainability in AI [45], particularly in enabling informed decision-making, since DL models are frequently perceived as “black boxes”.

The work presented in [26] dives into the relevance of zero-day [46] and adversarial [47] attacks. Particularly, AI models such as CNNs are vulnerable to adversarial attacks; indeed, by just changing one pixel of an image, the neural network can make a mistake in the prediction [47]. For example, [48] provides a perspective showing that the supervised learning paradigm may lead to the presence of adversarial attacks in smart classifiers due to input-label correlation issues while training (i.e., the mislabeling of samples and the presence of non-robust features). Regarding the protection techniques commented on in [26], we highlight the application of DRL since, as explained in [27], this learning strategy implies the use of a DL model as an approximation function to evaluate the best possible action and therefore maximize the overall reward. Although DRL seems promising in the cybersecurity domain for problems such as the evasion of anti-malware detection engines [49] and the mitigation of DDoS attacks [50], DRL agents do not perform well if the learned problem/scenario changes (even slightly), therefore presenting a lack of generalizability [27].

Macas et al. [29] identify three pivotal domains within DL pertinent to cybersecurity: FSL, attention-based models, and Graph Neural Networks (GNNs) [51,52,53]. FSL is particularly tailored for scenarios with limited training samples containing supervised information, thereby significantly reducing the necessity for large-scale supervised data collection [6]. This approach is further categorized into one-shot learning, where only a single supervised example is available [54], and zero-shot learning, which operates in the absence of any supervised example for the target task [55]. Notable models within the FSL framework include Siamese Networks [56] and Triplet Networks [57]. Attention-based models, such as Transformers [58], prioritize relevancy, enabling the model to focus on specific segments of the input that are crucial for the prediction task [51]. This mechanism enhances computational efficiency and allows for greater parallelization compared to RNNs by diminishing the sequence of operations required. GNNs, on the other hand, are instrumental in anomaly detection within graphs at various levels (node, edge, subgraph, and the entire graph) [59], and they are adeptly applied in modeling intricate challenges like detecting bots on social media platforms and identifying financial fraud.

Kaur et al. [31] highlight the prevention and analysis of data breaches, recommending the study of the dark web to discover any accidental exposure of sensitive data to recover it and save the reputation of both companies and customers. In this same topic, fake document generation is also a proposed research direction to protect critical files by creating several fake versions of the same document. Regarding AI, the paper points out the relevance of augmented intelligence, combining AI-based solutions with human operators, under the motivation of making better decisions in cybersecurity.

Lastly, in [32], the authors evaluate the importance of NLP tools such as ChatGPT [60] in cybersecurity, remarking that conversational AI can help by providing personalized awareness training sessions on information security. However, chatbots can be manipulated, with carefully crafted prompts to, for example, write sophisticated spear-phishing messages and deceive email recipients. See Figure 3 for a mind map of all the future research directions presented by the surveys discussed in this section.

Figure 3. Mind map of all the future research directions presented by the surveys discussed in Section 2.

The analysis presented herein highlights network intrusion, malware, and adversarial attacks as prominent themes within the domain of cybersecurity. We reveal the extensive application of various DNN architectures (e.g., AEs, CNNs, GANs, and LSTMs), particularly in the realm of anomaly detection. Concurrently, novel learning paradigms like DRL have surfaced. Additionally, there is an increasing focus on pivotal issues like explainability, alongside the exploration of the Internet of Things (IoT) [61] and botnets [62]. However, a notable gap in the literature is the relative oversight of S-SL as a training strategy for developing DL solutions to address cybersecurity challenges. Therefore, we firmly believe that the surveys discussed in this section would benefit from integrating insights on the applications of S-SL in anomaly detection [63], which have shown potential in areas such as fake news detection. In comparison to fully supervised learning techniques, S-SL offers the advantage of leveraging vast amounts of unlabeled data while simultaneously achieving state-of-the-art results and enhanced generalization capabilities [64].

The next section presents the taxonomy of S-SL and a general training pipeline using this novel learning strategy. In addition, the section reviews how S-SL has been implemented in malware and network intrusion detection.

3. Self-Supervised Learning for Cybersecurity

S-SL strategies are divided into two types: auxiliary pretext tasks and contrastive learning [5]. According to the problem domain, different pretext tasks can be chosen, such as predicting the angle of rotation of an image [65], as well as forcing a network to predict the original order of a shuffled audio and filling in missing words in a text or the missing edges in a graph [10]. These pretext tasks need to be carefully designed in an effort to learn useful representations with unlabeled data by automatically creating pseudo-labels from the dataset’s attributes [5] for solving application-specific downstream tasks [66], such as image classification, object detection, and action recognition [67]. On the other hand, contrastive learning focuses on the concept of similarity to bring similar samples closer (i.e., the positive samples), separating them from dissimilar samples (i.e., the negative samples) [67]. Here, data augmentation is relevant for achieving better performance since intra-class variability aids in finding more robust similarities [5]. See Figure 4 for a general training pipeline using S-SL with either auxiliary pretext tasks or contrastive learning. This pipeline starts with unlabeled data and the choice of a training S-SL strategy, which can imply data augmentation or pretext tasks; then, once the pre-trained model is built, labeled data is used for fine-tuning to adapt the model to specific downstream tasks.

Figure 4. General training pipeline using Self-Supervised Learning.

Choosing the appropriate architecture and pretext task is critical in S-SL [66]. CNNs can be used for S-SL. Nevertheless, in the state of the art, there are other methods whose design leverages this novel learning strategy, for example, Momentum Contrast (MoCo) [68], Vision Transformers (ViTs) [69], Bootstrap Your Own Latent (BYOL) [70], SimCLR [71], TabNet [72], and Barlow Twins [73]. See Figure 5 for a diagram of how the models presented in this survey are classified according to our taxonomy of S-SL strategies.

Figure 5. Classification of the Self-Supervised Learning models presented in this survey, by training strategy implemented.

A key difference between these two types of architecture lies in the cost function used in the pre-training phase. While TabNet and data2vec [74] utilize the reconstruction function, other architectures such as MoCo and SimCLR use the contrastive loss function. In this regard, InfoNCE [75] is an example where a normalized version of cross-entropy using $\tau$—as temperature hyperparameter to adjust randomness—is leveraged to accurately detect positive samples. This clearly indicates the training objective of the initial context extraction phase, where, instead of exploring similar and dissimilar samples as in contrastive learning, the algorithms under the auxiliary pretext tasks strategy aim, for example, to reconstruct missing parts of the input data using the L2 error.

To explain the proposed S-SL training pipeline for cybersecurity, we show two practical examples in Figure 6: the first case corresponds to network intrusion and the second to malware detection. In both scenarios, we divide an initial dataset into an unlabeled version with thousands of records and a labeled version with dozens of manually labeled records. For network intrusion, we select the LATAM-DDoS-IoT dataset [76,77], generate grayscale images from the normal and DDoS flows, define an augmentation policy with cropping and rotation, then pre-train SimCLR, and finally fine-tune the model for detecting an attack and classifying it per protocol (i.e., HTTP, UDP, TCP). Regarding the malware problem, using the CIC-MalMem-2022 dataset [78], we directly process the tabular input data and follow the auxiliary pretext tasks approach by masking a predefined percentage of features obtained from the software samples, then pre-train TabNet and fine-tune it to not only detect malware but also classify it per category (e.g., ransomware, spyware, trojan) and family (e.g., WannaCry, SpyEye, and Kovter).

Figure 6. Training pipelines using Self-Supervised Learning for network intrusion and malware detection. Both unlabeled and labeled datasets can be different, but need to be related.

In the following subsections, we present the applications of S-SL for the two most popular problems in cybersecurity, namely malware and intrusion detection.

3.1. Malware Detection

Malware is malicious software designed for purposes such as the collection of sensitive information from cellphones (e.g., calls and locations) in the form of Spyware [79], and the encryption of information to request money for its recovery, in the form of Ransomware [80,81]. Three feature extraction methods exist to design malware detection systems: static, dynamic, and hybrid [82]. The static method extracts features from the contents of the executables without running them [82]; the dynamic method runs the executable, for instance, in a virtual machine, to extract features from its behavior [82]; lastly, the hybrid method merges features from the static and dynamic methods, to achieve higher detection rates [82].

Xu et al. [83] proposed a dynamic analysis solution based on S-SL for the binary detection of Windows malware. The pre-trained model consisted of standard Transformer encoders that process tokenized API sequences. Similar to the two auxiliary pretext tasks used in the BERT [84] model for language understanding, the authors of [83] proposed the random masking of a number of tokens in the API sequences and the prediction of the correct order of two segments of an API call. The datasets proposed in [85,86,87] were used, and the custom S-SL model presented state-of-the-art results, outperforming other previous works without S-SL pre-training even above 16% in the F1 score.

In [88], S-SL is used for static detection of Android malware using ViTs and the MalNet dataset [89]. In ViTs, an image is split into fixed-size patches, then each of them is used to generate an embedding using a linear process. After that, position embeddings are added, and finally, the resulting sequence of vectors is fed to a standard Transformer encoder [69]. ViTs have demonstrated to be a competitive alternative to CNNs [69]. MalNet is a dataset of over 1.2 million binary images across 47 types and 696 families of Android APKs collected from AndroZoo [90]. For the ViT trained on [88], the authors used an architecture with 12 layers, 86 million parameters and patches of size 16 × 16. The images were of size 224 × 224 with three RGB channels. First, the model was self-supervised pre-trained with unlabeled images from the MalNet dataset, with masked autoencoding as the pretext task. This task forced the model to attempt recovering the input image based only on the unmasked 25% of the original image. After that, the ViT learned a generic representation of MalNet, which was further transferred to the downstream tasks of the same dataset for malware detection, and for malware family and type classification. The labels of the images were used for these downstream tasks of interest to adapt the self-supervised representation. Compared to popular DL architectures trained with supervised learning, namely ResNet [91], DenseNet [92], and MobileNetV2 [93], the ViT outperformed all of them, achieving a macro-F1 score of 49.7% and 49.1% for multiclass classification of malware type and family, respectively, and 97% of accuracy for binary classification.

Yang et al. [94] proposed a hybrid methodology for the detection and classification of Android malware, employing the BYOL approach and utilizing datasets such as CICMalDroid 2020 [95], which includes both dynamic and static features, encompassing app permissions, API calls, system calls, among others. BYOL, distinguished by its unique method for contrastive learning that obviates the need for negative samples, is structured around a dual neural network architecture comprising online and target networks. These networks mutually learn and are enhanced through data augmentation strategies [70]. Crucially, at the culmination of the training phase, the online network encoder is utilized for downstream tasks [70]. In their study, Yang et al. selected the ResNet encoder for training and evaluation, incorporating Cutoff [96] and Gaussian Noise [97] as data augmentation techniques. The performance of BYOL was contrasted with that of a fully supervised learning framework in two different configurations: one combined with a 2-layer Multi-layer Perceptron (MLP) and another with a single linear layer. Remarkably, in some datasets, such as AMGP [98] and Drebin [99], the BYOL-based approach surpassed the fully supervised model, while in others, like CICMalDroid 2020 and Praguard [100], the S-SL method exhibited a performance gap of less than 1% compared to its fully supervised counterpart [94].

Sern et al. [101] presented an innovative approach to classifying Windows malware binaries from [102,103], utilizing a modified version of the data2vec S-SL framework. The data2vec model is distinguished by its uniform learning objective applicable across various modalities, including vision, speech, and language [74]. It employs masked prediction and utilizes a standard Transformer network, operating in dual modes: teacher and student. In this framework, the teacher network is fed unmasked input to generate training targets, while the student network is tasked with predicting the representation of the original input using a masked version of the same [74]. In their study, Sern et al. preprocessed each malware binary into a grayscale image format, revealing that binaries from the same malware family exhibited similar visual patterns. Deviating from the conventional data2vec implementation that leverages Transformers, the authors opted for a CNN architecture. The resultant embeddings from the CNN were then input into an MLP for the purpose of multi-class classification of malware families. The outcomes of this novel approach were remarkable, achieving an average accuracy of 99.1% and surpassing traditional supervised training methods by nearly 4% [101].

Wang et al. [104] created an S-SL solution for the static detection of Windows executable binaries from the EMBER dataset [105]. The proposed approach used the pre-trained BERT model for feature embedding, exclusively with its auxiliary pretext task of the masking of words. The resulting representation vectors were then passed through a similarity score defined as the cosine of the angle between two vectors and finally contrasted using a temperature-scaled cross-entropy loss. A bidirectional LSTM (BI-LSTM) [106] with an attention layer was selected for the final classification module that receives as input the embedded sequences of words. This architecture outperforms the state of the art in some cases above 2.5% for the F1 score metric.

Ismail et al. [107] presented a benchmark between different S-SL models for two downstream tasks: Windows malware family classification and binary detection. These S-SL models were MoCo, SimCLR, SimSiam [108], and SwAV [109], compared to a modified version of SimCLR proposed by the authors using ResNet-18 and random cropping as the main augmentation technique. A smaller version of the original ImageNet dataset [110] called Imagenette [111] was used for the pre-training phase, whilst for the fine-tuning phase, Malimg [112] was used for the classification across 25 different families, and the Maldeb dataset [113] was proposed and used for the binary detection of software samples as benign or malicious. The experimentation was conducted over grayscale images of preprocessed software, achieving 98.4% accuracy for malware family classification and 96.2% accuracy for malware binary detection. An additional experiment transforming the Maldeb samples into spectrograms was performed, exploring audio representations for malware binary detection with the best accuracy of 85.9%.

As an extension to [107], the same authors worked on late fusion techniques to combine the extracted features after converting malware and benign binaries into two modalities, namely grayscale images and audio for spectrograms [114]. This multimodal learning used the Maldeb and BODMAS [115] datasets, both for training and evaluation. Moreover, adversarial examples were generated from Maldeb using GANs, producing a curated set with the best 1000 images for testing deception. When classifying these adversarial malware samples and comparing with other S-SL architectures such as MoCo and SwAV, the proposed multimodal method achieved the best accuracy of 95.1%.

Regarding IoT, Golmaryami et al. [116] and Wang et al. [117] explored the defense of different devices, such as smart door locks, smart lamps, and security cameras that are victims of malware attacks, including Mirai and BASHLITE. The authors of [116] created adversarial samples from the IoT-23 [118] and N-BaIoT [119] datasets using a GAN, perturbing legitimate samples with random noise. This pretext task of creating adversarial samples helped in the adversarial training process under the S-SL paradigm, improving robustness against maliciously intended crafted instances.

Wang et al. [117], based on SimCLR and using the Malimg, BIG 2015 [120], BenchMFC [121], IoTPOT [122], and X-POT [123] datasets, demonstrated that S-SL pre-training allows rapid convergence when training a model. Furthermore, S-SL pre-training improves common classification metrics such as the accuracy, precision, recall, and F1 score by at least 1.06% in the case of Malimg, when compared to just training directly with the same dataset (i.e., not pre-training at all) for malware family classification.

In Table 4, we show a breakdown of these different works around S-SL for malware detection in terms of the training strategy and datasets used, along with observations on the methodology conducted. As stated in [124], the Android platform has become the main target of mobile malware, and according to [125], Windows systems are the victims of most of the attacks launched by cybercriminals. Therefore, it is not surprising that the first applications of S-SL in the malware domain are focused on these two popular operating systems.

Table 4. Self-Supervised Learning works for malware detection.

References	Strategies	Datasets	Methodology	Observations
Xu et al. [83], 2021	Auxiliary Pretext Tasks	1. Windows API calls	Text analysis using standard Transformer encoders.	Malicious actors can intercept these API calls and alter their parameters, affecting the responses received by the invoked functions.
Seneviratne et al. [88], 2022	Auxiliary Pretext Tasks	1. MalNet	Masking of 2D colored malware images.	There is no standard on what color to assign to a specific characteristic of a malware. This research focuses on Android applications so the transfer of this color assignment knowledge is not clear for other files and Operating Systems.
Yang et al. [94], 2022	Contrastive Learning	1. CICMalDroid 2020 2. AMGP 3. Drebin 4. Praguard	Linear and non-linear evaluation heads appended to the BYOL architecture.	The size of the datasets is very low, typically between 10-20 thousand records. The only big dataset ideal for S-SL pre-training is Drebin, with above 100 thousand samples.
Sern et al. [101], 2022	Auxiliary Pretext Tasks	1. Windows binaries	data2vec implementation using CNNs for processing grayscale images.	Converting malware binaries into images is not a realistic natural transformation, but instead embeddings may be produced after the extraction of process variables from the running in a sandbox of those viruses.
Golmaryami et al. [116], 2022	Auxiliary Pretext Tasks	1. IoT-23 2. N-BaIoT	GANs for creating adversarial samples.	CNNs are used as classifiers, limiting the work to only capture the spatial correlations in the IoT data. By default, the two used datasets are in tabular format, therefore, it could have been modeled instead adversarial attacks for tabular data. These attacks are more complex for deceiving the neural network than just modifying one pixel.
Wang et al. [104], 2023	Contrastive Learning	1. EMBER	Semantics relation in malware binaries; uses pretext tasks such as word prediction and sentence order prediction, contrasted using temperature-scaled cross-entropy loss.	The EMBER dataset is massive in size, with above one million samples. However, the authors only used about 15 thousand records, loosing valuable information they could have leveraged for S-SL pre-training.
Ismail et al. [107], 2024	Contrastive Learning	1. Imagenette 2. Malimg 3. Maldeb	Grayscale images and spectrograms were used for representing malware samples.	Imagenette, a smaller version than ImageNet with around 13 thousand samples was used for pre-training, which is detrimental to the benefits of S-SL from massive volumes of data for the context extraction phase.
Ismail et al. [114], 2025	Contrastive Learning	1. Maldeb 2. BODMAS	Multimodal learning with late fusion techniques.	Although this technique is comprehensive regarding the combination of images and audio representations of malware samples, these transformations are less natural than directly working over the feature vectors provided in BODMAS.
Wang et al. [117], 2025	Contrastive Learning	1. Malimg 2. BIG 2015 3. BenchMFC 4. IoTPOT 5. X-POT	S-SL pre-training with malware images.	Different malware families can have a different amount of samples. This class representation can be unbalanced, which requires that more robust metrics such as macro F1 score are used for evaluating classification performance.

Every second, approximately 3.4 new samples of malware are discovered [126]. However, modern antiviruses still incorrectly flag some ransomware strains as spyware families. Therefore, to handle the demand and how sophisticated the malware problem becomes, it is also relevant that current studies not only deliver the detection rates achieved, but also provide a deeper definition of malware, with clearer feature sets and their explanation, which will help to obtain a better characterization and understanding.

Together with the previous observation, for a better explanation of the different malware categories, stronger patterns can be extracted by testing with a different dataset than in the pre-training or fine-tuning stages. Using more datasets in an S-SL pipeline is critical for measuring and improving the generalization capabilities.

3.2. Intrusion Detection

Intrusion detection is classified into network-based and host-based classification [28]. This classification relies upon the source of content that is examined. For instance, network-based intrusion detection considers the traffic gathered by multiple routers, and host-based intrusion detection considers the events that occur on a personal computer [28].

In [8], the NETFLOW [127] and Insider Threat Test [128] datasets were used to train an attention-based sequence-to-sequence architecture, emulating an autoencoder using an LSTM, to capture temporal relationships for modeling the benign and malicious users of a network. The implemented auxiliary pretext task was the alignment between the input and output sequences, by calculating the reconstruction error. When compared to a baseline autoencoder, the proposed sequence-to-sequence model presented a better precision–recall curve than unsupervised learning, showing the importance of temporal relationships and the attention mechanism when detecting anomalies.

Wang et al. [3] proposed a network-based intrusion detection system based on BYOL and grayscale images obtained from preprocessing the UNSW-NB15 dataset [129]. The data augmentation operations included horizontal and vertical flipping, random cropping, and a novel strategy proposed by the same authors, called random_shuffle, which consists of randomly changing the positions of the network traffic array before reshaping it into an image. Regarding the feature extraction encoder for BYOL, in this experiment, BoTNet [130] was the best encoder architecture in terms of classification performance when compared to other ones, such as ResNet, DenseNet, and SqueezeNet [131]. The generalization ability of the proposed model was evaluated with fine-tuning on the NSL-KDD [132], KDD CUP 99 [133], CIC-IDS2017 [134], and CIDDS_001 [135] datasets, outperforming some purely supervised learning models in the state of the art, for instance, in some cases by more than 5% in terms of accuracy.

Zhang et al. [136] presented a network-based intrusion detection solution combining S-SL and FSL, using preprocessed grayscale images of size 7 × 7 obtained from the NSL-KDD and the UNSW-NB15 datasets. The proposed model had three blocks. Each included a 2D convolution, batch normalization, and the ReLU activation function; at the end of these three blocks, a 2D max pooling was applied; finally, two fully connected layers were added to the model. In the pre-training phase, the pseudo-labels were not generated automatically but using different discrete features from the datasets (e.g., the protocol variable); for fine-tuning, only the two final fully connected layers were trained. In the case of the NSL-KDD dataset, when compared with the Deep Learning model XGBoost-DNN [137], which was trained in a supervised manner for binary classification, the S-SL model shows stronger performance, e.g., it results in an improvement of 3% and 1% for precision and recall, respectively. In this setting, both the supervised and the S-SL models used 1000 labeled samples for training (in the case of S-SL, for fine-tuning). Nevertheless, only 200 labeled samples from the NSL-KDD dataset were enough for training the S-SL model to obtain an accuracy of 95.2% in the downstream task adaptation process.

Based on the UNSW-NB15 dataset, ref. [138] used contrastive learning to train an S-SL model structurally similar to BYOL without the prediction stage. The encoder included convolution and max pooling layers, and for classification, the model used a fully-connected layer with the Softmax function. For data augmentation, the authors applied masking, which consisted in randomly selecting a predefined percentage of features of the input sample to assign them a value of zero. Compared to the BYOL model with BoTNet encoder from [3], the results in [138] presented a stronger classification performance, by over 4% in terms of accuracy and above 1.5% in F1 score.

Deng et al. [139] worked with the UNSW-NB15, CIC-IDS2017, and CSE-CIC-IDS2018 [140] datasets to create a network-based intrusion detection system using S-SL with contrastive learning. For the data augmentation strategy, the authors generated adversarial examples based on the idea proposed by [141]. This promising idea relies on adversarial attacks to produce more challenging positive and negative training pairs. For generating flow-based information, the authors of [139] applied instance-level discrimination and used the 43 features proposed in [142]. When selecting an MLP as the backbone for the S-SL model and unifying the three previously mentioned datasets, the accuracy for DoS attack detection was 97.63%, compared to the 54.34% accuracy from an MLP model without the S-SL pre-training process.

Mahmoud et al. [16] presented a 2-stage cyber–physical attack detection system for water distribution networks. The first stage consisted of implementing S-SL to inform about a potential attack, whilst the second stage used an Isolation Forest [143] model to confirm the warning. For training, the BATADAL dataset [144] was used, where data collected from different devices, such as water level sensors, are available. As the pretext task, the contrast of sensor readings was selected, and the time and classification performance were measured. Due to the dataset size of 61,000 records for training, the authors of [16] decided to code ML models for S-SL, like Random Forest and XGBoost. When considering the time taken to detect attacks, Random Forest achieved a strong performance in the first stage, demonstrating that S-SL may enable decision-making in near real-time.

In [145], a GNN-based system was created using the NetFlow protocol version of the UNSW-NB15 and CSE-CIC-IDS2018 datasets, namely NF-UNSW-NB15-v2 and NF-CSE-CIC-IDS2018-v2, both with a set of 43 standard features [142]. E-GraphSAGE [146] and Deep Graph Infomax (DGI) [147] were used together for capturing edge-level information and contrasting the generated positive and negative edge embeddings. The trained encoder was then transferred to four other anomaly detection methods, including Isolation Forest, where the macro F1 score values indicate increases of over 15% when the use of the generated edge embeddings via S-SL is compared to the use of raw features for binary attack detection.

The research by Almaraz-Rivera et al. [17] developed an advanced IDS for IoT network security, particularly against DDoS attacks, by employing the S-SL model MoCo v2 [148]. MoCo v2, an evolution of the original MoCo framework, incorporates a contrastive learning strategy that involves a dynamic queue of negative samples alongside queries and positive instances. In their research, Almaraz-Rivera et al. conducted experiments using visual representations derived from network flows, particularly focusing on data from the Bot-IoT [149] and LATAM-DDoS-IoT datasets. The results from these experiments demonstrate 4.83% improvement in the F1 score for attack detection compared to a fully supervised learning approach that utilized ResNet-34. This enhancement not only underscores the efficacy of MoCo v2 in the realm of IoT security but also highlights the potential of S-SL models in advancing cybersecurity measures.

The authors of [150] created an S-SL system based on tabular learning and the TabNet model for IoT domain data, pre-training a tabular MLP (TabMLP) [151]. TabNet follows an autoencoder structure comprised of multiple transformers and the pretext task of masking a part of the input features. It emulates and outperforms the behavior of DTs regarding automatic feature selection and the non-linear processing of these selected features in a sequential multi-step architecture [72]. Three datasets were used in [150], namely WUSTL-EHMS [152], WUSTL-IIoT [153], and Bot-IoT, with varied information ranging from healthcare sensing devices to smart home appliances. With only 5 and 10 random samples per class for the fine-tuning phase, the proposed S-SL system beats state-of-the-art fully supervised models, with cases above 20% for the F1 score metric.

Cao J. et al. [154] presented an S-SL solution based on hierarchical transformers for protecting the Controller Area Network (CAN) of intelligent vehicles against DoS, fuzzy, and spoofing attacks, which alter critical variables like the speed and steering of the cars. The masking pretext task was used in the training phase to obfuscate normal sequences of CAN messages from the Car-Hacking dataset [155]. In terms of F1 score, the proposed S-SL method outperforms the supervised learning method from [156] by 11.2% for the binary detection of DoS attacks, whilst for the other threats both show a similar performance. Regarding inference time performance, the S-SL system can detect each CAN message in an average of 0.1075 milliseconds.

In the same domain of in-vehicle security, Zhang et al. [157] evaluated the pretext task of reconstruction error with Transformers, LSTMs, and GRUs, in an encoder–decoder fashion. Each model was individually trained with only normal data and then tested in the four types of attacks in the Car-Hacking dataset. This S-SL approach using an LSTM gave the best detection rate, except for DoS, where the Transformer model excelled. Other experiments were conducted using the ROAD dataset [158], but not for S-SL.

Table 5 shows the collection of S-SL works for intrusion detection. As explained, these approaches are network-based, therefore, it is of interest to propose a host-based intrusion detection system. Although these host-based systems can present execution speed limitations, it is necessary that different security mechanisms collaborate to get a more efficient and effective detection of attacks [159]. Furthermore, host-based variables are critical for characterizing abnormal behavior, since examining process variables inside a computer is as rich as getting socket information on the communication between hosts [160].

Table 5. Self-Supervised Learning works for intrusion detection.

References	Strategies	Datasets	Methodology	Observations
Lee et al. [8], 2020	Auxiliary Pretext Tasks	1. NETFLOW 2. Insider Threat Test	Autoencoder-like LSTM architecture to capture temporal relationships and reconstruction errors.	The benchmark focused on a comparative against autoencoders, but other architectures were obviated, such as Transformers for attention-based modeling.
Wang et al. [3], 2021	Contrastive Learning	1. UNSW-NB15 2. NSL-KDD 3. KDD CUP 99 4. CIC-IDS2017 5. CIDDS_001	The $random\_shuffle$ pretext task was proposed, changing input network traffic arrays before converting them into images.	Fine-tuning process on older datasets is only useful for benchmark against previous methodologies, but not for evaluating if more recent attacks are correctly detected or not.
Zhang et al. [136], 2022	Auxiliary Pretext Tasks	1. UNSW-NB15 2. NSL-KDD	Few-Shot Learning and the processing of grayscale images.	S-SL pre-training should be conducted over rich and novel network traffic, not on the NSL-KDD dataset.
Deng et al. [139], 2022	Contrastive Learning	1. UNSW-NB15 2. CIC-IDS2017 3. CSE-CIC-IDS2018	Adversarial sample creation for harder (better) contrastive learning.	This framework proposal should have been compared against the original SimCLR architecture, and others like BYOL and MoCo.
Mahmoud et al. [16], 2022	Auxiliary Pretext Tasks	1. BATADAL	S-SL in tandem with One-Class Classification for attack detection on water distribution networks.	The dataset size of 61 thousand records is worth it for implementing Deep Learning, not only ML models like Random Forest.
Caville et al. [145], 2022	Contrastive Learning	1. NF-UNSW-NB15-v2 2. NF-CSE-CIC-IDS2018-v2	Graph Neural Networks for producing edge-level embeddings.	Experiments without the NetFlow protocol version of the used datasets, are missing. This turns out to be relevant for justifying the usage of NetFlow.
Lotfi et al. [138], 2023	Contrastive Learning	1. UNSW-NB15	BYOL + MLP prediction head.	Evaluation with a linear classification head, like Logistic Regression, is worth it for indicating if strong patterns were extracted in the pre-training phase. This predictor is missing in the paper.
Almaraz-Rivera et al. [17], 2023	Contrastive Learning	1. Bot-IoT 2. LATAM-DDoS-IoT	MoCo v2 for visual representations of DDoS attacks traffic.	Recent IoT traffic is learned using the LATAM-DDoS-IoT dataset, encompassed by physical and simulated devices.
Ayesha S. et al. [150], 2023	Auxiliary Pretext Tasks	1. WUSTL-EHMS 2. WUSTL-IIoT 3. Bot-IoT	Self-Supervised Tabular Learning for IoT network traffic.	Most of the network datasets in this literature review are in tabular format. Therefore, the usage of TabNet becomes more straightforward with this type of relations in the data, rather than capturing spatial correlations.
Cao J. et al. [154], 2024	Auxiliary Pretext Tasks	1. Car-Hacking	Protection of the Controller Area Network of smart vehicles.	The measurement of inference time performance is relevant for critical tasks such as the security of passengers inside an intelligent car.
Zhang et al. [157], 2025	Auxiliary Pretext Tasks	1. Car-Hacking 2. ROAD	Anomaly detection using Transformers, LSTMs, and GRUs.	The experimentation is limited to only training with normal traffic, emulating more a One-Class Classification approach.

Another relevant observation is the transformation of tabular data into 2D images. While this is useful for explaining to different audiences how several types of attacks resemble and can be visually identified, this extra step makes real-time detection harder in production classification pipelines. Another limitation is still the training process using datasets such as KDD CUP 99 and NSL-KDD, which do not represent recent traffic behaviors, and beyond obtaining standardized results, this potentially delays the creation and exploration of new data for our IDSs.

3.3. Contrastive Learning vs. Auxiliary Pretext Tasks in Cybersecurity Contexts

The reason for choosing either one of the two S-SL training strategies is heavily dependent on the data modality, although there is always the influence of the researcher or the practitioner. That being said, both techniques can be applied, but one excels over the other depending on key factors such as classification speed and the interpretability of the predictions.

For example, network transactions that represent fraudulent payments or reconnaissance traffic can be intuitively conceived as a table with attributes and records. In this scenario, it is better in terms of processing speed to handle this information with a Tabular Network following the auxiliary pretext task strategy of masking cells, than following the contrastive learning strategy using SimCLR to model the inter-class difference. However, for some people, it can be easier to explain the network traffic as an image and therefore decide on the former strategy. See the approaches in [17,150] for this comparison of spatial and tabular correlations in the data for the detection of DDoS attacks.

Instead of paying more attention to the training strategy implemented by the S-SL architecture, we suggest opting for those S-SL algorithms that process the data modality of interest in a more natural way. This further motivates the discussion of S-SL for CV, NLP, and tabular data domains, presented in the next section.

4. Other Applications and Future Research Directions in Cybersecurity for Self-Supervised Learning

The papers explained thus far are crucial for a proactive response to malware and network intrusion. However, there exist other opportunities where S-SL can also be applied. Therefore, in this section, we present some of these future research directions in S-SL for cybersecurity. See Figure 7 for the corresponding diagram.

Figure 7. Diagram of the research directions identified as future work in cybersecurity for Self-Supervised Learning.

4.1. Network Intrusion

The performance of AI-based methods is strongly dependent on the available datasets [29], and according to the different network-based intrusion detection systems presented in this survey, there are outdated collections of data, such as NSL-KDD and KDD CUP 99, that are still used, even when they are not representative of real existing networks. Therefore, it is necessary to train and test new models with datasets that include current attack behavior and a large amount of records. For example, in the IoT domain, the recently created LATAM-DDoS-IoT dataset integrates millions of flows and has shown strong performance in a production-simulated scenario.

Furthermore, to the best of our knowledge, there is not yet a host-based intrusion detection system based on S-SL trained models. The Leipzig Intrusion Detection Dataset (LID-DS) [161], due to its varied information, including system calls, process names, process IDs, and the tested performance using Siamese Networks and FSL [162], can be of potential interest for research.

Another opportunity area is the encrypted traffic analysis on networks where sensitive data is transmitted through security protocols like Transport Layer Security (TLS). With the high increase in attacks over the years using encrypted communication channels on the Internet, AI-based technologies are mandatory to examine encrypted traffic for anomaly detection [163].

4.2. Malware

Regarding malware detection, the available works about S-SL are focused on Android and Windows targets. Even though these two platforms are the most popular malware victims, there are other operating systems, such as iOS and macOS, that also require special attention.

Additionally, S-SL, together with RAM analysis [164], can provide robust protection against fileless malware to address ransomware. These memory-based antiviruses work by reading, in constant intervals of time, the RAM of the infected device to detect obfuscated malware that uses coding techniques such as string encryption and packing to remain hidden [165,166]. To carry on this detection task, the CIC-MalMem-2022 appears in the state of the art as a promising dataset.

Malware is in constant evolution, thus having up-to-date datasets is complicated. However, building a large initial collection of processed fileless malware [167] is a strong step toward creating sophisticated AI-based protection systems. VirusTotal [168] and theZoo [169] are two repositories that can help to obtain the binaries necessary to create this collection of data.

4.3. Misinformation

Fake news and rumors can exist in articles or social media posts that claim unverified statements (potentially false), which are consumed by several users [170]. The detection of image manipulation, as a step toward identifying anomalies that result in this misinformation, was proposed in [171], where an S-SL model was trained using metadata from real photographs.

Furthermore, Ref. [172] presented an S-SL approach for image splicing detection and localization, training the model with the frequency transform of images. With respect to detecting anomalies in text, Ref. [173] introduced a novel pretext task on text sequences for S-SL based on hiding parts of the input tokens and predicting the masking pattern that was used.

4.4. Graph Anomaly Detection

The detection of anomalies on attributed networks using S-SL [174] helps model complex problems such as misinformation on social networks. Although GNNs for anomaly detection are still an under-explored topic [175], relevant works are leveraging S-SL to identify normal and fraudulent users, for example, with Deep Cluster Infomax (DCI) [176], Graph Convolutional Networks (GCNs) [177,178], and the mix of contrastive learning and the pretext task of reconstruction error in the framework proposed in [179].

For example, Chen et al. [178] proposed injecting features in graphs using the attribute and adjacency matrices to create anomalies by performing feature exchange, multiplication, and division. This idea helped the representation learning process, where a reconstruction phase was trained using a GCN, calculating the original and injected features. This framework was tested for different downstream applications, including the detection of banned and suspicious users on social networks.

4.5. Financial Fraud

Detecting malicious activities in the financial sector is crucial. Some applications for fraud detection are the identification of the misuse of credit or debit cards and the manipulation of insurance claiming processing systems by submitting forged documentation [180]. In [181], a GNN-based fraud classifier was proposed, which was trained using contrastive learning S-SL to detect fraudulent and benign nodes (i.e., normal and suspicious users). Schreyer et al. [182] used the SimCLR architecture to learn invariant representations of accounting data for audit tasks. Ref. [183] implemented the contrastive learning strategy with a DNN to perform representation learning using spatial and temporal information from input payment sequences from two different datasets, including real private data from a bank in Taiwan.

In the state of the art, the advancements in AI-based financial fraud detection systems are limited. The main reason is the lack of publicly available datasets, since those currently used are either fully private or present obfuscation of features due to user privacy and confidentiality [184,185]. As stated in [186], the IEEE-CIS dataset [187] appears as a suitable data collection for experimentation over the financial fraud domain, with varied information from major credit card providers, namely Visa, Mastercard, Discover, and American Express.

Almaraz-Rivera et al. [188], using the IEEE-CIS dataset, created a fraud detection system for online shopping based on S-SL and tabular learning, adding a non-linear layer using MLP to the SubTab architecture [189]. Compared to supervised and unsupervised learning models, the proposed system outperformed XGBoost by 2.14% AUROC, and the original SubTab implementation with Logistic Regression by 1.33% in the same metric.

Nevertheless, there is still a need to develop more varied datasets to measure the generalization of fraud detection solutions that model credit card user behavior. Using temporal and spatial information remains critical, as provided by the amount of money, timestamp, and location of the different transactions.

4.6. Cyber–Physical System Protection

As stated in [24], security in smart grid systems is an area that requires further exploration. The smart grid concept refers to the transition of current electricity distribution networks toward a modern, efficient, and automated energy supply and management [190]. Different attacks can affect the reliability of smart grids, such as DDoS attacks and the modification of control signals and sensor data [190]. Therefore, in the transition to smart and sustainable cities [191,192], S-SL can help with anomaly detection and data traffic control in critical infrastructures, as shown by the authors of [193], with a solution against False Data Injection Attacks (FDIAs) and Time Delay Attacks (TDAs).

Moreover, self-driving cars, by design, exhibit serious vulnerabilities in the CAN protocol with the lack of encryption and authentication mechanisms [194], allowing the injection of intentionally fabricated messages for manipulating the behavior of the vehicles, like the gear status and the victim’s tire pressure monitoring system [155]. Therefore, in-vehicle networks require the development of sophisticated protection systems, where S-SL accelerates the creation process since it does not need the aid of experts for labeling in the pre-training phase, saving a tremendous amount of time.

Lastly, transportation systems, such as the metro, can also benefit from integrating passenger flow information with S-SL for user security. Classifying the number of passengers on a certain line and metro station in a specific hour has tremendous advantages. If detecting the inflow or outflow as abnormal (e.g., below or above the usual), the model can trigger alarms for digging deeper into potential strikes, sports events, accidents, affecting the user traffic, and therefore take real-time decisions to handle these special cases. In this regard, we can find two state-of-the-art works leveraging ViTs and custom TabNet architectures, for the cities of Paris, France [195] and Medellin, Colombia [196], respectively.

4.7. Adversarial Attacks

Adversarial attacks aim to fool neural networks by adding perturbations to the input data. In cybersecurity, an example is incorrect predictions by manipulating signals in wireless networks [197]. In [198], S-SL was conducted using an autoencoder structure built with an LSTM to detect Sybil attacks on GPS trajectories. Leveraging four perturbation techniques, which encompass moving and removing parts of each trajectory to emulate traffic from one road to another and faster traffic speeds, the author injected 5% of adversarial trajectories into the Porto [199], Beijing [200], and Cologne [201] datasets. The results indicate that the proposed model outperforms the state-of-the-art baselines on all three datasets to detect adversarial trajectories crafted by Sybil attacks.

Another study is the detection of 40,000 adversarial samples generated using the Compound Attack technique [202] to transform legitimate URLs into phishing URLs by randomly inserting hyphens between subwords. For this, Liu et al. [203] performed a post-training phase of the CharBERT language model [204], defining three different auxiliary pretext tasks, namely: predicting masked tokens within URL sequences; modifying entire words by deleting, adding, or shuffling characters to then reconstruct the original words; and determining whether two text segments are connected coherently. The datasets used were GramBeddings [202], Mendeley [205], and two Kaggle repositories [206,207]. When testing this adapted model, it defeated the state of the art for adversarial attack detection, indicating an AUC value of 99.41%.

See Table 6 for a summary of all the different applications of S-SL in cybersecurity that have been commented on so far in this section.

Table 6. Other applications of Self-Supervised Learning in cybersecurity.

Cybersecurity Field	Applications of Self-Supervised Learning	Summary
Misinformation	[171,172,173]	Detection of image manipulation and anomalies in text to combat fake news.
Graph Anomaly Detection	[176,178,179]	Detection of banned and suspicious users on social networks.
Financial Fraud	[181,182,183,188]	Detection of forged documentation for insurance claiming and audit tasks. Moreover, the detection of credit and debit cards misuse.
Cyber–Physical System Protection	[193,195,196]	Detection of False Data Injection Attacks and Time Delay Attacks against smart grid systems. Furthermore the detection of abnormal passenger flow behaviors in metro systems.
Adversarial Attacks	[198,203]	Detection of Sybil attacks on GPS trajectories and the usage of the Compound Attack technique for transforming legitimate URLs into phishing URLs.

4.8. Secure and Safe AI

This survey has focused on the applications of S-SL for cybersecurity. However, we have not mentioned the reverse approach, where security mechanisms are implemented at the data and model levels to protect the smart solutions created against problems such as data poisoning, backdooring, and model extraction [208,209,210]. This field is called Secure AI, and it is not trivial for S-SL, where, for example, the existence of vulnerabilities in the pre-training phase can be directly transferred to the post-training phase, affecting the different downstream tasks [211,212]. Another complex example is the difficulty of creating a protection against encoder extraction in S-SL, because of the high-dimensional output of the representations produced, which differs from the low-dimensional output of scores produced by classifiers [213].

Safe AI is another interesting field and refers to the responsible use of the created S-SL solutions, where the decisions made are aligned with an ethical framework [214]. We can highlight two important aspects for trustworthy AI: reproducibility and privacy. Regarding reproducibility, current and future papers must provide clear procedures or programs to verify their research, which facilitates putting into practice the latest approaches or conducting follow-up research [215]. For privacy, data memorization is a critical issue and refers to the memorization that DL models can make of sensitive personal information, leaking these individual data points when the models are consumed by end users [216]. Localizing where memorization occurs in S-SL encoders is an interesting research direction [217].

See Table 7 for a breakdown of these different solutions toward secure and safe AI in S-SL models.

Table 7. Secure and Safe AI for Self-Supervised Learning.

Field	Solutions for Self-Supervised Learning	Summary
Secure AI	[209,211,212,213]	Backdoor attacks for vision models, as well as the evaluation of the transferability of vulnerabilities in the pre-training phase to the post-training phase and downstream tasks. Furthermore, the defense against model extraction, particularly against encoder extraction.
Safe AI	[217]	Localization of data memorization in S-SL vision encoders, at different granularities.

4.9. Self-Supervised Tabular Learning

The need for extending the success of S-SL to the tabular domain was stated in [218], where the authors indicated that the most common data type in the real world is in tabular format, proposing two novel auxiliary pretext tasks and a novel data augmentation method for tabular learning. Since the correlation structure for tabular data is harder to identify than the spatial correlations in images, the sequential correlations in text, or the temporal correlations in audio, extending the success of S-SL to the tabular domain is not straightforward. Nevertheless, promising works are trying to solve this challenge, such as VIME [218], TabNet, and SubTab.

Self-Supervised Tabular Learning (S-STL) can significantly improve the training pipelines of some of the works presented in Section 3, by reducing the steps required to preprocess the input data, as in [114], where the authors transformed malware binaries to grayscale images and spectrograms instead of directly working on the feature engineering of different software samples distributed into tables.

However, according to Figure 8, the number of available documents in the state of the art is extremely low for S-STL, with only nine, compared to the fields of CV and NLP, where the numbers surpass the hundreds of papers. Potentially, this limited progress is due to the fact that tree-based models already provide strong classification performance that surpasses DL architectures in tabular data [219]. The extension of S-SL to tabular learning will not just accelerate the classification pipelines by working directly over the input tabular data, but also close the existing performance gap between DL and ML models in this domain, where popular baselines such as XGBoost are still dominant [219,220]. The work in [188] is an example of how S-SL can be used to achieve these two goals.

Figure 8. Cumulative frequency of the total number of documents published in Scopus from 2019 to 2025, according to the three main data domains in Self-Supervised Learning: Tabular Learning, Computer Vision, and Natural Language Processing.

Suitable pretext tasks for tabular data work around the notion of corrupting feature vectors. By applying a mask to the features of a sample x to produce an obfuscated vector $\tilde{x}$, the goal can be not only the prediction of missing values, but also the prediction of the features that have been completely masked [218]. Capturing the correlation in tabular data is not the same as learning the spatial relations that can be obtained from images or the temporal relations from audio and video, requiring specific architectures that can leverage the structured nature in tables.

4.10. Tiny Machine Learning

Once an AI-based solution has been created for any cybersecurity problem, the next step is to deploy this application in a real-world scenario. For this, as discussed in [29], two crucial aspects must be considered: online learning and knowledge distillation. Online learning aids in the continuous update of the models from new data coming in the form of streams [221,222] to handle concept drift [223]; on the other hand, knowledge distillation is an effective technique for the compression and acceleration of DNNs [224], which is helpful for the reduction of the parameter size of S-SL models [11] and the deployment on resource-constrained devices as those in edge computing [225].

In this regard, deploying an AI-based solution can occur and is necessary in some cases for embedded systems. Tiny ML [226] is presented here as a direction for future research when integrating S-SL models into resource-constrained devices, such as microcontroller units, where the RAM capacity is limited to a few KBs. However, knowledge distillation is not the only technique for handling architecture complexity.

While knowledge distillation reduces the parameter size following the teacher–student behavior, where a more efficient student model is derived from the complex teacher architecture, another technique called quantization deals with compressing these neural network’s parameters, for example, from 32-bit floating-point values to 8-bit [227]. Nevertheless, these accelerations also reduce the inference power of the models, making it also necessary to explore additional methods for S-SL, such as SSQL [228].

5. Summary of the Key Findings

Table 8 summarizes and compares the focus of the surveys analyzed in this article with our literature review:

Table 8. Comparison of this literature review with the selected surveys from the state of the art.

References	Self-Supervised Learning as Training Strategy	Main Cybersecurity Problems	Main Future Research Directions
Mahdavifar and Ghorbani [23], 2019	✗	Network intrusion and malware	Model training more human-like (i.e., without relying on pure supervised learning).
Aldweesh et al. [24], 2020	✗	Network intrusion	Novel datasets with recent attack scenarios and traffic behaviors.
Gümüşbaş et al. [25], 2021	✗	Network intrusion	Study of the adequacy of features in known benchmark datasets.
Dixit and Silakari [26], 2021	✗	Network intrusion and malware	Attention on adversarial attacks and the analysis of the speed and memory consumption of cybersecurity applications.
Sewak et al. [27], 2022	✗	Network intrusion, malware, and adversarial attacks.	Explainable Artificial Intelligence and Cooperative AI.
Halbouni et al. [28], 2022	✗	Network intrusion	Parameter and architecture optimization of deep neural networks.
Macas et al. [29], 2022	✗	Malware and network intrusion	Few-Shot Learning, Knowledge distillation, and Online learning.
Pawlicki et al. [30], 2022	✗	Network intrusion and adversarial attacks	Ethics and bias in AI, data protection and privacy.
Kaur et al. [31], 2023	✗	Network intrusion and malware	Zero-day attacks and multi-lingual threat intelligence.
Ozkan-Okay et al. [32], 2024	✗	Network intrusion, malware and financial fraud	Adversarial attacks against Generative AI.
This work	✔	Network intrusion and malware	Self-Supervised Tabular Learning, Tiny ML, and Secure and Safe AI.

• First, we are the only ones to present S-SL as a training strategy for the creation of AI-based detection solutions in the cybersecurity field.
• Regarding the main attacks considered, we are in tandem with the two most relevant problems in the state of the art, namely malware and network intrusion.
• Finally, we shed awareness on the creation of efficient tabular learning architectures able to natively leverage S-SL, as well as the protection of these AI models against problems such as encoder extraction, data memorization, and the transferability of vulnerabilities from the pre-training to the fine-tuning phase.

Regarding the proposed taxonomy of contrastive learning vs auxiliary pretext tasks in cybersecurity:

• Although standard [2,5,229], it is suitable for describing the different S-SL applications presented here. Nevertheless, it becomes important to clarify that this categorization may be too coarse, so AI architectures should not be evaluated alone but in tandem with the nature of the input training data.
• Therefore, S-STL emerges in this discussion when deciding which architecture type to leverage for our data modality.
• Lastly, cybersecurity is well-known to be a heavily imbalanced domain, where the ratios between the normal class and the attack categories are big. The pre-training phase of S-SL, along with the fine-tuning part using FSL, tackle this issue, reducing the bias in the created detection systems.

As a matter of guidance, for anyone interested in conducting a benchmark using S-SL, here are three takeaways:

• First, it is crucial to avoid outdated datasets such as NSL-KDD for network intrusion detection or BIG 2015 for malware. Although useful for comparison with the available classification performance presented in the literature, relying on old data harms the transfer of the created solutions to production scenarios.

  –

  Even though it is advisable to have extremely large datasets of more than a million samples for the pre-training phase [230], smaller data just above 100,000 instances can still provide competitive classification performance [231]. The reason is that pre-training directly on the target task data is more optimal than on a general data regime. That being said, when selecting a dataset, you can prioritize those that strongly relate to your downstream tasks. For example, if working on IoT traffic, do not pre-train on general network traffic, but focus on IoT devices instead.

• Secondly, it is important to decide the training strategy to be conducted, either contrastive learning or auxiliary pretext tasks. Nevertheless, we suggest paying more attention to choosing the S-SL algorithm that processes the data modality of interest in a more natural way.
• Finally, do not overlook protecting the protector. Evaluate the robustness of your models. Do not only prioritize having a better F1-score, but also a high stability.

Lastly, S-SL solutions can sound complex for implementation, but it should not be that way. There are a couple of things that favor the usage of S-SL in production scenarios:

• The pre-training phase is time-consuming, especially when dealing with images, but the final encoders are connected to standard non-linear and linear classification heads, alleviating costly inference tasks. An MLP and a logistic regression can be used as these predictors, respectively.
• Furthermore, as discussed in the previous section, Tiny AI, with knowledge distillation and quantization techniques, allows for feasible real-time deployments.

In the next section, we present the final comments of this survey, along with concrete followup research questions.

6. Conclusions

Self-Supervised Learning is a training paradigm that leverages the well-known unsupervised and supervised learning strategies to train robust ML and DL models that exhibit high generalization capabilities and strong time and classification performance. These models do not require large amounts of labeled data for the fine-tuning process, but instead, FSL is a popular alternative. S-SL can be classified into two strategies, namely contrastive learning and the usage of auxiliary pretext tasks, where both types have demonstrated that robust representation learning from the knowledge domain is as crucial as the architecture/framework choice.

The benefits of S-SL are listed as follows:

• Cheaper and faster training without the need for human experts for the massive labeling of input data.
• Long-term scalable AI solutions because of the easy fine-tuning (adaptation) of the base pre-trained models.
• Improved detection rates, compared to purely supervised and unsupervised learning models.

In areas such as cybersecurity and medicine, where annotated data is scarce but domain knowledge is constantly evolving, S-SL represents a strong suitor for creating AI solutions [232]. Despite the mentioned advantages, the current literature on DL for cybersecurity applications obviates S-SL as a strategy for model training. The low volume of work in this area can be a reason. Indeed, this low volume of publications opens a wide range of applications that can benefit from S-SL, as graph anomaly detection for the identification of bots on social networks, the detection of fake news considering image and text manipulations, the protection of in-vehicle networks against data injection attacks, and the detection of adversarial attacks that affect legitimate URLs and transform them into phishing.

The results presented in this survey highlight the potential of S-SL to surpass popular training strategies such as supervised learning, but it is important to clarify that all the discussed studies employ a different methodology, varying the amount of data used, the evaluation metrics, and the baseline architectures. Therefore, we advise readers not to be swayed by the impression that S-SL is the master learning paradigm that will solve all our problems, but it is certainly true that it is an advancement in how we conceive AI nowadays.

Each of the three research questions stated in Section 1 guided the content of this literature review, allowing us to conclude that S-SL, and more specifically Self-Supervised Tabular Learning, is a suitable strategy in cybersecurity. However, additional research questions emerge for future exploration:

• With the observed progress in AI, is awareness alone enough limitation to explore novel architectures besides autoencoders for anomaly detection in cybersecurity?
• All models learn the data in different ways. Therefore, what are the specific inductive biases [219] that could make S-STL outperform standard S-SL algorithms?
• Finally, from this survey, we can state that mitigation strategies do not receive the same attention as detection systems receive. Once an IDS flags a problem, how can S-SL support the automation of the corresponding remediation processes?

We hope that this article serves as the basis for any scientist or practitioner interested in studying and implementing S-SL in a highly volatile field such as cybersecurity.