SSCL-TransMD: Semi-Supervised Continual Learning Transformer for Malicious Software Detection

Round 1

Reviewer 1 Report

The authors should consider these comments before guaranteeing to be published

1.      The abstract need to be rewritten long statements are existed.

2.      The list of contributions  are very long need to paraphrasing and there is some points need to be clarify as “It improves the lifelong unsupervised mix up algorithm(LUMP)”

3.      How the dataset splits in training and testing?

4.      Many important steps in the proposed model should be presented in Fig.1 and Fig2

5.      Discuss the overfitting problem

Extensive editing of English language required

Author Response

Reviewer #3:

1. The abstract need to be rewritten long statements are existed.

The author's answer: Thank you for your comment. We have reorganized the content of the abstract.

2. The list of contributions are very long need to paraphrasing and there is some points need to be clarify as “It improves the lifelong unsupervised mix up algorithm(LUMP)”

The author's answer: We have also realized that the list of contributions in the introduction is quite lengthy. Therefore, we have refined it to make it more concise.

3. How the dataset splits in training and testing?

The author's answer: We apologize for missing the key information like this. We split the whole dataset into 8:2, for training and testing respectively. And we have added this information in the experiment section.

4. Many important steps in the proposed model should be presented in Fig.1 and Fig2

The answer's answer: Thank you for your comment. We redrew our Fig1 and Fig2 to present our proposed method.

5. Discuss the overfitting problem

The author's answer: We acknowledge your comment on the discussion of the overfitting problem. In the context of continuous learning and constantly acquiring new samples, our main objective is to fit the new data as accurately as possible. However, it is generally challenging to achieve a perfect fit with new data, making overfitting less likely to occur. Consequently, we did not specifically address the issue of overfitting in our paper.

Author Response File: Author Response.docx

Reviewer 2 Report

This work considers a Semi-Supervised Continuous Learning Model for Malware Detection based on Transformer (SSCL-TransMD) for malware detection. For the study to reach the desired quality, the authors must correct some deficiencies.

1. The article should generally be written in journal format. The format of this article needs to be corrected.

2. The abstract part of the study emphasizes that SSCL-TransMD performs well in the semi-supervised continuous learning scenario. However, some numerical data regarding this performance were not shared. It would be appropriate to share the numerical results of the performance values of this model.

3. A comparison should be made between the prediction data of the models discussed in the study and the actual data, and the figures representing these data should be included in the article.

4. This study used the CICMalDroid 2020 dataset to evaluate the proposed model. The dataset includes more than 17,341 Android samples from multiple sources from December 2017 to December 2018. The study mentioned that the dataset consists of five categories: adware, banking malware, SMS malware, risky software, and harmless software. However, there is no descriptive statistical data for the data set. A table is needed showing statistical data such as mean, standard deviation, variance, number of samples, kurtosis, and skewness for each category of data collected from five different categories.

5. The authors handled the methodology and results part of the study well, but the figures in the results part are not descriptive.

6. A discussion section should be added to the study, and this study should be examined under many subheadings, such as its advantages, disadvantages, and features that distinguish it from other studies.

7. The conclusion part of the study should be enriched.

8. The reference part of the study should be arranged according to the journal format.

Minor editing of English language required

Author Response

1. The article should generally be written in journal format. The format of this article needs to be corrected.

The author's answer: We apologize for the misuse of format in this article. We have used the MDPI template from submitting site to re-format this paper into MDPI journal style.

2. The abstract part of the study emphasizes that SSCL-TransMD performs well in the semi-supervised continuous learning scenario. However, some numerical data regarding this performance were not shared. It would be appropriate to share the numerical results of the performance values of this model.

The author's answer: We have added several specific numerical data to the end of the abstract to prove the performance of our method in the semi-supervised continuous learning scenario.

3. A comparison should be made between the prediction data of the models discussed in the study and the actual data, and the figures representing these data should be included in the article.

The author's answer: We appreciate your suggestion to include a comparison between the prediction data of the models discussed in our study and the actual data, along with figures representing these data. However, we believe that incorporating such a comparison would be of limited significance and would significantly increase the redundancy of the article. Therefore, we have chosen to utilize F1 and Micro F1 scores to represent the prediction performance of the models.

4. This study used the CICMalDroid 2020 dataset to evaluate the proposed model. The dataset includes more than 17,341 Android samples from multiple sources from December 2017 to December 2018. The study mentioned that the dataset consists of five categories: adware, banking malware, SMS malware, risky software, and harmless software. However, there is no descriptive statistical data for the data set. A table is needed showing statistical data such as mean, standard deviation, variance, number of samples, kurtosis, and skewness for each category of data collected from five different categories.

The author's answer: Thank you for your valuable comment. We have taken note of the lack of descriptive statistical data for the dataset used in our study. In response to this, we have provided a table in the results section that presents statistical information for each category of data collected from the five different categories. The table includes measures such as mean, standard deviation, variance, number of samples, kurtosis, and skewness. We believe that this addition will enhance the clarity and comprehensiveness of our results.

5. The authors handled the methodology and results part of the study well, but the figures in the results part are not descriptive.

The author's answer: We appreciate your comment about figures in the results part. We also acknowledge that the figures in the experiment section lack descriptive details. Therefore, we have provided more detailed explanations of the content within the captions of the figures.

6. A discussion section should be added to the study, and this study should be examined under many subheadings, such as its advantages, disadvantages, and features that distinguish it from other studies.

The author's answer: We agree that proving a comprehensive discussion section is crucial to better present the contributions and characteristics of our study. In response to your comment, we included a discussion section in the revised version of the paper. This section is right after the experiment section and before the conclusion section. We analyze the advantages of our proposed SSCL-TransMD model, and we also discussed the potential limitations or disadvantages in our approach.

7. The conclusion part of the study should be enriched.

The author's answer: We acknowledge the suggestion to enrich the conclusion part of the study. In light of this, we have taken into account our experimental results and summarized them accordingly. As a result, we have expanded the content of the conclusion to address this concern.

8. The reference part of the study should be arranged according to the journal format.

The author's answer: Apology again for the misuse of format in the paper. And we have used the MDPI template from submitting site to make it right.

Author Response File: Author Response.docx

Reviewer 3 Report

The paper claims to address the issue of catastrophic forgetting in Android malware detection using a semi-supervised continual learning approach based on Transformer models. However, the paper does not adequately demonstrate the novelty of this approach compared to existing techniques. It would be helpful if the authors could provide a more comprehensive review of related work and clarify how their proposed model distinguishes itself from previous research in this field. 

The paper introduces an "improved lifelong semi-supervised mixture algorithm" and a semi-supervised algorithm for obtaining pseudo-labels. However, it lacks a strong theoretical foundation or mathematical justification for these algorithms. It is essential to provide a clear explanation of the underlying principles and how these methods address the catastrophic forgetting problem in Android malware detection.

While the paper claims that SSCL-TransMD outperforms existing deep learning models in the semi-supervised continual learning scenario, the experimental evaluation lacks critical details. Reviewers need more information on the specific evaluation metrics used, statistical significance tests, and the overall experimental setup. Additionally, providing baseline results with standard datasets and clear comparisons with existing approaches would strengthen the paper's claims.

The paper mentions conducting experiments on three different datasets, but it does not provide adequate information about these datasets. Reviewers need more details about the characteristics of these datasets, such as their sizes, sources, and diversity, to assess the generalizability of the proposed model. Furthermore, it would be beneficial to benchmark SSCL-TransMD against state-of-the-art models on well-established Android malware detection datasets.

The paper's conclusion lacks depth and does not adequately summarize the key contributions and limitations of the proposed approach. A more in-depth discussion of the results, implications, and potential future directions would enhance the paper's overall quality.

In conclusion, the paper lacks sufficient novelty, theoretical justification, and comprehensive experimental evaluation to support its claims. The presentation of the proposed model and experimental results needs improvement, and a more critical discussion of the findings is required.

Needs improvement

Author Response

1. The paper claims to address the issue of catastrophic forgetting in Android malware detection using a semi-supervised continual learning approach based on Transformer models. However, the paper does not adequately demonstrate the novelty of this approach compared to existing techniques. It would be helpful if the authors could provide a more comprehensive review of related work and clarify how their proposed model distinguishes itself from previous research in this field.

The author's answer: We appreciate your comment. To demonstrate the novelty of our proposed method, we add a discussion section right after the experiment section and before the conclusion. We discussed the advantage, disadvantage and potential development direction in the future in this newly added section.

2. The paper introduces an "improved lifelong semi-supervised mixture algorithm" and a semi-supervised algorithm for obtaining pseudo-labels. However, it lacks a strong theoretical foundation or mathematical justification for these algorithms. It is essential to provide a clear explanation of the underlying principles and how these methods address the catastrophic forgetting problem in Android malware detection.

The author's answer: We apologize for the lack of strong foundation of our proposed method. To address this, we have supplemented the theoretical underpinnings of LUMP to elucidate how it addresses catastrophic forgetting. Please check it in (2)Memory replay method, section 3.1.1.

3. While the paper claims that SSCL-TransMD outperforms existing deep learning models in the semi-supervised continual learning scenario, the experimental evaluation lacks critical details. Reviewers need more information on the specific evaluation metrics used, statistical significance tests, and the overall experimental setup. Additionally, providing baseline results with standard datasets and clear comparisons with existing approaches would strengthen the paper's claims.

The author's answer: In the experiment section, we have provided detailed information about the metrics we used. The detailed experimental procedure is presented in the pseudocode. Unfortunately, in the field of malware detection, there is currently no widely recognized baseline. Therefore, we did not compare our approach with a baseline. Instead, we compared our approach with four existing methods.

4. The paper mentions conducting experiments on three different datasets, but it does not provide adequate information about these datasets. Reviewers need more details about the characteristics of these datasets, such as their sizes, sources, and diversity, to assess the generalizability of the proposed model. Furthermore, it would be beneficial to benchmark SSCL-TransMD against state-of-the-art models on well-established Android malware detection datasets.

The author's answer: We apologize for the misunderstandings caused by our incorrect statements presented in the text. We evaluated our proposed method on three different datasets, and we found that the experimental results were very close. Therefore, considering the length of the article and to avoid superfluous data redundancy, we have selected the data that we deem most representative to showcase in the paper. Please allow us to apologize once again for the inappropriate expressions used in the article.

5. The paper's conclusion lacks depth and does not adequately summarize the key contributions and limitations of the proposed approach. A more in-depth discussion of the results, implications, and potential future directions would enhance the paper's overall quality.

The author's answer: Specifically, we have added a new section before the conclusion called "Discussion." In this section, we have provided a more comprehensive analysis of the strengths and limitations of our proposed approach. Additionally, we have also included a discussion on the potential future directions for further research. We believe that these additions have significantly enhanced the depth of our conclusion. They now adequately summarize the key contributions and limitations of our proposed approach, as well as provide a more in-depth discussion of the results, implications, and potential future directions.

6. In conclusion, the paper lacks sufficient novelty, theoretical justification, and comprehensive experimental evaluation to support its claims. The presentation of the proposed model and experimental results needs improvement, and a more critical discussion of the findings is required.

The author's answer: Thank you for your valuable feedback on our academic paper. We have carefully considered your suggestion and made the necessary revisions to address the concern. We have expanded and revised the content of the paper according to the aforementioned comments.

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

 Accept in present form

Reviewer 3 Report

The responses are okay. 

Still, I found a few grammatical mistakes.