Knowledge Distillation Based Recommendation Systems: A Comprehensive Survey

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This article provides a review of knowledge distillation systems. Basically an approach by which some of the knowledge of a large complex models, such as deep neural networks, is transferred to a simpler model, such a linear model. This type of approach can be useful in several different situations. One of the most obvious applications is when the recommendation system s run in a device with limited computational capabilities (compared to the model requirements) such as in mobile applications

 

The authors do a thorough review including 115 references. The authors followed a logical approach described the main categories (section 3) and main applications (section 4). Followed by future directions and conclusions. I have some comments.

 

• There should be a bit more focus on the limitations of these approaches. In this type of review is important to have a critical assessment of the current approaches, including the limitations (with details).
• Section 3.3 Teacher-student architecture. This is an important part of the paper. The authors first go through the basic concepts and then in subsections 3.3.1 and 3.3.2… go through the concept of one teacher-one student, and many teacher-one student…. For this type of paper (review) there needs to be more detail explaining the general concept (with details). I am referring to the need of expanding the first part (section 3.3), which is key to then understand the following subsections.
• Line 211. It is mentioned that they are using the L2 norm (the most frequently used), any reason to make this explicit? i.e. other norms considered?
• There should also be a bit more comment regarding model appropriates i.e. when (in which type of situation) to use each type of model. Any heuristics to know when to use these types of different approaches?
• While the structure of the paper is logic it would seem reasonable to have the future directions section after the conclusions. The conclusions should organically lead the author to the future directions section.

 

Minor/style comments

I think that the authors forgot to change the acknowledgements section (still the same wording as template). Need to be careful with these things.  

Author Response

Please refer to the attached word document.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The main question of this article is "How can knowledge distillation (KD) be effectively integrated into recommendation systems to address performance, efficiency, and scalability issues?". The topic is highly relevant and original. Although KD and recommendation systems have been separately explored in previous research, this paper explicitly identifies and addresses a clear gap by systematically reviewing and categorizing the integration of KD within recommendation systems—a topic that had not yet received comprehensive coverage in existing surveys. 

This article significantly advances the field by:

• Providing a comprehensive categorization of KD recommendation methods based on knowledge types, distillation schemes, and Teacher-Student architectures.
• Offering detailed insights into how KD methods can be integrated specifically into the recommendation pipeline, such as recall, pre-ranking, and ranking phases.
• Identifying clear future research directions, especially emphasizing practical applications in industry contexts.

The conclusions presented are consistent with the arguments and evidence given. They effectively summarize the categories of KD-based recommendation systems and highlight potential future research directions, clearly addressing the main research question posed by the survey.

The references are comprehensive, up-to-date, and highly relevant to the topic.

As suggestions for improvement I can point the following:

1. Clarify the criteria used for selecting the included literature (databases, search terms, inclusion/exclusion criteria).
2. Explicitly describe the systematic process (e.g., PRISMA methodology) used for literature identification and screening.
3. Provide a brief statistical overview (e.g., charts or a flow diagram) summarizing the screening process and showing the number of papers excluded at each stage.
4. Include more recent references (2023-2025) explicitly linked to cutting-edge developments in KD, particularly in relation to large language models and diffusion-based methods, to enhance the currency of the paper.
5. Simplifying Figure 2 (categories of knowledge distillation recommendation systems) for better readability, as it currently appears somewhat dense.
6. Adding short explanatory captions or legends for each figure, ensuring they can be fully understood independently of the main text.

Author Response

Please refer to the attached word document.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The authors addressed the topic of knowledge distillation-based recommendation systems, presenting a comprehensive survey. Below are the main comments and concerns:

1. The abstract should provide basic information about the rationale for the research and its results. A detailed description of the objectives is not necessary in this part of the manuscript.

2. The introduction provides a brief overview of the research topic. Lines 49-58 contain references to several publications, but there is no indication of their relevance to the research topic. In addition, this section lacks an indication of the objectives or research questions, along with a rationale.

3. The manuscript lacks a literature review to identify research gaps. The gaps, in turn, would identify the research objectives.

4. Sections 2 and 3 contain information on Knowledge Distillation Recommendation Systems, but it is difficult to determine the nature of this part of the manuscript. It is not a formal literature review, nor is it a description of the research. Without clearly defined objectives, it is impossible to assess whether the authors have achieved their goals.

5. In section 4, the authors review the applications of knowledge distillation in industrial recommendation systems. According to the data in Tables 1-3, the analysis covered about 40 publications, but due to the lack of a described literature survey methodology, it is not possible to assess whether the results presented are representative. In addition, only a few applications are mentioned in the following section; there is no justification as to why these were selected for detailed description.

6. The summary is short and very general. There is no reference to the research objectives, no description of the limitations of the work carried out, and no information about the authors' contributions.

To summarize the most important shortcomings:

1. Lack of clearly defined research objectives derived from identified gaps.

2. No description of the methodology of the research conducted.

3. Description of results too general.

4. Lack of clear theoretical and practical contributions.

5. Lack of identification of limitations of the research conducted.

Author Response

Please read the attached word document.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Comment

Overall
(1) The theme is very interesting and eye-catching. However, the abstract is missing some of the authors' new discoveries and results that they would like to emphasize.
Rather than the content of the discussion, please clarify what is newly discussed and what are the points to be emphasized.

(2) Explanation of the KD model
This discussion is quite common among LLMs experts such as Hinton, so please write a bit more about the contents of the journal you submitted, such as a discussion of the treatment of the architecture of the data, assuming the contents and readership of the journal. In particular, there is a huge computational cost in having this study done. If the discussion is tailored to this journal, I think it should include a discussion of the specifications, the computational cost as a bottleneck, and the cost of preparing the data set and other work costs.

(3) How to summarize the previous studies in chapters 2 to 3
If the discussion is based on the nature of the submitted journals mentioned above, this previous research part should be able to significantly reduce the number of pages by creating about two types of tables.
In other words, Tables 1 and 2 should be brought first. The reader may overlook them. The author should also be able to make appropriate suggestions as to which methods are ultimately the most sophisticated, which will allow for appropriate development with reduced costs in the technical areas, and their limitations.

(4)Chapter 5.
Regarding suggestions and discussions based on the final survey, these should also be summarized in a table, with additions sliding into a discussion of the content of the submitted journal and its intended audience, including a discussion of advantages, disadvantages and development costs from the author's perspective.

(5) Conclusion
The author's emphasis is a bit fluffy, so it would be desirable if the author could provide more information on issues, disadvantages, and discussion of architectural and computational cost issues, etc., in line with the content of the journal.

Author Response

Please read the attached word document.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

Thank you for addressing my comments and concerns.

Author Response

Please read the attached word document.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Comments 
Overall 

Very sincere response to your previous comment, and very easy to read with your corrections. However, I think you should briefly discuss the difference between Knowledge Distillation Learning, Deep Learing and ML for the reader. A description of this would help keep the reader's attention.

As for the references, please survey and discuss additionally a few more examples from other countries and other prior studies.
That is all.

Author Response

Please read the attached word document.

Author Response File: Author Response.pdf