A Flexible Session-Based Recommender System for e-Commerce

Round 1

Reviewer 1 Report

- Title should be changed, perhaps "Session-Based Recommendation Systems" 

- No reference for "Flexible Recommender Systems".

- The introduction requires a summary of the main objectives/contributions of this paper. 

- There is no proper literature review to see if there is similar research in the literature. 

- Section 2 needs a paragraph at the beginning to describe the subsections. '

- There is no proper discussion section to summarise the work, etc. 

Author Response

Comment:

• Title should be changed, perhaps "Session-Based Recommendation Systems"

Response: Thank you for the comment. The title has been changed to “A Flexible Session-Based Recommender System for e-Commerce"

Comment:

• No reference for "Flexible Recommender Systems".

Response: A link to the project webpage is now given in section 1.

Comment:

• The introduction requires a summary of the main objectives/contributions of this paper.

Response: Thank you for the comment. We have now added in section 1 a paragraph discussing the problems and challenges addressed by this work.

Comment:

• There is no proper literature review to see if there is similar research in the literature.

Response: There is a very detailed literature review in Section 2. This Section has been renamed to make clear cut that represents not only a presentation of the methods used, but also a very large literature review.

Generally, the topics covered are very broad, however in every subsection of Section 2 discussing a specific method there are all the major publications and many more. In total, there are 38 references covering, in our opinion, all the important and relevant literature in SBRS systems.

Comment:

• Section 2 needs a paragraph at the beginning to describe the subsections. '

Response: Thank you for the comment. We added a paragraph.

Comment:

• There is no proper discussion section to summarise the work, etc.

Response: Thank you for the comment. We have actually already a lengthy summarization paragraph in the conclusion. See bullets 1-7 in section 6 as well as the text before and after that.

Reviewer 2 Report

The topic is interesting, and the problem addressed is important but can be more well motivated by corelating the research gaps in the literature. The methods are well-explained. The experiments are convincing. However, some minor improvements are required, as suggested below.

·         Abstract needs improvements, as its not indicating the research gap and novelty of the technique clearly

·         Generally the RNNS (LSTM) are applied to non-linear problems such as time series forecasting etc. How do you related the graph theory data used in recommender systems as a non-linear input? Pls clarify.

·         LSTM has greater space state complexity. In recommender systems it can eventually grow more. How do you address this issue?

·         There is a single equation in the section 5.1 that too is not numbered. Also change it to non-bold font. If possible, add some more formulas to support the arguments.

·         A table of Notations and Definitions should be added that explains the symbols used in the manuscript.

·         The authors provided some insights from the existing literature related to their problem in the introduction section and subsequent sections. However, a separate literature review section is strongly recommended to add that must explain and compare the recent works.

·         What are the limitations of your proposal model? Pls add in the manuscript and highlight some future works in it.

·         The conclusion section is too lengthy. It is suggested to concise it.

·         Since the authors are not native English speakers. It is suggested that 1-2 native English experts should proofread the article for corrections.

Author Response

Comment:

• The topic is interesting, and the problem addressed is important but can be more well motivated by corelating the research gaps in the literature. The methods are well-explained. The experiments are convincing. However, some minor improvements are required, as suggested below.

Response: Thank you for the comments. We have now added in section 1 a paragraph discussing the gaps covered by this work.

Comment:

• Abstract needs improvements, as its not indicating the research gap and novelty of the technique clearly

Response: Thank you for the comment. We have modified the abstract according to your suggestions.

Comment:

• Generally the RNNS (LSTM) are applied to non-linear problems such as time series forecasting etc. How do you related the graph theory data used in recommender systems as a non-linear input? Pls clarify.

Response: Thank you for your comment. We added a few sentences in the relevant section (Section 2.2) to clarify this concern.

Comment:

• LSTM has greater space state complexity. In recommender systems it can eventually grow more. How do you address this issue?

Response: To address the growing computational complexity of RNN we have employed GPU for training and inference. It responds in real time for user session browsing.

Comment:

• There is a single equation in the section 5.1 that too is not numbered. Also change it to non-bold font. If possible, add some more formulas to support the arguments.

Response: Unfortunately, we cannot find any equation in section 5.1, probably you mean Section 5.4. We removed the equation because the formula of MRR is already presented in previous section.

Comment:

• A table of Notations and Definitions should be added that explains the symbols used in the manuscript.

Response: Thank you for the comment. We have added a table of abbreviations in the introduction.

Comment:

• The authors provided some insights from the existing literature related to their problem in the introduction section and subsequent sections. However, a separate literature review section is strongly recommended to add that must explain and compare the recent works.

Response: The topics covered are very broad so there is no specific section covering the literature review but there are proper references in every subsection discussing a specific topic. There are a total of 38 references covering, in our opinion, the most important literature in the diverse subjects.

Comment:

• What are the limitations of your proposal model? Pls add in the manuscript and highlight some future works in it.

Response: Thank you for the comment. We have discussed the limitations of our approaches in the conclusion section and described possible future directions.

Comment:

• The conclusion section is too lengthy. It is suggested to concise it.

Response: Thank you for the comment. Since other reviewers asked us to add discussion, we have decided to keep the size of the conclusion section and change the title of the section to “Discussion and Conclusions”.

Comment:

• Since the authors are not native English speakers. It is suggested that 1-2 native English experts should proofread the article for corrections.

Response: Thank you for the comment. We have corrected the errors in English.

Reviewer 3 Report

Section 1 must be improved.

-       Authors need to revise the title of the paper, as it stands it is too succinct. The title must contain the information necessary to make the reader understand what he will find in the work.

-       the abstract must be reviewed, first present the problem, then briefly describe what you have done and finally anticipate some of the results of your study. do it in this order.

-       References must be numbered in order of appearance in the text (including citations in tables and legends) and listed individually at the end of the manuscript. Use this format [1].

-       Authors should emphasize contribution and novelty, the introduction needs to clarify the motivation, challenges, contribution, objectives, and significance/implication. 

-       You should add more information in the introductory part, you should add other works that have also addressed the problem.

-       You must properly introduce your work, specify well what were the goals you set yourself and how you approached the problem.

-       At the end of the section you introduce FRES project, but you don't clarify if in this work you present all the activities performed in the project or only some.

Section 2 must be improved.

-       In this section you should introduce the materials and methods used in this study.

-       Instead you just did a bibliographic analysis of how other authors have approached the problem. This is fine, but after focusing the reader's attention on how the problem has traditionally been approached you should detail how you have approached the problem.

-       Start with a description of the data you used to test your methodology.

-       Then describe in detail the technology you used

-       You said you used LSTM, so start with a detailed description of the RNN and then move on to describing LSTM

-       I could not find a detailed description of the evaluation metrics you have adopted. How will you measure your model's performance? This section is essential in order to demonstrate the effectiveness of your methodology. Furthermore, only by adopting adequate metrics will it be possible to compare your results with those obtained by other researchers.

Section 3 must be improved.

-       This section contains what I would have expected to find in the previous section.

-       Merge this section with the previous one as a subsection

-       Add a flow chart of your methodology

Section 4 must be improved.

-       That being said in the previous section, it would be good if you reorganized the sections in a more effective way

Section 5 must be improved.

-       In this section presenting the results, I think it is appropriate to rename the section to Results and Discussion

-       A description of the hardware and software used for data processing is completely missing. Describe in detail the hardware used:  Extract this data from the datasheet of the hardware manufacturer. To make reading the specifications of the hardware more immediate, you can insert them in a table, listing the instruments used and the specific characteristics for each.

-       Also, you should describe in detail the software platform you used. Also describe the machine learning-based libraries you used.

-       A detailed discussion of the results obtained is missing. Try to summarize what was obtained and try to extract useful information from the work carried out. Also add bibliographic references to support your conclusions, to give more weight to your statements.

-       The section relating to the methodologies based on Machine Learning must be enriched. You must summarize the essential characteristics of the methods you have used and justify your choices. Try to summarize what are the strengths and weaknesses of the methods, in this way you can make the reader understand why you have chosen these methodologies.

-       Add a table with the parameters of the LSTM used

Section 6 must be improved.

-       Paragraphs are missing where the possible practical applications of the results of this study are reported. What these results can serve the people, it is necessary to insert possible uses of this study that justify their publication.

-       They also lack the possible future goals of this work. Do the authors plan to continue their research on this topic?

41) Don’t leave a blank line after the period

61) POI explain the meaning of

63) Do not use abbreviation such as i.e. I have seen that you often use this abbreviation, so I will not repeat this advice again, it also applies to the other occurrences.

82) Do not use abbreviation such as e.g. I have seen that you often use this abbreviation, so I will not repeat this advice again, it also applies to the other occurrences.

145) “SVD” Do not use acronyms until you have presented the full definition, I will not repeat this advice again, it also applies to the other occurrences.

152) Remove e.g.

157) “Neo4j” Add references to allow the reader to learn more about the topic

167-170) Do not anticipate the results, in this section you only have to describe the methodologies

227” Doc2Vec” Introduce adequately the topic and add references to allow the reader to learn more about the topic

236) Word2Vec Introduce adequately the topic and add references to allow the reader to learn more about the topic

327) Figure 1 must be improved: The text is too small and appears blurry, making it difficult for the reader to follow the flow of information.

341) Figure 2 must be improved: The text is too small and appears blurry, making it difficult for the reader to follow the flow of information. , I will not repeat this advice again, it also applies to the other occurrences.

530-531) you already said it

Author Response

Comment:

Section 1 must be improved.

• Authors need to revise the title of the paper, as it stands it is too succinct. The title must contain the information necessary to make the reader understand what he will find in the work.

Response: Thank you for the comment. The title has been changed to “A Flexible Session-Based Recommender System for e-Commerce and Services" which better represents the content of the paper.

Comment:

• the abstract must be reviewed, first present the problem, then briefly describe what you have done and finally anticipate some of the results of your study. do it in this order.

Response: Thank you for the comment. We have updated the abstract according to your suggestions.

Comment

• References must be numbered in order of appearance in the text (including citations in tables and legends) and listed individually at the end of the manuscript. Use this format [1].

Response: We have used the format [1].

Comment:

• Authors should emphasize contribution and novelty, the introduction needs to clarify the motivation, challenges, contribution, objectives, and significance/implication.

Response: Several sentences in the abstract and introduction have been added to address this comment

Comment:

• You should add more information in the introductory part, you should add other works that have also addressed the problem.

Response: We have added quite a number of most relevant references in every section of the paper. We believe that this information is enough for the reader to understand the problems discussed in every section.

Comment:

• You must properly introduce your work, specify well what were the goals you set yourself and how you approached the problem.

Response: We have added a paragraph at the end of the introduction that explains the goals and the methodology presented in the paper.

Comment:

• At the end of the section you introduce FRES project, but you don't clarify if in this work you present all the activities performed in the project or only some.

Response: The paper summarizes the majority of the work carried out during the project. This is now mentioned in the introduction.

Comment:

Section 2 must be improved.

• In this section you should introduce the materials and methods used in this study.
• Instead you just did a bibliographic analysis of how other authors have approached the problem. This is fine, but after focusing the reader's attention on how the problem has traditionally been approached you should detail how you have approached the problem.

Response: Section 2 indeed discusses materials and methods which is the title of the section.

Comment:

• Start with a description of the data you used to test your methodology.
• Then describe in detail the technology you used

Response: We believe that the description of the particular datasets that we used is second in importance. It is more important to first describe the problems we addressed and the methodology we used for each problem and then present the data and finally the results.

Comment:

• You said you used LSTM, so start with a detailed description of the RNN and then move on to describing LSTM
• I could not find a detailed description of the evaluation metrics you have adopted. How will you measure your model's performance? This section is essential in order to demonstrate the effectiveness of your methodology. Furthermore, only by adopting adequate metrics will it be possible to compare your results with those obtained by other researchers.

Response: Thank you for the comment. We have added a description of RNN in section 2.1. The evaluation metrics used in our experiments are now described in section 2.4

Comment:

Section 3 must be improved.

• This section contains what I would have expected to find in the previous section.
• Merge this section with the previous one as a subsection

Response: Done.

Comment:

• Add a flow chart of your methodology

Response: Thank you for the comment. We have now added a schematic diagram of the tasks, methods and data representation tools studied in our approach (Figure 1).

Comment:

Section 4 must be improved.

• That being said in the previous section, it would be good if you reorganized the sections in a more effective way

Response: Section 4 describes in detail all the datasets that have been used in our experiments and it’s organization in paragraphs corresponds to Section 2 that describes the methods used. Section 2 and Section 4 are effectively aligned.

Comment:

Section 5 must be improved.

• In this section presenting the results, I think it is appropriate to rename the section to Results and Discussion

Response: What this reviewer proposed is done.

Comment:

• A description of the hardware and software used for data processing is completely missing. Describe in detail the hardware used: Extract this data from the datasheet of the hardware manufacturer. To make reading the specifications of the hardware more immediate, you can insert them in a table, listing the instruments used and the specific characteristics for each.
• Also, you should describe in detail the software platform you used. Also describe the machine learning-based libraries you used.

Response: this cannot be realistically done, because many and different machines  have used to execute the experiments. We will need many pages to give such details.

Comment:

• A detailed discussion of the results obtained is missing. Try to summarize what was obtained and try to extract useful information from the work carried out. Also add bibliographic references to support your conclusions, to give more weight to your statements.

Response: There is a very detailed discussion of the results obtained in Section 5. There are more than 5 pages discussing the results, together with references to tables, figures and literature when necessary. For example Section 5.2 states: “This finding is in accordance with other works on sequence modeling (Chung et al, 2014) where combinations of LSTM and GRU variants outperform standard RNNs.”. Also further down it states: “Also, the results are comparable to the accuracy results that SotA methods have achieved (Section 2.3).” making explicit references to the state-of-the-art that has been presented in Section 2.3

Comment:

• The section relating to the methodologies based on Machine Learning must be enriched. You must summarize the essential characteristics of the methods you have used and justify your choices. Try to summarize what are the strengths and weaknesses of the methods, in this way you can make the reader understand why you have chosen these methodologies.

Response: The Sections 2.1 , 2.3  and 2.5 (more than 6 pages) in total present in great detail how Machine Learning are used in our work. We also summarize strengths and weaknesses of each method.

Comment:

• Add a table with the parameters of the LSTM used

Response: In Section 2.5.6 we state that the number of units for the LSTM model is 200.

Comment:

Section 6 must be improved.

• Paragraphs are missing where the possible practical applications of the results of this study are reported. What these results can serve the people, it is necessary to insert possible uses of this study that justify their publication.

Response: The practical results of the study are reported extensively in Section 5.5 (2 pages) asn also in Section 5.6 (2 pages). Also, practical results and implication are discussed in the conclusions. So, in total there are more than 5 pages reporting practical applications, and we believe this comment is addressed sufficiently , if not perfectly.

Comment:

• They also lack the possible future goals of this work. Do the authors plan to continue their research on this topic?

Response: Several sentences have been added in Section 6. Conclusions and future directions, discussing our plans for future work.

Comment:

41) Don’t leave a blank line after the period

Response: We have fixed it.

61) POI explain the meaning of

Response: POI = Point of Interest. It has now been explained in the text.

Comment:

63) Do not use abbreviation such as i.e. I have seen that you often use this abbreviation, so I will not repeat this advice again, it also applies to the other occurrences.

Response: the abbreviation “e.g.” has been removed in almost all cases.

Comment:

82) Do not use abbreviation such as e.g. I have seen that you often use this abbreviation, so I will not repeat this advice again, it also applies to the other occurrences.

Response: the abbreviation “e.g.” has been removed in almost all cases.

Comment:

145) “SVD” Do not use acronyms until you have presented the full definition, I will not repeat this advice again, it also applies to the other occurrences.

Response: Thank you for the comment. We have now added a table of abbreviations at the beginning of the paper so the acronym “SVD” is now explained.

Comment:

152) Remove e.g.

Response: Actually, the abbreviation “e.g.” is pretty standard and is used to convey our intended meaning so we are keeping it.

Comment:

157) “Neo4j” Add references to allow the reader to learn more about the topic

Response: Thank you for the comment. We have added references [39],[40]

Comment:

167-170) Do not anticipate the results, in this section you only have to describe the methodologies

Response: Thank you for the comment, we have removed the phrase “however evaluation of the results showed that this simple graph-based method 168 is less effective than machine learning models.”

227” Doc2Vec” Introduce adequately the topic and add references to allow the reader to learn more about the topic

236) Word2Vec Introduce adequately the topic and add references to allow the reader to learn more about the topic

Response: Thank you for the comment. We've added a paragraph about it in Section 2.1

Comment:

327) Figure 1 must be improved: The text is too small and appears blurry, making it difficult for the reader to follow the flow of information.

Response: Done

Comment:

341) Figure 2 must be improved: The text is too small and appears blurry, making it difficult for the reader to follow the flow of information. , I will not repeat this advice again, it also applies to the other occurrences.

Response: Done

Comment:

530-531) you already said it

Response: We kept the sentence “The results reported here are the average of the results produced from each split.” because it conveys the important information about the results.

Round 2

Reviewer 1 Report

my comments have been addressed, thanks

Author Response

Thank you.

Reviewer 3 Report

The authors addressed the reviewer's comments with attention and modified the paper with the suggestions provided. The new version of the paper has improved both in the presentation and in the contents.

Minor revision

- The format of the text is not consistent with the format recommended by the journal. It is advisable to reformat all elements of the paper according to the template of the journal.

-Equations must be numbered

-497) After section 2 there must be 3 not 4

-Table 1.2 are not formatted consistently with journal recommendations

-There are a lot of empty spaces, so journal space is wasted

-Add information about the hardware and software used

Author Response

• The format of the text is not consistent with the format recommended by the journal. It is advisable to reformat all elements of the paper according to the template of the journal.

The entire paper has been reformatted using the template of the journal (applsci-template.dot)

-Equations must be numbered

Equations have been numbered.

-497) After section 2 there must be 3 not 4

Section numbering changed

-Table 1.2 are not formatted consistently with journal recommendations

Table has been formatted

-There are a lot of empty spaces, so journal space is wasted

Since reformatting has done, the empty spaces have been removed

-Add information about the hardware and software used

A paragraph is added to describe hardware and software used