Experimental Evaluation of Possible Feature Combinations for the Detection of Fraudulent Online Shops

Round 1

Reviewer 1 Report

Hypothesis is not clearly presented.

It needs to be improved to be considered for publication.

hypothesis that large feature sets are not necessarily 65 required for the detection of malicious stores, and in order to achieve high accuracy, it is 66 sufficient to evaluate a few essential publicly available features.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The topic of the manuscript is interesting but the following points might add value to the manuscript:  

2.2. Primary Dataset Preparation

1. Simplify Sentences: To improve comprehension, some long sentences should be made shorter. Reading comprehension may be enhanced by segmenting long sentences into smaller ones.

2. Explicit declaration: Though you give a thorough explanation, you might want to begin with a succinct and clear technique declaration. For readers, this opening sentence can act as a guide.

3. Additional Information on Dataset Creation: Provide more details about the manual and Python script-based procedures used to create the main dataset. Transparency is improved by giving precise information about the extraction procedure.

4. Consistent vocabulary: Make sure that vocabulary is consistent, especially when referring to characteristics. For instance, you include TrustPilot and SiteJabber in the feedback and rating features (F16 and F17), but then you refer to them as "feedback and rating platforms." Keep the naming consistent from beginning to end.

5. Describe Feature F18: Give a little more background information or an explanation of the Tranco list and the connection between being on it and the identification of fraudulent internet stores.

6. Brief Explanations: Take into account giving succinct explanations for specific words or ideas, including combinatorial formulas. Readers who might not be familiar with the terms can benefit from this.

7. Rationale for Machine Learning Algorithms: Clearly state the reasons for your selection of particular machine learning algorithms. Further context could be added with a brief justification of their applicability to your research question or their frequent use in related studies.

2.3. Experimental Setup

1. Justification for Library Selection: Take a moment to quickly explain why particular libraries were chosen. For example, why was 'Scikit-learn' selected for machine learning and 'BeautifulSoup' selected for web scraping? This can assist readers in comprehending the reasoning behind your decisions.

2. Make the 'Combination' Library clear: Describe the 'combination' library in brief or include a reference for it so that others can create secondary datasets. Readers who might not be familiar with this particular library will benefit from this.

3. Extra Information on Splitting Techniques: You discuss a variety of splitting techniques; however, you should think about including a sentence or two that explains the selection process and the possible effects various tactics may have on model evaluation.

4. Explicit Declaration of Machine Learning Framework: Although Scikit-learn was used for machine learning, making it clear to readers that it is the main machine learning framework can help them understand it better.

5. Python Version: Take into Account the version of Python that you were using for your research. This data may be significant to repeatability.

6. Integration of Citations: To provide further context to your experimental decisions, think about mentioning pertinent sources if some certain configurations or settings have been impacted by best practices or existing research.

3. Results

1. Explicit Declaration of Major Findings: Take into consideration offering a succinct synopsis or explicit declaration outlining the major conclusions drawn from the data. This can point readers toward the most crucial conclusions.

2. Trade-offs and Considerations: Talk about any trade-offs or factors related to selecting a particular number of features. For example, if adding features just slightly improves accuracy, talk about the possible computational complexity or cost.

3. Explanation of Significance: You report in Figure 3 that the number 'X' in the feature columns indicates the significance of the features. It could be helpful to go into more detail about the process used to identify its relevance or to cite the technique for significance assessment.

4. Provide Figure captions: If they aren't there already, think about including captions that succinctly explain the meaning and content of each figure (such as Figures 2 and 3).

5. Consistent vocabulary: Make sure that vocabulary is used consistently. In this section, for example, you reference "accuracy" and "classification accuracy"; please ensure that these terms are utilized consistently.

6. Discussion of Feature Significance: Although you identify features F9 and F15 as significant, it would be helpful to include a brief explanation of these features' significance for the classification job in order to give your results more context.

7. Generalization Consideration: Talk about how broadly applicable your results are. Mention, for instance, if the findings are unique to the dataset that was used or if they might be useful in other similar situations.

8. References to Relevant Figures: To help readers get the context, when referencing figures in the text (e.g., "see Figure 2"), think about providing a summary of the most important details from the cited figure.

4. Discussion

1. Visual Aids: To enhance your written descriptions, think about adding extra visual aids like charts or graphs. Complex relationships can be better understood when shown visually.

2. Emphasizing Important lessons learned: To reaffirm for readers the most crucial themes, provide a summary of the discussion section's main conclusions or implications at the conclusion.

3. Prospective Routes: Talk about possible future paths for this field's research. Examine if adding more features could increase accuracy, for instance, or think about how reliable the suggested approach is in various situations.

4. Classifier Comparison: As you analyze the performance of various classifiers, quickly go over the reasons for the superior performance of some classifiers. This might offer more information on the characteristics of the data.

5. Consistent Terminology: Make sure that terms are used consistently throughout the conversation. You refer to "feature sets," "combinations of features," and "features," for instance, so keeping your language consistent will help the reader understand what you're saying.

6. Statement Quantification: Whenever feasible, give a numerical value to claims like "a very high accuracy of 0.9605." Giving readers a benchmark or reference point can make it easier for them to understand the relevance

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

I was grateful for the opportunity to read this paper. The paper is publishable but I think it would be much stronger if authors better set the paper in a broader academic debate on the effectiveness of counter-fraud initiatives. I highlight some of the key comments and suggestions below.

1.     The authors should better clarify into which academic discussion or body of knowledge this paper contributes; and better introduce a broader counter-fraud context. I think that the paper can contribute to a wider academic discussion on the effectiveness of fraud disruption. Ideally, authors should include several paras in their Introduction or include a contextual section on the problem of effective fraud disruption. Consider discussing the following works (see also references to other works in the same papers).

a.     Sutherland, E.H., 1949. White collar crime. New York: Dryden Press.

b.     See issues around evidence based policing in counter-fraud and money laundering: Hock, B., Button, M., Shepherd, D. W. J., & Gilmour, P. M. (2023). What works in policing money laundering? Journal of Money Laundering Control. https://doi.org/10.1108/JMLC-07-2023-0109, https://doi.org/10.1108/JMLC-07-2023-0109

2.     The above would allow the authors to contrast their findings in Section 4 with a broader academic literature and engage in a more critical discussion.

3.     Consider including a very short “Conclusion” section

4.     Page 5 – the links does not read well; use the citation standard/footnoting consistently

English is fine

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

1.      Abstract is wordy and lacks conclusive summarization of this study. Also, this manuscript uses present and past tense alternatively in the same paragraph, and this situation is happened throughout entire contents of this manuscript. This implies technical writing should be much improved before accepting for publication.

2.      In this work, authors evaluated the influence of the selected 18 features and their number on the accuracy of the detection of fraudulent online shops. They even made conclusion that only 7 most significant features allowed to achieve 0.9605 accuracy, while the best classification result was achieved with fifteen features, reaching an 24 accuracy of 0.9693 using a random forest classifier. From researching point of view, the interesting readers might expect not only the results, but prefer to know more about this study, e.g. what are the reasons causing this result, the summarization, utilization, or what’s the insight and conclusion remarks of this study, etc.

3.      There is no much novelty of this topic; however, authors might claim their contributions from the application point of view. I feel that “discuss” section should be the core part of this study, which might fulfill this goal from providing answers to those demands depicted in comment 2. However current Section 4 is weak, which consists of only two pages, and this section is not well-organized, where one paragraph after another paragraph is presented to describe author’s opinions. It is difficult for readers to know the results or author’s contributions. Authors might set subsection in this section to analyze and discuss systematically the results of this work to meet reader’s demand.   

4.      Additional conclusion section is suggested.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 5 Report

The study's focus on a custom dataset and the use of multiple machine learning models to test combinations of up to 18 features across different categories (URL-based, content-based, and third-party services-based) is commendable for its thoroughness.

The methodology is robust, involving the creation of a primary dataset with 18 features and the exploration of feature subsets of various lengths. The combinatorial approach to feature selection and the use of a range of classifiers (Decision Tree, Random Forest, SGD, Logistic Regression, Gaussian Naive Bayes, Multilayer Perceptron, XGBoost) add to the study's depth. The methodology section is well-structured, providing clarity on the process of feature extraction and dataset preparation.

The results are significant, especially the finding that even a limited set of 7 features can achieve an accuracy of 0.9605. The highest accuracy achieved (0.9693) with fifteen features using a random forest classifier is noteworthy. These results provide valuable insights into the relative importance of different features in detecting fraudulent online shops. The detailed presentation of these results, including the table showing feature combinations and their corresponding accuracies, is both informative and easy to comprehend.

The observation that a smaller number of significant features can achieve high detection accuracy is particularly useful for practical applications, such as developing security plugins for web browsers. The study's confirmation of known attributes of fake e-shops and the identification of less obvious dependencies are valuable contributions to the field.

Suggestions:

Feature Analysis: Further analysis of the less significant features could provide insights into why they have a lesser impact on accuracy. Understanding the limitations or irrelevance of certain features could inform future feature selection processes.

Algorithm Comparison: While the study effectively compares various feature combinations, a deeper analysis comparing the performance of each machine learning algorithm used could provide additional insights into their suitability for this type of fraud detection.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 4 Report

Authors have made great modifications to the revised manuscript, especially the discussion section. I have no further comments. 

Reviewer 5 Report

The authors improved the manuscript as suggested.