Methodology for AI-Based Search Strategy of Scientific Papers: Exemplary Search for Hybrid and Battery Electric Vehicles in the Semantic Scholar Database

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review the article entitled "Methodology for AI-based search strategy of scientific papers: Exemplary search for hybrid and battery electric vehicles in the Semantic Scholar database".

Despite its potential contributions to the field of AI utilisation in literature review processes, the study presents several shortcomings and negative aspects that should be addressed in order to meet publication standards.

1. Although the paper provides an overview of existing methodologies for systematic literature review across various fields (medicine, social sciences, engineering), it does not offer a sufficiently detailed analysis of previous research on the use of AI in the search and selection process for academic publications. In my view, it is essential for the authors to reference more recent studies (as research from 2003, 2005, 2009, 2012, 2013, etc., may already be obsolete), better explain the limitations of existing methodologies, and highlight how their method brings clear improvements;

2. The methodology presented is applied exclusively to data retrieved from Semantic Scholar, which constitutes a significant limitation. In my opinion, reliance on a single database raises concerns regarding the exhaustiveness and quality of the results. A validation of the methodology would require expanding the search to other databases, such as Scopus or Web of Science, to demonstrate the robustness of the method and its applicability in a broader context;

3. I believe the methodology is not explained in sufficient detail, particularly regarding the quality control mechanisms for the results. The authors admit that AI cannot control the filters in the database interface, nor how certain search terms are interpreted. While this limitation is mentioned, it is not addressed by incorporating additional methods to verify data quality;

4. Although the authors present the AI-based search and selection method for publications, there is no adequate validation process for the proposed methodology. Without a clear comparison between the AI methodology and a traditional manual approach, it remains unclear whether the AI method provides significant improvements in terms of efficiency and the quality of results;

5. In my opinion, the paper’s conclusions are too general and do not offer clear recommendations for the practical use of the proposed method. While the authors mention that AI can improve the systematic review process, they do not provide specific details on how this method can be applied in other fields or how the identified limitations, such as the risk of excluding relevant papers, can be overcome.

In conclusion, while the paper addresses an interesting and potentially innovative topic regarding the use of AI in reviewing scientific literature, it is, in my opinion, not yet ready for publication in its current form. I believe that major revisions are required at this stage.

Author Response

Dear Reviewer,

Thank you for taking the time to review our manuscript and for providing valuable feedback. We greatly appreciate your input on all five points. The Editor indicated that Comment #3 is the most relevant, and thus we have focused our response on addressing it thoroughly. We have made corresponding adjustments to the manuscript and provide detailed responses to each of your comments below:

#1: In the original manuscript, we focused on describing existing methodologies for literature search and additionally provided a review of current research on the use of AI in literature review. However, while previous studies have generally offered an overview of available tools, they have neither provided a clear methodology for exploiting all possible search result within the database nor tailored it for use in an engineering-related context.

#2: We would like to clarify that automated API queries are legally permissible, and Semantic Scholar provides access to a substantial collection of scientific papers. The study we cited in Section 2.1.4 reports that Semantic Scholar covers 98.8% of relevant scientific literature, underscoring the reliability of this database. Consequently, we did not conduct additional validation with other databases (other publications also worked with e.g. PubMed only).

#3: We recognize that our initial statement “For the automated part, it must be assumed that the limitations and filters beyond the interface, as well as AI interpretation, cannot be controlled”—may have been unclear and led to some misunderstanding. This statement refers to two existing limitations in our methodology:

 

1. Uncertainty in Term Interpretation: Terms like “review” and “study” may yield different results within the Semantic Scholar database. This variation arises from the system’s interpretation of terms, which we cannot fully control, requiring us to adapt to these inherent limitations.

 

2. Database Filter and Pre-selection Constraints: Our exemplary search was limited to engineering-related fields by configuring the API query accordingly.

 

To improve the transparency of our methodology, we have revised the manuscript to clarify and expand upon our quality control approach, which consists of two primary stages:

 

1. Quality Control of Search Terms: We employed a broad search strategy and monitored for duplications to ensure relevance and accuracy, verifying that our search terms were precise enough to produce relevant, consistent results. This step has been emphasized in the revised manuscript.

 

2. Quality Control of Retrieved Literature: We used filters in our query (as described in Section 2.1.4), and we have further highlighted this step in the manuscript. In our exemplary search, we first filtered the results as discussed in Section 4.2. The final selection of 713 papers then involved a manual review to confirm their relevance to the topic.

 

This two-pronged quality control process strengthens the validity of our findings and establishes a robust foundation for future research. We believe that a follow-up review focusing on these 713 papers would not only affirm the efficacy of our methodology but also underscore the value of our approach in achieving such a comprehensive set of papers.

#4: We agree that a validation comparing the AI-based method with a traditional manual approach would provide additional insights. However, the sheer volume of over 10,000 papers analyzed in our study would not be feasible to manage manually within a comparable timeframe. The AI-based approach enables us to handle and process extensive datasets efficiently, which would be impractical with manual methods. And we are confident that due to our set-up of the search-terms and neighborhood analysis we will include also papers we would not have searched for.

First quality assessment was performed manually within the scope of the methodology. A detailed evaluation will take place in a follow-up publication, which is planned to focus on the papers identified as relevant in the exemplary search.

#5: While our study demonstrates the method within an engineering context, it is indeed easily adaptable to other fields by simply changing the keywords and API filters. We designed the methodology to be adaptable, enabling researchers in various disciplines to apply it effectively with minimal adjustments.

To address the concern of potentially excluding relevant papers, we mitigate this risk through a carefully defined set of keywords and iterative use of the Citrus search tool. By conducting multiple search loops, we ensure a more comprehensive capture of the literature. This approach allows us to refine search parameters to maximize inclusivity while maintaining focus on relevant results. This was demonstrated in the manuscript with the convergence analysis (Section 4.3).

Thank you again for your insightful feedback, which has helped us enhance the clarity and rigor of our manuscript.

Best regards,

Florian Wätzold

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

Overall excellent work. The language is clear, concise. I rarely see manuscripts of this quality. The fact that I also have no feedback can be seen as a compliment. Well done. 

One small suggestion, perhaps you can take a look at the headers (see 2 and 2.1). The indent makes it a bit odd to read. Furthermore, in section 2.2 the bullet points are aligned in an odd way. Can you take a look at this? 

The appendices might be added as supplementary materials (rather than as appendices).

Author Response

Thanks for your feedback! We have changed the wording of the appendix, and adjusted the formatting.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

In my opinion, the manuscript entitled "Methodology for AI based search strategy of scientific papers: Exemplary search for hybrid and battery electric vehicles in the Semantic Scholar database" has been sufficiently improved. As such, I recommend publication in present form.