ChatGPT Research: A Bibliometric Analysis Based on the Web of Science from 2023 to June 2024

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. **Comprehensive Understanding**: "The paper demonstrates a comprehensive understanding of ChatGPT's architecture and its potential applications. The depth of analysis is impressive and shows a solid grasp of the underlying principles."

2. **Clear and Concise Writing**: "The writing is clear and concise, making complex concepts accessible to a broad audience. The explanations are well-structured, and the flow of the paper is logical and easy to follow."

3. **Innovative Insights**: "The paper provides innovative insights into the practical applications of ChatGPT. The examples and case studies used are relevant and effectively illustrate the capabilities of the model."

4. **Thorough Research**: "The research is thorough and well-documented. The paper references a wide range of sources, which adds credibility and depth to the discussion. The use of up-to-date references is particularly commendable."

5. **Engaging Content**: "The content is engaging and thought-provoking. The authors have done an excellent job of highlighting both the strengths and potential challenges of ChatGPT, providing a balanced perspective."

6. **Practical Implications**: "The discussion on the practical implications of ChatGPT is particularly strong. The paper does a great job of connecting theoretical concepts to real-world applications, making the research highly relevant and impactful."

7. **Visual Aids**: "The use of visual aids such as diagrams and tables is highly effective. They enhance the reader's understanding and break down complex information into easily digestible parts."

8. **Contribution to the Field**: "This paper makes a significant contribution to the field of AI and natural language processing. It not only advances our understanding of ChatGPT but also opens up new avenues for future research."

9. **Critical Analysis**: "The critical analysis of ChatGPT’s performance and limitations is thorough and insightful. The authors provide a balanced view, acknowledging both the strengths and areas for improvement."

10. **Future Directions**: "The section on future research directions is particularly valuable. It offers a clear roadmap for further exploration and highlights important questions that remain to be answered."

Author Response

Comment 1:

**Comprehensive Understanding**: "The paper demonstrates a comprehensive understanding of ChatGPT's architecture and its potential applications. The depth of analysis is impressive and shows a solid grasp of the underlying principles."

 

Response 1:

The author sincerely appreciates the reviewer’s positive feedback and recognition of the efforts to provide a comprehensive understanding of ChatGPT's architecture and potential applications.

 

----------------------------------------------

 

Comment 2:

**Clear and Concise Writing**: "The writing is clear and concise, making complex concepts accessible to a broad audience. The explanations are well-structured, and the flow of the paper is logical and easy to follow."

 

Response 2:

The author is grateful for the reviewer’s comment regarding the clarity and conciseness of the writing.

 

----------------------------------------------

 

Comment 3:

**Innovative Insights**: "The paper provides innovative insights into the practical applications of ChatGPT. The examples and case studies used are relevant and effectively illustrate the capabilities of the model."

 

Response 3:

The author appreciates the reviewer’s acknowledgment of the innovative insights presented in the manuscript.

 

----------------------------------------------

 

Comment 4:

**Thorough Research**: "The research is thorough and well-documented. The paper references a wide range of sources, which adds credibility and depth to the discussion. The use of up-to-date references is particularly commendable."

 

Response 4:

The author thanks the reviewer for recognizing the thoroughness of our research and the effort we put into ensuring a well-documented manuscript.

 

----------------------------------------------

 

Comment 5:

**Engaging Content**: "The content is engaging and thought-provoking. The authors have done an excellent job of highlighting both the strengths and potential challenges of ChatGPT, providing a balanced perspective."

 

Response 5:

The author thanks the reviewer’s feedback.

 

----------------------------------------------

 

Comment 6:

**Practical Implications**: "The discussion on the practical implications of ChatGPT is particularly strong. The paper does a great job of connecting theoretical concepts to real-world applications, making the research highly relevant and impactful."

 

Response 6:

The author sincerely thanks the reviewer for highlighting the strength of the discussion on the practical implications of the manuscript.

 

----------------------------------------------

 

Comment 7:

**Visual Aids**: "The use of visual aids such as diagrams and tables is highly effective. They enhance the reader's understanding and break down complex information into easily digestible parts."

 

Response 7:

The author appreciates the reviewer’s positive feedback on the visual aids used in the manuscript.

 

----------------------------------------------

Comment 8:

**Contribution to the Field**: "This paper makes a significant contribution to the field of AI and natural language processing. It not only advances our understanding of ChatGPT but also opens up new avenues for future research."

 

Response 8:

The author is deeply grateful for the reviewer’s recognition of the manuscript’s contribution.

 

----------------------------------------------

Comment 9:

**Critical Analysis**: "The critical analysis of ChatGPT’s performance and limitations is thorough and insightful. The authors provide a balanced view, acknowledging both the strengths and areas for improvement."

 

Response 9:

The author sincerely thank the reviewer for acknowledging the thoroughness and balance of the critical analysis of ChatGPT’s performance and limitations.

 

----------------------------------------------

 

Comment 10:

**Future Directions**: "The section on future research directions is particularly valuable. It offers a clear roadmap for further exploration and highlights important questions that remain to be answered."

 

Response 10:

The author is pleased that this section has been recognized as a meaningful contribution to guiding future work.

Reviewer 2 Report

Comments and Suggestions for Authors

Regarding the manuscript with the title "ChatGPT in Biomedical Research: A Bibliometric Analysis Based on the Web of Science from 2023 to June 2024"

I want to inform you that this manuscript is not innovative in terms of the methodology of this study. The presented materials are not sufficient and clear, and the research questions are not well formulated and the answers are not well done.

Author Response

Comment 1:

I want to inform you that this manuscript is not innovative in terms of the methodology of this study. The presented materials are not sufficient and clear, and the research questions are not well formulated and the answers are not well done.

 

Response 1:

The author thanks the reviewer for their valuable feedback.

Regarding the concern about the methodology’s innovativeness, the author acknowledges that bibliometric analysis is a well-established approach rather than a novel methodology. The primary aim of this study was to apply this widely-used method to deliver a timely and comprehensive overview of ChatGPT-related research, given the rapid developments in this field.

In terms of the presentation of materials and clarity, the findings have been presented using tables and figures that are standard in bibliometric studies. These visual aids were designed to enhance the accessibility and interpretability of the data.

Finally, regarding the formulation of research questions and their answers, the objective of this study is to provide an up-to-date bibliometric analysis reflecting the latest developments in ChatGPT research, using data from the Web of Science database. The study also incorporates thematic mapping to identify and visualize key research themes and trends, which we believe offers valuable insights into the evolving landscape of this field.

Reviewer 3 Report

Comments and Suggestions for Authors

Overall, I find this paper interesting. This bibliometric study provides a comprehensive overview of the burgeoning research on ChatGPT in biomedical fields, reflecting its interdisciplinary reach and potential influence, particularly within healthcare. The analysis highlights key research contributions and prolific institutions, emphasizing the rapid growth of interest in this AI-driven technology. However, one limitation in the study is the lack of reference to established benchmarking and evaluation metrics specifically tailored to medical applications. Please see the following references and discuss: 

1. Ching Nam Hang, Pei-Duo Yu, et al: MEGA: Machine Learning-Enhanced Graph Analytics for Infodemic Risk Management. IEEE J. Biomed. Health Informatics 27(12): 6100-6111 (2023)

2. Zhe Fei, Yevgen Ryeznik, et al: An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic. IEEE Trans. Big Data 8(6): 1463-1480 (2022)

3. Md. Tahmid Rahman Laskar, Sawsan Alqahtani, M. Saiful Bari, et al: A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations. CoRR abs/2407.04069 (2024)

4. Davy Tsz Kit Ng et al: Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study. Br. J. Educ. Technol. 55(4): 1328-1353 (2024) 

I find that the metrics proposed are critical for assessing the reliability, safety, and practical efficacy of ChatGPT in healthcare contexts. Incorporating some discussions with respect to recent benchmarks (see above with regards to ChatGPT and LLMs) would provide a stronger foundation for evaluating ChatGPT's role in this sensitive field, supporting more rigorous analysis and guiding its responsible implementation in clinical settings.

Comments on the Quality of English Language

English is OK.

Author Response

 

Comment 1:

Overall, I find this paper interesting. This bibliometric study provides a comprehensive overview of the burgeoning research on ChatGPT in biomedical fields, reflecting its interdisciplinary reach and potential influence, particularly within healthcare. The analysis highlights key research contributions and prolific institutions, emphasizing the rapid growth of interest in this AI-driven technology. However, one limitation in the study is the lack of reference to established benchmarking and evaluation metrics specifically tailored to medical applications. Please see the following references and discuss:

 

1. Ching Nam Hang, Pei-Duo Yu, et al: MEGA: Machine Learning-Enhanced Graph Analytics for Infodemic Risk Management. IEEE J. Biomed. Health Informatics 27(12): 6100-6111 (2023)

 

2. Zhe Fei, Yevgen Ryeznik, et al: An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic. IEEE Trans. Big Data 8(6): 1463-1480 (2022)

 

3. Md. Tahmid Rahman Laskar, Sawsan Alqahtani, M. Saiful Bari, et al: A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations. CoRR abs/2407.04069 (2024)

 

4. Davy Tsz Kit Ng et al: Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study. Br. J. Educ. Technol. 55(4): 1328-1353 (2024)

 

I find that the metrics proposed are critical for assessing the reliability, safety, and practical efficacy of ChatGPT in healthcare contexts. Incorporating some discussions with respect to recent benchmarks (see above with regards to ChatGPT and LLMs) would provide a stronger foundation for evaluating ChatGPT's role in this sensitive field, supporting more rigorous analysis and guiding its responsible implementation in clinical settings.

 

Response 1:

The author thanks the reviewer for the thoughtful and encouraging feedback, as well as for bringing attention to the references that highlight benchmarking and evaluation metrics relevant to healthcare applications. To address this valuable point, we have incorporated all four suggested references into the Study implications section of the revised manuscript and discussed their relevance to benchmarking and evaluation in healthcare settings. The author hopes these revisions address the reviewer’s concerns and enhance the manuscript’s contribution to the field:

“It should also be mentioned that robust evaluation metrics are essential to assess the reliability and safety of large language models (LLMs) such as ChatGPT. Recent research highlights the challenges of evaluating LLMs, emphasizing the importance of reproducibility, reliability, and robustness. Factors such as prompt engineering, de-coding parameters, and data integrity issues (e.g., contamination and outdated labels) significantly impact evaluation outcomes [40]. In healthcare, where data analytics is critical, integrating LLMs requires careful assessment of their capabilities [41]. Fur-thermore, the rise of multi-modal LLMs necessitates extending evaluation methodolo-gies to include visual and linguistic processing [42]. Incorporating these advancements will enable rigorous analysis, support responsible clinical implementation, and miti-gate risks such as bias, hallucinations, and inaccuracies.” (page 12, line 380-390)

and

“This includes not only teaching students about the ethical implications of AI but also ensuring they understand the importance of data privacy and responsible data usage [43].” (page 12, line 404)

Reviewer 4 Report

Comments and Suggestions for Authors

1. The authors stated that "The SCI-Expanded in the Web of Science Core Collection was queried using the Topic (TS) field: TS=(chatgpt) OR TS=(chat-gpt) OR TS=(chat gpt)." However, since the research topic is "ChatGPT in Biomedical Research: A Bibliometric Analysis Based on the Web of Science from 2023 to June 2024," it would be more appropriate to include biomedical-specific keywords in your query. This would ensure the data retrieved is more relevant to biomedical research.

 

2. The diagram in Figure 1 resembles the PRISMA flow diagram/method. If this is the case, please provide a reference to the PRISMA guidelines to ensure proper attribution.

 

3. The Discussion and Conclusion sections do not sufficiently address the relevance of ChatGPT in Biomedical Research. This limitation seems to arise because the search query did not specifically include biomedical-related terms. Expanding the query to focus on biomedical research would likely improve the relevance of your findings, the summary, and the abstract.

Author Response

Comment 1:

The authors stated that "The SCI-Expanded in the Web of Science Core Collection was queried using the Topic (TS) field: TS=(chatgpt) OR TS=(chat-gpt) OR TS=(chat gpt)." However, since the research topic is "ChatGPT in Biomedical Research: A Bibliometric Analysis Based on the Web of Science from 2023 to June 2024," it would be more appropriate to include biomedical-specific keywords in your query. This would ensure the data retrieved is more relevant to biomedical research.

 

Response 1:

The author appreciates the reviewer for the constructive suggestion regarding the inclusion of biomedical-specific keywords in the search query.

While the SCI-Expanded™ database in the Web of Science Core Collection extensively covers biomedical sciences, it is not exclusively limited to this area, encompassing a wide range of scientific disciplines. This interdisciplinary scope was considered a strength of the study, allowing us to capture the broad research landscape related to ChatGPT. However, we acknowledge that the generality of the initial search query may have limited the focus on specifically biomedical-related studies.

To ensure consistency with the scope of the data analyzed, we have revised the manuscript title and relevant sections of the text to remove the term "biomedical." These changes reflect the broader interdisciplinary focus of the study while maintaining relevance to ChatGPT research.

We hope this adjustment aligns the manuscript more closely with the scope of the data and addresses the reviewer’s concerns effectively.

 

----------------------------------------------

 

Comment 2:

The diagram in Figure 1 resembles the PRISMA flow diagram/method. If this is the case, please provide a reference to the PRISMA guidelines to ensure proper attribution.

 

Response 2:

We thank the reviewer for pointing out the resemblance of Figure 1 to the PRISMA flow diagram. In fact, the diagram was constructed based on the format suggested by a proposed reporting guidelines for bibliometric studies. To ensure proper attribution and alignment with best practices, we have added the following in the Materials and Methods section (page 2, line 80-81)

"A study flow diagram, constructed with reference to a proposed reporting guidelines for bibliometric studies, is shown in Figure 1 [26]."

26. Koo, M.; Lin, S.C. An analysis of reporting practices in the top 100 cited health and medicine-related bibliometric studies from 2019 to 2021 based on a proposed guidelines. Heliyon 2023, 9, e16780.

----------------------------------------------

Comment 3:

The Discussion and Conclusion sections do not sufficiently address the relevance of ChatGPT in Biomedical Research. This limitation seems to arise because the search query did not specifically include biomedical-related terms. Expanding the query to focus on biomedical research would likely improve the relevance of your findings, the summary, and the abstract.

Response 3:

The author thanks the reviewer for their insightful comment regarding the relevance of the Discussion and Conclusion sections to ChatGPT in biomedical research. We acknowledge that the generality of the initial search query may have limited the focus on specifically biomedical-related studies. In Web of Science, while biomedical sciences are indeed extensively covered, the SCI-Expanded™ database is by no means limited to this field.

To address the reviewer’s concern, the manuscript title and relevant sections of the text are modified to remove the term "biomedical," ensuring alignment with the broader scope of the study. These changes reflect the interdisciplinary nature of the data analyzed while maintaining relevance to the key themes of ChatGPT research.

 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Accpeted

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed my comments and greatly improved the paper for publication.

Author Response

We sincerely appreciate the reviewer’s positive feedback and recognition of the improvements made in our manuscript.

Reviewer 4 Report

Comments and Suggestions for Authors

The authors have carefully addressed all the reviewers' concerns, and I appreciate the detailed, point-by-point revisions that have significantly improved the quality of the research. However, I recommend revising the conclusion section to make it more impactful. Specifically, consider summarizing the key findings and highlighting the study’s contributions—such as addressing research gaps, validating findings, or introducing new methods. Additionally, transparently acknowledge the study's limitations, framing them as opportunities for future research. Finally, conclude with a broader perspective, connecting the findings to real-world applications or their potential broader impact.

Author Response

The reviewer's suggestions is greatly appreciated. I have revised the conclusion section accordingly. In the updated conclusion, I have emphasized the key findings, highlighted the study's contributions to addressing research gaps and advancing the field, and transparently acknowledged the limitations while framing them as opportunities for future research. Below is the revised conclusion:

This bibliometric analysis provides an overview of the collaboration patterns and key research themes in ChatGPT-related studies from January 2023 to June 2024. Our findings reveal the multidisciplinary nature of this field, with significant intersections across technological innovation, healthcare applications, educational advancement, and ethical considerations. We identified emerging trends, including enhancements in chatbot functionalities, advancements in AI system capabilities, and expanded applications in deep learning and task analysis.

By synthesizing global research trends, this analysis addresses notable research gaps and validates the relevance of advanced AI applications. In doing so, it also highlights areas that require further methodological refinement, which we acknowledge as limitations that open avenues for future investigations. Moreover, the findings support the responsible integration of ChatGPT into patient education and clinical decision-making and inform the development of AI-enhanced learning platforms, particularly for transforming medical training and enabling personalized education.

In a broader context, our work contributes to a deeper understanding of ChatGPT's evolving role across multiple disciplines and establishes a foundation for future research aimed at optimizing the societal and practical applications of chatbot technology.