Review Reports
- Magdalena Trillo-Domínguez 1,*,
- Juan Ignacio Martin-Neira 2 and
- María Dolores Olvera-Lobo 1
Reviewer 1: Patrick Pang Reviewer 2: Federico Pennestrì
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis study provides a timely and balanced scoping review comparing ChatGPT and Google Search as intermediaries in non-professional health self-consultation, highlighting key advantages, structural limitations, and research gaps. However, the current research exhibits limitations in methodological transparency, comprehensiveness of the search, and presentation of synthesized evidence. Specific comments are detailed below:
Major Comments:
1. The search was limited to PubMed/MEDLINE and Scopus, omitting Web of Science (WoS), a major multidisciplinary database commonly used in health informatics and AI-related reviews. While Scopus overlaps substantially with WoS, including WoS could enhance comprehensiveness and reduce the risk of missing unique interdisciplinary records. It is recommended that the authors add WoS to the search strategy, with duplicates removed subsequently using tools such as EndNote.
2. No full electronic search strategy was presented for any database, including specific keywords and Boolean operators. PRISMA-ScR Item 8 requires presenting the complete search string for at least one database to enable reproducibility. Authors must add this in the Methods section or as supplementary material.
3. NotebookLM was used to systematise variables and generate word clouds and idea analyses, with outputs cross-checked via NVivo. However, no details are provided on specific prompts, iteration cycles, handling of AI biases or hallucinations, or inter-rater agreement metrics for NVivo validation. Word clouds and idea analysis outputs are not reported or displayed. Greater transparency is needed in the Methods or supplementary files to support reproducibility.
4. The Results section presents only descriptive statistics on publication year, country, and specialty distribution in tabular form, while performance differences in accuracy, hallucinations, actionability, readability, and other key dimensions are described narratively without a dedicated summary table. Adding such a table in the main text would greatly improve clarity and evidence synthesis.
5. The manuscript repeatedly states that ChatGPT's actionability is approximately 40% and "lower than that observed for traditional search engines." However, no specific quantitative figures are provided for Google Search or traditional search engines in any section. Authors are encouraged to supplement comparable metrics to clarify the basis for the "lower" assertion.
6. The following recent articles related to ChatGPT should be included to enrich the discussion:
J Li, Y Yang, R Chen, D Zheng, PCI Pang, CK Lam, D Wong and Y Wang. 2025. Identifying Healthcare Needs with Patient Experience Reviews Using ChatGPT. PLOS One, 20(3), e0313442. doi:10.1371/journal.pone.0313442
J Li, Y Yang, C Mao, PCI Pang, Q Zhu, D Xu and Y Wang. 2025. Revealing Patient Dissatisfaction with Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis. Journal of Medical Internet Research, 27, e66344. doi:10.2196/66344
Author Response
REVIEWER 1
Review Report Form
Open Review
( ) I would not like to sign my review report
(x) I would like to sign my review report
Quality of English Language
(x) The English could be improved to more clearly express the research.
( ) The English is fine and does not require any improvement.
Is the work a significant contribution to the field?
Is the work well organized and comprehensively described?
Is the work scientifically sound and not misleading?
Are there appropriate and adequate references to related and previous work?
Comments and Suggestions for Authors
This study provides a timely and balanced scoping review comparing ChatGPT and Google Search as intermediaries in non-professional health self-consultation, highlighting key advantages, structural limitations, and research gaps. However, the current research exhibits limitations in methodological transparency, comprehensiveness of the search, and presentation of synthesized evidence. Specific comments are detailed below:
Major Comments:
- The search was limited to PubMed/MEDLINE and Scopus, omitting Web of Science (WoS), a major multidisciplinary database commonly used in health informatics and AI-related reviews. While Scopus overlaps substantially with WoS, including WoS could enhance comprehensiveness and reduce the risk of missing unique interdisciplinary records. It is recommended that the authors add WoS to the search strategy, with duplicates removed subsequently using tools such as EndNote.
Response:
We thank the reviewer for this valuable suggestion. The decision to limit the search to PubMed/MEDLINE and Scopus was taken deliberately, based on the specific objectives and scope of this scoping review. PubMed/MEDLINE was selected due to its centrality in biomedical and health-related research, while Scopus was chosen for its broad multidisciplinary coverage, which substantially overlaps with Web of Science, particularly in the fields of health informatics and artificial intelligence.
Given the exploratory and comparative nature of the study, the restricted temporal window (2023–2025), and the focus on empirical evaluations of ChatGPT and Google Search in non-professional health self-consultation contexts, the combination of these two databases was considered sufficient to ensure comprehensive coverage of the relevant literature while maintaining methodological coherence.
Nevertheless, the authors acknowledge that the inclusion of Web of Science could potentially retrieve additional records and recognize this as a limitation of the review, which has now been explicitly noted in the manuscript.
- No full electronic search strategy was presented for any database, including specific keywords and Boolean operators. PRISMA-ScR Item 8 requires presenting the complete search string for at least one database to enable reproducibility. Authors must add this in the Methods section or as supplementary material.
Response:
We thank the reviewer for highlighting this important point. In accordance with PRISMA-ScR Item 8, the full electronic search strategy, including all keywords and Boolean operators, has now been provided as Supplementary File S1. In addition, an explicit reference to this supplementary material has been incorporated into Section 2.2 (Inclusion and exclusion criteria and search strategy) of the Methods section to ensure transparency and reproducibility of the search process.
- NotebookLM was used to systematise variables and generate word clouds and idea analyses, with outputs cross-checked via NVivo. However, no details are provided on specific prompts, iteration cycles, handling of AI biases or hallucinations, or inter-rater agreement metrics for NVivo validation. Word clouds and idea analysis outputs are not reported or displayed. Greater transparency is needed in the Methods or supplementary files to support reproducibility.
We thank the reviewer for this comment. We acknowledge that the description of the use of NotebookLM in the original manuscript may have suggested a more central analytical role than intended. To avoid methodological ambiguity and ensure transparency, the Methods section has now been revised to clarify that NotebookLM was employed solely as an auxiliary exploratory tool to support the initial organisation and systematisation of variables identified in the reviewed studies, and not as a primary analytical instrument.
The core processes of data extraction, categorisation and synthesis were conducted by the authors through manual qualitative analysis, and the mention of word clouds and idea analyses—whose outputs were not essential to the results reported—has been removed from the manuscript. As a result, issues related to prompt specification, iteration cycles, AI bias management or inter-rater agreement metrics are not applicable to the analytical framework ultimately used in this study. This revision enhances methodological clarity and aligns the description of tools with their actual role in the research process.
- The Results section presents only descriptive statistics on publication year, country, and specialty distribution in tabular form, while performance differences in accuracy, hallucinations, actionability, readability, and other key dimensions are described narratively without a dedicated summary table. Adding such a table in the main text would greatly improve clarity and evidence synthesis.
We thank the reviewer for this constructive suggestion. To improve clarity and facilitate the synthesis of evidence, a new summary table has been added to the Results section. This table consolidates the main comparative dimensions reported across the included studies—such as accuracy, hallucinations, actionability, readability, and information quality—providing an integrated overview of the relative performance of ChatGPT and Google Search. The table complements the existing narrative synthesis without altering the analytical approach or the results reported.
In addition, to enhance the transparency of the process and support the replicability of the analysis, the data extraction table used in the review has been made available as Supplementary File S2 and deposited in Zenodo: https://doi.org/10.5281/zenodo.18510248
EN EL TEXTO:
“Table X provides a comparative synthesis of the main performance dimensions reported across the reviewed studies.”
|
Dimension |
ChatGPT (as reported in the reviewed studies) |
Google Search (as reported in the reviewed studies) |
Evidence base (as described in the corpus) |
|
Factual accuracy |
Consistently higher factual accuracy, with DISCERN scores of 4–5/5 and moderate to strong correlations with expert clinical evaluations |
Lower factual accuracy in comparative assessments |
Multiple comparative empirical studies (e.g., Kolac et al., 2025; Fushimi et al., 2025) |
|
Information quality |
Higher overall information quality, characterised by coherent and structured responses |
More heterogeneous information quality, frequently described as fragmented in comparative assessments |
Comparative and observational studies |
|
Hallucinations |
Presence of fabricated or non-verifiable references reported in approximately 31–45% of cases |
Not directly reported as a comparable phenomenon; information relies on externally linked sources that may contain inaccuracies. |
Several empirical studies focusing on reference validity (e.g., Burns et al., 2024; Mendel et al., 2025) |
|
Readability |
High linguistic complexity, typically corresponding to grades 12–14 on the Flesch–Kincaid scale |
Variable readability depending on source and webpage |
Observational and comparative studies (e.g., Alamleh et al., 2025) |
|
Actionability |
Limited actionability, with approximately 40% of responses providing clearly actionable guidance |
Higher actionability reported for traditional search engines in comparative studies |
Comparative studies (e.g., Dubin et al., 2023) |
|
Source transparency |
Opaque source provenance and limited traceability of information sources |
Greater source traceability and verifiability through linked webpages |
Narrative evidence across the reviewed studies |
|
User perception / satisfaction |
Generally positive user evaluations due to clarity, coherence and perceived empathy |
Not systematically assessed in the reviewed corpus |
Selected empirical studies reporting user perceptions |
- The manuscript repeatedly states that ChatGPT's actionability is approximately 40% and "lower than that observed for traditional search engines." However, no specific quantitative figures are provided for Google Search or traditional search engines in any section. Authors are encouraged to supplement comparable metrics to clarify the basis for the "lower" assertion.
We thank the reviewer for this clarification request. The statement regarding the lower actionability of ChatGPT responses was based on comparative findings reported in the reviewed studies, rather than on a single standardised quantitative metric applied uniformly across systems. To avoid ambiguity, the manuscript has now been revised to explicitly indicate that the comparison refers to relative tendencies reported in comparative studies, rather than to directly comparable percentage values for Google Search. No additional quantitative figures were introduced, as such metrics were not consistently reported across the included studies.
- The following recent articles related to ChatGPT should be included to enrich the discussion:
J Li, Y Yang, R Chen, D Zheng, PCI Pang, CK Lam, D Wong and Y Wang. 2025. Identifying Healthcare Needs with Patient Experience Reviews Using ChatGPT. PLOS One, 20(3), e0313442. doi:10.1371/journal.pone.0313442
J Li, Y Yang, C Mao, PCI Pang, Q Zhu, D Xu and Y Wang. 2025. Revealing Patient Dissatisfaction with Health Care Resource Allocation in Multiple Dimensions Using Large Language Models and the International Classification of Diseases 11th Revision: Aspect-Based Sentiment Analysis. Journal of Medical Internet Research, 27, e66344. doi:10.2196/66344
Response:
We thank the reviewer for these relevant references. Upon careful examination, we determined they focus primarily on the use of ChatGPT for patient experience review mining and sentiment analysis rather than on comparative evaluation between ChatGPT and traditional search engines in health self-consultation contexts. Because the objective of this scoping review is to synthesise empirical evidence that explicitly compares ChatGPT and Google Search as intermediaries in health information seeking. we did not integrate these studies into the revised manuscript.
Reviewer 2 Report
Comments and Suggestions for AuthorsDear authors,
I have read your paper with interest.
I think it deservese publication, provided some major revisions especially regarding the time range (Google search is likely to have information published before 2023) and improving the classification in fields. For example, not only orthopedics from surgery, but also orthopedics from muscoloskeletal disease in general, or "musculoskeletal disorders" and surgery, including orthopedic in the former.
The next need to be shortened in many points.
Best wishes
These are my comments:
L. 36 please remove "From ... Chat GPT"".
L. 51 I could not understand what "(Authors)" mean.
L. 62 I would replace "patients" with "parties". Were the materials generated only by patients?
L. 131 Why did you start from 2023?
LL. 132-135 This part is redundant, it should be strongly summarized. For instance, the aim of the study is mentioned more than once and also is the time range of the literature review. In general, the introduction should ne strongly made more concise.
LL. 170-173 Now it is clear why did you start from 2023. It must be moved to the first time you mention the time range. Moreover, if starting from 2023 is due to ChatGPT being officially launched that year, the same does not work for the comparer Google Search, therefore you could miss information published on the latter before 2023.
LL. 242-243 Why was orthopedics put together with surgery? I do not agree with this classification, it should be at least motivated clearly.
L. 273 a "higher" performance.
L. 295 It would be interesting to read some examples of these non-existent words or terms, putting them in context.
L. 361 I would stop mentioning the time range. Just mention it once at the beginning.
LL. 418-421 please remove the final sentence, it is already written in the introduction.
LL. 458-471 These considerations on public health communication should be made in the discussion with a dedicated section like those on future research and health communication. The conclusion is too long.
Author Response
Reviewer 2
Review Report Form
Open Review
(x) I would not like to sign my review report
( ) I would like to sign my review report
Quality of English Language
( ) The English could be improved to more clearly express the research.
(x) The English is fine and does not require any improvement.
|
Is the work a significant contribution to the field? |
|
|
Is the work well organized and comprehensively described? |
|
|
Is the work scientifically sound and not misleading? |
|
|
Are there appropriate and adequate references to related and previous work? |
Comments and Suggestions for Authors
Dear authors,
I have read your paper with interest.
I think it deservese publication, provided some major revisions especially regarding the time range (Google search is likely to have information published before 2023) and improving the classification in fields. For example, not only orthopedics from surgery, but also orthopedics from muscoloskeletal disease in general, or "musculoskeletal disorders" and surgery, including orthopedic in the former.
The next need to be shortened in many points.
Best wishes
RESPONSE:
We sincerely thank the reviewer for the careful reading of the manuscript and for the detailed, line-specific suggestions. All comments have been addressed as follows.
These are my comments:
- 36 please remove "From ... Chat GPT"".
- 51 I could not understand what "(Authors)" mean.
- 62 I would replace "patients" with "parties". Were the materials generated only by patients?
- 131 Why did you start from 2023?
See lines 170–173
- 132-135 This part is redundant, it should be strongly summarized. For instance, the aim of the study is mentioned more than once and also is the time range of the literature review. In general, the introduction should ne strongly made more concise.
- 170-173 Now it is clear why did you start from 2023. It must be moved to the first time you mention the time range. Moreover, if starting from 2023 is due to ChatGPT being officially launched that year, the same does not work for the comparer Google Search, therefore you could miss information published on the latter before 2023.
RESPONSE:
The restriction of the review to the 2023–2025 period was a deliberate and conceptually grounded decision, which has now been explicitly stated at the first mention of the temporal range in the Introduction, as suggested by the reviewer. The objective of the study is not to analyse Google Search or ChatGPT in isolation, but to examine their coexistence and comparability as intermediaries in online health self-consultation. Public access to ChatGPT began in late 2022, and it is only from 2023 onwards that empirical studies systematically address scenarios in which both systems are simultaneously available and directly compared.
We acknowledge that Google Search has been in use for a much longer period and that relevant literature exists prior to 2023. However, including such studies would introduce an inherent asymmetry into the comparative framework, as ChatGPT was not yet part of the information ecosystem. For this reason, earlier literature on Google Search was intentionally excluded, as it does not address the comparative context that constitutes the focus of this scoping review. This decision ensures conceptual coherence and internal validity for the specific comparative purpose of the study.
- 242-243 Why was orthopedics put together with surgery? I do not agree with this classification, it should be at least motivated clearly.
We thank the reviewer for this relevant observation. We agree that the initial classification of medical specialties—particularly the grouping of orthopaedics and surgery—required clearer justification. In response, the manuscript has been revised to include an explicit explanation of this decision (see revised lines 248–258). The grouping reflects a substantial overlap in clinical focus (e.g., trauma and joint replacement) and aligns with the way several reviewed articles self-categorised under general surgery while primarily addressing musculoskeletal conditions. Although we acknowledge the disciplinary distinction, this decision was made to facilitate comparative synthesis within the framework of the present scoping review. This clarification improves methodological transparency without altering the analytical approach or findings.
- 273 a "higher" performance.
Response:
Thank you for this observation. The wording has been revised as suggested, and “a high performance” has been changed to “a higher performance” to reflect the comparative context more accurately.
- 295 It would be interesting to read some examples of these non-existent words or terms, putting them in context.
Response:
We thank the reviewer for this useful suggestion. In response, illustrative examples of non-existent terms or medically implausible concepts generated by ChatGPT have been added to the manuscript. These examples, drawn from the reviewed studies, help contextualize the nature of lexical hallucinations and improve the transparency and interpretability of the results.
- 361 I would stop mentioning the time range. Just mention it once at the beginning.
Response:
Thank you for this observation. The reference to the time range in line 361 has been removed, as suggested. The temporal scope of the review is now mentioned only at its first and most relevant occurrence, to avoid redundancy and enhance conciseness.
- 418-421 please remove the final sentence, it is already written in the introduction.
Response:
Thank you for this observation. The final sentence in lines 418–421 has been removed to avoid redundancy with the Introduction and to improve conciseness.
- 458-471 These considerations on public health communication should be made in the discussion with a dedicated section like those on future research and health communication. The conclusion is too long.
Response:
Thank you for your suggestion. We have moved the considerations on public health communication to the Discussion section and included them in a dedicated subsection, as suggested. As a result, the Conclusion is now more concise.
Author Response File:
Author Response.docx
Round 2
Reviewer 1 Report
Comments and Suggestions for Authors1. The authors acknowledge that including Web of Science could potentially retrieve additional relevant records and have documented this as a limitation in the revised manuscript. However, they maintain their original decision to search only PubMed/MEDLINE and Scopus, asserting that these two databases were sufficient to cover the relevant literature. By merely classifying the risk of omission as a limitation without implementing remedial measures, the methodological rigour and evidentiary completeness of the review are compromised. This approach does not satisfy the core requirements of PRISMA-ScR regarding the comprehensiveness and reproducibility of the search strategies.
2. The authors argue that the two recommended articles primarily focus on using ChatGPT for mining and sentiment analysis of patient experience texts, rather than on direct comparative evaluation between ChatGPT and Google Search in the context of non-professional health self-consultation, and therefore were not included. Nevertheless, the discussion sections of both articles explicitly address how patients utilise AI tools to understand and articulate healthcare needs, as well as the potential role of AI in enhancing patients’ health information acquisition and experience. These aspects are conceptually related and complementary to the central theme of the present review, which concerns the role of AI as an intermediary in health information seeking. They would be particularly valuable in the Discussion section to deepen the exploration of ChatGPT’s broader implications within the ecosystem of patient-initiated health information seeking. It is disappointed that the authors have chosen a narrow perspective for their work and ignored the relevant research in the broader health disciplines.
Although the authors have responded positively to several other comments, the two fundamental issues outlined above remain unresolved. These shortcomings directly affect the completeness and scholarly contributions, particularly in a rapidly developing and highly competitive research domain. In view of the above, I cannot recommend acceptance at this time.
Author Response
Response to Reviewer 1
We would like to sincerely thank the reviewer for their thoughtful and constructive comments, which have substantially strengthened the rigor and comprehensiveness of our manuscript. We carefully considered the observations and engaged in internal discussion before implementing significant revisions to address the two main concerns raised:
- The inclusion of Web of Science (WoS) as a primary database in the search strategy.
- The incorporation of recently published 2025 articles directly connected to the topic.
Below, we detail the modifications introduced in response to these recommendations.
- Inclusion of Web of Science (WoS)
In response to the reviewer’s suggestion, we expanded the search strategy to include Web of Science (WoS) as a core database alongside PubMed/MEDLINE and Scopus. This inclusion strengthens the interdisciplinarity and exhaustiveness of the review by incorporating additional indexed literature from communication, information science, and technology domains.
The search was conducted in WoS between February 13 and 15, 2026. As a result of this expanded search strategy, the final corpus increased to 63 empirical studies.
The following sections of the manuscript were updated accordingly:
- Methodology section (line 72): Explicit inclusion of WoS in the database selection and search description.
- Study selection process: Updated numbers reflecting the new total of 63 included records.
- Results section 3.1 (lines 248–253): Revised quantitative description of publication distribution and corpus characteristics.
- Discussion section (lines 297–301 and line 456): Adjusted interpretation of findings in light of the expanded dataset.
- Conclusions (lines 471–480): Reformulated to reflect the strengthened empirical basis.
- Inclusion of 2025 Articles Relevant to the Topic
Following the reviewer’s recommendation, we incorporated the two recently published 2025 empirical studies directly related to the comparison between Google Search and ChatGPT in health self-consultation contexts.
These studies have been fully integrated into:
- The quantitative synthesis of the corpus (now n = 63).
- The analytical interpretation in the Results section.
- The Discussion and Conclusions, where their findings contribute to reinforcing observed patterns regarding accuracy, hallucinations, and actionability.
- Updated Supplementary Materials
To ensure transparency and reproducibility:
- Supplementary File S1 (search strategy) has been updated to reflect the WoS inclusion.
- Supplementary File S2 (data extraction table) has been revised to include all 63 records.
- The Excel database uploaded to Zenodo has been replaced with a new version reflecting the expanded corpus and updated metadata. The Zenodo access link has been updated accordingly.
- Updated Figures
All three figures have been revised to reflect the updated corpus:
- PRISMA workflow diagram (Figure 1)
- Corpus characteristics (Figure 2)
- Comparative synthesis diagram (Figure 3)
Both the graphical files and the in-text references have been updated to ensure internal consistency.
- References Section
The final section of references (lines 514–705) has been expanded and reorganized to incorporate all newly included studies, ensuring bibliographic completeness and coherence with in-text citations.
Final Statement
We believe these substantial modifications directly address the reviewer’s concerns and significantly enhance the methodological robustness, comprehensiveness, and transparency of the review. We sincerely appreciate the reviewer’s guidance, which has strengthened the scientific contribution of the manuscript.
Reviewer 2 Report
Comments and Suggestions for AuthorsDear Authors,
I am not convinced about the classification and distinction between orthopedics, surgery and so on, but I think this is not a reason to stop the paper revision at this stage, considering that the other recommendations were all met.
Best wishes
Author Response
Response to Reviewer 2
We sincerely thank the reviewer for their careful reading of the manuscript and for their comment regarding the classification of medical specialties, particularly the grouping of orthopedics and surgery.
In the first revision round, we provided an initial clarification; however, we acknowledge that the explanation may not have sufficiently justified the analytical decision. Following both the reviewer’s observation and the substantial update of the corpus after the inclusion of Web of Science (as recommended by Reviewer 1), we reconsidered the classification strategy in greater depth.
With the expanded dataset (n = 63), we re-examined the distribution of specialties and implemented a revised categorization. In the updated version of the manuscript and in the revised Figure 2, the category has been consolidated under “Orthopedics”, without maintaining “Surgery” as a parallel label.
This decision was based on the following considerations:
- The studies initially grouped under “orthopedics and surgery” primarily focused on musculoskeletal conditions, trauma-related pathologies, and orthopedic surgical procedures (e.g., joint replacement, spinal surgery, arthroscopy), rather than general surgery as a broader discipline.
- Within the expanded corpus, strictly surgical studies unrelated to musculoskeletal conditions represented a minor proportion.
- To enhance conceptual clarity and avoid disciplinary ambiguity, the revised classification now agglutinates musculoskeletal injuries, orthopedic operations, and related skeletal conditions under the broader and more coherent specialty of Orthopedics.
We believe that this reclassification resolves the reviewer’s concern by providing a clearer and more internally consistent taxonomy of specialties. Importantly, this adjustment does not alter the analytical conclusions of the study, as the purpose of the specialty categorization is descriptive rather than inferential.
We appreciate the reviewer’s comment, which prompted a more precise categorization and improved the methodological transparency of the manuscript.
Author Response File:
Author Response.docx
Round 3
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors have properly addressed my concerns and therefore I have no further comments. Thank you for their efforts and the opportunity for reviewing this paper.