Evaluation of Replies to Voice Queries in Gynecologic Oncology by Virtual Assistants Siri, Alexa, Google, and Cortana

Round 1

Reviewer 1 Report

Paper deals with important topics in BioMedInformatics. The authors have reported on the validity of audible replies to voice queries in gynecologic oncology.    

However, I have a number of suggestions:   

1. Abstract should be extended by the obtained results in part of the performance evaluation of the proposed approach entities described in 2018. 

2. Paper does not fit Bioinformatics journal guidelines.

3. Please optimize image sizing in the paper and table guidelines.  

4. I would suggest visualize comperison of the result, as right now it's not clearly presented.  

5. The paper hasn’t any clear future description and plans how to develop this current research.  

6. Please provide a link to an open-access repository with the dataset you attached at the end of the paper, and add them into references, it'll be a far better way, to organize it.

7. A lot of references do not meet the guideline and standarts.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

1. The abstract is poorly written. 

2. The whole manuscript sways away from one point to another. The entire manuscript needs excessive reconstruction.

3. There is no statistical power used. The statistical methods and results should be properly discussed.

4. The results and conclusion did not properly explain the outcomes of the study.

5. Why there are references in the conclusion region? please add that text to the discussion portion.

6. What is the significance of this study in the medical field, please add text to ensure the importance of this current study.

The manuscript contains typographical, grammar and sentence structure issues.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Thanks for allowing me to review this paper. This is a very interesting topic, but the way authors have conducted and written is unacceptable for publication. Here are my comments:

1. Method: Need to rewrite it again. They have started writing with the data analysis process. Later on, they provided a statistical analysis section. It is really confusing.

2. What was the questionnaire validation process?

3. Who has evaluated the answer to those questions?

4. Why did authors not ask the same amount of questions to Google, Alexa, SiRI, and Cortona?

Please rewrite the methods and results section.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

In this study, authors compared replies of voice queries in gynecologic oncology by several virtual assistants. It is very useful for pateints to choose suitable virtual assistants for gynecologic oncology. Below are a number of issues that the authors shall address: 

1. Why did you choose these 24 general queries rather than others. Do you try other sets of queries?

2. There are also other questions which are proposed by patients. For example, what is IVB ovarian cancer with liver metastasis or What is the survival rates of ovarian cancer.

3. Did you try other questions but with bad replies?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 5 Report

The authors Land et al have done a great job critically analyzing the response of virtual assistants to common questions in the field of gynecological oncology. The study is very interesting. The questions asked are simple, only about the definition of certain concepts in gynecological oncology. VAs are able to answer the questions to various extent. This modern study addresses the question whether patients should rely on VAs for answering simple questions regarding gynecological oncology. Unfortunately, the VAs are able to correctly answer only a small fraction of gynecological oncology questions. It suggests a strong need for improvement in artificial intelligence and VA systems. I have only a few comments for the authors.

1.     While the VAs are able to answer general questions with good accuracy, they lack accuracy while answering gynecological oncology questions. Could this be attributed to the difficulty in finding the information about gynecological oncology on internet and lack of a good server to analyze/interpret scientific literature? Authors could comment on these differences in ability.

2.     In the materials and methods authors mention VAs can understand different languages and dialects, but what about different accents? Same words in English may be pronounced differently in different accents, would the VA recognize that? Would accuracy of the analysis change depending on accent? Authors could comment on this.

3.      Another important question to ask would be if these results hold true if searches are carried out in different language? Would the accuracy of answers depend on language used?

4.     Authors could comment on the correlation between accuracy of answering gynecological oncology questions and search engine in use. Should the patients preferentially rely on one search engine over others?

5.     Authors could comment on whether the accuracy of answers reduce if more complex questions about gynecological oncology are asked. If the questions are not mere definition and are more complex, the accuracy will be very low, and patients should absolutely avoid using VAs.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 6 Report

Dear Authors,

The manuscript titled “Evaluation of Replies to Voice Queries in Gynecologic Oncology by Virtual Assistants Siri, Alexa, Google, and Cortana” is interesting however manuscript has lots of potential for improvement. Here are few suggestions.

1.    In section 2, line 107, the scoring system does not consider the relevancy of the question. Could you please elaborate more on the scoring system for the relevancy, specifically when the information and guidelines are regularly reviewed and updated.

Comment: Should relevancy of the information be considered when evaluating VAs? Current update vs dated information?

2.    Large language model like Google’s PaLM2 and OpenAI’s GPT-4 through user interface as BARD and ChatGPT has evolved as convenient way to access information for public. The output information in these models is based on the quality of the prompt provided however the model itself can generate prompts based on the topic or keywords. Models like BARD also provides references for its information.

Comment:  In conclusion section, authors mentioned about the BARD and ChatGPT, however it is recommended that author expand the discussion/conclusion section to discuss more about the VAs and its incorporation with large language model.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The revised manuscript addressed all the points.

Reviewer 3 Report

Can be accepted for publication.

Need moderate revision.

Reviewer 6 Report

Dear Authors,

All the comments are addressed.