Voice Assistants as Consultants for Male Patients with Sexual Dysfunction: A Reliable Option?

Round 1

Reviewer 1 Report

Dear Authors, thank you for your article, which is quite good in general. However I have a few comments to make:

1) There is a repetition line 220 ("it was")

2) I would have liked to have access to the answers given by the VAs, and the reasons their answers were classified as "low" or "intermediate" or "high". We have access to the questions in the Supplementary Material but no to the answers and, more important, to the criterion you used ot classify them as low or high. Because, without these data, it is hard to make my own opinion (though I'm not a urologist)

3) I'm a bit skeptical about the category "Use of empathic language": what is an "empathic language"? What does it mean when it applies to VA and so to computers? Are we allowed to expect from a machine an "empathic language"? Is it relevant to evaluate the presence of "empathic language" and how do you measure it?

4) From a more general point of view, I don't know if this article is some kind of commercial test between 3 VA or a test about the relevance of the use of VA as consultants for male sexual dysfonction. In other words, I find that the question of recognition of the questions, though important and prior to the answers, should not be the main point of your study. But this is a personal opinion and I do not suggest that you modify that point.

Thanks a lot again, I enjoyed reading your article, though I'm a bit frustrated not to have access to the answers.

 

Author Response

Dear Reviewer,

First of all, we wish to thank you for reviewing our paper entitled “Voice Assistants As Consultants For Male Patients With Sexual Dysfunction: A Reliable Option?”. We realize that your time is valuable, and we are very grateful for your efforts.

We have carefully read all your comments and we have tried to clarify all the concerns expressed, improving the quality of the paper according to your suggestions. For your convenience, in addition to the revised manuscript, we report below our responses to your comments and the changes made in the manuscript.

 

 

Reviewer 1

Dear Authors, thank you for your article, which is quite good in general. However, I have a few comments to make:

>>> Thank you for this appreciation.

 

1) There is a repetition line 220 ("it was")

>>> Thank you for this suggestion. We corrected the sentence.

 

2) I would have liked to have access to the answers given by the VAs, and the reasons their answers were classified as "low" or "intermediate" or "high". We have access to the questions in the Supplementary Material but no to the answers and, more important, to the criterion you used ot classify them as low or high. Because, without these data, it is hard to make my own opinion (though I'm not a urologist).

>>> Thank you for this comment. We specified in the Methods section how the answers were classified (the criteria for the classification). For each recognized question several response characteristics (domains) were subjectively rated on a scale from 0 to 10 (according to the quality). The evaluation of all answers was performed by two authors independently, for each characteristic the mean of the two assigned scores was considered. Arbitrarily, a mean domain score of 0-3 was associated with low quality, > 3 but < 7 with intermediate quality, and 7-10 with high quality. The authors involved in the evaluation were expert urologists who knew the correct answers to the questions formulated and therefore they can judge the responses of the VAs.

We respectfully disagree that adding the VA response list would add value to our research. First of all, the specific questions are not standardized and the answers could change over time and on the basis of the location/language (we specified this in the limitation section of our paper); therefore, the specific list of answers it would make little sense (less than the list of questions, provided because at least it is constant). Then, the readers should not be interested in the single answers, but in the reliability of the answers and in the methodology used by for this evaluation (the aim of our study). If the reader is not an expert, he/she has no tool/knowledge to evaluate the single answers (the list of answers would be really of little use); if the reader is an expert, despite having the tools to evaluate the individual responses, his/her evaluation remains subjective and hardly comparable with the evaluation of the authors (the list of answers would still be of little use). Finally, we hope that the reviewer takes into account that all the answers (hundreds) were recorded and played back but not transcribed. The transcription of all answers would take a long time and it was not requested by either the other reviewers or the Editor.

 

3) I'm a bit skeptical about the category "Use of empathic language": what is an "empathic language"? What does it mean when it applies to VA and so to computers? Are we allowed to expect from a machine an "empathic language"? Is it relevant to evaluate the presence of "empathic language" and how do you measure it?

>>> Thank you for this comment. Of course, we are aware that a machine cannot be “empathic” in the strict sense. With term “empathetic language” we meant the ability of the VAs to vary the "tone of the answer" (eg: happy, sad, reassuring, worried) based on the potential feelings (positive or negative) associated with the specific question asked by the user. This characteristic was evaluated on the basis of the subjective perception of the authors, according to the method explained in the previous comment.  We specified this points in the discussion.

 

4) From a more general point of view, I don't know if this article is some kind of commercial test between 3 VA or a test about the relevance of the use of VA as consultants for male sexual dysfunction. In other words, I find that the question of recognition of the questions, though important and prior to the answers, should not be the main point of your study. But this is a personal opinion and I do not suggest that you modify that point.

>>> Thank you for this valuable opinion.

 

 

Author Response File: Author Response.docx

Reviewer 2 Report

Thank you for this seminal work. Several suggestions:

Introduction

This has too much detail for the average reader because he/she knows what a VA is.

 

Materials - 

Line 7 "quite" should be "quiet."

You have put the entire survey in the appendix, thank you, but give an example in the text, perhaps the first question, for each domain of inquiry. 

The scale is not a "qualitative scale," that's an oxymoron. Use "analog" or "semi-quantitative" or "semi-objective" or some such term as preferable. 

Mean of 2 raters scores: for the value used to be meaningful, you need to tell us how much variance there was between the 2 raters.

Results

The detailed table is too long to be easily intelligible. Isn't there some graphic that could convey this information more intelligibly and concisely than running a table over 2 pages with color used to convey quality. A "star" chart or "tree plot" or something?

Discussion

Add something about how relatively poor the info from the VA was on PE compared to ED and how this is the more important because PE is the most frequent form of male sexual dysfunction (30%!).

Add that the VA should have a built-in "I don't know enough to help you; go see your doctor" kind of answer for medical questions.

Conclusions

Main conclusion should be that such VA need improvement to reach a middling level of quality, and that is not stated. 

Author Response

Dear Reviewer,

First of all, we wish to thank you for reviewing our paper entitled “Voice Assistants As Consultants For Male Patients With Sexual Dysfunction: A Reliable Option?”. We realize that your time is valuable, and we are very grateful for your efforts.

We have carefully read all your comments and we have tried to clarify all the concerns expressed, improving the quality of the paper according to your suggestions. For your convenience, in addition to the revised manuscript, we report below our responses to your comments and the changes made in the manuscript.

 

Reviewer 2

Thank you for this seminal work. Several suggestions:

>>> Thank you for this appreciation.

 

Introduction

This has too much detail for the average reader because he/she knows what a VA is.

>>> Thank you for this comment. We shortened and simplified the Introduction.

 

Materials -

Line 7 "quite" should be "quiet."

>>> Thank you for this suggestion. We corrected the sentence.

 

You have put the entire survey in the appendix, thank you, but give an example in the text, perhaps the first question, for each domain of inquiry.

>>> Thank you for this comment. Since all the questions are already reported in the supplementary materials and the list of all the conditions are already mentioned in the Methods section (easily available), we respectfully believe that giving examples (e.g., first question for each condition) would not add scientific value to our paper, unjustifiably lengthening the text.

 

The scale is not a "qualitative scale," that's an oxymoron. Use "analog" or "semi-quantitative" or "semi-objective" or some such term as preferable.

>>> Thank you for this suggestion. The term qualitative referred to the answers not to the scale. However, we removed the term “qualitative scale” to avoid confusion. We wrote: “the response characteristics (domains) were rated on a scale from 0 to 10 (according to the quality)”.

 

Mean of 2 raters scores: for the value used to be meaningful, you need to tell us how much variance there was between the 2 raters.

>>> Thank you for this comment. Sorry we weren't clear enough. The maximum difference in score tolerated for each evaluation between the two authors was 3 points. For larger differences, a third senior author assigned the final score. However, this occurred in a minority of answers (<5%). We added this detail in the Methods section.

 

Results

The detailed table is too long to be easily intelligible. Isn't there some graphic that could convey this information more intelligibly and concisely than running a table over 2 pages with color used to convey quality. A "star" chart or "tree plot" or something?

>>> Thank you for this comment. We think that Figure 1 and the text in the paragraph 3.2 are sufficient to clearly describe the quality and characteristics of the answers. So, we kept these two elements and moved the large Table 2 to the Supplementary Materials (Table S2).

 

Discussion

Add something about how relatively poor the info from the VA was on PE compared to ED and how this is the more important because PE is the most frequent form of male sexual dysfunction (30%!).

>>> Thank you for this comment. We added this relevant point in the Discussion section.

 

Add that the VA should have a built-in "I don't know enough to help you; go see your doctor" kind of answer for medical questions.

>>> Thank you for this suggestion. We added this this reasonable proposal in the Future perspectives section.

 

Conclusions

Main conclusion should be that such VA need improvement to reach a middling level of quality, and that is not stated.

>>> Thank you for this suggestion. We included this sentence in the Conclusion section: “The potential of VAs is huge, but a significant improvement in their technology is still needed to make them a reliable tool to search information on sexual health”

Author Response File: Author Response.docx