Review Reports - Quantib Prostate Compared to an Expert Radiologist for the Diagnosis of Prostate Cancer on mpMRI: A Single-Center Preliminary Study

Round 1

Reviewer 1 Report

General comments:

This paper is a revised version of a manuscript previously submitted to Diagnostics that I have reviewed. It discusses the employment of AI in assisting the radiologist with a more accurate diagnosis of prostate cancer and better patient stratification for treatment.

The paper has scientifically improved, and the authors have considered most comments raised in the previous report.

However, the article would benefit from an English language revision – see some corrections in Special comments, but the list is ampler than this; thus, I recommend the authors to invite a native speaker to revise the article.

Also, please be consistent with the referencing style. The authors have used three different referencing styles in the paper: [^{ref nr}], [ref nr] and ^{ref nr}.

Specific comments:

Line 47 – “Recent studies have shown…” or “A recent study has shown….”

Line 53 - …clinically significant…’

Line 59 – replace 2,6% with 2.6%

Line 80 – …while the remaining….

Line 83 – …as showed in Table 1.

Line 84 – Table 1 caption should read “Patient classification as a function of biopsy and radiology results”

Line 103 – obtained from each patient

Lines 118-119 – reword the sentence so it reads: “where scores 1 and 2 correspond to benign tissue, score 3 shows inconclusive results and scores 4 and 5 suggest malignant tissue”.

Line 181 – define PPV

Line 190 – …Table 2 and Table 3, respectively.

Lines 209-213: The authors state that “Evaluating group A, the expert radiologist reached a sensitivity of 71.7 and a PPV of 211 84.4%, while the AI-assisted-radiologist reached a sensitivity of 92.3% and a PPV of 90.1% 212 (Figure 2).” This result does not show in Figure 2 as the figure illustrates the combined (group A + group B) results. Please clarify this aspect in the text.

Line 282 – ‘higher and greater’ have the same meaning here; please remove one of them.

Line 288 – remove ‘completely’ (…which were identified as negative…)

Line 306 – “..so, to calculate the sensitivity and PPV…”

Line 325 – what do you mean by “and possibly specify to specific use-cases”? – please reword for clarity

Line 334 – “The role of mp-MRI in the management of prostate…”.

Author Response

Reviewer 1 reply

General comments:

The paper has scientifically improved, and the authors have considered most comments raised in the previous report.

Thanks to the reviewer for the comments. The whole manuscript has been reviewed by a native English author (LH).

Also, please be consistent with the referencing style. The authors have used three different referencing styles in the paper: [^{ref nr}], [ref nr] and ^{ref nr}.

Thanks for the comment, we modified the references according to journal guidelines.

Specific comments:

Line 47 – “Recent studies have shown…” or “A recent study has shown….”

Line 53 - …clinically significant…’

Line 59 – replace 2,6% with 2.6%

Line 80 – …while the remaining….

Line 83 – …as showed in Table 1.

Line 84 – Table 1 caption should read “Patient classification as a function of biopsy and radiology results”

Line 103 – obtained from each patient

Lines 118-119 – reword the sentence so it reads: “where scores 1 and 2 correspond to benign tissue, score 3 shows inconclusive results and scores 4 and 5 suggest malignant tissue”.

Line 181 – define PPV

Line 190 – …Table 2 and Table 3, respectively.

Line 282 – ‘higher and greater’ have the same meaning here; please remove one of them.

Line 288 – remove ‘completely’ (…which were identified as negative…)

Line 306 – “..so, to calculate the sensitivity and PPV…”

Line 325 – what do you mean by “and possibly specify to specific use-cases”? – please reword for clarity

The sentence was eliminated.

Line 334 – “The role of mp-MRI in the management of prostate…”.

All the comments has been modified/clarified following the reviewer indications.

Reviewer 2 Report

This study used an AI software to detect prostate cancer, comparing an inexperienced rater ussing the software with an experienced rater without the software. The AI software greatly outperformed the experienced rater in all outcomes. The presentation of the article could be greatly improved, especially the figures. The methods need to be explained in greater detail.

Extensive edit for grammar is needed throughout. I have not commented further on this below.
Patients only underwent biopsy if the mp-MRI was positive. In that case, how can the authors consider this as a gold standard if it is not possible to tell whether the patients who had negative MRI would have positive biopsies? In other words, false-positives were measured but false negatives were not. Thus group C could potentially be a mixture of true and false negatives.
“A written informed consensus was obtained by each patient who performed the mp- 103 MRI.” should read “consent” not “consensus”.
Table 1 would be better presented as a flowchart diagram than a single-row table.
Figure 1 should either be split into multiple Figures, or the most important parts selected-out and shown together in a single panel, instead of split across 3 different parts. Figure 1c seems to show a copy-pasted screenshot of the anatomy, which has not been properly cited.
Was the expert radiologist also blinded to diagnosis? If not, it is not meaningful to compare the experienced and inexperienced radiologists’ performance.
Methods: please give more detail on the specific statistical tests used for each comparison.
It would be useful to define each group A,B,C in terms of false/true-positive/negative (line 179).
Line 179: how were true negatives defined? Is the true negative group required for calculation of PPV, sensitivity and specificity?
Can the authors provide any post-hoc analysis of the falsely-identified cases and the falsely-missed cases to understand why their classification was wrong?

Author Response

Reviewer 2 reply

Comments and Suggestions for Authors

Extensive edit for grammar is needed throughout. I have not commented further on this below.
Thanks to the reviewer for the comment. The whole manuscript has been reviewed by a native English author (LH).
Patients only underwent biopsy if the mp-MRI was positive. In that case, how can the authors consider this as a gold standard if it is not possible to tell whether the patients who had negative MRI would have positive biopsies? In other words, false-positives were measured but false negatives were not. Thus group C could potentially be a mixture of true and false negatives.
- Thanks to the reviewer for this comment which allow us to be clearer. “false negative” was added in line 317, as well as the wording of gold standard was changed to ground truth set for this study in line 70. This should clarify that this clinical implementation study has indeed it’s limitations, but does show the possible clinical value of using the AI software.
“A written informed consensus was obtained by each patient who performed the mp- 103 MRI.” should read “consent” not “consensus”.
Table 1 would be better presented as a flowchart diagram than a single-row table.
- We did not perform any exclusion process; so we thought that the single-row table was easier to be understood
Figure 1 should either be split into multiple Figures, or the most important parts selected-out and shown together in a single panel, instead of split across 3 different parts. Figure 1c seems to show a copy-pasted screenshot of the anatomy, which has not been properly cited.

Thanks for this suggestion. We split the figure and modifies Fig1c caption for a better explanation

Was the expert radiologist also blinded to diagnosis? If not, it is not meaningful to compare the experienced and inexperienced radiologists’ performance.

Yes he was. We added a sentence to line 161 to address this

Methods: please give more detail on the specific statistical tests used for each comparison.
- Thanks to the reviewer for this comment. The following sentience has been added as last paragraph of M&M: “One-sided student t-test was used to compare age, PSA levels, Prostate volume, lesion volume and lesion axial dimension between the three groups of patients. The statistical significance was set at 0.05.”
It would be useful to define each group A,B,C in terms of false/true-positive/negative (line 179).

Thanks to this comment. We followed this suggestion to be clearer

Line 179: how were true negatives defined? Is the true negative group required for calculation of PPV, sensitivity and specificity?
- Following this suggestion, we defined sensitivity and PPV in M&M section (“The statistical analysis for the prediction of sensitivity and positive predictive value (PPV) was performed by SPSS v. 22 using traditional computational analysis. We calculated sensitivityas number of true positives/(number of true positives+number of false positives) and PPV as number of true negatives/(number of true negatives+number of false positives), to calculate PPV”). Even more we specified that “for this study, specificity and negative predictive value could not be assessed” in last paragraph of the discussion.
Can the authors provide any post-hoc analysis of the falsely-identified cases and the falsely-missed cases to understand why their classification was wrong?
- Thanks to the reviewer for this comment. A post-hoc analysis is currently not possible due to the retrospective nature of the study. This is a re-analysis of a years worth of data and clinical practice was not set-up specifically for the study, as not all patients are followed-up. In particular we did not have any future information of some patients. However this is a good point for future studies. We commented it in the last part of the discussion to be more specific about this topic.

Reviewer 3 Report

Section 2.2, last paragraph format issue.

Section 2.3, Incorrect labelling on citation "Scores <= ... 9 and 9-10 [12]"

Discussion, "For the group of patients for whom no biopsies..." Missing paragraph indentation

Author Response

Reviewer 3 reply

Comments and Suggestions for Authors

Section 2.2, last paragraph format issue.

Thanks to the reviewer for this suggestion. The paragraph was re-written as following: “A written informed consent was obtained from each patient who performed the mp-MRI.”

Section 2.3, Incorrect labelling on citation "Scores <= ... 9 and 9-10 [12]"

Thanks for this comment. All the reference style was checked and modified according to journal reference style.

Discussion, "For the group of patients for whom no biopsies..." Missing paragraph indentation

The indentation was added as suggested

Round 2

Reviewer 2 Report

The authors have addressed most of my concerns including the major ones.

However, Fig 1 looks low quality and the source is not cited. Please re-draw the Figure from scratch, or cite the original source.

Author Response

Thanks to the reviewer for the comment.
The software automatically reports the localization of the indicated ROI on prostate anatomy image.
The Fig1c is a screenshot of the software's interface. The image is exactly as represented in the software and automated report
The figure quality was improved as suggested.

Author Response File: Author Response.docx