Computerized Decision Support for Bladder Cancer Treatment Response Assessment in CT Urography: Effect on Diagnostic Accuracy in Multi-Institution Multi-Specialty Study

Round 1

Reviewer 1 Report

The manuscript describes a study evaluating a computerized AI and image-based decision support system for evaluating response to therapy in bladder cancer. This topic holds high clinical significance to help reduce over and under treatment, and to support surgical de-escalation strategies. The decision support tool developed presents a nice example of incorporating objective quantitative imaging into clinical assessments using a practical and simple interface. Authors performed a large reader study including a high number of physician readers, varying by institution, medical specialty, and years of training. Each reader reviewed and assessed response in 157 bladder cancer lesions in 123 patients, with and without the CDSS-T aid. The manuscript is well-written and the presented findings show promising value of the CDSS-T system to improve reproducibility and accuracy of pCR response assessments. However, there are several questions and clarifications to address that would improve the manuscript:

1. Introduction – suggest rewording last sentence of first paragraph to something like ‘It is of great importance to evaluate the response of bladder lesions to chemotherapy treatment to spare the patient the toxicities of further unnecessary chemotherapy or to support surgery de-escalation’. Also sentence ‘Patients with complete response may be considered for…’ remove the word ‘otherwise’.
2. Methods – Please add a table with patient and lesion characteristics to describe the cohort (e.g. age, cancer grade, size, subtype, pathologic response, etc)
3. Methods – please provide more information on ROIs or segmentations used as input for the deep learning and radiomics analyses leading to the CAD score. (By whom, approach, methods used for lesion localization and segmentation). Were they regenerated by each physician, or the same predetermined CAD score used for all reads? An illustration would be helpful.  
4. Methods – how was SD = 25 selected as cutoff to define easy vs difficult cases? Suggest providing some clarification to justify this. In results Section 3.2, please provide resulting n clearly for each group, as well as number of pCRs in each group.
5. Methods – statistics – it is not clear what metric was used from observer reads for calculation of ROC curves (e.g. percentage response, response category, likelihood of pCR, etc). This should be described for all AUCs reported in the results.
6. Results – AUC for CDSS-T numeric CAD score alone should be stated clearly in text at the beginning of results for context (currently given only in tables but not explained).
7. Results Section 3.1 – statement ‘all observers except #6 improved their AUCs with CDSS-T’ is not clear as most do not have significant p-values in Table 2 comparisons. Presumably authors refer to the AUC curve values increasing slightly, but this should be stated more clearly and did not reach significance. Also ‘Observer #9 with CDSS-T aid performed better than CDSS-T alone’ needs more explanation as CDSS-T alone was not reported at this point in results and no AUCs or p-values are given for context to support this statement.
8. Results – Table 3, suggest remove separate row corresponding to ‘8 radiologists from UM’ or better justify the need for analyzing this subgroup separately (vs. 9 total radiologist group) 
9. Results Section 3.6 - Intra-reader variability. While authors present interesting findings of intr-reader reproducibility based on overall reader AUCs, it would be preferable to see impact on actual individual lesion response assessments (using Bland-Altman, wCV or other comparisons as appropriate). In this way, inter-reader variability could also be characterized at the lesion level (n=157 lesions, 17 readers) and presented here, both with and without CDSS-T.
10. Discussion – authors are encouraged to add some discussion on the reader experience using the software tool, and potential challenges and areas to improve there in order to incorporate it into routine clinical workflow or near-term clinical trials. For example, who would generate CAD scores in clinical implementation, how long would it take, would it require a centralized analysis or onsite, etc. Some thoughts regarding downstream integration would be interesting.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper deals with the effect of an AI-based decision support system on the accuracy of bladder cancer diagnosis.and treatment response. 

In general, the paper is well written and informative. Due to the MANY abbreviations, it is sometimes hard to read and the reader has to go back and forth to remember what such an abbreviation stands for. The Result section is very detailled, and the many details sometimes make it difficult to keep the overview. Some surprising effects (at least for me) are not discussed, for example the fact that the influence of easy/difficult setups is completely "inverse" for oncologists.

The work obviously is doing a retrospective analysis of data/diagnosis (at least that is my impression). Of course it would be also interesting to see the results of a prospective use of the project results. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf