Skip to Content
You are currently on the new version of our website. Access the old version .
  • Comment
  • Open Access

6 August 2026

2 Pages

Beyond the Single Prompt: Rethinking How We Study AI Chatbots as Patient Information Sources in Urological Oncology. Comment on Oberoi et al. Artificial Intelligence Chatbots as Patient Information Sources in Penile Cancer: A Multi-Platform Evaluation. Soc. Int. Urol. J. 2026, 7, 47

,
and
1
Surgical Outcomes Research Centre (SOuRCe), Royal Prince Alfred Hospital, Sydney 2050, Australia
2
Department of Urology, Royal Prince Alfred Hospital, Sydney 2050, Australia
*
Author to whom correspondence should be addressed.
Oberoi and colleagues have produced a timely study that adds penile cancer to the growing catalogue of urological conditions against which artificial intelligence (AI) chatbot performance has been benchmarked [1]. Their findings are consistent with the existing literature: responses are broadly understandable, information quality is comparable across platforms, readability consistently exceeds recommended thresholds, and actionability is suboptimal on most platforms [2,3,4]. The identification of bleomycin recommendations by two platforms, in direct contradiction to a strong European Association of Urology-American Society of Clinical Oncology (EAU-ASCO) guideline, introduces a patient-safety dimension that elevates this article beyond the current standard.
Clinicians will recognise a shift already underway in consulting rooms. Patients no longer arrive having run their symptoms through a search engine; instead, they arrive having had a conversation with an AI. The “Dr Google” phenomenon that shaped a generation of patient counselling has given way to something qualitatively different: a personalised and authoritative conversation that patients engage with before, during, and after clinical contact. This shift demands a corresponding evolution in how researchers study these tools.
We do not take issue with the methodology. Rather, we use this publication as an occasion to ask whether the questions under study are still the most important ones.
The dominant methodology in the existing literature involves submitting standardised questions to each platform, scoring responses against validated instruments, and comparing platforms statistically. This approach is reproducible and quantitative, but it rests on the assumption that a patient formulates one discrete question, reads the response, and stops. This is not how patients use these tools. Interactions are conversational and directed. Furthermore, the conversation is further influenced by the AI tool’s memory/history with the user.
A patient with a new penile cancer diagnosis does not ask a single question. He asks many, each shaped by what the previous response contained and the totality of other conversations. Whether this form of communication would amplify or temper a critical safety error like the bleomycin recommendation is a question single-prompt methodology cannot answer.
Studying multi-turn interactions is methodologically challenging but achievable. Designing a fixed set of clinical scenarios with predetermined follow-up questions would allow consistent testing across platforms while still reflecting how real conversations unfold. Red-teaming protocols, in which chatbots are subjected to escalating or emotionally pressured follow-up queries, offer a complementary adversarial approach and are directly transferable from general health chatbot evaluation. One such protocol found that error rates on advice queries spiked to 50% under multi-turn stress conditions compared to zero on single-turn assessment of the same system, demonstrating that single-prompt methodology does not merely simplify evaluation but actively conceals clinically meaningful failure modes [5].
Importantly, the field should shift from evaluating information to studying the patient experience of receiving it. Patient-centred research would open clinically urgent questions that remain unstudied. How patients actually use these tools remains unstudied: as preparation before a consultation, reinforcement of what was discussed in clinic, or a substitute for clinical contact altogether? The last of these carries the greatest clinical stakes. The threshold at which a patient decides to ask an AI rather than contact their surgical team is shaped by perceived accessibility, anxiety about burdening staff, and the ready availability of AI relative to a busy specialist clinic. What drives that decision and which features of AI responses either reinforce or erode the impulse to seek timely clinical review have direct implications for service design and patient counselling.
The finding that AI responses consistently exceed recommended reading thresholds is now an established fact. The field does not need another study confirming responses are written at grade 10 rather than grade 8. It needs evidence on whether this affects patient comprehension and behaviour. The next generation of research should move from asking what information AI provides to how patients seek, receive, and act on it. That reorientation is where the clinical value of this literature will ultimately be determined.

Author Contributions

Conceptualization, D.G. and N.J.; methodology, N.J. and N.K.; writing—original draft preparation, D.G.; writing—review and editing, N.J. and N.K.; supervision, N.J.; project administration, N.K. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Oberoi, K.; Hassan, S.; Woon, D.; Sethi, K. Artificial Intelligence Chatbots as Patient Information Sources in Penile Cancer: A Multi-Platform Evaluation. Soc. Int. Urol. J. 2026, 7, 47. [Google Scholar] [CrossRef] [Scilit]
  2. Gibson, D.; Jackson, S.; Shanmugasundaram, R.; Seth, I.; Siu, A.; Ahmadi, N.; Kam, J.; Mehan, N.; Thanigasalam, R.; Jeffery, N.; et al. Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: Multimetric assessment. J. Med. Internet Res. 2024, 26, e55939. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Musheyev, D.; Pan, A.; Loeb, S.; Kabarriti, A.E. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Eur. Urol. 2024, 85, 13–16. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  4. Santucci, J.; Stapleton, P.; Ibrahim, J.; Johns-Putra, L.; Elmer, S.; Sathianathen, N. Quality of patient information on interstitial cystitis from artificial intelligence chatbots. BJU Int. 2026, 138, S57–S63. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  5. Hussain, S.-A.; Jackson, D.I.; Lewis, A.; Fosler-Lussier, E.; Sezgin, E. Toward trustworthy chatbots: A protocol for red teaming for health related conversations. Sci. Rep. 2026, 16, 15550. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.