Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant
Abstract
1. Introduction
2. Materials and Methods
3. Results
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Fox, S.; Duggan, M. Health Online 2013. Pew Research Center 2013. Available online: https://www.pewresearch.org/internet/2013/01/15/health-online-2013/ (accessed on 22 July 2024).
- Land, J.M.; Pavlik, E.J.; Ueland, E.; Ueland, S.; Per, N.; Quick, K.; Gorski, J.W.; Riggs, M.J.; Hutchcraft, M.L.; Llanora, J.D.; et al. Evaluation of Replies to Voice Queries in Gynecologic Oncology by Virtual Assistants Siri, Alexa, Google, and Cortana. BioMedInformatics 2023, 3, 553–562. [Google Scholar] [CrossRef]
- Brandl, R.; Ellis, C. Tooltester. ChatGPT Statistics 2024—All the Latest Statistics about OpenAI’s Chatbot. Available online: https://www.tooltester.com/en/blog/chatgpt-statistics/ (accessed on 22 July 2024).
- Rao, A.; Pang, M.; Kim, J.; Kaminen, M.; Lie, W.; Prasad, A.K.; Landman, A.; Dreyer, K.; Succi, M.D. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. J. Med. Internet Res. 2023, 25, e48659. [Google Scholar] [CrossRef] [PubMed]
- Kan, M. ChatGPT’s Knowledge Base Finally Extends Beyond 2021. PC 11/06/2023. Available online: https://www.pcmag.com/news/chatgpts-knowledge-base-finally-extends-beyond-2021 (accessed on 22 July 2024).
- Ortiz, S. ZDNET What Is Google Bard? Here’s Everything You Need to Know. 9 February 2024. Available online: https://www.zdnet.com/article/what-is-google-bard-heres-everything-you-need-to-know/ (accessed on 22 July 2024).
- Microsoft Copilot. Wikipedia. Available online: https://en.wikipedia.org/wiki/Microsoft_Copilot (accessed on 22 July 2024).
- UpToDate. Wikipedia. Available online: https://en.wikipedia.org/wiki/UpToDate (accessed on 22 July 2024).
- Gemini Apps Privacy Notice. Your Data and Gemini Apps. Last Updated: 8 February 2024. Available online: https://support.google.com/gemini/answer/13594961?visit_id=638450180634226006-3120607282&p=privacy_help&rd=1&collected_data#your_data (accessed on 22 July 2024).
- What Happens to My Data When I Use Copilot? Available online: https://learn.microsoft.com/en-us/power-platform/faqs-copilot-data-security-privacy (accessed on 22 July 2024).
- Chen, S.; Kann, B.H.; Foote, M.B.; Aerts, H.J.W.L.; Savova, G.K.; Mak, R.H.; Bitterman, D.S. Use of Artificial Intelligence Chatbots for Cancer Treatment Information. JAMA Oncol. 2023, 9, 1459–1462. [Google Scholar] [CrossRef] [PubMed]
- Janopaul-Naylor, J.R.; Koo, A.; Qian, D.C.; McCall, N.S.; Liu, Y.; Patel, S.A. Physician Assessment of ChatGPT and Bing Answers to American Cancer Society’s Questions to Ask About Your Cancer. Am. J. Clin. Oncol. 2024, 47, 17–21. [Google Scholar] [CrossRef] [PubMed]
- Shea, Y.F.; Lee, C.M.Y.; Ip, W.C.T.; Luk, D.W.A.; Wong, S.S.W. Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis. JAMA Netw. Open. 2023, 6, e2325000. [Google Scholar] [CrossRef] [PubMed]
- Schubert, M.C.; Wick, W.; Venkataramani, V. Performance of Large Language Models on a Neurology Board-Style Examination. JAMA Netw. Open. 2023, 6, e2346721. [Google Scholar] [CrossRef] [PubMed]
- Hoang, Y.N.; Chen, Y.L.; Ho, D.K.N.; Chiu, W.C.; Cheah, K.J.; Mayasari, N.R.; Chang, J.S. Consistency and Accuracy of Artificial Intelligence for Providing Nutritional Information. JAMA Netw. Open. 2023, 6, e2350367. [Google Scholar] [CrossRef] [PubMed]
- Irwin, K. Adobe’s AI Assistant Can Summarize PDFs, PowerPoints for You. PC Magazine 02. 20 February 2024. Available online: https://www.pcmag.com/news/adobes-ai-assistant-can-summarize-pdfs-powerpoints-for-you?utm_source=email&utm_campaign=whatsnewnow&zdee=gAAAAABlzjXaj__BrV5eo-f2qF7sDrWWKEtkmH2G19SKAH7kdCyO2QZWMpWIrsyK4J9MUmTylA0dKjizA-4gORPeq8yq9vplTV5ka-cM6LR2CdQFveA56U4%3D (accessed on 22 July 2024).
- Koller, D.; Beam, A.; Manrai, A.; Ashley, E.; Liu, X.; Gichoya, J.; Holmes, C.; Zou, J.; Dagan, N.; Wong, T.Y.; et al. Why We Support and Encourage the Use of Large Language Models in NEJM AI Submissions. NEJM AI 2023, 1. [Google Scholar] [CrossRef]
- Cecco, L. Air Canada Ordered to Pay Customer Who Was Misled by Airline’s Chatbot. The Guardian. 21 February 2024. Available online: https://www.theguardian.com/world/2024/feb/16/air-canada-chatbot-lawsuit (accessed on 22 July 2024).
- Miao, H. Alphabet Stock Drops 8% after Google Rollout of AI Search Features. The Wall Street Journal. Available online: https://www.wsj.com/livecoverage/stock-market-news-today-02-08-2023/card/alphabet-stock-drops-after-google-parent-introduces-ai-search-features-wgCJG3IDoSbfL3SgyrNI (accessed on 22 July 2024).
- Mihalcik, C. Google ChatGPT Rival Bard Flubs Fact About NASA’s Webb Space Telescope. Available online: https://www.cnet.com/science/space/googles-chatgpt-rival-bard-called-out-for-nasa-webb-space-telescope-error/ (accessed on 22 July 2024).
- Hao, K. What Is ChatGPT? What to Know About the AI Chatbot That Will Power Microsoft Bing. The Wall Street Journal. 16 May 2023. Available online: https://www.wsj.com/articles/chatgpt-ai-chatbot-app-explained-11675865177?st=q4wbp2 (accessed on 22 July 2024).
- Quach, K. Microsoft’s AI Bing Also Factually Wrong, Fabricated Text During Launch Demo. The Register. Available online: https://www.theregister.com/2023/02/14/microsoft_ai_bing_error/ (accessed on 22 July 2023).
Query # | Question | Correct Answer Link |
---|---|---|
1. | What is stage I ovarian cancer? [308.6 ± 15.6 (247,330)] | Answer |
2. | What is stage II ovarian cancer? [349.6 ± 15.5 (310,394)] | Answer |
3. | What is stage III ovarian cancer? [344.6 ± 20 (307,421)] | Answer |
4. | What is stage IV ovarian cancer? [338.6 ± 27.7 (252,425)] | Answer |
5. | What is stage IC1 ovarian cancer? [309.6 ± 22.1 (223,347)] | Answer |
6. | What is stage IIIA1 ovarian cancer? [355 ± 16.2 (325,414)] | Answer |
7. | What is stage IVB ovarian cancer? [371.8 ± 25.1 (303,436)] | Answer |
8. | What are the subtypes of epithelial ovarian cancer? [296.8 ± 15.7 (365,350)] | Answer |
9. | What is screening for ovarian cancer? [324 ± 23.2 (264,384)] | Answer |
10. | What are the screening recommendations for ovarian cancer? [318.4 ± 15.3 (284,359)] | Answer |
11. | What are ways to prevent ovarian cancer? [387.8 ± 23 (318,439)] | Answer |
12. | What are the symptoms of ovarian cancer? [313.8 ± 22.9(237,365)] | Answer |
13. | What is hereditary ovarian cancer? [320 ± 24.9 (256,386)] | Answer |
14. | What is ovarian cancer risk reduction? [372.2 ± 43.1 (252,518)] | Answer |
15. | What is screening for cervical cancer? [335.4 ± 38.2 (236,423)] | Answer |
16. | What are the screening recommendations for cervical cancer? [260.8 ± 15.2 (219,350)] | Answer |
17. | What are the options for a 20-year-old woman requesting a Pap smear? [383 ± 23.8 (334,473)] | Answer |
18. | What is the HPV vaccine? [355.8 ± 19.2 (301,412)] | Answer |
19. | What are the ages for HPV vaccination? [229.6 ± 21.4 (186,304)] | Answer |
20. | What are the three dose HPV vaccine recommendations? [258.2 ± 24.8 (197,344)] | Answer |
21. | What are borderline epithelial tumors of the ovary? [349.6 ± 21.9 (294,413)] | Answer |
22. | What is carcinosarcoma of the ovary? [322.6 ± 7.4 (303,344)] | Answer |
23. | What are high-grade serous tumors of the ovary? [311.4 ± 15.5 (277,368)] | Answer |
24. | What is stage IB endometrial cancer? [338.4 ± 13.4 (295,366)] | Answer |
Question: | Google-Bard/Gemini Correct (YES/NO) | MS-Bing-CoPilot Correct (YES/NO) |
---|---|---|
| YES (3.3%) | YES (5.3%) |
| YES (1.4%) | YES (0.4%) |
| YES (8.7%) | YES (14.5%) |
| YES (4.2%) | YES (4.2%) |
| YES (28%) | YES (25.6%) |
| YES (9.1%) | YES (19%) |
| YES (7.1%) | YES (11.6%) |
| YES (8.4%) | YES (27.6%) |
| YES (9.9%) | YES (16%) |
| YES (0.3%) | YES (0.5%) |
| YES (32%) | YES (24.4%) |
| YES (12.8%) | YES (25.9%) |
| YES (4.6%) | YES (8.2%) |
| YES (22%) | YES (28.3%) |
| YES (18.2%) | YES (13.7%) |
| NO | NO |
| YES (9.7%) | YES (10.2%) |
| YES (0.3%) | YES (0.5%) |
| NO | NO |
| NO | NO |
| YES (3.9%) | YES (17.9%) |
| YES (13.5%) | NO |
| YES (17.3%) | YES (10.9%) |
| YES (13.7%) | YES (0.5%) |
Total number of correct responses | 21 | 20 |
Percentage of correct number of questions (% correct queries) | 87.5% (82.5%) | 83.3% (78.3%) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Pavlik, E.J.; Ramaiah, D.D.; Rives, T.A.; Swiecki-Sikora, A.L.; Land, J.M. Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant. BioMedInformatics 2024, 4, 1773-1782. https://doi.org/10.3390/biomedinformatics4030097
Pavlik EJ, Ramaiah DD, Rives TA, Swiecki-Sikora AL, Land JM. Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant. BioMedInformatics. 2024; 4(3):1773-1782. https://doi.org/10.3390/biomedinformatics4030097
Chicago/Turabian StylePavlik, Edward J., Dharani D. Ramaiah, Taylor A. Rives, Allison L. Swiecki-Sikora, and Jamie M. Land. 2024. "Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant" BioMedInformatics 4, no. 3: 1773-1782. https://doi.org/10.3390/biomedinformatics4030097
APA StylePavlik, E. J., Ramaiah, D. D., Rives, T. A., Swiecki-Sikora, A. L., & Land, J. M. (2024). Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant. BioMedInformatics, 4(3), 1773-1782. https://doi.org/10.3390/biomedinformatics4030097