Can ChatGPT Guide Parents on Tympanostomy Tube Insertion?

Background: The emergence of ChatGPT, a state-of-the-art language model developed by OpenAI, has introduced a novel avenue for patients to seek medically related information. This technology holds significant promise in terms of accessibility and convenience. However, the use of ChatGPT as a source of accurate information enhancing patient education and engagement requires careful consideration. The objective of this study was to assess the accuracy and reliability of ChatGPT in providing information on the indications and management of complications post-tympanostomy, the most common pediatric procedure in otolaryngology. Methods: We prompted ChatGPT-3.5 with questions and compared its generated responses with the recommendations provided by the latest American Academy of Otolaryngology–Head and Neck Surgery Foundation (AAO-HNSF) “Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)”. Results: A total of 23 responses were generated by ChatGPT against the AAO-HNSF guidelines. Following a thorough review, it was determined that 22/23 (95.7%) responses exhibited a high level of reliability and accuracy, closely aligning with the gold standard. Conclusion: Our research study indicates that ChatGPT may be of assistance to parents in search of information regarding tympanostomy tube insertion and its clinical implications.


Introduction
Chat GPT is a conversational artificial intelligence (AI) tool designed to be a resource for individuals seeking information on a wide range of topics, including medical issues.One of the tool's fundamental strengths is its ability to elucidate complex medical terminology and describe complex physiology in a succinct manner [1][2][3][4].However, concerns pertaining to the accuracy and reliability of the instruction provided by ChatGPT, specifically in matters relating to healthcare, have been raised [5][6][7].As several parents of children undergoing myringotomy tube insertion in our practice have recently admitted using ChatGPT, the goal of our study was to evaluate the accuracy and efficacy of ChatGPT as a resource for patients seeking information on the most performed otolaryngology surgery: tympanostomy tube insertion.
We hypothesized that ChatGPT is capable of providing comprehensive answers to questions commonly asked by parents about tympanostomy tube insertion.In order to assess the performance of the application, we evaluated the responses generated by ChatGPT in comparison with the highest standards of healthcare practice.
The findings of this study can contribute significantly to the ongoing efforts to leverage AI in patient-centered care while also ensuring the provision of high-quality medical Children 2023, 10, 1634 2 of 6 information to patients and caregivers.We hope to promote its responsible use as a valuable resource guiding parents to make informed decisions about the care of their child.
It is more important than ever to explore the incorporation of supplementary resources such as AI to enhance the effectiveness of patient-provider communication and the quality of services delivered in the healthcare system.

Materials and Methods
A list of predetermined questions regarding surgical indications and complications post-tympanostomy insertion was presented to ChatGPT by two users on separate devices using the ChatGPT-3.5version, which is free of charge and readily available to the majority of parents.To prevent any biases from the prior question, a new ChatGPT session was started for each prompt.Responses collected by the two users were recorded in our database to be compared with the statements published in the latest "American Academy of Otolaryngology-Head and Neck Surgery Foundation Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)" [8].Two independent otolaryngologists evaluated if the answer provided by the chatbot was aligned with the latest guidelines, cited the verbatim guidelines, and explicitly referenced the AAO-HNSF, as well as communicated the importance of seeking a clinician's input.The inter-rater reliability was assessed using Cohen's Kappa test.To confirm the consensus on all 4 outputs, the responses of both parties were reviewed by the senior author (SJD).A visual representation of the workflow is depicted in Figure 1.
assess the performance of the application, we evaluated the responses generated by ChatGPT in comparison with the highest standards of healthcare practice.
The findings of this study can contribute significantly to the ongoing efforts to leverage AI in patient-centered care while also ensuring the provision of high-quality medical information to patients and caregivers.We hope to promote its responsible use as a valuable resource guiding parents to make informed decisions about the care of their child.
It is more important than ever to explore the incorporation of supplementary resources such as AI to enhance the effectiveness of patient-provider communication and the quality of services delivered in the healthcare system.

Materials and Methods
A list of predetermined questions regarding surgical indications and complications post-tympanostomy insertion was presented to ChatGPT by two users on separate devices using the ChatGPT-3.5version, which is free of charge and readily available to the majority of parents.To prevent any biases from the prior question, a new ChatGPT session was started for each prompt.Responses collected by the two users were recorded in our database to be compared with the statements published in the latest "American Academy of Otolaryngology-Head and Neck Surgery Foundation Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)" [8].Two independent otolaryngologists evaluated if the answer provided by the chatbot was aligned with the latest guidelines, cited the verbatim guidelines, and explicitly referenced the AAO-HNSF, as well as communicated the importance of seeking a clinician's input.The inter-rater reliability was assessed using Cohen's Kappa test.To confirm the consensus on all 4 outputs, the responses of both parties were reviewed by the senior author (SJD).A visual representation of the workflow is depicted in Figure 1.

Results
A total of 23 responses generated by ChatGPT were assessed by two otolaryngologists from distinct clinical settings (Table 1).There was perfect agreement between the two judges (Cohen's K = 100%) that ChatGPT provided comprehensive recommendations that adequately matched the information highlighted by the guidelines.

Results
A total of 23 responses generated by ChatGPT were assessed by two otolaryngologists from distinct clinical settings (Table 1).There was perfect agreement between the two judges (Cohen's K = 100%) that ChatGPT provided comprehensive recommendations that adequately matched the information highlighted by the guidelines.Both raters agreed that 22/23 responses (95.7%) presented high-quality information closely aligning with the guidelines.Recommendations of one response, pertaining to the need of tympanostomy tube insertion in children with recurrent acute otitis media (AOM) who have unilateral or bilateral middle ear effusion (MEE), differed from the guidelines.The AI tool was not committal as to the treatment provided, stating: "Decision for tympanostomy tube insertion in these cases may depend on factors such as the severity and duration of effusion, impact on hearing, associated symptoms, and overall clinical picture", whereas the guidelines stipulated that children should be offered bilateral tympanostomy tubes.
On 2/23 occasions, ChatGPT cited the verbatim AAO-HNSF guideline in its response.Regarding the action point concerning acute tympanostomy tube otorrhea, the question asked on ChatGPT, "Should clinicians prescribe topical antibiotic ear drops for children with uncomplicated acute tympanostomy tube otorrhea?",cited verbatim the guideline.According to the guidelines from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS), clinicians should consider topical antibiotic ear drops for children with uncomplicated acute tympanostomy tube otorrhea.On a second occasion, regarding the action point concerning water precautions, the question asked on ChatGPT, "Should clinicians encourage routine prophylactic water precautions (earplugs or headbands or avoidance of swimming) for children with tympanostomy tubes?", cited verbatim the guideline.According to the guidelines from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS), routine prophylactic water precautions, such as the use of earplugs or headbands or avoidance of swimming, are generally not recommended for children with tympanostomy tubes.
In total, 3/23 of ChatGPT's responses referenced the AAOHNS guidelines explicitly.ChatGPT's responses made at least one direct reference to consult with a specialist or healthcare provider when considering all available treatment options and inherent risk factors.These references include the following key examples: "It's important to have open and thorough discussions with your child's healthcare provider, usually an ear, nose, and throat specialist (otolaryngologist), to determine the most appropriate course of action for your child".Another frequently encountered recommendation goes as follows: "It is important for parents to have open communication with the child's healthcare provider, discuss concerns, and ask questions to ensure a comprehensive understanding of the available options and make informed decisions about their child's care".
The evaluators found that the information provided by ChatGPT exceeded the depth of content found in the guidelines and was easily understandable and tailored to meet the needs of parents who may not be familiar with medical terminology.
Institutional Review Board Statement: Ethical review and approval were waived for this study by the McGill University Faculty of Medicine and Health Sciences Research Ethics Office on 18 April 2023, in accordance with the national legislation and the institutional requirements.Not applicable.