Comparative Assessment of AI Chatbot Responses to Parental Questions on Pediatric Dental Anxiety and Behavior Guidance: Accuracy, Reliability, Readability, and Day-to-Day Consistency

Highlights

What are the main findings?

• Four AI chatbots generally provided accurate responses to parental questions about pediatric dental anxiety and behavior guidance, with high semantic consistency across three consecutive days.
• Significant differences were found in accuracy, reliability, and readability. Source transparency was limited across models, while Claude Sonnet 5.5 produced the most readable responses.

What are the implications of the main findings?

• AI chatbots may support parental education but should not replace individualized guidance from pediatric dentists, particularly regarding sedation and general anesthesia.
• Improving source transparency and using parent-friendly language are essential to making AI-generated dental health information more accessible and verifiable.

Abstract

Background/Objectives: The performance of artificial intelligence (AI) chatbots in providing information to parents about pediatric dental anxiety and behavior guidance remains insufficiently established. This study compared four AI chatbots regarding accuracy, overall quality, reliability, readability, and day-to-day consistency. Methods: Twenty frequently asked parental questions were submitted to ChatGPT-5.6 Luna, Gemini 3.6 Flash, Claude Sonnet 5.5, and Grok 4.5 over three consecutive days, generating 240 responses. Two blinded pediatric dentists independently evaluated accuracy using a five-point Likert scale, overall quality using the Global Quality Scale (GQS), and reliability using modified DISCERN (mDISCERN). Readability was assessed using the Flesch Reading Ease Score (FRES), and day-to-day semantic consistency using Sentence-BERT. Model comparisons were performed using the Friedman test with Holm-adjusted pairwise comparisons. Results: Significant differences were identified among chatbots in accuracy, reliability, and readability (all p < 0.001). Accuracy was generally high across models (median 4.5–5), although Gemini demonstrated significantly lower accuracy than ChatGPT and Grok. Reliability remained limited (median mDISCERN scores: 1.00–3.00/5), particularly regarding source transparency. Claude demonstrated the highest readability (median FRES: 71.90), whereas Grok had the lowest (47.47). No significant differences were observed in overall quality (p = 0.051) or day-to-day semantic consistency (p = 0.602). Conclusions: Although AI chatbots generally provided accurate and semantically consistent responses, limitations in reliability and readability were evident. These findings support their potential use as supplementary parental education tools while emphasizing the importance of source transparency, accessible language, and individualized counseling by pediatric dentists.

1. Introduction

Chatbots based on large language models (LLMs) are rapidly becoming a common means of accessing health information because they can generate natural-language, personalized responses to questions. In Australia, 9.9% of adults used ChatGPT for health questions within six months, and 61% of these users asked questions that required clinical evaluation [1]. Most parents also search the internet and social media for information about their children’s health, yet they do not always share this information with their physician [2,3]. However, a substantial proportion of the references in medical texts generated by chatbots have been reported to be fabricated or inaccurate [4], and users rarely verify the information they receive [5]. Even accurate information is of limited benefit when it is presented above the reading level recommended for patient materials [6].

The accuracy and comprehensibility of online health information are particularly important when parents evaluate treatment options and make decisions about their children’s care. In pediatric dentistry, dental anxiety is one of the main factors affecting these processes. Dental anxiety affects approximately one-quarter of children and adolescents and may lead to treatment avoidance and reduced oral health-related quality of life [7]. Behavior guidance techniques are used to achieve treatment compliance in anxious children. These techniques range from non-pharmacological approaches, such as tell-show-do and positive reinforcement, to nitrous oxide, sedation, and general anesthesia [8,9]. Their success depends largely on parents preparing the child for the visit and, particularly for pharmacological techniques, providing informed consent. Parents may seek information about how to prepare an anxious child for dental visits, how dentists manage uncooperative behavior, and when sedation or general anesthesia may be necessary. In this context, chatbot-generated responses could influence parents’ understanding of available behavior guidance approaches, their expectations regarding dental treatment, and the questions they discuss with pediatric dentists. However, incomplete, difficult-to-understand, or poorly sourced information may create misconceptions, particularly when parents are considering procedures involving sedation or general anesthesia. Therefore, evaluating the accuracy, reliability, and readability of chatbot responses is clinically relevant to determining their potential value and limitations as supplementary sources of parental information [9].

In pediatric dentistry, chatbot responses have been evaluated for general pediatric dentistry questions [10], space maintainers [11], zirconia crowns [12], traumatic dental injuries [13], pediatric dental sedation [14], and guideline adherence [15]. These studies reported marked differences among models in reliability and readability. However, evidence remains limited regarding chatbot responses to parental questions that encompass the broader spectrum of pediatric dental anxiety and behavior guidance, including non-pharmacological techniques, sedation and general anesthesia, and parental involvement in dental care. Responses to the same question have also been shown to vary across days [16]. Nevertheless, the day-to-day semantic consistency of responses to parental questions on these topics remains insufficiently investigated. To the best of our knowledge, no study has evaluated chatbot responses to parental questions on behavior guidance and dental anxiety by simultaneously assessing accuracy, overall quality, reliability, readability, and day-to-day semantic consistency.

This study aimed to compare the responses of ChatGPT (GPT-5.6 Luna; OpenAI), Gemini (3.6 Flash; Google), Claude (Sonnet 5.5; Anthropic), and Grok (4.5; xAI) to frequently asked parental questions on behavior guidance and dental anxiety in terms of accuracy, overall quality, reliability, readability, and consistency across three consecutive days. These four systems were selected based on three practical criteria: availability through free-access interfaces, representation of different AI developers, and suitability for standardized comparisons using identical parental questions. Their inclusion allowed the evaluation of four established conversational AI platforms from different providers rather than multiple models from a single developer. The use of free-access interfaces was intended to reflect a readily accessible source of information for parents, while the inclusion of models from different developers allowed the assessment of variation across the selected systems. The selection was purposive rather than exhaustive and was not intended to represent all available AI chatbots. The null hypothesis was that the responses of the four chatbots would not differ in accuracy, overall quality, reliability, readability, or day-to-day consistency.

2. Materials and Methods

2.1. Study Design

In this cross-sectional comparative study, the responses of four LLM-based chatbots to parental questions on behavior guidance and dental anxiety in pediatric dentistry were evaluated in terms of accuracy, overall quality, reliability, readability, and day-to-day consistency. The study was conducted in accordance with the Declaration of Helsinki. Ethics committee approval was not required because only the outputs of publicly available chatbots were used and no human participants were involved. The study was designed and reported in consideration of the Chatbot Assessment Reporting Tool (CHART) statement developed for chatbot health advice studies [17] and the METRICS checklist for generative artificial intelligence studies [18].

2.2. Development of the Question Set

To identify questions frequently asked by parents about behavior guidance and dental anxiety, a frequently asked questions (FAQ)-based approach used in dentistry [12,19] was followed. Google searches were conducted between 15 and 20 August 2026, using three search terms: “frequently asked questions dental anxiety children behavior management”, “pediatric dentist FAQ sedation nitrous oxide children behavior guidance questions parents”, and “nitrous oxide OR sedation FAQ children dentist parents questions answered”. The first two pages of Google search results for each query were screened for eligibility. Web pages were considered eligible if they contained parent-oriented questions related to pediatric dental anxiety, behavior guidance, sedation, or general anesthesia. A total of 174 questions were compiled from 37 websites identified through the Google searches. Duplicate, non-English, promotional, and irrelevant questions were excluded. Questions addressing similar parental concerns were grouped. Two pediatric dentists (P.C., M.P.) independently evaluated the remaining questions based on their recurrence across different websites and clinical relevance. No numerical frequency threshold was applied. Disagreements regarding question selection were resolved through discussion until consensus was reached, resulting in a final set of 20 questions. The use of multiple search terms and websites was intended to capture a range of commonly presented parental concerns rather than to establish a statistically representative sample of parents’ information needs. Because the questions were obtained from publicly available online sources and were not directly collected from parents, their representativeness could not be independently verified. The questions were grouped into four categories: techniques used by the dentist (5 questions), dental anxiety and fear management (6 questions), sedation and general anesthesia (6 questions), and parental role and communication (3 questions). The questions were prepared in English in lay language without technical terms, consistent with the language used by parents (Table 1).

Table 1. Parental questions submitted to the chatbots and their categories.

Category	No	Question (as Submitted to the Chatbots)
Techniques used by the dentist	Q01	Does the dentist explain and show my child what will happen before a procedure, and does this actually help?
	Q02	What can I do at home to reduce my child’s fear of the dentist?
	Q03	Does rewarding my child for good behavior (stickers, toys, etc.) help them get used to treatment?
	Q04	What distraction methods does the dentist use (video, music, toys, etc.)?
	Q05	Does showing my child a video of another child visiting the dentist before treatment help?
Dental anxiety and fear management	Q06	My child is very afraid of the dentist—how can I make the first visit easier?
	Q07	My child cries and resists during the exam—is this normal?
	Q08	How can I rebuild my child’s trust after a previous negative dental experience?
	Q09	What should and shouldn’t I tell my child before a dental visit?
	Q10	How can I tell that my child is afraid of the dentist (signs such as stomachache, sleeplessness, repeating the same question)?
	Q11	Does the approach the dentist uses change as my child gets older?
Sedation and general anesthesia	Q12	What is laughing gas (nitrous oxide), and is it safe for my child?
	Q13	What is the difference between calming medication (sedation) and full anesthesia (general anesthesia), and when is each preferred?
	Q14	What should I pay attention to if my child will be given calming medication (fasting, aftercare, etc.)?
	Q15	What happens next if my child does not cooperate at all?
	Q16	What are the risks of dental treatment under full anesthesia (general anesthesia)?
	Q17	Does laughing gas have any long-term side effects in children?
Parental role and communication	Q18	Should I stay with my child during the exam, or wait outside?
	Q19	Can showing my own fear of the dentist in front of my child harm them?
	Q20	My child is afraid of the dentist after hearing about a sibling’s bad experience—what should I do?

The questions were submitted to the chatbots in English exactly as shown in this table.

2.3. Chatbots and Data Collection

Four widely used, publicly and freely available LLM-based chatbots developed by different companies were included. Systems from different developers were selected to compare, using the same standardized questions, chatbots that may differ in their development processes, training data, response-generation mechanisms, and safety approaches. This approach was intended to ensure that the findings were not specific to a single developer and to evaluate differences in response quality, accuracy, and safety across systems. Free versions were chosen because parents are most likely to use these versions when seeking information. Model selection was guided by accessibility, availability of free versions, and suitability for standardized comparisons, rather than the geographical origin of the developers. Consequently, all four selected chatbots were developed by US-based companies.

The versions offered by default in the interface on the study dates were used: ChatGPT (GPT-5.6 Luna; OpenAI, San Francisco, CA, USA), Gemini (3.6 Flash; Google, Mountain View, CA, USA), Claude (Sonnet 5.5; Anthropic, San Francisco, CA, USA), and Grok (4.5; xAI, Palo Alto, CA, USA). The model version was verified in the interface and recorded in each session, and the reported versions correspond to the model labels displayed by the interfaces on the query dates. The default settings of the models were not changed, and no optional reasoning mode, web search, or additional tool features were activated. Considering that LLM responses may change over time [20], the questions were submitted on 30 September, 1 October, and 2 October 2026. Three consecutive days were selected to assess short-term, day-to-day variability in chatbot responses under standardized data collection conditions. This repeated-measures approach allowed each model to be evaluated on the same questions across three independent daily sessions, providing three pairwise comparisons (Days 1–2, 1–3, and 2–3) for assessing semantic similarity. The consecutive-day design was intended to capture short-term variation while limiting the observation period during which changes in model availability or configuration might occur. However, the three-day period was not intended to establish long-term response stability. On each of the three consecutive days, queries were performed between 09:00 and 10:00 via a web browser. All chatbot queries were performed using a single computer to maintain consistent data collection conditions and minimize potential variability associated with the use of different devices. To prevent previous interactions from influencing the responses, a new account was created for each model each day, memory features were disabled, and each question was asked in a new chat session. The first response obtained without any follow-up or additional prompt was recorded, and a screenshot of each response was taken. In total, 240 responses (4 models × 20 questions × 3 days) were obtained.

2.4. Blinding

Before scoring, the model names, interface elements, and any information that could reveal the model were removed from the responses. The 12 responses to each question were randomly ordered and coded. The coded responses were compiled into a single scoring file and sent separately to each rater. The raters were unaware of which model and which day each response belonged to until scoring was completed. The FRES was also calculated using only the coded response texts, without model or day information.

2.5. Assessment of Accuracy, Overall Quality, and Reliability

The blinded responses were scored independently by two pediatric dentists (P.C., M.P.) based on the behavior guidance guideline of the American Academy of Pediatric Dentistry (AAPD) [9]. Accuracy was assessed using a 5-point Likert scale rating the agreement of the response with current clinical knowledge from 1 (completely incorrect) to 5 (correct and complete) [21]. In evaluating accuracy, the raters considered both the correctness and completeness of the information provided. Responses containing a combination of correct and incorrect statements were evaluated by taking into account the nature and potential clinical significance of the inaccuracies. Minor omissions or incomplete explanations were distinguished from potentially misleading information, particularly when inaccuracies could affect parental understanding or treatment-related decisions. Thus, responses containing clinically significant misinformation were not considered fully accurate, even if other parts of the response were correct. Overall information quality was scored with the Global Quality Scale (GQS), which was developed by Bernard et al. [22] to evaluate internet-based patient information resources. On this scale, a score of 1 indicates poor quality and information not useful for patients, and a score of 5 indicates excellent quality and information highly useful for patients. Reliability was assessed with the modified DISCERN (mDISCERN), a five-item adaptation by Singh et al. [23] of the DISCERN instrument developed by Charnock et al. [24]. The modified version was chosen because the items of the original instrument focusing on the benefits and risks of treatment options are not applicable to most chatbot outputs, which provide short answers to a single question. Each item was scored as 0 (no) or 1 (yes), giving a total score of 0–5, with higher scores indicating more reliable information. Responses with a difference of ≥2 points between the two raters on at least one scale were re-evaluated jointly by both raters, and a consensus score was determined. In the analyses, the consensus score was used for these responses and the mean score of the two raters for all other responses.

2.6. Readability Assessment

Readability was assessed with the Flesch Reading Ease Score (FRES) [25]. The FRES was calculated in Python (version 3.11) using the textstat library (version 0.7.3). Before calculation, the same rules were applied to all responses: markdown symbols, bullet points, URLs, and reference numbers were removed, and heading and bullet lines were treated as separate sentences. As a sensitivity analysis, the FRES was also calculated without counting line breaks as sentence boundaries. The FRES ranges from 0 to 100, with higher values indicating easier readability. Values were interpreted according to Flesch’s classification [25]; scores of 60 or above were considered to indicate a standard readability level, whereas the level recommended for patient materials corresponds to approximately 80 or above [6]. The scales and scoring criteria used are presented in Table 2.

Table 2. Scales used to assess accuracy, quality, reliability, and readability.

Score/Item	Definition/Interpretation
Likert scale—accuracy (1–5)
1	Completely incorrect
2	Outdated or misleading information
3	Partially correct; superficial
4	Mostly correct; adequate
5	Correct and complete
Global Quality Scale (GQS)—quality (1–5)
1	Poor quality; most information missing; not useful for patients
2	Generally poor quality; some information present but many important topics missing; of very limited use
3	Moderate quality; some important information adequately discussed, but some points missing
4	Good quality; most important information present; useful for patients
5	Excellent quality; highly useful for patients
Modified DISCERN (mDISCERN)—reliability (0–5)
Item 1	Are the aims clear and achieved?
Item 2	Are reliable sources of information used?
Item 3	Is the information presented balanced and unbiased?
Item 4	Are additional sources of information listed for patient reference?
Item 5	Are areas of uncertainty mentioned?
Scoring	Each item: 0 = No, 1 = Yes; total score 0–5 (higher score = more reliable)
Flesch Reading Ease Score (FRES)—readability (0–100)
90–100	Very easy
80–89	Easy
70–79	Fairly easy
60–69	Standard
50–59	Fairly difficult
30–49	Difficult
0–29	Very difficult

GQS: Global Quality Scale; mDISCERN: modified DISCERN; FRES: Flesch Reading Ease Score. Likert, GQS, and mDISCERN scores were assigned independently by two blinded pediatric dentists; the FRES was calculated automatically.

2.7. Day-to-Day Consistency Assessment

To assess day-to-day consistency, the semantic similarity of the responses given by the same model to the same question on different days was calculated. For this purpose, the cosine similarity between vector representations generated with the Sentence-BERT (SBERT; all-mpnet-base-v2) model using the sentence-transformers library (version 6.1.0) was used [26]. SBERT was selected because it generates contextualized sentence embeddings that allow the semantic similarity of texts to be assessed beyond exact word matching. This approach was considered appropriate for comparing chatbot responses that may differ in wording or structure while conveying similar overall meanings. For each question–model pair, the similarities for Days 1–2, 1–3, and 2–3 were averaged. Values range from 0 to 1, with values closer to 1 indicating greater semantic similarity between responses. Importantly, this measure was used to assess similarity in overall meaning rather than agreement on individual clinical statements. Clinical accuracy was evaluated separately by two pediatric dentists using the AAPD behavior guidance guideline [9].

2.8. Statistical Analysis

Inter-rater agreement was assessed with the average-measures intraclass correlation coefficient (ICC) [ICC(2,2)], calculated using a two-way random-effects model with absolute agreement, together with its 95% confidence interval. ICC calculations were based on the two raters’ original independent scores, prior to consensus assessment. Values < 0.50 were interpreted as poor, 0.50–0.75 as moderate, 0.75–0.90 as good, and > 0.90 as excellent [27,28]. Agreement on the mDISCERN items was examined with Cohen’s kappa coefficient.

Descriptive statistics for the main outcome measures are presented as medians and interquartile ranges (Q1–Q3), calculated using all 60 responses per model (20 questions × 3 days). Likert, GQS, and mDISCERN are ordinal scales, and the same 20 questions were submitted to all four models; therefore, non-parametric tests for dependent samples were used for comparisons among models. The Friedman test was selected because it allows comparisons among more than two related groups without assuming normally distributed outcomes. Each question served as a matched block, ensuring that model comparisons were based on responses to identical questions rather than independent observations. To account for the repeated assessments across three consecutive days, the scores obtained for each question–model combination were averaged across the three days before conducting the model comparisons. Thus, each model contributed one aggregated value per question, and the 20 questions, rather than the 60 individual daily responses per model, constituted the matched observational units for the Friedman analysis. This approach prevented the three daily responses to the same question from being treated as independent observations and provided a summary of model performance over the study period. Because the primary objective was to compare overall chatbot performance rather than temporal changes, and the three-day observation period was primarily intended to assess short-term semantic consistency, separate analyses of day-specific effects and model-by-day interactions were not performed. Short-term semantic consistency was evaluated separately using SBERT-based cosine similarity. When a significant difference was found, pairwise differences were examined with the Wilcoxon signed-rank test, and p values were adjusted with the Holm method [29]. Effect size was calculated as Kendall’s W [30] and interpreted according to Cohen’s criteria as small (0.1–0.3), moderate (0.3–0.5), or large (≥0.5) [31]. Analyses were performed in Python (version 3.11; SciPy 1.17.1 and pingouin 0.7.0 libraries), and statistical significance was set at p < 0.05.

3. Results

3.1. Inter-Rater Agreement

Agreement between the two raters who scored the chatbot responses for accuracy, overall quality, and reliability ranged from moderate to excellent (ICC = 0.71–0.91; Table 3). Agreement was excellent for reliability (mDISCERN), good for accuracy (Likert), and moderate for overall quality (GQS). Cohen’s kappa values for the mDISCERN items ranged from 0.634 (item 5) to 1.000 (items 1 and 4). The relatively lower agreement for Likert and GQS was consistent with one rater systematically assigning higher scores than the other (mean Likert score 4.75 vs. 4.47). The raters’ scores differed by no more than 1 point for 99.6% (Likert) and 98.8% (GQS) of the responses. Nine responses (3.8%) with a difference of ≥2 points on at least one scale were rescored by consensus.

Table 3. Inter-rater reliability (intraclass correlation coefficient; n = 240).

Scale	ICC	95% CI	Interpretation
Accuracy (Likert)	0.76	0.59–0.85	Good
Reliability (mDISCERN)	0.91	0.80–0.95	Excellent
Quality (GQS)	0.71	0.52–0.81	Moderate

ICC: intraclass correlation coefficient, two-way random-effects model, absolute agreement, average measures [ICC(2,2)]; CI: confidence interval. Single-measures ICC(2,1) values were 0.62 for Likert, 0.84 for mDISCERN, and 0.55 for GQS. ICC values were interpreted as poor (<0.50), moderate (0.50–0.75), good (0.75–0.90), and excellent (>0.90), according to the criteria described by Koo and Li [28].

3.2. Accuracy, Overall Quality, and Reliability

The accuracy, overall quality, reliability, readability, and day-to-day consistency results for the four models are presented in Table 4. In all models, the median accuracy and overall quality scores were 4.5 or 5. Nevertheless, accuracy scores differed significantly among the models (p < 0.001; W = 0.28). In pairwise comparisons, the accuracy score of Gemini 3.6 Flash was lower than those of ChatGPT-5.6 Luna (p = 0.009) and Grok 4.5 (p = 0.015); the other pairwise differences were not significant (p ≥ 0.16). No significant difference was found among the models in overall quality scores (p = 0.051). Reliability scores were moderate to low (median mDISCERN scores ranged from 1.00 to 3.00) and differed significantly among the models (p < 0.001; W = 0.42). Gemini 3.6 Flash scored lower than all three other models (ChatGPT-5.6 Luna p = 0.001; Grok 4.5 p = 0.002; Claude Sonnet 5.5 p = 0.030). The score of Claude Sonnet 5.5 was also lower than that of Grok 4.5 (p = 0.029). The difference between ChatGPT-5.6 Luna and Claude Sonnet 5.5 was borderline (p = 0.052), whereas ChatGPT-5.6 Luna and Grok 4.5 were similar (p = 0.660).

Table 4. Accuracy, overall quality, reliability, readability, and day-to-day consistency by model (median [Q1–Q3]).

Measure	ChatGPT-5.6 Luna	Gemini 3.6 Flash	Claude Sonnet 5.5	Grok 4.5	Friedman χ2(3)	p	Kendall W
Accuracy (Likert, 1–5)	5.00 [4.50–5.00]	4.50 [4.00–5.00]	5.00 [4.50–5.00]	5.00 [4.50–5.00]	16.95	<0.001	0.28
Overall quality (GQS, 1–5)	5.00 [4.38–5.00]	4.50 [4.50–5.00]	5.00 [4.50–5.00]	5.00 [4.50–5.00]	7.75	0.051	0.13
Reliability (mDISCERN, 0–5)	3.00 [2.50–3.50]	1.00 [1.00–2.50]	2.50 [1.00–3.00]	3.00 [2.50–3.12]	25.39	<0.001	0.42
Readability (FRES, 0–100)	52.85 [41.87–62.07]	56.61 [45.58–62.48]	71.90 [63.14–82.57]	47.47 [35.93–57.60]	39.66	<0.001	0.66
Day-to-day consistency (cosine similarity, 0–1)	0.923 [0.894–0.966]	0.942 [0.912–0.958]	0.932 [0.898–0.948]	0.941 [0.922–0.953]	1.86	0.602	0.03

Values are presented as median [Q1–Q3], calculated from all 60 responses per model. Friedman tests were performed using the three-day mean for each question–model combination, with the 20 questions serving as matched blocks (n = 20).

3.3. mDISCERN Items

Across the mDISCERN items, clear and achieved aims were observed at high rates in all models (96.7–100.0%). In contrast, the use of reliable sources and the provision of additional sources for patients were the items with the lowest rates of fulfillment (Table 5). The highest rate for the use of reliable sources was observed for ChatGPT-5.6 Luna (29.2%), whereas this item was not fulfilled in any response from Gemini 3.6 Flash. Additional sources for patients were provided in only one of the 240 responses (0.4%). Regarding balanced and unbiased presentation of information, ChatGPT-5.6 Luna and Grok 4.5 had the highest rates (73.3%) and Gemini 3.6 Flash the lowest (41.7%). Similarly, mention of areas of uncertainty was highest for Grok 4.5 (75.0%) and lowest for Gemini 3.6 Flash (38.3%).

Table 5. Mean fulfillment of mDISCERN items by chatbot (%; n = 60 per model).

mDISCERN item	ChatGPT-5.6 Luna	Gemini 3.6 Flash	Claude Sonnet 5.5	Grok 4.5
1. Are the aims clear and achieved?	100.0	96.7	98.3	98.3
2. Are reliable sources used?	29.2	0.0	5.0	27.5
3. Is the information balanced and unbiased?	73.3	41.7	63.3	73.3
4. Are additional sources listed for patients?	1.7	0.0	0.0	0.0
5. Are areas of uncertainty mentioned?	70.8	38.3	61.7	75.0

Percentages were calculated from the final scores (for responses without consensus scoring, the mean of the two raters; e.g., 0.5 = one rater answered “yes”).

3.4. Readability Results

Readability differed significantly among the models (p < 0.001; W = 0.66). The FRES of Claude Sonnet 5.5 was higher than those of all three other models (median: 71.90 [63.14–82.57]; all p < 0.001). Grok 4.5 had the lowest median FRES (47.47 [35.93–57.60]), whereas Claude Sonnet 5.5 had the highest median FRES among the four models. ChatGPT-5.6 Luna and Gemini 3.6 Flash were similar (p = 0.81) (Table 4 and Table 6).

Table 6. Distribution of FRES levels and response length by model (n = 60 responses per model).

Criterion	ChatGPT-5.6 Luna	Gemini 3.6 Flash	Claude Sonnet 5.5	Grok 4.5
Responses with FRES ≥ 60, n (%)	18 (30.0)	18 (30.0)	47 (78.3)	12 (20.0)
Easy–fairly easy (70–89), n	3	4	32	4
Standard (60–69), n	15	14	15	8
Fairly difficult (50–59), n	19	22	5	12
Difficult–very difficult (0–49), n	23	20	8	36
Word count, mean ± SD	324.6 ± 126.9	351.8 ± 77.3	390.0 ± 81.9	488.9 ± 140.7

3.5. Day-to-Day Consistency Results

All four models provided highly consistent responses in terms of meaning across the three days (median cosine similarity 0.923–0.942), with no difference among the models (p = 0.602; W = 0.03) (Table 4; Figure 1). The lowest similarity was observed for questions that could be interpreted differently depending on the context: Q07 (the child crying and resisting), Q15 (the child not cooperating at all), and Q18 (the parent staying with the child during the examination).

Figure 1. Mean semantic similarity (cosine similarity) across the three days for each question and model. Darker shades indicate higher consistency; the lowest values were observed for Q07, Q15, and Q18.

4. Discussion

4.1. Principal Findings

In this study, the responses of four current chatbots (ChatGPT-5.6 Luna, Gemini 3.6 Flash, Claude Sonnet 5.5, and Grok 4.5) to frequently asked parental questions on behavior guidance and dental anxiety were compared in terms of accuracy, overall quality, reliability, readability, and day-to-day consistency. Significant differences were found among the models in accuracy, reliability, and readability, whereas no differences were found in overall quality or day-to-day consistency. Therefore, the null hypothesis was rejected for accuracy, reliability, and readability but could not be rejected for overall quality and day-to-day consistency. Although the evaluated chatbots generally achieved high accuracy scores under the study conditions, these findings should be interpreted cautiously, as accuracy was assessed by two pediatric dentists and potentially clinically relevant errors were not systematically categorized. Furthermore, limitations in source transparency and readability were observed across the evaluated models.

4.2. Accuracy and Overall Quality

In this study, all four chatbots generally provided highly accurate responses to parental questions on behavior guidance and dental anxiety in pediatric dentistry (median 4.5–5). However, the observed differences suggest that similarly high accuracy ratings do not necessarily indicate equivalent completeness or clinical usefulness of the information provided. This finding is broadly consistent with previous studies evaluating parental questions in pediatric dentistry [11,15,32]. In their study of 30 parental questions, Kapoor et al. [32] reported mean accuracy scores of 4.97 for ChatGPT and 4.60 for Gemini. Similarly, the lower accuracy scores of Gemini 3.6 Flash compared with ChatGPT-5.6 Luna and Grok 4.5 in the present study point to a comparable trend. In contrast, Karadeniz et al. [11] found no significant difference in accuracy among models for questions on space maintainers. These discrepancies suggest that the clinical topics evaluated, the content of the questions, and the model versions used may influence the results.

The literature is more variable regarding the pharmacological steps of behavior guidance. It has been reported that no model was fully consistent with guidelines across 32 clinical questions on pediatric dental sedation [14], whereas Claude and Gemini achieved accuracy above 90% on guideline-based true/false questions [33]. Similarly, a previous study evaluating AI-generated responses to questions on local anesthesia, sedation, and general anesthesia reported differences among chatbot models in accuracy and safety [34]. In the present study, the relatively small differences in accuracy suggest that the models may differ more in the completeness of their explanations than in the correctness of the information provided. Nevertheless, an incomplete response may make it difficult for parents to fully weigh risks and alternatives, particularly for topics requiring consent, such as sedation and general anesthesia.

The similarly high overall quality ratings across models suggest that the responses were generally perceived as useful for parental education. This finding is consistent with the study by Karadeniz et al. [11], which reported high GQS scores for chatbot responses to parental questions on space maintainers. However, the GQS assesses the overall usefulness of a response with a single score and does not separately measure reliability elements such as source citation. Therefore, high-quality scores should be interpreted together with the low reliability scores.

Nevertheless, the generally high accuracy scores should be interpreted cautiously, particularly for questions involving sedation and general anesthesia. Although the expert-based accuracy assessment considered the correctness and completeness of responses, the study did not separately categorize or quantify potentially clinically relevant errors. Therefore, high overall accuracy scores do not exclude the possibility of individual inaccuracies, omissions, or misleading recommendations that could have clinical implications. This distinction is particularly important for sedation and general anesthesia, where recommendations must account for patient-specific factors and professional clinical assessment.

4.3. Reliability and Source Citation

The coexistence of generally high accuracy and overall quality scores with relatively low reliability scores highlights the distinct dimensions assessed by these instruments. While accuracy reflects agreement with current clinical knowledge and the GQS evaluates overall information quality, the mDISCERN assesses aspects of information reliability, including source transparency. Thus, clinically appropriate and well-structured responses may still lack sufficient supporting information to allow independent verification. As chatbots are increasingly used for health information, the traceability of the sources of the information provided becomes more important [1,4]. In this study, limited source citation and the near absence of additional references for parents emerged as major weaknesses, suggesting that even otherwise accurate responses may provide insufficient opportunities for independent verification. Similarly, none of the five LLMs evaluated with parental questions on pediatric anesthesia reached a “good” level on DISCERN, and source citation was very limited [35]. Considering that most of the references provided by chatbots may be fabricated or inaccurate [4] and that only a small proportion of users verify the information they receive [5], information without a stated source deprives parents of the opportunity to verify it. This is clinically important, given that parents may rely on these responses when making decisions about sedation, general anesthesia, and nitrous oxide.

Gemini 3.6 Flash received the lowest mDISCERN score among the four models and did not cite a source in any of its responses. This finding contrasts with studies on space maintainers [11] and zirconia crowns [12], in which Gemini 2.5 Flash received the highest DISCERN scores. Pu et al. [35] noted that source citation scores largely depend on whether the model performs a real-time web search while generating a response. One possible explanation for the differences in citation behavior between Gemini versions is variation in the availability or use of real-time web search; however, this remains a hypothesis, as real-time search functionality was not directly evaluated in the present study. It has also been shown that the behavior of the same service can change markedly within a short period [20]; therefore, chatbot evaluations should be interpreted in relation to a specific version and date.

The high reliability score of Grok 4.5 is consistent with the study by Li et al. [36], in which Grok-3 received the highest mDISCERN score and more frequently supported its responses with sources in orthodontic questions from patients with periodontitis. In the same study, Claude-Sonnet-4 was the model with the lowest reliability; in the present study, source use by Claude Sonnet 5.5 was likewise limited. The higher reliability scores of ChatGPT-5.6 Luna and Grok 4.5 were associated with more frequent inclusion of sources and statements of uncertainty in their responses.

4.4. Interpretation of Readability Findings

Readability is decisive for parents to understand information and apply it correctly [6]. Although Claude Sonnet 5.5 generated more readable responses than the other models, even its responses did not consistently meet the readability level recommended for patient education materials. This suggests that relatively better readability does not necessarily ensure that AI-generated information is sufficiently accessible to parents with different levels of health literacy. Previous studies in pediatric anesthesia, dental anesthesia and sedation, and endodontic patient education have similarly reported limited readability of AI-generated health information [34,35,37]. However, the readability performance of individual models appears to vary across clinical topics and model versions. Notably, the comparatively favorable readability of Claude Sonnet 5.5 contrasts with findings from studies evaluating earlier Claude versions in traumatic dental injuries, pediatric dental sedation, endodontic complications, and orthodontic consultations [13,14,36,38]. This discrepancy may reflect differences in model versions, response-generation characteristics, and the use of questions formulated in everyday parental language rather than technical terminology.

Importantly, readability and reliability should be considered complementary dimensions of information quality. In the present study, Claude Sonnet 5.5 produced more readable responses but showed limited source transparency, whereas Grok 4.5 demonstrated relatively higher reliability despite generating more difficult-to-read responses. This pattern highlights the importance of evaluating whether parents can both understand and verify AI-generated information. In pediatric dentistry, misunderstanding information about pre-sedation fasting, general anesthesia, or postoperative care may have direct implications for patient safety. Therefore, AI-generated parental education materials should combine accessible language with verifiable sources and clinically appropriate recommendations rather than prioritizing readability or reliability alone.

4.5. Interpretation of Day-to-Day Consistency Findings

All four models were highly consistent in meaning across the three days (median cosine similarity 0.92–0.94), with no difference among the models. This finding suggests that the models generally maintained the meaning of their responses over short periods, although semantic stability does not necessarily indicate clinical accuracy or long-term reliability. Lisboa et al. [39] reported good consistency in responses to dental trauma questions over 30 days, while Karakaya et al. [37] observed acceptable internal consistency in repeated responses to endodontic questions. In contrast, another study reported differences in consistency among models when queries were performed at different times of day [16]. The standardized daily query window used in the present study may have reduced variability associated with the timing of data collection, although this possibility was not directly tested. Furthermore, unlike studies that assessed consistency based on accuracy scores, the present study evaluated semantic similarity between response texts.

The lower similarity observed for context-dependent questions suggests that ambiguous parental queries may lead to variations in the explanations provided by chatbots. However, semantic similarity reflects the overall meaning of responses and does not establish whether specific clinical recommendations remain unchanged or are clinically correct. Two responses may exhibit high semantic similarity while containing the same inaccurate recommendation or differing in clinically important details. Therefore, the high short-term semantic similarity observed in this study should not be interpreted as evidence of clinical equivalence or correctness, and the stability of chatbot responses over longer periods remains uncertain [20].

4.6. Clinical Implications

From a clinical perspective, the relevance of chatbot-generated information lies in its potential to address parents’ practical concerns before and after dental appointments. Questions about managing a child’s fear, understanding behavior guidance techniques, and evaluating sedation or general anesthesia options require information that is not only clinically accurate but also understandable and sufficiently transparent to support informed discussions with pediatric dentists. The findings indicate that chatbots should be positioned as tools that complement, rather than replace, the dentist in informing parents. Given that parents do not always share information obtained online with their physician, it may be useful for dentists to ask parents at the first visit which sources they have used and to review the information obtained from chatbots together with them. In particular, instructions that directly concern the child’s safety, such as pre-sedation fasting times and post-procedural care, should be provided through written, individualized instructions from the clinician rather than on the basis of chatbot responses. The absence of sources in most responses means that parents cannot verify the basis of the information; therefore, clinicians are advised to direct parents to institutional guidelines and reliable patient information resources. For developers, making source citation and plain-language communication default features of health-related responses could increase the usefulness of these tools for informing parents.

4.7. Strengths and Limitations

To the best of our knowledge, this is the first study to examine the performance of the four evaluated model versions on behavior guidance and dental anxiety. In addition, the day-to-day consistency of chatbot responses in pediatric dentistry was evaluated through the semantic similarity of the response texts rather than through accuracy scores. In these respects, the findings provide a benchmark for future studies with newer model versions.

This study has several limitations. The question set was developed from publicly available online FAQs rather than questions collected directly from parents. Therefore, although the selected questions covered several clinically relevant aspects of dental anxiety and behavior guidance, they may not fully represent the frequency, diversity, or priorities of concerns expressed by parents in clinical practice. Queries were performed only in English and with the free versions of the models; results may differ in other languages or with paid versions. Because the models are frequently updated, the findings reflect only the versions and dates evaluated [20]. Each question was asked with a single prompt; interactions in which parents might explore a response further with follow-up questions in real-world use were not evaluated. How parents understood the responses and how the responses affected their decision-making were also not measured. The FRES measures the structural features of a text but does not directly reflect reader comprehension. Although clinical accuracy was assessed by pediatric dentists, potentially clinically relevant errors were not systematically classified or analyzed as a separate outcome. Consequently, the findings cannot establish the absence of clinically important misinformation, particularly in responses concerning sedation and general anesthesia. Semantic similarity reflects the overall meaning of responses but does not verify the accuracy or consistency of individual clinical recommendations. Consequently, the SBERT-based analysis cannot exclude clinically important differences between responses with high similarity scores. Finally, the responses were evaluated by two raters. Although inter-rater agreement was good for accuracy and moderate for overall quality, one rater tended to assign higher scores than the other. This systematic difference in scoring tendencies may have introduced some variability into the subjective assessments, particularly for the GQS. However, most individual ratings differed by no more than one point, and larger discrepancies were resolved through consensus. Future studies involving additional independent raters and more standardized scoring procedures may help minimize rater-related variability. Future studies should verify the findings in other languages, with paid versions, and over longer follow-up periods, include more raters, and directly measure the extent to which parents understand the responses. Furthermore, all four chatbots were developed by US-based companies, which may limit the generalizability of the findings to AI systems developed in other geographical and linguistic contexts. Differences in training data, language representation, cultural perspectives, and alignment with regional clinical guidelines may influence the responses generated by other models. Future studies should therefore include chatbots developed in different countries and evaluate their performance across multiple languages and clinical guideline frameworks.

Although averaging the scores across three days allowed model comparisons without treating repeated daily responses as independent observations, this approach did not explicitly assess day-specific effects or interactions between chatbot models and assessment days. Therefore, potential temporal changes in accuracy, overall quality, reliability, and readability, including differences in these changes across models, could not be determined. In addition, no a priori power analysis was performed; the number of questions was determined in line with the numbers used in similar studies. Because no accepted clinical threshold exists for cosine similarity, the consistency values were interpreted comparatively across models rather than in absolute terms. Furthermore, the three-day observation period was designed to assess short-term semantic variability and does not allow conclusions regarding the long-term stability of chatbot responses. Future longitudinal studies involving repeated assessments over several weeks or months are needed to determine whether the observed consistency is maintained over time.

5. Conclusions

In this study, the four evaluated chatbots (ChatGPT-5.6 Luna, Gemini 3.6 Flash, Claude Sonnet 5.5, and Grok 4.5) generally achieved high accuracy and overall quality scores in their responses to parental questions on behavior guidance and dental anxiety under the study conditions, although Gemini 3.6 Flash demonstrated significantly lower accuracy than ChatGPT-5.6 Luna and Grok 4.5, and reliability remained limited across all four models. No significant differences were observed among the models in overall quality. Regarding reliability, ChatGPT-5.6 Luna and Grok 4.5 received higher mDISCERN scores, whereas Gemini 3.6 Flash received the lowest score and did not cite a source in any response. In terms of readability, Claude Sonnet 5.5 achieved the highest readability scores, whereas Grok 4.5 achieved the lowest scores. However, high accuracy scores and semantic similarity do not necessarily establish clinical safety or actual parental comprehension, neither of which was directly assessed in this study. These findings suggest that the specific chatbot versions evaluated may serve as supplementary sources of parental information, but their responses should not replace individualized guidance from pediatric dentists, particularly for decisions involving sedation and general anesthesia. The findings are limited to the evaluated model versions, English-language questions, free-access interfaces, and the specific data collection period; they should not be generalized to other AI chatbots or different versions and settings of the same systems.