Can AI Models like ChatGPT and Gemini Dispel Myths About Children’s and Adolescents’ Mental Health? A Comparative Brief Report

Abstract

Background: Dispelling myths is crucial for policy and health communication because misinformation can directly influence public behavior, undermine trust in institutions, and lead to harmful outcomes. This study aims to assess the effectiveness and differences between OpenAI’s ChatGPT and Google Gemini in dispelling myths about children’s and adolescents’ mental health. Methods: Using seven myths about mental health from the UNICEF & WHO Teacher’s Guide, ChatGPT-4o and Gemini were asked to “classify each sentence as a myth or a fact”. Responses of each LLM for word count, understandability, readability and accuracy were analyzed. Results: Both ChatGPT and Gemini correctly identified all 7 statements as myths. The average word count of ChatGPT’s responses was 60 ± 11 words, while Gemini’s responses averaged 60 ± 29 words, a statistically non-significant difference between the LLMs. The Flesch–Kincaid Grade Level averaged 11.7 ± 2.2 for ChatGPT and 10.2 ± 1.3 for Gemini, also a statistically non-significant difference. In terms of readability, both ChatGPT and Gemini’s answers were considered difficult to read, with all grades exceeding the 7th grade level. The findings should nonetheless be interpreted with caution due to the limited dataset. Conclusions: The study adds valuable insights into the strengths of ChatGPT and Gemini as helpful resources for people seeking medical information about children’s and adolescents’ mental health, although the content may not be as easily accessible to those below a college reading level.

1. Introduction

According to the United Nations Children’s Fund (UNICEF) report The State of Children in the European Union 2024 and its accompanying Policy Brief on child and adolescent mental health, more than 11 million children and adolescents under the age of 19, or roughly 13 percent, are estimated to experience mental health issues [1]. The occurrence of these conditions grows with age, starting at around 2 percent in children younger than 5 and reaching around 19 percent in teenagers aged 15 to 19 [1]. As summarized by Shim et al. [2], in the USA, 20 percent of children are diagnosed with a mental health condition each year, and by the age of 18, 40 percent will have met the criteria for such conditions. Despite the high prevalence of mental health issues, treatment rates remain low [2].

Over the years, the number of people seeking health information online has increased significantly, with many parents turning to the internet to research their child’s symptoms and guide healthcare decisions [3]. This trend is also evident in the search for mental health information, which has become progressively more common among both adolescents and parents [4].

World Health Organization & United Nations Children’s Fund (UNICEF) [5] claim that reducing myths about mental health helps clear up confusion and creates a safe space where people of all ages feel comfortable seeking help when needed. To support this, they have developed a Teacher’s Guide for social and emotional learning, freely available online, designed for professionals working with adolescents aged 10–14 in educational settings. This includes teachers, school counselors, and mental health professionals [5].

Recently, prominent artificial intelligent (AI) large language models (LLMs), two well-known examples being OpenAI’s ChatGPT and Google Gemini, have gained significant attention for their ability to generate human-like responses in conversations [6]. ChatGPT is recognized for its creativity and depth, while Gemini focuses on accuracy and brevity [7]. A recently published study demonstrated ChatGPT’s ability to generate educational materials for patients on common public health issues [8]. Another study highlighted its usefulness in supporting parents of children with pediatric oncological diseases [9], while another demonstrated its accuracy in delivering information about autism to parents [10]. Nonetheless, notable challenges persist. LLMs can sometimes produce misleading or factually incorrect information, particularly when interpreting complex medical concepts [11,12]. Moreover, AI systems often replicate existing social biases because they are trained on data that underrepresents marginalized groups such as women, racialized communities, and Indigenous peoples, leading to unequal care and outcomes [13]. Many AI models also overlook social determinants of health that are crucial to equity. Accessibility concerns further arise, as AI technologies can be expensive and difficult to implement in rural or low-resource settings, while cultural and linguistic barriers may also limit equitable access [13]. As LLMs gain popularity among patients seeking health information online, understanding their strengths and constraints is crucial. This study is a preliminary analysis aimed at assessing the effectiveness and differences between OpenAI’s ChatGPT and Google Gemini in dispelling myths about children’s and adolescents’ mental health.

2. Methods

Using the seven myths about mental health included in the Teacher’s Guide, developed as part of the Helping Adolescents Thrive package by the WHO Departments of Mental Health and Substance Use, and of Maternal, Newborn, Child and Adolescent Health and Aging, along with the UNICEF Maternal Newborn Adolescent Health Unit [5]:

• If a person has a mental health condition, it means the person has low intelligence.
• You only need to take care of your mental health if you have a mental health condition.
• Poor mental health is not a big issue for teenagers. They just have mood swings caused by hormonal fluctuations and act out due to a desire for attention.
• Nothing can be done to protect people from developing mental health conditions.
• A mental health condition is a sign of weakness; if the person were stronger, they would not have this condition.
• Adolescents who get good grades and have a lot of friends will not have mental health conditions because they have nothing to be depressed about.
• Bad parenting causes mental conditions in adolescents.

ChatGPT-4o (15 September 2024, version 4o) and Gemini (16 September 2024, version App) were asked to “classify each sentence as a myth or a fact”. No adjustment to a predetermined reading level was prompted.

A descriptive analysis evaluated the responses of each LLM for word count, understandability using the Flesch–Kincaid Grade Level, and readability using the Flesch–Kincaid Reading Ease Score (translated into educational level). Although some readability formulas are more specialized, researchers and healthcare professionals commonly use the Flesch tests to evaluate whether health information is written at a reading level appropriate for patients [14]. For this analysis, seventh grade was defined as the school grade at which the average adult should find LLM responses simple to read.

The accuracy of each LLM (measured dichotomously as Yes/No) in dispelling the seven myths was assessed based on whether their responses included the following arguments (or similar in content), adapted from the World Health Organization & United Nations Children’s Fund (UNICEF) [5] Teacher’s Guide:

• Mental illness can affect anyone.
• Everyone can benefit from taking proactive steps to improve their mental health.
• Mental health issues in teenagers are real.
• Various factors can protect individuals from developing mental health conditions.
• Mental health conditions are not a sign of weakness or lack of willpower.
• Depression can affect anyone, regardless of socioeconomic status or how good their life may seem externally.
• A range of factors can influence the mental health of adolescents, their caregivers, and the relationship between them.

These arguments correspond to the seven myths previously stated and were evaluated by a single reviewer—the author of the present manuscript.

To assess differences in word count or understandability between OpenAI’s ChatGPT and Google Gemini responses, Wilcoxon signed-rank test was used. Data analyses were performed using SPSS Statistics (Version 21). The significance level was p < 0.05.

This study did not involve the analysis of human research participants or sensitive data. Therefore, clearance from the Institutional Ethics Committee was not required, in accordance with the prevailing guidelines. LLMs were accessed under free-use conditions.

3. Results

Both ChatGPT and Gemini correctly identified all 7 statements about children’s and adolescents’ mental health as myths (false statements) (Table 1). The average word count of ChatGPT’s responses was 60 ± 11 words, while Gemini’s responses averaged 60 ± 29 words, a statistically non-significant difference between the LLMs. The Flesch–Kincaid Grade Level averaged 11.7 ± 2.2 for ChatGPT and 10.2 ± 1.3 for Gemini, also a statistically non-significant difference. In terms of readability, both ChatGPT and Gemini’s answers were considered difficult to read, with all grades exceeding the 7th grade level.

Table 1. Mental health knowledge misconceptions and responses from ChatGPT and Gemini.

	Large Language Model	Large Language Model Responses	WC	FKG	Reading Level	Dispelling Myth?
Myth #1	GPT-4o	That sentence is a myth. Mental health conditions are not related to intelligence. People with mental health conditions can have varying levels of intelligence, just like anyone else. Having a mental health condition does not affect a person’s intellectual capabilities.	40	10.7	College	Yes
	Gemini	Myth. This is a harmful stereotype. Mental health conditions are not related to intelligence. Many people with mental health conditions are highly intelligent and successful.	25	9.3	College	Yes
Myth #2	GPT-4o	That sentence is a myth. Everyone should take care of their mental health, regardless of whether they have a diagnosed mental health condition. Mental health is an essential part of overall well-being, and maintaining it involves managing stress, emotions, and relationships, just as physical health involves taking care of your body.	51	11.1	College	Yes
	Gemini	Myth. Mental health is important for everyone, regardless of whether they have a diagnosed condition. Taking care of your mental health can help you cope with stress, build resilience, and improve your overall well-being.	34	8.9	10th to 12th grade	Yes
Myth #3	GP-4o	That sentence is a myth. Poor mental health is a significant issue for teenagers. While hormonal changes can affect mood, many teens experience serious mental health challenges, such as anxiety, depression, and stress. These issues should not be dismissed as mere mood swings or attention-seeking behavior. It’s important to address their mental health needs with care and attention.	58	9	10th to 12th grade	Yes
	Gemini	Myth. Teenagers experience a wide range of emotions, including mood swings, due to hormonal changes and developmental factors. However, these experiences do not equate to poor mental health. Teenagers can also suffer from serious mental health conditions such as depression, anxiety, and eating disorders. It’s important to take their mental health concerns seriously and provide support.	56	11.2	College	Yes
Myth #4	GPT-4o	That sentence is a myth. While some factors that contribute to mental health conditions, such as genetics, cannot be controlled, many strategies can help protect and promote mental health. These include maintaining strong social connections, managing stress, practicing self-care, fostering healthy coping mechanisms, seeking early treatment, and promoting supportive environments. Prevention and early intervention can significantly reduce the risk or severity of mental health conditions.	65	14.4	College	Yes
	Gemini	Myth. While there may not be a guaranteed way to prevent mental health conditions, there are many strategies that can help reduce the risk and promote mental well-being. These include: Healthy lifestyle: Regular exercise, a balanced diet, and sufficient sleep can contribute to good mental health. Stress management: Techniques like mindfulness, meditation, and deep breathing can help manage stress and anxiety. Social support: Building strong relationships with friends and family can provide emotional support and a sense of belonging. Seeking help: If you or someone you know is experiencing symptoms of a mental health condition, seeking professional help is important. There are many effective treatments available, including therapy and medication.	110	11	College	Yes
Myth #5	GPT-4o	That sentence is a myth. A mental health condition is not a sign of weakness. Mental health disorders can affect anyone, regardless of their strength, willpower, or character. They are often caused by a combination of genetic, biological, environmental, and psychological factors. Seeking help for a mental health condition is a sign of strength, not weakness, as it requires courage to acknowledge and address one’s challenges.	66	9.6	College	Yes
	Gemini	Myth. This is a harmful stereotype that can lead to stigma and discrimination. Mental health conditions are not a sign of weakness or lack of character. They are medical conditions that can affect anyone, regardless of their strength or resilience. Just like physical illnesses, mental health conditions require treatment and support.	51	8.4	10th to 12th grade	Yes
Myth #6	GPT-4o	That sentence is a myth. Mental health conditions can affect anyone, regardless of external factors like academic success or social life. Adolescents with good grades and many friends can still experience mental health issues such as depression or anxiety, as these conditions are influenced by various factors, including genetics, brain chemistry, and personal experiences. Mental health challenges are not always visible from the outside and can affect people in all circumstances.	71	12.6	College	Yes
	Gemini	Myth. This is a harmful stereotype that ignores the complexity of mental health. Adolescents can experience mental health conditions, even if they seem to have everything going for them. Factors like genetics, life experiences, and brain chemistry can all play a role in the development of mental health conditions. It’s important to recognize that success in school and social life does not guarantee mental well-being. Adolescents may be struggling with internal pressures, insecurities, or traumatic experiences that are not visible to others.	82	11	College	Yes
Myth #7	GPT-4o	That sentence is a myth. While parenting styles and family dynamics can influence a child’s mental health, mental health conditions in adolescents are usually caused by a complex interaction of factors, including genetics, brain chemistry, trauma, environment, and life experiences. Bad parenting alone does not cause mental health conditions, though supportive parenting can help promote emotional well-being and resilience. Mental health conditions often involve factors beyond just parenting.	68	14.6	College graduate	Yes
	Gemini	Myth. While parental factors can contribute to the development of mental health conditions in adolescents, they are not the sole cause. Mental health is a complex issue influenced by a variety of factors, including genetics, environmental factors, and individual experiences. It’s important to avoid blaming parents for their children’s mental health struggles. Instead, focus on providing support and seeking professional help when needed.	63	11.7	College	Yes
p value			>0.05	>0.05	-	-

4. Discussion

Overall, the results suggest that both ChatGPT and Gemini successfully dispelled all seven myths about children’s and adolescents’ mental health, providing accurate and similar information on common mental health misconceptions. The ability of LLMs to set straight health-related myths has also been validated in other areas, such as cancer [15], Alzheimer’s disease [16], and diabetes [17]. The results of the present study align with those of Mondal, Panigrahi, Mishra, Behera and Mondal [8], who also found that the LLM’s answers were at an ideal difficulty level for college students. The finding that all readability scores exceeded the 7th-grade level has critical health literacy implications, as both parents and adolescents may struggle to comprehend materials written at a high school or college reading level. Individuals with inadequate health literacy face significant challenges in accessing and effectively using healthcare services [18]. They may struggle to register for insurance or understand medical forms and instructions, often leaving appointments without fully comprehending their care. Misunderstanding prescription directions can lead to improper medication use, and limited comprehension contributes to poorer management of chronic conditions, resulting in higher hospitalization rates [18]. In the present study, no adjustment to a predetermined reading level was prompted, as most individuals would lack the insight to request such changes, making the present results more representative. Nevertheless, another study by Spallek et al. [19] focused on answering users’ direct queries about mental health and substance use education noted that without engineered prompting, the outputs reading level was poor, highlighting that LLM’s responses still needed human oversight. Several limitations should be noted. The present study is limited by the lack of information on how target populations, such as parents, adolescents, and educators would evaluate the responses of ChatGPT and Gemini, as well as by the absence of citations in these LLMs’ responses to support evidence-based information. These limitations have been noted in other studies as well (e.g., Huang, Chen, Huang, Cai, Lin, Wu, Zhuang and Jiang [17]). The sample size was also limited (reflecting the small number of myths tested), which reduced statistical power; consequently, the lack of statistically significant differences between models may reflect insufficient data rather than a true absence of effect. The study was conducted exclusively in English, leaving unclear how the findings translate to multilingual or low-resource settings, where misinformation and health literacy challenges may differ substantially. Nevertheless, the strength of this work lies in the relevance of the issue addressed and the use of WHO and UNICEF-developed tool as a reference point. Future research should address these gaps by incorporating larger and more diverse sets of myths and conducting qualitative studies to assess perceived comprehension and user trust. Cross-linguistic research would also provide valuable insight into the performance and accessibility of LLMs across diverse linguistic and cultural contexts.

Despite its limitations, this study adds valuable insights into the strengths of ChatGPT and Gemini as helpful resources for people seeking medical information about children’s and adolescents’ mental health, although the content may be less accessible to those below a college reading level. From a practical perspective, educators and clinicians should treat LLM-generated content as a complementary resource rather than an authoritative source. These models can assist in countering misinformation, but their responses should be verified against trusted, evidence-based references. Furthermore, it should not be overlooked that excessive work and overload of responsibilities themselves constitute significant risk factors for adolescents’ mental well-being, highlighting the need for a balanced approach that combines accurate information, supportive environments, and responsible use of AI tools.