Adolescents’ Experience with a Conversational Agent for Depression

Abstract

Conversational Agents have been showing promise for depression in adults in the short-term. Although, there has been little research done for conversational agents (CAs) with depression in adolescents. This study aimed to determine adolescents’ user experience with Athenabot, a behavioral activation CA for depression. The study included 66 participants who interacted with Athenabot. Participants were aged 13 to 18 (mean = 14.12) and predominantly identified as female (56.1%). Participants’ confidence in the CA’s utility to improve mood significantly increased from baseline to post-intervention (p < 0.001). Adolescents provided an acceptable Net Promoter Score of 6.73. Positive themes from feedback included the CA being helpful and favorably viewed, while negative themes included its perceived audience-dependency and impersonal nature. Recommendations for improvement included reducing repetitive questions and enhancing personalization. Adolescents significantly preferred multiple-choice questions over typed response questions (p < 0.05). However, there were no significant differences in preference for emojis, memes, or GIFs. Adolescents reported an increased confidence that the CA could improve their mood. While the CAs received acceptable support, feedback highlighted a need for improved engagement and personalization. Adolescents favored multiple-choice button questions over typed responses and preferred GIFs over memes and emojis, with no significant demographic differences.

1. Introduction

An estimated 5 million adolescents in the United States (U.S.) aged 12 to 17 encountered at least one major depressive episode during the last year [1]. Research indicates that many adolescents diagnosed with Major Depressive Disorder (MDD) remain untreated, with 37.9% receiving no treatment and only 36.9% obtaining care from mental health specialists [2]. Moreover, only 39% of adolescents and children had a positive response to treatment for depression, and this positive response tends to diminish gradually over time [3].

There is a need for effective interventions and support systems to address the mental health challenges faced by adolescents. Traditional therapy may be costly and hard to scale. The use of digital therapeutics, including mobile apps, teletherapy, video games, and conversational agents, can be beneficial in addressing mental health issues, especially among the youth population, due to their accessibility and scalability. Conversational Agents (CAs) are automated software that provides interactive dialogue to users [4]. CAs offer greater accessibility than traditional therapy services, as they are not constrained by time or location, and they are low-cost and scalable. Thus, CAs may represent a potential solution to address global mental health needs. However, there is limited research on their efficacy with adolescents. CAs can be programmed to respond with text, spoken language, live videos, or emojis. Regarding text-based CAs, there are two primary categories: rule-based and AI-based. Rule-based CAs use decision trees that determine the message to send based on the user’s response pattern. A systematic review and meta-analysis of CAs have shown significant improvements in adults with depression in the short term, but not in the long term [5]. However, there is limited literature available for CAs targeted towards adolescent mental health issues.

Only a few studies were conducted with adolescents using CA: Tess for adolescents with diabetes [6]; “Layla’s Got You” about contraception, sexually transmitted infections (STIs), and sexual health for Black and Hispanic young women [7]; and KIT, a positive-body-image CA developed to provide psychoeducation and coping strategies [8]. To date, only two studies have focused on CAs tailored for depression in adolescents. BethBot is a text-based CA delivered through Facebook messenger that provides psychoeducation and teaches coping strategies for adolescents with depression [9]. Users recommended incorporating more emojis and slang into the conversation as well as creating a CA that was more personal and allowed for preference adjustment. A total of 54.5% of the users had a positive view of BethBot, and they believed that the CA could improve symptoms as well as offering someone to talk to. The study highlighted a limitation due to the small sample size of 23 participants, of which only 13 completed the full experience. The second study focuses on Woebot, a CBT-based CA developed to deliver support for various psychological challenges for emerging adults aged 16 to 21 [10]. This study found that users generally acquired social support from the CA. Users stated that the CA can readily offer access to social support and that many users viewed fewer barriers to communicating with a CA than with a human. Furthermore, numerous users stated that it was easier to self-disclose to a CA than a person because CAs offer anonymity and privacy.

Due to the limited number of CAs available for adolescents, there is uncertainty in the methods of interactions preferred by the users. In addition, the longer-term outcomes of these interactions are unknown. Communication preferences in adolescents for CAs, such as Emojis, GIFs or Memes, require further research. Mariamo et al. [11] found that questions with GIFs were viewed favorably by adolescent users but did not significantly affect the probability of replies. Mostafavi and Porter [12] showed that using emojis in online messaging can aid emotional expression in response to the absence of body language and vocal cues. Adolescents’ perception of the features of the CA can be better understood through the Technology Acceptance Model (TAM) and the Unified Theory of Acceptance and Use of Technology (UTAUT) [13]. Understanding the factors that influence adolescents’ initial adoption of and engagement with a CA is key to influencing continued use. It is possible that adolescents are more likely to use a CA when it is perceived as useful and easy to use, and when others use it. Additionally, adolescents may be a good fit for interventions delivered through CA due to their familiarity with text-based communications. Within this framework, the design features of the CA are expected to promote engagement in a personalized format, which, in turn, may influence adolescents’ perceived confidence in the agent’s ability to support mood-related outcomes.

2. Materials and Methods

This study used a single-group pretest–posttest design to examine changes in outcomes following interaction with the chatbot intervention.

Athena Bot

Athena Bot is a text-based CA designed for this study delivered via Chatfuel. Scripts were modified from a CBT behavioral activation perspective CA [9]. Athena is rule-based, so the order of messages and the CA’s responses were not dependent on the users’ comments. The Athena CA was designed before the advent of AI, and as a result, the design used a rule-based system that proceeded based on pre-chosen paths. The CA script also included open-ended, multiple choice, Likert scale, and net promoter score (NPS) questions to assess user experience. The scripts were modified to include three questions to ascertain the user’s preference for questions that include a meme, GIF, or emoji. Athena’s messages included emojis, internet memes, and GIFs related to concepts from Behavioral Activation. Each message had three versions with a meme, GIF, or emoji (See Figure 1).

Memes, Emojis, and GIFs

Emojis, which means “picture words” in Japanese, are graphic symbols representing facial expressions, common objects, and actions among others [14]. Internet memes, coined by Mike Godwin in 1993, are images with text overlaid that incorporate elements from popular culture that are remixed by numerous participants to serve as a form of public commentary [15]. The Graphics Interchange Format (GIF) is a bitmap image design issued in 1987 by computer scientist Steve Wilhite at CompuServe. GIFs are animated images that have become culturally relevant in social media [16].

Measures

Confidence measure: The users’ Confidence score was calculated based on their answer to “On a scale from 1–10, how confident are you that a bot can teach you something to improve your mood?” before and after interacting with Athena.

Net Promoter Score: The users’ net promoter score (NPS) was calculated based on their answer to “On a scale of 1 to 10, with 1 being not at all and 10 being definitely, would you recommend the CA you used today to a friend?”

Preferences for question formatting: The question for open-ended versus multiple choice answer options was “How much do you like the way this question was presented to you?” Preference for question formatting was ascertained using Likert scales with 1 = I did not like it at all and 10 = I liked it very much after each of the six formatting questions.

Preferences for emojis, GIFs or memes in responses: The question for different emoji, GIF, and meme information presentations was, “How much do you like the way this text is presented to you?” Preference for emojis, GIFs, or memes was ascertained using Likert scales with 1 = I did not like it at all and 10 = I liked it very much after each type of question. See Figure 1.

Procedure

Recruitment was conducted through Instagram and Facebook with a link to the CA and snowball sampling. Participants were recruited in the Bay Area Region of California, USA. Participants needed to have a device that was able to access Chatfuel to participate in the study. The assent form was included at the beginning of the conversation. Upon obtaining participant assent, adolescents began messaging with the CA through Chatfuel. The CA text consisted of six modules, followed by a series of questions on user experience. The study was intended to be completed in a single session. Given the exploratory nature of the study, the potential for dropout, and the limitations of snowball sampling, a target sample size of 66 participants was selected. As part of ethical consideration for minors, the data was deidentified and participants were given permission to withdraw at any point during the study. Furthermore, support in the form of crisis lines was provided.

Data Analysis

Descriptive statistics were provided on all measures for quantitative data. For confidence, a paired samples t-test was used for a pre-and post-analysis of the confidence question. The distribution of NPSs was calculated by determining the frequency of each potential score [1,2,3,4,5,6,7,8,9,10]. A thematic analysis using Braun and Clarke’s [17] method was used to identify themes of adolescents’ perceptions and user experience. The themes were identified by the first author, who created a codebook. A second blind coder received the definitions of each theme and was asked to code the responses. Inter-rater reliability between the coders was assessed using Cohen’s kappa, which is frequently used to assess agreement between coders on categorical variables. A paired samples t-test determined adolescents’ preference for multiple-choice versus open-ended questions. A factorial ANOVA was conducted to determine preferences for emojis, memes, or GIFs descriptive statistics were used.

3. Results

3.1. Demographics

A total of 66 participants messaged with Athenabot; of those, 75% completed the study. The age range was 13–18 year olds (M = 14.12, S = 2.20, Mo =16). For gender, 56.1% identified as female, 42.4% male, and 1.5% non-binary. For ethnicity, 40.9% of participants identified as Hispanic. For race, 87.9% identified as White, 10.6% identified as Asian, and 1.5% identified as Black.

3.2. Confidence and Net Promoter Score

Adolescents’ confidence in the utility of the bot to improve their mood increased from baseline (M = 5.12, SD = 2.34) to post-intervention (M = 7.04, SD = 2.50) at a significant level with a large effect size (p < 0.001, dz = 0.76). Adolescents provided a Net Promoter Score of 6.73 (range = 1–10). A rating of 10 was the mode, and only one adolescent rated the CA as 1.

3.3. Adolescents’ Perceptions of Athena

3.3.1. Thematic Analysis of Perceptions

The main positive themes of adolescents’ responses were: the CA was helpful (n = 11) and they had a favorable view of the CA (n = 11). The main negative themes were: the CA was audience-dependent (n = 12) and felt impersonal (n = 8). Interrater reliability was calculated to be Cohen’s Kappa = 0.90, indicating almost perfect agreement. See

Table 1. Thematic Analysis of Perceptions.

Theme	Definition	Example	Frequency (%)
Audience Dependent	The user mentions people are different, or people like different things, or that the CA is not for everyone.	“I probably wouldn’t recommend the CA to a friend unless they said something about how they like to chat with ai bots”	12 (24.5)
Helpful	The user expresses that they were helped by the CA, or that the CA could help others.	“Because it can help my friend whenever she needs someone”	11 (22.4)
Favorable Rating	The user mentions a positive indication of their view of the CA.	“Because this is a good example”	11(22.4)
Impersonal/Limited	The user mentions that the bot is limited in its variety of responses,	“I gave that score because it feels more like you’re taking a test”	8 (16.3)
Uncertain/Ambiguous	The user mentions that they don’t know or are not sure if or why they would recommend the CA.	“idk”	7 (14.3)
Total			49 (100)

.

3.3.2. Thematic Analysis of Recommendations

Adolescents responded that the main recommendations for improving the CA were: to reduce the repetitive questions, improve interactions so they feel more personal, and include fewer emojis, GIFS, and memes. Interrater reliability was calculated as Cohen’s Kappa = 0.84, indicating almost perfect agreement. See

Table 2. Thematic Analysis of Recommendations.

Improvement Theme	Subtheme	Definition	Example	Frequency (%)
Positive Comments		The user mentions that they liked everything about the CA, that there is nothing they would change, or mentions something that they liked about interacting with Athena.	“nothing really”	11 (22.4)
Limited and Impersonal Responses		The user mentions the CA was limited in responses, that it felt robotic or mechanical, or that they felt the CA missed opportunities to respond	“Maybe a more responsive and understand text results and easier toUnderstand sentence that a normal person would respond properly to a question or concept”	7 (14.3)
Content	Expanded Content	The user mentions that they want more content, such as questions, situations, etc.	“adding in different details more questions about mental health”	7 (14.3)
	Less Content	The user mentions they want less content, such as questions, situations, etc.	“being more specific, not repeating questions, and be a little shorter”	2 (4.1)
Preference for Simplicity and Less Repetition		The user mentions that the CA is repetitive	“Asking me over and over if I like things”	5 (10.2)
Emoji, Gifs, and Memes	More emojis, gifs, and/or memes	The user mentions that they think there should be more emojis, gifs, and/or memes in their message.	“1.more emoji 2. more memes 3. less press buttons”	2 (4.1)
	Less emojis, gifs, and/or memes	The user mentions that there should be less or no emojis, gifs, and/or memes in their message.	“no memes, no emojis, take it a little more serious”	6 (12.2)
I don’t know		The user mentions that they don’t know what they would want to improve about the CA.	“I don’t know”	4 (8.2)
Text Issue		The user mentions an issue with messaging or typing	“not send many messages at once, that’s it”	3 (6.1)
Uncodable		Response was not codable	“Giving money giving food giving more money”	2 (4.1)
Total				49 (100)

.

3.4. Adolescents Preference for Question Formatting and Emojis, Memes, or GIFs

Adolescents rated multiple-choice button questions (M = 7.62) significantly higher than typed response questions (M =5.74) (p < 0.05). To determine if adolescents prefer emojis, memes, or GIFs of a CA for depression in adolescence, descriptive statistics were used. Although adolescents provided a higher rating for GIFs (M = 6.32), followed by memes (M = 5.95) and then emojis (M = 5.44), there were no significant differences.

4. Discussion

There is scant research on CAs designed explicitly for adolescents [6,9,18]. This study aimed to investigate various aspects of adolescents’ interactions with the behavioral activation CA Athenabot. The most relevant finding was that adolescents reported a significant increase in their confidence in the utility of the bot to improve their mood baseline to post-intervention. This finding highlights the potential of a CA for adolescents with mood-related problems, and that they found the information useful.

Interestingly, adolescents reported a Net Promoter Score (NPS) of 6.73 (mode = 10), indicating acceptable support. This score was similar to the one reported in the previous version of this CA (i.e., 6.04; [9]). Two factors may explain the acceptable NPS of both studies. On one hand, some users perceived the CA as very useful. The fact that a rating of 10 was the mode supports this hypothesis. On the other hand, adolescents are constantly exposed to the latest technologies, especially through video games and immersive environments. In contrast, many studies, including the current one, often rely on rudimentary tools and methodologies such as survey instruments. Adolescents who are accustomed to those immersive experiences may find the methods used in this study less relevant or engaging. As a result, there may be challenges in capturing their attention and interest in participating in studies conducted with less advanced tools. These findings underscore the importance of considering target users’ unique characteristics and preferences when designing and evaluating CA interventions, as well as the need for further research to understand factors influencing user satisfaction and engagement across diverse populations.

Regarding the comments supporting the adolescents’ NPS, the most frequent theme reported was that Athena needed to adjust better to the audience, highlighting the need for a more personalized CA. Additionally, there were several comments about Athena being helpful; adolescents found that it provided useful information, guidance or emotional support. Adolescents also reported a favorable view of Athena’s functionality, usability and quality of interactions. On the other hand, some users felt Athena was impersonal and lacked warmth and empathy, which may have contributed to adolescents’ feelings of frustration, disengagement or dissatisfaction. The lack of warmth and empathy may be due to Athena being scripted instead of generative AI; scripted bots do not respond in as empathic a manner as the newer versions of CA utilizing large language models. The quantitative findings showed an increase in the adolescents’ confidence in the capability of the CA to improve their mood, which is supported by the qualitative themes that describe Athena as helpful. This highlights the capability of a more primitive CA that is rule based in helping with building rapport and confidence in the user’s regarding the CA.

Regarding preference for type of questions, adolescents rated multiple-choice button questions significantly higher than typed response questions. This is consistent with a previous study where adolescents responded positively to the presence of buttons in conversational interfaces and appreciated the guidance in conversational topic flow provided by button-directed conversations [8]. Similarly, Inkster et al. [19] found that adult users generally prefer responding through preformatted options. The preference for multiple-choice buttons aligns with cognitive load theory [20], which posits that reducing cognitive load by offering structured formats can enhance engagement and comprehension. This may be particularly true for adolescents navigating various cognitive and emotional challenges.

GIFs had the highest scores when asked about their preferences for GIFs, emojis or memes, but there were no significant differences. This finding is consistent with Mariamo et al. [11], who reported favorable views toward GIFs in CAs by adolescent users but which did not affect engagement. While there were no differences in terms of preferences, emojis, GIFs and memes may play a role in contributing to perceiving the CA as more human. According to Rapp et al. [21], giving a CA a human name and using emojis might make users perceive it as human. Furthermore, Rapp et al. [21] suggest that giving CAs human names can foster a perception of the CA as more personable and approachable, thus increasing users’ willingness to engage with it. Interestingly, studies have indicated that users prefer female CAs [22]. Finally, Mostafavi and Porter [12] also highlighted the role of emojis and other multimedia content in online messaging as aids for emotional expression, especially in the absence of nonverbal cues like body language and vocal tone. Future studies can explore preferences for conversation style in terms of media usage across different CA platforms and the effectiveness of these styles in building rapport with users.

The sample is selected with a limited age group in the US and as such the generalizability of the findings may be limited. Future research can explore these topics across different geographical regions and age groups. This study was designed as a preliminary study on the effects of conversational agents and conducted as a single session, which poses a limitation in finding the long-term effects of an intervention using conversation agents. Future direction for studies can explore the more continuous and long term effects of conversational agents on adolescents with depression. Given that there were no significant differences between the different media types, future research can explore different determinants of engagement to see if there are differences in preferences for adolescents.

5. Conclusions

The findings of this study highlighted that adolescents showed increased confidence in the CA’s ability to improve their mood, providing evidence for its potential to support mental well-being. Despite an acceptable Net Promoter Score, dissatisfaction themes like repetitive questioning and perceived impersonality underscore the importance of refining the CA’s design. This study deepens our understanding of adolescent engagement with mental health CAs. It underscores the importance of continuous innovation and refinement in digital intervention strategies, with far-reaching implications for improving adolescent mental health outcomes in the digital age.