Development of a Comprehensive Evaluation Scale for LLM-Powered Counseling Chatbots (CES-LCC) Using the eDelphi Method

Abstract

Background/Objectives: With advancements in Large Language Models (LLMs), counseling chatbots are becoming essential tools for delivering scalable and accessible mental health support. Traditional evaluation scales, however, fail to adequately capture the sophisticated capabilities of these systems, such as personalized interactions, empathetic responses, and memory retention. This study aims to design a robust and comprehensive evaluation scale, the Comprehensive Evaluation Scale for LLM-Powered Counseling Chatbots (CES-LCC), using the eDelphi method to address this gap. Methods: A panel of 16 experts in psychology, artificial intelligence, human-computer interaction, and digital therapeutics participated in two iterative eDelphi rounds. The process focused on refining dimensions and items based on qualitative and quantitative feedback. Initial validation, conducted after assembling the final version of the scale, involved 49 participants using the CES-LCC to evaluate an LLM-powered chatbot delivering Self-Help Plus (SH+), an Acceptance and Commitment Therapy-based intervention for stress management. Results: The final version of the CES-LCC features 27 items grouped into nine dimensions: Understanding Requests, Providing Helpful Information, Clarity and Relevance of Responses, Language Quality, Trust, Emotional Support, Guidance and Direction, Memory, and Overall Satisfaction. Initial real-world validation revealed high internal consistency (Cronbach’s alpha = 0.94), although minor adjustments are required for specific dimensions, such as Clarity and Relevance of Responses. Conclusions: The CES-LCC fills a critical gap in the evaluation of LLM-powered counseling chatbots, offering a standardized tool for assessing their multifaceted capabilities. While preliminary results are promising, further research is needed to validate the scale across diverse populations and settings.

1. Introduction

1.1. Background

Counseling chatbots are conversational agents designed to provide mental health support, guidance, and therapeutic conversations [1]. These chatbots simulate human-like interactions to help individuals manage their emotional well-being, particularly in situations where immediate access to human counselors is unavailable. By offering scalable and accessible mental health services, counseling chatbots address key barriers such as cost and geographical limitations, making them an appealing solution for both users and healthcare systems [2]. The adoption of counseling chatbots has accelerated due to increasing mental health awareness and the growing demand for scalable solutions, a trend that has been further amplified by the COVID-19 pandemic [3]. In many regions, shortages of mental health professionals and long wait times for therapy [4] further highlight the need for alternative support systems. Counseling chatbots can bridge this gap by offering immediate, on-demand assistance. As mental health issues become more pressing globally, the role of these technologies is expected to expand in the next years. Traditionally, counseling chatbots relied on rule-based systems and Natural Language Processing (NLP) techniques to interpret and respond to user inputs [5]. These systems often used decision trees and pattern matching to generate responses, which limited their ability to handle complex or nuanced conversations. However, recent advancements in Large Language Models (LLMs), such as GPT-3 and GPT-4, represent a significant upgrade. LLMs leverage deep learning techniques, particularly transformer [6] architectures to process and generate text with a much deeper understanding of context and semantics. Unlike traditional NLP systems, LLMs are trained on vast amounts of data, which enables them to generate more coherent, personalized, and contextually aware responses. Consequently, where implemented, LLMs have vastly improved the quality of text generated, the level of personalization, and the contextual awareness of chatbot interactions, allowing more natural and emotionally resonant conversations [7]. As counseling chatbots become more sophisticated due to the introduction of LLMs, so must the methods used to evaluate them. Traditional evaluation scales, centered on user satisfaction and related concepts, are valuable but may no longer suffice in capturing the full spectrum of these systems’ capabilities. Their measures are useful but may not fully capture the unique strengths and challenges of modern LLM-powered counseling chatbots. For example, while traditional scales might assess whether a chatbot provides relevant answers, they often overlook aspects like the chatbot’s ability to express empathy, remember previous interactions, or maintain a natural and engaging conversation over time. The integration of LLMs introduces new aspects, such as enhanced empathy, seamless conversational flow, and contextually appropriate emotional responses, which require more nuanced and multifaceted evaluation tools.

1.2. Related Works

The evaluation of mental health chatbots is a complex task that involves assessing multiple aspects of their performance and impact on users. Effective mental health chatbots must engage users, provide a positive user experience, be easy to use, offer helpful and empathetic support, foster trust and alliance, and demonstrate strong technical performance in terms of language quality [8]. As such, researchers and developers have identified several key aspects that are crucial to the success of these systems. Engagement [9] and user experience [10], for example, can influence users’ motivation to continue using the chatbot and their overall satisfaction with the system. Usability [11] is critical to ensuring that users can effectively interact with the chatbot and access the support they need. Perceived helpfulness, empathy, trust, support, and alliance are all essential components of a therapeutic relationship and are critical to establishing a sense of rapport and connection between the user and the chatbot [8]. Technical performance and language quality, meanwhile, are fundamental to ensure that the chatbot can provide accurate and informative responses to users’ queries.

To evaluate these dimensions, researchers have employed a range of scales and metrics. Engagement, for instance, has been evaluated [12,13] using the User Engagement Scale (UES) [14]. This scale provides insights into users’ emotional and cognitive investment in interacting with the chatbot. In contrast, user experience has been assessed [15,16,17] through the User Experience Questionnaire (UEQ) [18], which captures users’ subjective experience of using the chatbot in terms of attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty. To evaluate usability, instead, researchers have utilized [19,20,21,22] several scales, including the System Usability Scale (SUS) [23], Chatbot Usability Questionnaire (CUQ) [24], and Bot Usability Scale (BUS) [25]. These scales provide a comprehensive understanding of users’ perceptions of the chatbot’s ease of use. Perceived helpfulness, which refers to users’ beliefs about the chatbot’s ability to provide effective support, has been evaluated [26,27] using frameworks such as the Unified Theory of Acceptance and Use of Technology (UTAUT) [28] and Perceived Usefulness and Ease of Use (PEOU) [29]. These frameworks help researchers understand the factors that influence users’ intentions to use mental health chatbots. Empathy, a critical component of human–computer interaction in mental health contexts, has been assessed [30,31] using scales such as the Perceived Empathy of Technology Scale (PETS) [32] and Empathy Scale for Human–Computer Communication (ESHCC) [33]. These scales capture users’ perceptions of the chatbot’s ability to understand and respond to their emotional needs. In addition to empathy, perceived trust, support, and alliance are essential aspects of mental health chatbots. The Virtual Therapist Alliance Scale (VTAS) [34] has been used [35] to evaluate these dimensions, providing insights into users’ perceptions of the chatbot as a supportive and trustworthy therapeutic agent. Technical performance and language quality instead are often evaluated [36,37,38,39] using automated metrics such as perplexity, BLEU [40], and ROUGE [41]. These metrics provide quantitative insights into the chatbot’s ability to generate coherent, contextually appropriate, and grammatically accurate responses.

Despite the availability of these tools, many studies still rely on custom evaluation grids tailored to their research needs [42,43], introducing variability across studies. These grids are often designed to cover all relevant aspects in a compact form, as using a full set of scales for the evaluations would be too lengthy and impractical for many studies. This lack of standardization, however, can hinder comparisons between studies and limit the generalizability of findings. As LLM-powered counseling chatbots become increasingly sophisticated, an integrated evaluation approach is essential to ensure that all aspects are adequately assessed while keeping the length of the scale manageable.

1.3. Aim

To address the limitations of current evaluation methods and provide a compact comprehensive tool designed for the unique demands of LLM-powered counseling chatbots, we aim to develop a novel scale (Comprehensive Evaluation Scale for LLM-Powered Counseling Chatbots (CES-LCC)) using the eDelphi method [44]. This approach is particularly well-suited to emerging fields like AI-driven counseling, where expert knowledge is still evolving and consensus on best practices has not yet been fully established.

2. Materials and Methods

2.1. Participants

Following the Delphi methodology [45], this study employed purposive sampling to assemble a panel of experts in counseling psychology, Artificial Intelligence (AI), and Human-Computer Interaction (HCI). The Delphi method does not mandate a statistically representative sample, thereby affording flexibility in panel size. Conventionally, panel sizes range from 15 to 30 participants [46,47]. Given this flexibility, invitations were extended to 22 experts, selected for their professional experience with counseling technologies, LLM-powered systems, and chatbot development. Of these, 16 experts agreed to participate. Eligibility criteria included a minimum of three years of professional experience, relevant academic publications or industry contributions, and familiarity with LLM technologies. Recruitment took place in a single round between 26 June 2024 and 4 July 2024. Invitations were distributed via email, outlining the study’s objectives and the participants’ role in defining and refining the evaluation scale. Participation was voluntary, with an estimated time commitment of 30 min per round. Experts were given a two-week time window to complete the first-round survey, with a reminder sent at the halfway point to encourage response. Following the completion of the first round, a second round of the eDelphi process was conducted to refine and consolidate the initial feedback. Participants were presented with a summary of the first-round results, including aggregated ratings and qualitative comments, and were invited to reassess their responses based on the group’s collective insights. The second-round survey was distributed on 20 July 2024, with a two-week response window and similar reminders to encourage participation.

2.2. Procedure

For our eDelphi study, we followed the four steps proposed by [48], which consist of (1) a preparatory phase, (2) eDelphi rounds, (3) data processing and analysis, and (4) conclusion and reporting. The procedure for the eDelphi study is visually summarized in Figure 1, which provides an overview of the steps and corresponding activities carried out at each stage.

2.2.1. Preparatory Phase

The development of CES-LCC began with the identification of key dimensions critical to the assessment of LLM-powered counseling chatbots. Given that LLM-powered counseling chatbots represent a relatively new and rapidly evolving field, we adopted a targeted approach to the literature. Instead of conducting a comprehensive review across a broad range of sources, we focused on key publications and resources that specifically address chatbot evaluation in the context of mental health [42,43,49,50] and LLM technologies [51,52,53,54] as well as on the scales described in Section 1.2. This targeted analysis allowed us to concentrate on the most relevant aspects to be evaluated, leading to the identification of recurring themes reflecting the technical, emotional, and linguistic dimensions relevant to the evaluation of LLM-powered counseling chatbots.

The initial pool of items was generated by M.B. and S.G., who have interdisciplinary expertise in AI, digital health system design, and psychology. The dimensions identified for the first draft of the scale included: Understanding my Requests, Providing Helpful Information, Clarity and Relevance of Responses, Ease of Use and Interaction, Language Quality, Trust, Emotional Support, Guidance and Direction, and Overall Satisfaction. Based on these dimensions, a total of 18 items (see Appendix A.1) were generated, with 2 items allocated to each category. We deliberately chose to limit the number of items per dimension to 2 to avoid guiding the experts too heavily and to allow for more open-ended feedback during the Delphi process.

2.2.2. eDelphi Rounds

The eDelphi process was structured into two iterative rounds aimed at refining and validating the scale. Both rounds were administered electronically via the Qualtrics XM platform [55].

First Round

During the initial round, experts were provided with the preliminary version of the scale comprising the eighteen items developed in the preparatory phase. Participants assessed each item’s relevance using a five-point Likert scale ranging from “Not relevant at all” to “Very relevant” and its priority on a separate five-point Likert scale from “Very low” to “Very high”. Additionally, they offered qualitative feedback regarding each item’s clarity and comprehensiveness and proposed additional items or dimensions. Following the first round, the original English scale was translated into Italian to ensure accessibility and applicability across both local and international contexts. The translation process followed a rigorous two-step approach to ensure linguistic and conceptual accuracy. Initially, the original scale (in English) was translated into Italian using a multilingual LLM, specifically Mistral Large [56]. Subsequently, two independent bilingual translators, both with expertise in artificial intelligence and psychology, undertook the refinement process. Each translator worked independently, leveraging their specialized knowledge to enhance the linguistic precision and conceptual clarity of the translation. Finally, the translators collaborated to consolidate their refinements into a cohesive and accurate final version.

Second Round

The revised scale, incorporating modifications from the first round, was inspected in the second eDelphi round by the same panel of experts. In this round, participants reviewed the updated scale, which included retained, revised, and newly added items or dimensions. Experts re-evaluated each item’s relevance using the same five-point Likert scale and assessed priority by ranking the items in order of importance. This ranking method facilitated a more precise determination of each item’s relative significance, helping future efforts to develop shorter versions of the scale by identifying the most critical items. Additionally, experts were asked to flag any items they deemed redundant and, if flagged, to specify which other item(s) the redundant item overlapped with. Participants also provided additional qualitative feedback to confirm whether the revisions effectively addressed prior concerns and were encouraged to suggest further enhancements or provide specific recommendations for improvement. Experts whose mother tongue is Italian were also asked to evaluate the translation quality, using both a five-point Likert scale from “Very poor” to “Excellent” and responding to the open-ended question: “Is there a better way to translate this item into Italian? If so, please provide the improved version below”.

2.2.3. Data Processing and Analysis

Throughout both eDelphi rounds, data processing involved detailed quantitative and qualitative analyses. All computations were performed using Python 3.11.5, with the Pandas library [57] (version 2.1.1) utilized for data processing and statistical analysis. For the first round, descriptive statistics (mean, median, interquartile range, and standard deviation) were calculated for both relevance and priority ratings of each item. In contrast, priority assessment in the second round was approached differently, utilizing rankings derived through the Borda count method [58]. This method aggregated participant rankings by assigning points inversely proportional to rank positions, providing a more structured framework for determining the collective prioritization of items. As in [48], items were retained if over 75% of participants rated them as 4 or 5 in relevance and if the interquartile range was below 2. Items with a mean relevance score below 3 or those failing to meet agreement criteria were excluded, along with any dimensions devoid of remaining items post-removal. Qualitative feedback from both rounds underwent thematic analysis [59] to extract common themes and suggestions related to item clarity, comprehensiveness, and potential oversights. This analysis was performed independently by M.B. and S.G. and subsequently consolidated through consensus. The insights derived from this analysis informed the necessary revisions and additions to the scale, thereby enhancing its overall quality and comprehensiveness. In the second round, an additional analysis was also performed to address redundancy among items. Since no established guidelines for redundancy were found in the literature, a statistical criterion was applied. Items flagged as redundant were analyzed, and those exceeding the third quartile of redundancy flags were systematically removed. This redundancy-focused refinement ensured a more concise and efficient evaluation scale, aligned with expert consensus.

To systematically guide and organize the refinement of the evaluation scale, we introduced the Add, Modify, Drop (AMD) approach, as summarized in the form of an algorithm in Algorithm 1. This framework consolidates established methods for decision-making regarding items/dimensions in the context of the development of new scales using the Delphi method [60,61,62,63]. For each dimension, items were dropped based on the aforementioned quantitative criteria, new items were added in response to qualitative feedback to address identified gaps, and existing items were modified according to qualitative feedback to improve clarity, comprehensiveness, and relevance.

Algorithm 1. Add, Modify, Drop (ADM) algorithm

Require: D: Dimensions, $I_d$: Items in d, Quantitative thresholds: $t_{\mathrm{agree}}$, $t_{\mathrm{IQR}}$,       $t_{\mathrm{mean}}$, $t_{\mathrm{redundancy}}$, Q: Qualitative feedback. Ensure: Refined scale with updated dimensions and items. 1: Initialize $D_{\mathrm{refined}}\leftarrow D$, $I_d^{\mathrm{refined}}\leftarrow I_d$ for all $d\in D$. 2: for each $d\in D_{\mathrm{refined}}$ do 3:     for each $i\in I_d^{\mathrm{refined}}$ do 4:         Compute:      % agree, mean, IQR. 5:         if % $\ge t_{\mathrm{agree}}$ and mean $>t_{\mathrm{mean}}$ and $\mathrm{IQR}<t_{\mathrm{IQR}}$ then 6:             Retain i. 7:         else 8:             Drop i. 9:         end if 10:     end for 11:     Refine $I_d^{\mathrm{refined}}$ with Q: improve clarity, address gaps, and add missing items. 12:     Perform redundancy analysis: 13:     for each $i\in I_d^{\mathrm{refined}}$ do 14:         Compute the number of flags for i based on redundancy checks. 15:         if flags for $i\ge t_{\mathrm{redundancy}}$ then 16:             Remove i from $I_d^{\mathrm{refined}}$. 17:         end if 18:     end for 19:     if  $I_d^{\mathrm{refined}}=\varnothing$ then 20:         Remove d from $D_{\mathrm{refined}}$. 21:     end if 22: end for 23: return $D_{\mathrm{refined}}$ with updated $I_d^{\mathrm{refined}}$.

2.2.4. Conclusion and Reporting

Upon completing the two eDelphi rounds, the final version of CES-LCC was assembled, documented, and prepared for dissemination (a full version of the scale is provided in Section 3.3).

2.3. Initial Validation in Real-World

To assess the reliability of CES-LCC, we conducted an initial validation in a real-world setting. This stage focused on testing the internal consistency of the scale items both globally and across its dimensions. Data collection involved 49 users (participants details in Appendix C) engaging with an LLM-powered chatbot that delivered the first session of the Self-Help Plus (SH+), an Acceptance and Commitment Therapy (ACT) based intervention for stress management and prevention originally developed by the World Health Organization (WHO) [64]. In this session, participants were introduced to the chatbot and received information about stress, emotional storms, and some exercises people can use to manage these situations (e.g., grounding, focused attention). Participants filled the CES-LCC after a single interaction with the chatbot. Items were rated on a 5-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree”. The reliability of the evaluation scale was assessed by calculating Cronbach’s alpha [65] for each of the nine dimensions, as well as for the overall scale. Additionally, both item-total correlations [66] and inter-item correlations were examined for each dimension to verify that individual items contribute meaningfully to their respective constructs without introducing strong redundancy.

3. Results

3.1. Demographic Description of Experts

The expert panel consisted of 16 professionals with expertise in psychology, AI, HCI, and digital therapeutics (DTx) (

Table 1. Demographic and professional characteristics of the expert panel.

Characteristic		Value or % (n)
Age		M = 34.50 (SD = 10.66)
Gender	Male	56.25% (9)
	Female	43.75% (7)
Education	Bachelor	12.50% (2)
	Master	50.00% (8)
	Doctorate	31.25% (5)
	PsyD Specialization	6.25% (1)
Area of expertise	Psychology	31.25% (5)
	Artificial Intelligence	31.25% (5)
	Human–Computer Interaction	18.75% (3)
	Digital Therapeutics	18.75% (3)
Occupation	Researcher	50.00% (8)
	Developer (AI)	37.50% (6)
	Psychologist	12.50% (2)
Job seniority	3–5 years	50.00% (8)
	6–10 years	18.75% (3)
	11–15 years	12.50% (2)
	16–20 years	0.00% (0)
	21+ years	18.75% (3)
Country	Italy	100% (16)

). Gender representation was balanced, with 56.25% male and 43.75% female participants, and academic qualifications were predominantly at the master’s (50%) and doctoral (31.25%) levels. Participants’ average age was 34.5 years (SD = 10.66), indicating moderate age variability, while professional roles ranged from researchers (50%) to AI developers (37.5%) and psychologists (12.5%). The group included a broad spectrum of professional experience, with half of the participants having junior roles (3–5 years in the field) and others contributing more senior-level expertise (18.75% with 21+ years of experience). All participants were based in Italy. While the homogeneous geographical location offers cultural homogeneity at the same time it might limit the views offered by the participants involved (more is discussed in Section 4).

3.2. First Round

During the first round of the eDelphi process, a thorough assessment of the original 18-item scale was undertaken. Item-level analysis revealed that 6 items did not meet the predetermined agreement criteria based on relevance and interquartile range (IQR) thresholds, leading to their exclusion from the scale (see Appendix A.1). Concurrently, experts provided qualitative feedback that resulted in the addition of 12 new items, enhancing the scale’s ability to capture significant aspects that need to be evaluated in LLM-powered counseling chatbots. Additionally, five existing items were split into separate subitems to more effectively address distinct aspects, as some items were initially found to cover multiple, overlapping areas. To improve clarity, two items were rephrased based on the qualitative suggestions provided (see Appendix A.3). The dimension “Ease of use and interaction” was removed due to the absence of remaining items after the exclusion process. Meanwhile, a new dimension called “Memory” was introduced to evaluate the chatbot’s ability to retain and utilize prior interactions effectively.

Experts recommended (see Appendix A.2) that each dimension should contain at least three items to meet the psychometric requirement for assessing internal consistency and to ensure the scale’s reliability (“Psychometrically, factors should have at least 3 items to be considered reliable, with 2 items it is not even possible to calculate internal consistency”). Consequently, after initial revisions, new items were added to dimensions with fewer than three items to meet this requirement. Moreover, qualitative feedback highlighted the necessity of assessing privacy and security concerns related to the chatbot. However, experts concluded that these aspects pertain more to production nonfunctional requirements rather than intrinsic characteristics of the chatbot’s functionality and, therefore, were excluded from the evaluation scale (“Items related to data privacy and security might be relevant in this scenario. However, in my experience, these items are more aligned with production or implementation processes and might be better addressed under regulations like the EU AI Act and GDPR”.). Moreover, in relation to the “Trust” dimension, experts raised concerns about anthropomorphizing chatbots. They cautioned that attributing human-like qualities could skew the assessment of trustworthiness. As a result, all items containing any form of anthropomorphism were removed and replaced with new items proposed by the experts to better align with the construct while avoiding any reference to human characteristics. The priority assessment of the items did not provide a clear picture, as priority values ranged from 3.13 to 4.88, with an average value of 3.88 (SD = 0.41).

3.3. Second Round

The second round of the eDelphi process focused on refining the revised evaluation scale, which at this point consisted of 34 items across nine dimensions, incorporating feedback provided during the first round. General feedback from experts gathered in the second round indicated that the scale was comprehensive and complete. However, experts also remarked on the presence of some redundancy in the scale, with certain items overlapping or duplicating information. The average relevance score of the items in this round was 4.32 (SD = 0.40), reflecting strong agreement among experts regarding the importance of the included items. Despite this general consensus, five items were removed due to low agreement, as they failed to meet the thresholds for relevance and IQR established in the methodology. Additionally, two items were excluded based on redundancy. The Italian translation of the scale received high ratings for quality, with an average score of 4.59 (SD = 0.31). However, comments from mother-tongue experts (n = 12) highlighted the need for minor adjustments to improve linguistic precision and conceptual clarity. Consequently, 14 items were revised to enhance their translation quality and to maintain consistency between the English and Italian versions. Furthermore, four items were rephrased in both languages based on qualitative feedback. To complement these refinements, the collective ranking of the items for each dimension was computed. This analysis not only provides a foundation for the potential development of shorter versions of the scale, which may improve its deployment in practical application settings but also makes it possible to identify the most relevant item within each dimension. These rankings, along with the finalized version of CES-LCC (27 items across 9 dimensions), are included in

Table 2. Final version of CES-LCC. Items in italics are the Italian version.

Dimension	Item	Priority
Understanding requests [UR]	The chatbot consistently understands what I am saying and asking. Il chatbot capisce ciò che sto dicendo e chiedendo.	1
	The chatbot is able to make adequate inferences based on my messages. Il chatbot è in grado di fare deduzioni appropriate basandosi sui miei messaggi.	2
	The chatbot asks specific questions to better understand my requests. Il chatbot fa domande specifiche per capire meglio le mie richieste.	3
Providing helpful information [PHI]	The chatbot provides accurate information. Il chatbot fornisce informazioni accurate.	1
	The chatbot provides helpful information. Il chatbot fornisce informazioni utili.	2
	The chatbot provides information grounded in theory and scientific literature. Il chatbot fornisce informazioni supportate da teorie e letteratura scientifica.	3
Clarity and relevance of responses [CRR]	The chatbot’s responses are clear, and easy to understand. Le risposte del chatbot sono chiare e semplici da capire.	1
	The chatbot’s responses are adequately concise. Le risposte del chatbot sono sufficientemente concise.	2
	The chatbot’s responses are irrelevant to my questions. Le risposte del chatbot non sono pertinenti alle mie domande.	3
Language quality [LQ]	The chatbot uses correct grammar and spelling in its responses. Il chatbot fornisce risposte grammaticalmente e ortograficamente corrette.	1
	The chatbot’s language is appropriate for the context. Il linguaggio del chatbot è appropriato al contesto.	2
	The chatbot’s language style sounds natural Lo stile linguistico del chatbot suona naturale.	3
Trust [T]	I feel safe sharing my personal matters with the chatbot. Mi sento al sicuro nel condividere questioni personali con il chatbot.	1
	I believe that the feedback and the information provided by the chatbot are trustworthy. Credo che i feedback e le informazioni fornite dal chatbot siano affidabili.	2
	I believe the chatbot is transparent about its limitations and capabilities. Credo che il chatbot sia trasparente riguardo ai suoi limiti e alle sue capacità	3
Emotional support [ES]	The chatbot makes me feel heard and understood. Il chatbot mi fa sentire ascoltato e capito.	1
	The chatbot’s responses feel empathetic and supportive. Le risposte del chatbot risultano empatiche e supportive.	2
	The chatbot’s responses can make me feel reassured Le risposte del chatbot sono in grado di farmi sentire rassicurato.	3
Guidance and direction [GD]	The chatbot provides adjusted guidance in coping with my problems. Il chatbot fornisce indicazioni personalizzate per aiutarmi a gestire i miei problemi.	1
	The chatbot encourages me to take positive steps. Il chatbot mi incoraggia a compiere azioni costruttive.	2
	The chatbot helps me set realistic and achievable goals. Il chatbot mi aiuta a stabilire obiettivi realistici e raggiungibili.	3
Memory [M]	The chatbot accurately recalls details from previous conversations. Il chatbot ricorda accuratamente i dettagli delle conversazioni precedenti.	1
	The chatbot maintains consistency by integrating past interactions into current responses. Il chatbot integra coerentemente le interazioni passate nelle risposte.	2
	The chatbot adapts its advice based on information provided in earlier sessions. Il chatbot adatta i suoi consigli in base alle informazioni fornite nelle sessioni precedenti.	3
Overall satisfaction [OS]	I am overall satisfied with the usability of this chatbot. Nel complesso, sono soddisfatto dell’usabilità di questo chatbot.	1
	Overall, I feel that my interactions with the chatbot were worthwhile. Nel complesso, trovo che le mie interazioni con il chatbot siano state proficue.	2
	I am overall satisfied with the support provided by this chatbot Nel complesso, sono soddisfatto del supporto offerto da questo chatbot.	3

.

3.4. Initial Validation

The initial validation of the scale in a real-world setting demonstrated its reliability both globally and across individual dimensions. A total of 49 participants completed the evaluation scale after interacting with the LLM-powered chatbot that delivers the first session of the Self-Help Plus (SH+) intervention. The overall scale exhibited excellent internal consistency, with a Cronbach’s alpha of 0.94, exceeding the generally accepted threshold of 0.70 for reliability [67]. Across the nine dimensions, Cronbach’s alpha values ranged from 0.47 to 0.91, with 8 out of 9 dimensions exceeding the 0.70 threshold (see

Table 3. Summary of inter-item correlations, item-total correlations, and Cronbach’s α values for all the dimensions of the scale.

Dimension	Inter-Item Correlation		Item-Total Correlation		Cronbach’s α
	Mean	Range	Mean	Range
UR	0.42	0.28–0.54	0.50	0.41–0.61	0.68
PHI	0.58	0.44–0.73	0.65	0.55–0.76	0.79
CRR	0.28	0.02–0.71	0.33	0.06–0.54	0.47
LQ	0.40	0.23–0.61	0.47	0.31–0.63	0.63
T	0.55	0.41–0.74	0.63	0.49–0.73	0.78
ES	0.77	0.70–0.86	0.82	0.76–0.88	0.91
GD	0.48	0.33–0.73	0.54	0.38–0.64	0.71
M	0.55	0.45–0.65	0.63	0.55–0.71	0.78
OS	0.75	0.72–0.77	0.80	0.78–0.82	0.90
Overall	N/A	N/A	N/A	N/A	0.94

). These results suggest strong consistency for all the dimensions except for CRR (Clarity and Relevance of Responses), which reached only a Cronbach’s alpha value of 0.47. Further investigation revealed that CRR3, the only reverse-coded item on the scale, was a key contributor to the lower reliability. Despite being recoded for the computation of Cronbach’s alpha, it may have introduced additional cognitive complexity for respondents, potentially affecting the consistency of responses within this dimension. Average item-total correlations across the dimensions ranged from 0.33 to 0.82, with most dimensions showing satisfactory alignment between items and their respective constructs. The Emotional Support (ES) and Overall Satisfaction (OS) dimensions achieved the highest item-total correlations, with means of 0.82 and 0.80, respectively, reflecting strong coherence within these constructs. By contrast, the CRR dimension showed the lowest mean item-total correlation at 0.33, further highlighting the misalignment of CRR3 with the rest of the items in this dimension. Inter-item correlations provided additional insights into the internal structure of the scale. Mean inter-item correlations ranged from 0.28 (CRR) to 0.77 (ES). While most dimensions demonstrated inter-item correlations within the acceptable range (0.20–0.70) [68], ES and OS displayed notably high mean inter-item correlations (0.77 and 0.75, respectively), suggesting a need to investigate potential residual redundancies in these dimensions. These preliminary findings indicate that while the scale overall and most of its dimensions demonstrate acceptable psychometric properties, specific dimensions, such as CRR need further investigation.

4. Discussion

This study aimed to develop a comprehensive evaluation scale for LLM-powered counseling chatbots (CES-LCC) leveraging the domain knowledge of a pool of experts using the eDelphi method. Through two rounds of expert feedback, the scale was refined to address a broad spectrum of aspects related to the evaluation of this type of counseling chatbot. The results, particularly the qualitative feedback obtained using open-ended questions, highlight the importance of a multidisciplinary approach to developing tools that effectively evaluate the different aspects and functionalities of modern digital health solutions. The final version of CES-LCC includes 27 items across nine dimensions and offers a robust framework to assess the unique challenges and capabilities of LLM-powered counseling chatbots. The structured eDelphi process facilitated the identification and integration of critical evaluation dimensions, leading to significant refinements. For instance, the addition of a “Memory” dimension underscores the need to assess chatbots’ abilities to retain and build upon previous interactions, a functionality critical for creating a coherent and personalized user experience with digital mental health interventions. The importance of this dimension is grounded in the scientific literature on the concepts of Memory Support Intervention [69,70] in which information on previous sessions is embedded by the therapist in the ongoing dialogue, session summaries, and skill-building exercises to enhance retention, facilitate continuity, and promote the practical application of therapeutic concepts in the patient’s daily life. In addition, recent research [71] on LLM-powered chatbots with long-term memory features suggests that embedding memories in interactions can improve engagement and user experience while fostering self-disclosure and a sense of familiarity. These findings underscore the significant role of memory in shaping the long-term user experience, distinguishing chatbots with memory capabilities from those without. The exclusion of a “Privacy and security” dimension from the scale instead reflects the need for a focused approach to evaluation in which chatbots and their infrastructural aspects (e.g., production choices like selecting encryption methods, implementing security protocols, adhering to privacy regulations) are evaluated separately. This is in line with ISO/IEC 25010 [72], which distinguishes between different quality characteristics in system and security evaluation. In this standard, security (encompassing attributes such as confidentiality, integrity, and authenticity) is treated as a distinct nonfunctional requirement, separate from usability or functional suitability [73]. The expert panel also emphasized the importance of avoiding anthropomorphic language when assessing trust to avoid misattributing human-like qualities to AI agents. This insight reflects the broader challenge related to both the design of transparent AI agents and the frameworks to assess such systems. Anthropomorphism is often used to increase retention and to promote self-disclosure [74], however using it in the context of mental health poses a great risk of exacerbating maladaptive behaviors and thoughts (e.g., social isolation) [75]. As a result, evaluation methods must address the unique capabilities of AI-driven systems while avoiding the promotion of anthropomorphic views, particularly in contexts where such perspectives could pose significant risks.

4.1. Implications

By addressing gaps in existing methodologies, CES-LCC offers a framework for comprehensively assessing LLM-powered counseling chatbots. Unlike traditional evaluation tools that often focus on single aspects, this scale captures a broader spectrum of dimensions, including emotional support, memory retention, and trustworthiness. This study highlights the potential value and utility of the developed evaluation scale, although its applicability and impact require further validation and exploration. For researchers, the scale provides an integrated tool that can facilitate systematic investigations into the effectiveness of counseling chatbots. By combining technical and relational dimensions, the scale encourages multidisciplinary studies, potentially fostering deeper insights into how these technologies interact with users in complex, emotionally charged scenarios. This could contribute to the development of more sophisticated chatbot designs and the refinement of LLM technologies in therapeutic settings. In practice, the scale may serve as a useful tool for developers, mental health practitioners, and policymakers to evaluate and improve counseling chatbots. Developers could use the scale to identify specific areas for enhancement, ensuring their chatbots meet the demands of the users. Mental health practitioners might find the scale helpful when selecting chatbots to integrate into their services, as it provides a structured way to assess their potential. Finally, policymakers, particularly those involved in healthcare technology regulation, could leverage the scale to establish benchmarks for chatbot performance and safety.

4.2. Limitations and Future Research

While this study provides valuable insights, several limitations must be acknowledged. First of all, the expert sample size was limited, and because the participant pool included only experts from Italy, the generalizability of the findings to broader cultural or professional contexts may be limited. The selection of Italian experts was a strategic choice, as we aimed to develop both the English scale and an Italian variant. Given our familiarity with the Italian landscape, we were able to engage highly relevant professionals whose expertise aligned closely with the study’s objectives. This approach ensured that the input collected was well-informed and contextually grounded. While including experts from multiple countries at this stage could have provided broader insights, it would have also introduced challenges in ensuring the relevance and comparability of expert contributions. To enhance the generalizability of the findings, future research should expand the expert panel internationally, allowing for cross-cultural validation and adaptation of the scale. Additionally, although an initial real-world validation was conducted, it involved filling CES-LCC after a single session delivered by a chatbot to a limited group of users (n = 49), which restricts the robustness of the conclusions regarding the scale’s practical application and reliability. Comprehensive real-world testing with diverse user groups is needed to assess the scale’s reliability and utility across various scenarios. The scale’s development is still in its nascent stages, and its psychometric properties require further investigation. In particular, while factor analysis was not employed in this study, it remains a crucial next step to reinforce the psychometric robustness of the scale. Given the study’s exploratory nature and reliance on expert consensus through the eDelphi method, factor analysis was deferred to a later stage when a larger and more diverse dataset would be available. Incorporating factor analysis in future research will help confirm the scale’s latent structure and enhance its validity, supporting its application across different contexts. Finally, the reliance on the eDelphi method, which depends on subjective expert judgment, introduces potential biases despite efforts to ensure diverse expertise and minimize the influence of individual perspectives. Future iterations should aim to integrate additional methodologies (e.g., factor analysis) to corroborate and enhance the objectivity of the findings.

5. Conclusions

This study presents the CES-LCC, a comprehensive evaluation scale developed to assess the unique challenges posed by evaluating LLM-powered counseling chatbots. Through an iterative eDelphi process involving multidisciplinary experts, the scale captures critical dimensions such as emotional support, trust, memory retention, and overall satisfaction. Initial validation in a real-world setting indicates strong reliability, emphasizing its potential utility for researchers, developers, and practitioners. The scale’s multidimensional approach encourages a holistic assessment of chatbot performance, facilitating the identification of areas for enhancement. Despite its limitations, including the reliance on a geographically restricted expert panel for its development and limited user validation, the CES-LCC represents a significant step forward in standardizing the evaluation of modern counseling chatbots. Future research should focus on broader validation efforts, integrating diverse user perspectives, exploring the scale’s psychometric properties, and examining its applicability in real-world contexts more extensively.