HEUXIVA: A Set of Heuristics for Evaluating User eXperience with Voice Assistants

Abstract

Voice assistants have become increasingly common in everyday devices such as smartphones and smart speakers. Improving their user experience (UX) is crucial to ensuring usability, acceptance, and long-term effectiveness. Heuristic evaluation is a widely used method for UX evaluation due to its efficiency in detecting problems quickly and at low cost. Nonetheless, existing usability/UX heuristics were not designed to address the specific challenges of voice-based interaction, which relies on spoken dialog and auditory feedback. To overcome this limitation, we developed HEUXIVA, a set of 13 heuristics specifically developed for evaluating UX with voice assistants. The proposal was created through a structured methodology and refined in two iterations. We validated HEUXIVA through heuristic evaluations, expert judgment, and user testing. The results offer preliminary but consistent evidence supporting the effectiveness of HEUXIVA in identifying UX issues specific to the voice assistant “Google Nest Mini”. Experts described the heuristics as clear, practical, and easy to use. They also highlighted their usefulness in evaluating interaction features and supporting the overall UX evaluation process. HEUXIVA therefore provides designers, researchers, and practitioners with a specialized tool to improve the quality of voice assistant interfaces and improve user satisfaction.

1. Introduction

As voice assistants become more common in smartphones, smart speakers, and other devices, understanding and improving user experience (UX) can significantly impact their use and effectiveness. UX evaluation can detect usability issues, identify user needs and preferences, and guide the design of more intuitive and effective voice interfaces. Heuristic evaluation is a widely used method for assessing UX with interfaces. It involves a group of UX/usability experts examining a system and evaluating whether its interface adheres to a set of principles known as heuristics [1]. This method is useful and effective for evaluating UX with voice assistants, as it allows for the quick and economical identification of usability problems. Nevertheless, it is crucial to have a set of heuristics that can detect specific issues related to these types of intelligent assistants, such as issues related to voice interaction, clarity and naturalness of the assistant’s responses, ease of use of voice commands, and the assistant’s ability to understand and effectively respond to user requests.

However, traditional UX/usability heuristics may not directly apply to voice interfaces, which rely on spoken language and audio feedback. Therefore, developing specialized heuristics adapted to the specific features and limitations of voice interaction can provide a more accurate and comprehensive evaluation for designers and researchers. For this reason, we proposed HEUXIVA: a set of 13 Heuristics for Evaluating User eXperience with Voice Assistants. The abbreviation HEUXIVA stands for Heuristics for Evaluating User eXperience with Voice Assistants. The acronym derives from the key concepts that define the purpose of the set: H for Heuristics, E for Evaluating, U for User, X for eXperience, V for Voice, and A for Assistants.

We used the methodology proposed by Quiñones et al. [2] to develop HEUXIVA. The proposed set was validated in two iterations through heuristic evaluation, user test, and expert judgments using Google Nest Mini. This allowed us to refine the proposed set and verify its effectiveness in detecting usability/UX problems related to voice assistants. Based on the results obtained, we conclude that, although preliminary, the findings indicate that HEUXIVA is a useful and reliable instrument for evaluating the user experience of voice assistants and that the research is progressing in a promising direction.

The article is organized as follows: Section 2 presents the background; Section 3 presentes the related work; Section 4 details the methodology applied to create HEUXIVA and explains the activities performed to develop, validate, and refine the set; Section 5 describes the validation process and its results; Section 6 presents HEUXIVA: a set of heuristics for evaluating user experience with voice assistants; Section 7 details the discussions; Section 8 presents the limitations of the study; and Section 9 discusses the conclusions and future work.

2. Background

The concepts of voice assistants, user experience and user experience evaluation are presented below. In addition, the related work is discussed.

2.1. Voice Assistants

Virtual assistants, including voice assistants and chatbots, are distinguished by their modes of interaction: voice and text, respectively [3]. These devices leverage artificial intelligence to perform various tasks based on user commands, such as sending emails, making phone calls, providing personalized recommendations, and controlling appliances [4,5,6]. In the literature, virtual assistants are referred to by several names, including Intelligent Personal Assistant (IPA) [4], Conversational Agents [7], and Virtual Personal Assistants (VPA) [8]. For the purposes of this study, the term IPA is used to denote these devices [9].

The characteristics of voice assistants vary depending on the device. Based on the literature review [9,10,11,12,13,14,15,16,17,18], we identified the following main features that describe these intelligent assistants:

• Effective Communication: The interaction between the user and the voice assistant is bidirectional, involving a continuous exchange of information and roles (sender and receiver) [13].
• Effective: Requests and responses are not restricted to a single topic and are coherent with the user’s environment [13,14].
• Activity Management: The voice assistant enables management actions, such as scheduling appointments, alarms, calls, sending messages, translations, among other tasks [14,18].
• Customizable: The device can be adapted according to the user’s preferences or needs, whether it be in language, voice of the assistant, voice commands for switches or smart plugs, news preferences, routines, among others [15,16].
• Multi-user: The device can recognize the voice of other people, making it usable by everyone present in the same location [17].

Additionally, we conducted a formal inspection to detect usability/UX issues with voice assistants (see Section 4.1). The results of this evaluation can be seen in Appendix D. As a result of this evaluation and considering the findings presented by [9,19,20,21], we consider the following features relevant when evaluating voice assistants:

• Security and Privacy: The device has a privacy policy that specifies what data they collect, why they collect it, and how the user can update, manage, export, and delete it.
• Multi-linkable: The device allows linking/integrating other devices and external services/apps and controlling their use, such as smart home devices (controlling lights, appliances, temperature, and air conditioning, etc.), music services, among others.
• Culturizable/Adaptable: The device recognizes/generates expressions and sets of words that cannot be deduced from the meanings of the words forming them, all according to the geographical location of the user.
• Voice Interface: The device provides the corresponding information through the voice interface.
• Guidance and Assistance: The device guides and assists the user with problems related to the use and installation/configuration of it.

2.2. User Experience

According to ISO 9241-10, the user experience (UX) is defined as “a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service” [22]. Several authors have proposed various attributes or factors to describe UX, such as Park et al. [23], Lykke et al. [24], and Morville [25]. To develop HEUXIVA, we selected the factors proposed by Morville [25], as they are concise and easy to consider for evaluating these devices:

• Useful: The product must be useful and satisfy a user’s need.
• Usable: The system must be easy to use and quick to learn.
• Desirable: The design elements should be attractive and interesting to the user to cause appreciation and emotion.
• Findable/locatable: Information should be navigable and easy to find within and outside a system.
• Credible: The image of the company or system must be trustworthy.
• Valuable: The system must provide added value or contribute to the user to satisfy his needs.
• Accessible: The system must be able to adapt to users with some type of disability.

2.3. User Experience Evaluation

UX evaluation methods provide information about a system interface and its users through several techniques, aiming to identify how users feel when interacting with software, devices, or applications. There are different evaluation methods, which can be classified into two main approaches: formative and summative [26,27]. Formative evaluations are conducted to improve the design of an interface, identify usability/UX problems, and detect elements that need improvement. Both qualitative and quantitative methods are used, such as usability inspections, usability tests, and heuristic evaluations [26,27]. On the other hand, summative evaluations are performed to measure whether a system meets expectations using quantitative and qualitative data, such as satisfaction surveys, performance metrics, and A/B tests [26,27]. To develop HEUXIVA, we used the following UX evaluation methods (throughout the stages of the applied methodology):

• Heuristic evaluation: A method in which a group of 3 to 5 evaluators analyze an interface, identifying positive and negative aspects according to a set of rules called heuristics [1].
• User testing (thinking aloud): A method involving representative users who navigate and interact with a system while performing predefined tasks, verbalizing their thoughts and actions aloud [28].
• Expert judgment: A method where UX experts and/or specialists apply their knowledge to review a system’s interface. They navigate the platform and identify elements that could improve or negatively affect the user experience [29,30].

3. Related Work

Langevin et al. [10] developed a set of usability heuristics to guide and evaluate the design of conversational agents, including both chatbots and voice assistants. They adapted Nielsen’s heuristics [31] and worked with experts to establish them. As a result, the authors proposed 11 heuristics, among which the “Context Prevention” and “Reliability” heuristics stand out as domain specific. Although this set includes voice assistants, it does not focus particularly on them, as chatbots are also considered (with a primary focus on evaluating mainly chatbots).

On the other hand, Sanchez-Adame et al. [11] created a set of 5 heuristics for chatbots. These devices, like voice assistants, use natural language processing to interact with users. The proposed heuristics are useful for evaluating chatbots but do not assess voice assistants. Moreover, they focus mainly on evaluating usability (efficiency, effectiveness, satisfaction) and not UX components.

Zwakman et al. [12] conducted research to verify if the System Usability Scale (SUS) [32] remains suitable for assessing usability aspects and end-user experiences. SUS was created to evaluate a system with an interface/GUI, and there are several differences between a voice-based environment and a GUI environment. The results indicated that using SUS might not be the best measure to evaluate the usability of smart assistants, as voice systems are interactive and more like human interaction. Additionally, SUS does not consider features unique to a voice environment, such as sound quality/clarity, ease of understanding, level of immersion, and speech monotony that may cause boredom. Based on the above, the authors adapted the 10 questions of SUS to evaluate voice assistants (VUS, [12,33]).

On the other hand, Cowan et al. [9] identified various problems related to the experiences of occasional users of Intelligent Personal Assistants (IPAs). Their research highlighted the following six main themes: (1) Challenges in supporting hands-free interaction; (2) Performance issues related to user accent and speech recognition overall; (3) Difficulties in integrating with third-party applications, platforms, and systems; (4) Social embarrassment as a barrier to using these devices in public settings; (5) The anthropomorphic nature of IPAs; and (6) Concerns regarding trust, data privacy, transparency, and ownership.

While there are interesting studies and sets of heuristics for evaluating aspects of smart assistants, none of them are directly focused on evaluating the UX with voice assistants. This reaffirms the need to create unique and specific heuristics for this domain that can effectively detect usability/UX problems, as these issues negatively affect users’ perception and interaction with these devices. Voice assistants have made the use of artificial intelligence more accessible, as users have been able to interact more intuitively with these devices. Given that HEUXIVA is a specific set for evaluating voice assistants, it will allow for establishing guidelines for the development and implementation of these technologies, making the interaction increasingly user-friendly.

4. Material and Methods

4.1. Methodology Applied for Developing HEUXIVA

The methodology proposed by Quiñones et al. [2] was used to develop the proposed set, since it establishes a methodical, iterative, and effective work plan using qualitative and quantitative methods. HEUXIVA was established and validated in two iterations through heuristic evaluations, expert judgments, and user testing (see Figure 1. Iterations are marked as “It. N”). In iteration 1, the eight stages of the methodology were conducted, and the first set of heuristics (HVA) was developed and validated. In iteration 2, the last three stages of the methodology were carried out to specify the final set of heuristics (HEUXIVA). Details of each iteration, including inputs, outputs, and activities performed, can be reviewed in Appendix A (iteration 1) and Appendix B (iteration 2). Appendix C presents the set of heuristics for voice assistants created in each iteration. Each set is abbreviated differently for each iteration: HVA (first version, iteration 1), HEUXIVA (second and final version, iteration 2). The iterations conducted are explained below.

Figure 1. Steps and iterations performed to develop HEUXIVA.

4.2. First Iteration: Development Process for HVA

In “Step 1: Exploratory Stage”, a literature review was conducted to obtain information on voice assistants and their features (see Section 2.1), user experience attributes (see Section 2.2), existing heuristic sets and related elements (see Section 2.3). To conduct this literature review, the digital databases Scopus, ScienceDirect, and Google Scholar were considered. With the results obtained, the most relevant documents whose topics were related to the area were analyzed. Although there are several related studies to intelligent assistants, none establishes a specific set for voice assistants. For this reason, studies related to UX in the interaction with these devices and studies that are based on heuristics [10,11] and/or questionnaires of virtual and/or voice assistants [12,33] were selected.

In “Step 2: Experimental Stage”, a formal inspection of a Google Assistant device was performed. The objective was to familiarize the user with the device and detect problems that may negatively affect the user experience. Two researchers from the team used the device for 11 days. The first days were to get to know and adapt to the device, the following days the device was used fluently with the objective of finding possible usability/UX problems. During these days, a total of 13 problems were detected. Scores were assigned on a 4-level scale to the detected problems using the severity scale used in heuristic evaluations, where severity is understood as the level of seriousness of the identified problem [34] (for more details about the problems detected, see Appendix D). Most of the identified problems have a value less than or equal to 2, even so two were identified that stand out as catastrophic problems (with a score of 4). These are “P1: Device ignores user” and “P2: Difficulty initializing device”. The first can generate problems with the user regarding the correct functioning of the device and its value to the user; the second causes frustration in the user and the desire or need to use it is lost.

In “Step 3: Descriptive Stage”, the relevant information obtained from the previous stages was selected. For this, we prioritized the information collected on a 3-level scale (1 being: slightly relevant information and 3: very important information) [2] and this was later organized into 5 categories: information about voice assistants, features of voice assistants, usability/UX attributes, sets of existing heuristics and other related elements, and usability problems of voice assistants (detected in the experimental stage through formal inspection). Finally, according to the prioritization, the following information was selected to develop the heuristics (the details of the selected information can be seen in the Appendix E):

• 5 types of information of voice assistants: 3 definitions of voice assistant [4,7,8], need to create a UX evaluation method for voice assistants, taxonomy of voice assistants.
• 10 voice assistants’ features: effective communication, effective, activity management, customizable, multi-user, security and privacy, multi-linkable, culturizable/adaptable [9], voice interface, guidance and assistance.
• 3 usability attributes: effectiveness, efficiency, and satisfaction [35].
• 7 UX attributes: useful, desirable, usable, findable/locatable, credible, and valuable, and learning capacity [25,36].
• 5 sets of existing heuristics or related elements: Nielsen [31], Langevin et al. [10], Sanchez-Adame et al. [11], Zwakman et al. [12,33] (VUS), Nowacki and Gordeeva [37].
• Voice Assistant Usability Issues: formal inspection made in Stage 2, Infrequent Users’ Experiences of Intelligent Personal Assistants by Cowan [9].

While usability is conceptually part of UX, in this study we considered its attributes (effectiveness, efficiency, and satisfaction [35]) as a distinct group of evaluative dimensions. This decision was made to emphasize their relevance in the design and validation of the proposed heuristics. Separating these dimensions allowed us to ensure a balanced coverage of both functional quality (usability) and experiential quality (UX). This distinction also facilitated the mapping of heuristics to specific measurable criteria during the development process.

In “Step 4: Correlational Stage”, the correlation between voice assistant features, usability/UX attributes, heuristic sets, associated usability issues (detected in the experimental stage) and items from the Voice Usability Seal (VUS) questionnaire was performed. During this process, it was discovered that there is no heuristic that fully covers the usability/UX attributes related to the features, these are partially covered. However, there are several heuristics and information that allowed the new set to be generated (this correlation process can be seen in Appendix F).

In “Step 5: Selection Stage”, the existing heuristics were selected based on the information collected in Stage 4. The following actions were determined: adapt, keep or eliminate. 1 heuristic was kept, and 39 heuristics were adapted. Of these 40 heuristics, 12 were identified as useful, 11 as important and 17 as critical. A total of 11 heuristics were eliminated because they are not related to the topic of voice assistants or are fulfilled by another selected heuristic. Even though there are no heuristics that fully cover the characteristics of voice assistants, mixing several proposals and adding more complete specifications helped to generate the new set (details on the selection process can be seen in Appendix G).

In “Step 6: Specification Stage”, the preliminary set of heuristics for voice assistants was proposed (see Appendix C). This version contains 12 heuristics which were documented using the template proposed in the methodology [2] including the following information: ID, name, definition, covered voice assistant feature, covered usability/UX attribute, and related heuristics. In “Step 7: Validation Stage”, the first version of heuristics (HVA) was validated through heuristic evaluation and expert judgment. Details of these validations can be seen in more detail in Section 5.1 and Section 5.2, respectively. In “Step 8: Refinement Stage”, the heuristics were refined based on the feedback obtained in Stage 7 as follows (for more details on the refinement see Appendix H):

• 12 heuristics need to be refined mainly by improving their checklists and definition.
• A new heuristic needs to be added to cover the user response aspects.
• It was decided to carry out a second iteration repeating the last three steps of the methodology.

4.3. Second Iteration: Development Process for HEUXIVA

In the second iteration, the last three stages of the methodology were performed again, starting from “Step 6: Specification Stage”. In this step, a second version of the set of heuristics that evaluate voice assistants (HEUXIVA) was proposed. The refinement made in the first iteration was considered along with the information matched in “Step 4: Correlational Stage” in the first iteration (see Appendix C).

In “Step 7: Validation Stage”, the second version of the set of heuristics was validated through expert judgment and user testing. Details of the validations performed can be reviewed in Section 5.3 and Section 5.4, respectively. Finally, in “Step 8: Refinement Stage”, 10 heuristics were modified based on the expert judgment and user testing performed in Stage 7. The final version of HEUXIVA is presented in Section 6.

5. Results

5.1. Results Obtained in the Iteration 1: Validation Through Heuristic Evaluation

We conducted a heuristic evaluation to validate the effectiveness of the first version of the proposed heuristics (HVA). For this purpose, we defined a control group, composed of 3 evaluators who used the existing set of heuristics for “conversational agents” (CAH) [10]; and an experimental group, composed of 3 evaluators who used the new set of proposed heuristics in its first version: HVA. Both groups conducted the evaluation using Google’s voice assistant Nest Mini [38]. Each group was composed of evaluators with a similar level of experience in interacting with voice assistants and performing heuristic evaluations. All were computer engineers with formal training in UX research and professional experience ranging from three to five years in the field of UX. Their ages ranged from 30 to 35 years, and the group consisted of four men and two women.

To evaluate the effectiveness of HVA, we used the criteria defined in the methodology applied [2] (the explanation of the formulas and the calculation of each criterion applied to evaluate effectiveness can be found in Appendix I). The results obtained by the control and experimental groups were compared in terms of:

• Numbers of correct and incorrect associations of problems to heuristics
• Number of usability/UX problems identified
• Number of specific usability/UX problems identified
• Number of identified usability/UX problems that qualify as more severe (how catastrophic the usability/UX problem detected is)
• Number of identified usability/UX problems that qualify as more critical (how severe and frequent the problem detected is)

Table 1 shows the results obtained in the heuristic evaluations performed by the experimental and control groups. In addition, the effectiveness of HVA in terms of the five criteria is shown. As shown in Table 1, HVA performed better than CAH on two of the five criteria. HVA detected more usability/UX problems than CAH and detected more specific problems related to voice assistants (ESS1 > ESS2). However, CAH had a higher percentage of correct associations than HVA (CA1 < CA2), and CAH detected more severe (ESV1 < ESV2) and critical (ESC1 < ESC2) problems than HVA. The above indicates that HVA requires refinement in terms of its specification both to improve clarity and increase the number of correct associations of problems to heuristics, as well as to increase the number of severe and critical problems detected (related to voice assistants).

Table 1. Effectiveness of HVA (first iteration).

	Experimental Group	Control Group	Observations
Number of evaluators	3	3	-
Set of heuristics used	Heuristics for evaluating voice assistants (HVA)	Conversational agents’ heuristics (CAH) [10]	-
Amount of heuristics	12	11	-
Total of problems identified	31	26	-
Total of the correct associations	15	17	-
Total of the incorrect associations	16	9	-
Percentage of the correct associations (CA)	CA1 = 48.8%	CA2 = 65.38%	CA1 < CA2, it is concluded that the control set performs better than the proposed set, as it has a higher percentage of correct associations (HVA requires refinement).
Percentage of the incorrect associations (IA)	IA1 = 51.62%	IA2 = 34.62%	IA1 > IA2, it is concluded that the control set is better, since the proposed set has a higher percentage of incorrect associations (HVA requires refinement).
Problems identified by both groups (P1)	7		(P2) identified more problems than (P3), it is concluded that the proposed set performs better than the control set.
Problems identified by the experimental group (P2)	24	-
Problems identified by the control group (P3)	-	19
Number of specific problems identified	19	14	-
Effectiveness in terms of number of specific problems identified (ESS)	ESS1 = 61.29%	ESS2 = 53.84%	ESS1 > ESS2, the proposed set identified more specific problems than the control set, then it works better.
Number of problems identified and qualified with a severity greater than 2	13	19	-
Effectiveness in terms of number of problems identified and qualified with a severity greater than 2 (ESV)	ESV1 = 41.93%	ESV2 = 70.07%	ESV1 < ESV2, it is concluded that the control set is better, since it finds more problems rated as severe than the proposed set (HVA requires refinement).
Number of problems identified and qualified with a criticality greater than 4	15	22	-
Effectiveness in terms of number of problems identified and qualified with a criticality greater than 4 (ESC)	ESC1 = 48.38%	ESC2 = 84.61%	ESC1 < ESC2, it is concluded that the control set encounters more problems rated as critical than the proposed set (HVA requires refinement).

5.2. Results Obtained in Iteration 1: Validation Through Expert Judgment

In addition to conducting a heuristic evaluation, in the first iteration we also conducted a survey with a group of three experts to evaluate HVA. These experts were those who participated as evaluators in the experimental group in the heuristic evaluation presented in Section 5.1. The survey was designed to obtain the evaluators’ perception of HVA along four dimensions: D1—Utility, D2—Clarity, D3—Ease of use, and D4—Need for a checklist. We used a five-point Likert-type scale (1 represents the worst rating and 5 the best for dimensions D1, D2, and D3. For dimension D4, a rating of 1 indicates a complete need for additional elements, while 5 signifies no need). Table 2 shows the average values obtained for each dimension per heuristic.

Table 2. Average perception scores for HVA set in the four evaluated dimensions (first iteration).

Heuristic	D1—Utility	D2—Clarity	D3—Ease of Use	D4—Need of Additional Elements
HVA1: System Status Visibility	5.0	5.0	4.0	4.3
HVA2: Feedback and Help Users Prevent Errors	4.6	4.6	3.6	4.3
HVA3: Brevity and Relevance of Information	4.0	4.6	2.6	4.6
HVA4: Natural Communication	4.6	4.0	3.3	5.0
HVA5: Match Between the System and the Real World	4.3	4.3	4.3	4.3
HVA6: Consistent Voice Interface	3.3	4.6	4.0	3.6
HVA7: User Control and Freedom	4.3	4.6	3.3	3.6
HVA8: Flexibility and Personalization	4.3	4.0	4.0	4.6
HVA9: Help Users Recognize, Diagnose, and Fix Errors	4.3	4.3	4.3	4.0
HVA10: System Guidance and Capabilities	4.6	4.3	4.6	4.0
HVA11: Reliability and Data Privacy	4.0	4.3	2.6	5.0
HVA12: Guides and Documentation	4.0	4.3	3.3	4.6
Average per dimension	4.3	4.4	3.7	4.4

Regarding dimension “D1—Utility”, the evaluators perceived all heuristics useful for evaluating voice assistants, except for the heuristic “HVA6: Consistent Voice Interface” (rated 3.3), indicating a need to review its utility by either improving its specification or removing it from the set. Nevertheless, all heuristics were perceived as clear by the evaluators (dimension “D2—Clarity”), with all ratings being 4.0 or higher. However, in terms of “dimension D3—Ease of use”, 6 out of the 12 heuristics were perceived by the evaluators as difficult to use in practice for detecting usability/UX issues in voice assistants (with ratings of 3.6 or lower), particularly the heuristics “HVA3: Brevity and Relevance of Information” and “HVA11: Reliability and Data Privacy” (both rated 2.6). This indicates a need to enhance the specification of these heuristics, either by adding more detail or improving their wording.

Finally, concerning dimension “D4—Need of additional elements”, the evaluators considered that 2 out of the 12 heuristics should incorporate additional information to improve their specification, the heuristics “HVA6: Consistent Voice Interface” and “HVA7: User Control and Freedom” (both rated 3.6). Both were perceived as useless and difficult to use, respectively, so their specifications could be significantly improved by incorporating additional elements. Based on the results obtained both from the heuristic evaluation and expert judgment, the heuristics were refined in the second iteration (step 6: specification stage).

5.3. Results Obtained in Iteration 2: Validation Through Expert Judgment

In the second iteration, we applied another survey to eight experts to validate the refined set of heuristics: HEUXIVA. We searched for experts through LinkedIn and contacted them via email. The experts had medium-to-high experience using heuristic sets and conducting heuristic evaluations. Specifically, three experts had high experience (more than 6 evaluations conducted); four experts had medium experience (4 to 5 evaluations), and one had low experience (3 evaluations conducted).

The survey had the same design as the one used in the first iteration (see Section 5.2), but this time it focused on evaluating the new heuristics proposal (HEUXIVA). The objective was to gather the evaluators’ perceptions of HEUXIVA along four dimensions: D1—Utility, D2—Clarity, D3—Ease of use, and D4—Need for a checklist. We used once again a five-point Likert-type scale (1 represents the worst rating and 5 the best for dimensions D1, D2, and D3. For dimension D4, a rating of 1 indicates a complete need for additional elements, while 5 signifies no need). Table 3 shows the average values obtained for each dimension per heuristic.

Table 3. Average perception scores for HEUXIVA set in the four evaluated dimensions (second iteration).

Heuristic	D1—Utility	D2—Clarity	D3—Ease of Use	D4—Need of Additional Elements
HEUXIVA1: System Status Visibility	5.0	4.9	4.8	5.0
HEUXIVA2: System Guidance and Capabilities	4.4	4.1	4.1	4.5
HEUXIVA3: Effective and Fluid Communication	4.6	4.3	4.3	4.5
HEUXIVA4: Environment Match Between Assistant and User Language	4.5	4.8	4.1	4.8
HEUXIVA5: Information Accuracy	4.4	4.1	3.9	4.9
HEUXIVA6: User Control and Freedom	5.0	4.0	4.8	4.9
HEUXIVA7: Consistent Voice Interface	4.3	4.3	3.9	4.8
HEUXIVA8: Voice Shortcuts. Flexibility and Personalization	4.5	4.3	4.0	4.8
HEUXIVA9: Error Prevention	4.6	4.8	4.4	4.6
HEUXIVA10: Help Users Recognize, Diagnose, and Fix Errors	4.9	5.0	4.9	4.9
HEUXIVA11: Data Privacy	4.4	4.9	4.3	5.0
HEUXIVA12: Voice Assistant Reliability	4.4	4.3	4.0	4.5
HEUXIVA13: Guides and Documentation	3.8	5.0	4.9	4.5
Average per dimension	4.5	4.5	4.3	4.7

As shown in Table 3, it is noticeable that most of the heuristics were well perceived by the experts in all four dimensions. Regarding dimension “D1—Utility”, 12 out of 13 heuristics received a rating above 4.3, indicating that they were perceived as very useful. Only the heuristic “HEUXIVA13: Guides and Documentation” received a “neutral” rating (3.8, “moderately useful”), suggesting that it could be refined to ensure it is considered useful for evaluating voice assistants. For the dimensions “D2—Clarity” and “D4—Need of additional elements”, all heuristics received ratings above 4.0, indicating that they were perceived as clear (easy to read) and do not require additional information for understanding and using them to detect usability/UX problems. Finally, concerning dimension “D3—Ease of Use”, 11 out of 13 heuristics were perceived as easy to use (with ratings above 4.0), except for the heuristics “HEUXIVA5: Information Accuracy” and “HEUXIVA7: Consistent Voice Interface”, which received a rating close to “neutral” (3.9), being perceived as “moderately easy to use”. This suggests that further improvements could be made to their specifications.

Compared to the results obtained in the expert judgment of the first iteration (see Section 5.2), we concluded that the specification of the heuristics has improved for all dimensions (see Table 4), particularly noting the positive enhancements in terms of “utility”, “ease of use”, and “need for additional elements” for the heuristics “HEUXIVA6: User Control and Freedom” and “HEUXIVA7: Consistent Voice Interface” (see Table 5).

Table 4. Comparison of results obtained in the expert judgment in the first and second iteration.

	D1—Utility	D2—Clarity	D3—Ease of Use	D4—Need of Additional Elements
Average first iteration	4.3	4.4	3.7	4.4
Average second iteration	4.5	4.5	4.3	4.7

Table 5. Improvements in the perception of HEUXIVA6 and HEUXIVA7 in the expert judgment of iteration 2.

Heuristic	ID	Iteration	D1—Utility	D2—Clarity	D3—Ease of Use	D4—Need of Additional Elements
User Control and Freedom	HV7	1º It.	4.3	4.6	3.3	3.6
	HEUXIVA6	2º It.	5.0	4.0	4.8	4.9
Consistent Voice Interface	HVA6	1º It.	3.3	4.6	4.0	3.6
	HEUXIVA7	2º It.	4.3	.3	3.9	4.8

5.4. Results Obtained in Iteration 2: Validation Through User Testing

We conducted a user test to: (1) verify whether the most severe and critical problems identified by evaluators in the heuristic evaluation conducted in iteration 1 (see Section 5.1) are perceived in the same way by users; and (2) identify usability/UX issues that arise during user interaction with a voice assistant and verify if these issues are covered by HEUXIVA (i.e., to check if it is possible to identify these problems detected during a user test using HEUXIVA). The user test was a thinking aloud type, moderated by the authors and synchronous.

5.4.1. User Test Design

The user test consisted of three parts (pre-test, test, and post-test). The first part (or pre-test) included an individual questionnaire with demographic questions to understand the participant’s profile and their experience using voice assistants, as well as a confidentiality agreement. The second part (test) involved a scenario with 8 tasks that participants had to perform individually (see Table 6). Additionally, during the test, participants were required to verbally express their opinions, experiences, emotions, and comments about the tasks and the use of the voice assistant. Finally, the third part (post-test) included a questionnaire to evaluate the participants’ perceptions and experiences using the voice assistant. For the tests, the Google Nest Mini voice assistant was used.

Table 6. Thinking aloud user test quantitative and qualitative results (second iteration).

Task (T)	Percentage of Task Fulfillment	Average Time	Observations	Most Expressed Emotions	Usability/UX Problems Related (P)	Heuristic Related (HEUXIVA)
T1: Make a call	100%	79.6 s	All users performed the task correctly. Users appreciated that their requests were carried out quickly and efficiently. It was evident that there is a need for the device to communicate to the user through voice, sound, and/or light that it is performing an action.	Neutral (41.7%) Happiness (33.3%)	P1: The user forgets the activation word P2: The device does not understand what the user says	P1 is covered by HEUXIVA6 P2 is covered by HEUXIVA4
T2: Check available flights	100%	131 s	All users performed the task correctly. Users become confused when they realize that sometimes the device cannot understand their requests. It is necessary to reconsider whether the activation command is too complex for users.	Irritation (41.7%) and Confusion (33.3%)	P3: The device does not effectively communicate the error P4: The user forgets the activation word	P3 is covered by HEUXIVA10 P4 is covered by HEUXIVA6
T3: Speak colloquially with the device	100%	77.8 s	All users performed the task correctly. Users expected the device to provide a response that matches their request. If the device performs an activity or delivers a response different from what was requested, users indicated that they tend to doubt both themselves and the device.	Neutral (41.7%) and Happiness (33.3%)	P5: The device does not respond immediately after the request is completed P6: The device provides incoherent and unrelated responses to what the user requested P7: The device does not understand the user’s idiolect	P5 is covered by HEUXIVA3 P6 is covered by HEUXIVA5 P7 is covered by HEUXIVA4
T4: Make queries in the area/field of History	100%	122.3 s	All users performed the task correctly. Based on user feedback, it highlighted the importance of the device communicating in the user’s language and providing information in an understandable manner.	Neutral (33.3%) and Happiness (33.3%)	P8: The device provides extensive and confusing information P9: The device does not follow instructions	P8 is covered by HEUXIVA5 P9 is covered by HEUXIVA1
T5: Set an alarm	100%	118.8 s	All users performed the task correctly. Users appreciated that the voice assistant responded to their requests; however, they became confused when they made a request, and the device did not carry out the specified action.	Happiness (41.7%)	P10: The device provides extensive information P11: The device interrupts the user while they are giving instructions	P10 is covered by HEUXIVA5 P11 is covered by HEUXIVA3
T6: Delete or edit an alarm	100%	85.3 s	All users performed the task correctly. Some users were confused and annoyed when they noticed that the device did not properly carry out the request they had just made.	Confusion (41.7%) and Irritation (33.3%)	P12: The device does not allow editing of an instruction P13: The device performs a function different from what was requested P14: The device does not effectively communicate the error P15: The device ignores the user P16: The device does not distinguish commands from questions	P12 is covered by HEUXIVA6 P13 is covered by HEUXIVA12 P14 is covered by HEUXIVA10 P15 is covered by HEUXIVA1 P16 is covered by HEUXIVA4
T7: Customize assistant attributes	100%	79.1 s	All users performed the task correctly. Users expected the voice assistant to allow them to perform the same actions they do when interacting with their mobile phone. This surprised some users when the device redirected them to the mobile interface.	Irritation (33%) and Neutral (33%)	P17: The device requests manual configurations to be made P18: The device provides lengthy instructions	P17 is covered by HEUXIVA7 P18 is covered by HEUXIVA5
T8: Find device	91.66% (11 of 12)	168.2 s	Most users completed the task. Users showed annoyance and/or frustration when they noticed that the device was not following their instructions, was delivering incorrect responses, and was also ignoring them.	Confusion (41.7%) and Irritation (33.3%)	P19: The device ignores the user (does not perform or respond to requests) P20: The device performs a function different from what the user requested	P19 is covered by HEUXIVA1 P20 is covered by HEUXIVA12

5.4.2. Participant Selection

Twelve users participated in the test, aged between 22 and 28 years. Four of the participants had never used a voice assistant before (inexperienced users), four had used a voice assistant at least once (medium-experienced users), and four used them daily (highly experienced users). We decided to seek three different user profiles (inexperienced or novice, medium experienced, and highly experienced) to obtain representative results and visualize how users interact with this type of device.

5.4.3. Results Obtained

Based on verbal comments made by users during the execution of tasks and the responses provided by users in the post-test, several usability/UX issues were identified and documented. Table 6 shows the tasks performed by the users in the thinking-aloud test and their results. Based on the users’ performance of the tasks, 20 usability/UX problems were identified. We reviewed whether HEUXIVA allows the detection of the identified problems (P1 to P20), concluding that HEUXIVA covers all problems detected in the test (see Table 6, last column). This allows to determine that the proposed set is effective in detecting usability/UX issues related to voice assistants.

As shown in Table 6, users completed all tasks with average times ranging from 78 to 168 s, reflecting a good performance for simple actions and greater effort for complex tasks. The most expressed emotions (neutral, happiness, confusion, and irritation) suggest that positive experiences were linked to successful and fluent interactions, while negative emotions appeared when the assistant failed to interpret or execute commands correctly.

On the other hand, it can be observed that the problems detected during the user test were related to the following 8 heuristics: HEUXIVA1, HEUXIVA3, HEUXIVA4, HEUXIVA5, HEUXIVA6, HEUXIVA7, HEUXIVA10, and HEUXIVA12 (see Table 7). Of these 8 heuristics, 5 were proposed to identify specific usability/UX problems directly related to voice assistants (HEUXIVA3, HEUXIVA4, HEUXIVA5, HEUXIVA7, and HEUXIVA12). Based on the results obtained in the user test, it is possible to highlight the utility of the HEUXIVA set, as several specific usability/UX problems were identified while the users were using the voice assistant and completing the tasks (12 specific problems detected, see Table 7).

Table 7. Problems detected in user testing and the related heuristics.

Heuristic	Number of Problems	Problems Related
HEUXIVA5: Information accuracy	4	− P6: The device provides incoherent and unrelated responses to what the user requested − P8: The device provides extensive and confusing information − P10: The device provides extensive information − P18: The device provides lengthy instructions
HEUXIVA6: User control and freedom	3	− P1: The user forgets the activation word − P4: The user forgets the activation word − P12: The device does not allow editing of an instruction
HEUXIVA4: Environment match between assistant and user language	3	− P2: The device does not understand what the user says − P7: The device does not understand the user’s idiolect − P16: The device does not distinguish commands from questions
HEUXIVA1: System status visibility	3	− P9: The device does not follow instructions − P15: The device ignores the user − P19: The device ignores the user (does not perform or respond to requests)
HEUXIVA3: Effective and fluid communication	2	− P5: The device does not respond immediately after the request is completed − P11: The device interrupts the user while they are giving instructions
HEUXIVA10: Help users recognize, diagnose, and fix errors	2	− P3: The device does not effectively communicate the error − P14: The device does not effectively communicate the error
HEUXIVA12: Voice assistant reliability	2	− P13: The device performs a function different from what was requested − P20: The device performs a function different from what the user requested
HEUXIVA7: Consistent voice interface	1	− P17: The device requests manual configurations to be made

These findings offer preliminary but consistent evidence supporting the effectiveness of HEUXIVA in identifying UX issues specific to voice assistants.

6. HEUXIVA: Heuristics for Evaluating User eXperience with Voice Assistants

Based on the iterations and validation described in the previous sections, the HEUXIVA set was refined and improved. We proposed a total of 13 heuristics that can be used to evaluate the user experience of voice assistants. Of the proposed heuristics, 7 are new and are defined to detect specific problems of voice assistants (HEUXIVA3, HEUXIVA4, HEUXIVA5, HEUXIVA7, HEUXIVA8, HEUXIVA11, and HEUXIVA12). These heuristics are presented in Table 8, including: ID, name, description, the voice assistant features evaluated with the heuristic; and the UX attributes evaluated with the heuristic.

Table 8. HEUXIVA: a set of Heuristics for Evaluating the User eXperience with Voice Assistants.

ID	Name	Description	Voice Assistant Feature	Usability/UX Attribute
HEUXIVA1	System status visibility	The device must indicate to the user via voice, sound or light every action that is being performed.	Effective communication, Voice interface, Activity management	Effectiveness, Efficiency, Useful, Valuable, Satisfaction
HEUXIVA2	System guidance and capabilities	The device must guide the user through dialog and activities using words that the user recognizes (and does not increase their cognitive abilities). It should also clarify in a simple way its capabilities.	Culturizable/adaptable, Voice interface, Effective communication, Activity management	Useful, Effectiveness, Wearable, Satisfaction, Desirable, Valuable
HEUXIVA3	Effective and fluid communication	The device must adapt to the context and situations that arise in the conversation, as well as remembering previous requests and conversations with the user.	Effective communication, Effective, Multi-user	Efficiency, Effectiveness, Useful, Wearable
HEUXIVA4	Environment match between assistant and user language	The device must understand the user’s particular way of speaking, in addition to interacting in their language with words, phrases and concepts familiar to the user.	Culturizable/adaptable, Multi-user, Voice interface	Useful, Effectiveness, Valuable, Desirable
HEUXIVA5	Information accuracy	The responses delivered by the device must be relevant, brief and according to what is requested by the user. Similarly, the device must provide truthful information during interaction with the user.	Effective communication, Effective, Voice interface	Effectiveness, Efficiency, Useful, Valuable, Satisfaction
HEUXIVA6	User control and freedom	The device allows the user to perform, redo, and undo actions or requests.	Activity management, Effective communication, Customizable	Credible, Valuable, Satisfaction, Learning capacity, Useful
HEUXIVA7	Consistent voice interface	The device must be able to provide information through voice and being consistent in its personality.	Effective communication, Voice interface, Culturizable/adaptable, Activity management	Satisfaction, Credible, Desirable, Useful, Valuable
HEUXIVA8	Voice shortcuts, flexibility and personalization	The device should answer depending on the environment in which the user is located, providing shortcuts according to the context, allowing customization and adapting according to the needs of the user.	Customizable, Multi-user, Multi-linkable	Effectiveness, Efficiency, Satisfaction, Usable, Learning capacity
HEUXIVA9	Error prevention	The device must provide the necessary information to warn the user when an error is about to occur.	Effective communication, Voice interface	Effectiveness, Efficiency, Useful
HEUXIVA10	Help users recognize, diagnose, and fix errors	Error messages should be expressed in simple language (not codes), accurately indicate the problem, and constructively suggest a solution that mostly uses voice commands or actions.	Culturizable/adaptable, Voice interface, Multi-linkable	Valuable, Useful, Effectiveness, Efficiency
HEUXIVA11	Data privacy	The device must inform the user about the privacy and use of personal data. Likewise, it must grant the possibility of rejecting the collection and analysis of their data, thus being transparent and truthful with the user.	Security and privacy, Activity management	Valuable, Satisfaction, Credible
HEUXIVA12	Voice assistant reliability	Reliability must be transmitted through the behavior of the device both in interaction with the user and when the user is inactive.	Customizable, Security and privacy	Valuable, Credible
HEUXIVA13	Guides and documentation	The device must provide simple and comprehensive physical or electronic documentation of the internal and external workings of the device, either through a request from the user or external search.	Guidance and assistance	Findable/locatable, Valuable, Useful, Effectiveness, Satisfaction

In addition, Appendix J presents each heuristic in detail using the template specified in the methodology applied [2]. Each heuristic is presented in a table containing: ID, name, definition, explanation, priority (how important the heuristic is: critical, important, or useful), usability and UX attributes evaluated with the heuristic, voice assistant features evaluated with the heuristic, set of heuristics related, a checklist, and a compliance example and non-compliance example.

A Supplementary Excel File (S1) is provided to support the practical application of HEUXIVA. This material includes five sheets: (1) a brief description of the 13 heuristics; (2) an extended description of each heuristic; (3) the corresponding checklist items for each one; (4) examples of usability/UX problems that can be identified using HEUXIVA and guidance on how to document them; and (5) a blank template for recording problems during heuristic evaluations. This resource aims to facilitate reproducibility and assist researchers and practitioners in applying the HEUXIVA.

7. Discussions

7.1. About the Results Obtained in Validation Stage (First and Second Iteration)

We performed four experiments to validate HEUXIVA, a heuristic evaluation and expert judgment in the first iteration, and another expert judgment and user testing in the second iteration. As shown in Table 1, the results from the first iteration demonstrate that the initial set of heuristics (HVA) achieved reasonable levels of effectiveness across the applied criteria. Although some variability was observed among evaluators, the findings show that the first version of heuristics permits to identify relevant usability/UX problems related to voice assistants. These early outcomes provided the empirical foundation for refining the heuristics into the final HEUXIVA set.

On the other hand, as shown in Table 4, the comparison of expert judgment results between the first and second iterations highlights the gradual refinement of HEUXIVA. Experts reported improvements in clarity, relevance, and applicability, particularly for those heuristics addressing user control and freedom, and consistent voice interface. The observed increase in agreement among evaluators suggests that the iterative process was effective in reducing ambiguity and improving the understanding of each heuristic’s purpose. Finally, the findings presented in Table 6 integrate quantitative and qualitative data from the thinking-aloud user test conducted during the second iteration. The results support the practical usefulness of HEUXIVA by demonstrating that the heuristics cover the types of usability/UX problems encountered by real users when interacting with voice assistants.

Although the current validations are preliminary, the results demonstrate good progress in the refinement of HEUXIVA through multiple iterations and experiments. The integration of expert and user perspectives provides evidence that the proposed set is useful and applicable for evaluating UX in voice assistants.

7.2. Comparative Analysis with Existing Heuristics and Evaluation Methods

To contextualize the contribution of HEUXIVA, Table 9 compares the proposed set with existing studies related to voice assistants’ evaluation, including the heuristics proposed by Langevin et al. [10] and Sánchez-Adame et al. [11], the Voice Usability Scale (VUS) proposed by Zwakman et al. [12], ergonomics criteria for voice user interface proposed by Nowacki and Gordeeva [37], and usability problems related to intelligent personal assistants (IPAs) identified by Cowan et al. [9]. While these previous studies represent important advances, most of them primarily address usability aspects and do not fully integrate user experience (UX) dimensions.

Table 9. Comparison between studies related to voice assistants.

Study	Domain	Description	Number of Elements	Validation	Limitations
Nielsen heuristics (1990) [1,31]	General desktop applications	Set of heuristics. Focus on usability.	10 heuristics	Expert review, heuristic evaluation.	Not specific to voice assistants, limited to usability.
Cowan et al. (2017) [9]	Intelligent personal assistants (IPAs)	6 main areas related to usability/UX problems. Focus on user experience.	6 key themes	No reported	Only qualitative analysis, does not propose heuristics.
Langevin et al. (2021) [10]	Conversational agents	Set of heuristics, adapted from Nielsen [31]. Focus on usability.	11 heuristics	Expert review, heuristic evaluation.	Not specific to voice assistants; limited to usability.
Sánchez-Adame et al. (2021) [11]	Chatbots	Set of heuristics. Focus on usability.	5 heuristics	Expert review, heuristic evaluation.	Only for text-based devices, limited to usability.
Zwakman et al. (2020) [12]	Voice assistants	Survey (scale), adapted from SUS [32]. Focus on usability.	10 items	Quantitative validation (exploratory factor analysis).	Does not propose heuristics; limited to perceived usability.
Nowacki and Gordeeva [37]	Voice user interface (VUIs)	Ergonomic criteria, based on [31,39]. Focus on usability and ergonomics.	8 criteria and 20 sub-criteria	Preliminary user testing	Preliminary validation, does not propose heuristics, limited to ergonomics.
HEUXIVA	Voice assistants	Set of heuristics, based on [9,10,11,12,31,37]. Focus on user experience.	13 heuristics	Heuristic evaluation, expert judgment, user testing.	Preliminary validation scope (single device).

Existing sets of heuristics are focused on chatbots and conversational agents, rather than voice assistants specifically. The sets by Langevin et al. [10] and Sánchez-Adame et al. [11] adapt Nielsen’s general usability principles but overlook essential aspects of voice interaction. Cowan et al. [9] provides an empirical characterization of user challenges with IPAs, highlighting several usability/UX problems. However, their study focuses on describing user problems rather than translating them into heuristics. The ergonomic criteria by Nowacki and Gordeeva [37] provide valuable guidance for voice user interfaces (VUIs) but lack empirical validation. Finally, the Voice Usability Scale (VUS) by Zwakman et al. [12] offers a quantitative perspective on usability through a concise ten-item survey adapting SUS [32] but does not provide heuristics or focus on user experience.

In contrast, HEUXIVA integrates insights from these works while addressing their main gaps. It includes usability and UX perspectives, incorporates features unique to voice assistants (such as effective communication, voice interface, guidance and assistance, adaptation, among others), and introduces practical checklists to guide evaluations. Moreover, HEUXIVA was developed through a structured, iterative methodology that combines literature synthesis, heuristic evaluation, expert judgment, and user testing, resulting in a comprehensive set. Although the current validation of HEUXIVA is preliminary (limited to a single device and small participant samples) the comparative analysis indicates that HEUXIVA advances the evaluation of voice assistants by addressing specific interaction issues not captured by existing heuristic sets. This early validation nonetheless provides promising evidence of its potential to guide more comprehensive and context-aware UX evaluations in future studies.

7.3. Novel Contributions and Creation of New Heuristics

Of the HEUXIVA set, six heuristics were adapted from existing proposals (Nielsen [31], Langevin et al. [10], Sánchez-Adame et al. [11], Nowacki and Gordeeva [37]). This adaptation, however, was not a direct translation but part of a systematic integration process defined in the correlational and selection stages of the applied methodology. During these stages, each heuristic was evaluated for its relevance to voice-based interaction, its coverage of UX attributes, and its ability to identify usability/UX problems.

Through this process, we also defined new heuristics. Specifically, seven heuristics (HEUXIVA3, HEUXIVA4, HEUXIVA5, HEUXIVA7, HEUXIVA8, HEUXIVA11, and HEUXIVA12) are considered entirely novel, as the aspects they address were only partially covered by existing proposals. These heuristics go beyond traditional usability/UX principles by integrating elements that are unique to voice assistants, such as conversational fluidity, linguistic adaptability, information accuracy, personalization, privacy transparency, and reliability in autonomous interactions. HEUXIVA extends the conceptual and practical boundaries of prior heuristic sets, offering a more comprehensive instrument for evaluating user experience in voice assistants. Table 10 shows the origin of each HEUXIVA heuristic and its contribution to UX evaluation.

Table 10. Origin of each HEUXIVA heuristic and its contribution to UX evaluation.

ID	Name	Type	Origin	Novel Aspect Introduced for UX Evaluation
HEUXIVA1	System status visibility	Adapted heuristic	Heuristics: Nielsen and Langevin et al.	Focus on feedback (voice, light, sound)
HEUXIVA2	System guidance and capabilities	Adapted heuristic	Heuristics: Nielsen and Langevin et al.	Guidance through dialog and capability explanation
HEUXIVA3	Effective and fluid communication	New heuristic	Heuristics: Sánchez et al. Ergonomics criteria: Nowacki and Gordeeva	Conversational fluidity, contextual continuity, memory
HEUXIVA4	Environment match between assistant and user language	New heuristic	Heuristics: Langevin et al.	Adaptation to user language and linguistic environment
HEUXIVA5	Information accuracy	New heuristic	Heuristics: Nielsen Ergonomics criteria: Nowacki and Gordeeva	Accuracy, brevity, and contextual relevance of responses
HEUXIVA6	User control and freedom	Adapted heuristic	Heuristics: Nielsen and Langevin et al. Ergonomics criteria: Nowacki and Gordeeva	Undo/redo through conversational commands
HEUXIVA7	Consistent voice interface	New heuristic	Heuristics: Nielsen and Langevin et al. Ergonomics criteria: Nowacki and Gordeeva	Voice consistency and coherence
HEUXIVA8	Voice shortcuts, flexibility and personalization	New heuristic	Heuristics: Nielsen and Langevin et al. Ergonomics criteria: Nowacki and Gordeeva	Voice shortcuts, customization, and adaptability
HEUXIVA9	Error prevention	Adapted heuristic	Heuristics: Nielsen and Sánchez et al. Ergonomics criteria: Nowacki and Gordeeva	Preemptive voice feedback before execution
HEUXIVA10	Help users recognize, diagnose, and fix errors	Adapted heuristic	Heuristics: Nielsen and Langevin et al. Ergonomics criteria: Nowacki and Gordeeva	Constructive voice-based error communication
HEUXIVA11	Data privacy	New heuristic	Heuristics: Langevin et al.	Data transparency, privacy management, user consent
HEUXIVA12	Voice assistant reliability	New heuristic	Heuristics: Langevin et al. Ergonomics criteria: Nowacki and Gordeeva	Reliability and trust in autonomous voice behavior
HEUXIVA13	Guides and documentation	Adapted heuristic	Heuristics: Nielsen	Simplified physical and digital documentation

To ensure transparency of coverage and to avoid redundancy across heuristics, a matrix was developed linking each voice assistant feature with its corresponding HEUXIVA heuristic, checklist item, problem type, and representative example (see Appendix K). To prevent overlaps among heuristics, three rules were applied: (1) each voice assistant feature was mapped to a primary heuristic that best represents its evaluative focus; (2) checklist items covering similar UX aspects were grouped and assigned to the most specific heuristic; and (3) overlapping items were merged under the heuristic with the broader or more integrative scope.

8. Limitations

This study has several limitations; however, we believe it represents a valuable contribution toward advancing the evaluation of UX with voice assistants. First, the validation scope was narrow, as all experiments were conducted using Google Nest Mini. While this restricts generalization to other devices, it allowed to maintain experimental control and consistency across iterations. Focusing on a single device enabled a more precise identification of domain-specific UX issues, which can later be contrasted with other platforms in future research.

Second, the sample sizes were relatively small and homogeneous. The heuristic evaluation involved two independent groups of three experts each (HVA vs. CAH), which may not fully eliminate group-composition effects. However, this setup provided valuable initial evidence about the effectiveness and clarity of the proposed heuristics. A larger, cross-over design is recommended for future studies to enhance statistical robustness, but the current results already show consistent tendencies that support the validity of HEUXIVA.

Third, in the first iteration, the same specialists who conducted the heuristic evaluation also participated in the expert judgment survey. This overlap may introduce a degree of confirmation bias. Nevertheless, it also ensured continuity and a deep understanding of the heuristics under review, resulting in meaningful, expert-informed feedback that guided the refinement of the set. Future iterations will address this by engaging independent evaluators for each stage.

Finally, the user testing in the second iteration was limited to 12 participants aged 22–28, using a single device and eight scenarios. While this constitutes a limited sample, it provided an appropriate pilot exploration that successfully verified the applicability and coverage of HEUXIVA. Broader studies including diverse devices (e.g., Siri or Alexa), languages, and usage contexts are planned to expand the external validity of the results.

Overall, despite these limitations, the study presents a domain-specific heuristic set that fills a gap in the UX evaluation of voice assistants. The iterative process, combination of multiple validation methods, and findings demonstrate that HEUXIVA is both rigorous and promising, serving as a strong basis for future refinement and broader application.

9. Conclusions and Future Work

Voice assistants are designed to support users in their daily activities by performing tasks such as setting alarms, retrieving information, or managing smart devices through natural voice interaction. However, due to the diversity of existing platforms and their conversational limitations, they often present usability and UX issues that negatively affect user satisfaction and continued adoption. Establishing specific heuristics to evaluate the user experience with these devices is therefore essential to identify interaction problems and improve their overall quality.

In this study, we proposed HEUXIVA, a set of 13 heuristics specifically developed to evaluate the user experience of voice assistants. The heuristics were created through a structured, iterative methodology and validated through heuristic evaluation, expert judgment, and user testing. The results—although preliminary—suggest that HEUXIVA is a useful and reliable instrument for identifying usability and UX issues specific to voice assistants, indicating that this research is progressing in a promising direction.

As a limitation of this study, the experiments were conducted exclusively using the Google Nest Mini device, with small and homogeneous samples. These constraints provided control and consistency in early stages but also limit the generalizability of the findings. As future work, we plan to address these aspects through several actions: broaden the validation scope by including multiple platforms (e.g., Amazon Alexa and Apple Siri) and varied acoustic, linguistic, and environmental contexts; increase and diversify the participant samples, incorporating users from different age groups, linguistic backgrounds, and profiles to improve external validity; and separate evaluator roles by engaging independent expert groups for heuristic evaluation and for subsequent judgment or surveys, thereby reducing bias and improving result independence.

We expect that the proposed heuristic set will support researchers and industry practitioners in developing and refining new voice assistants, facilitating the detection of usability and UX problems and improving users’ overall interaction experience. By improving user experience, these systems can better ensure quality, satisfaction, and alignment with user expectations.