User Experience and Usability of Voice User Interfaces: A Systematic Literature Review

Abstract

As voice user interfaces (VUIs) rapidly transform the landscape of human–computer interaction, their potential to revolutionize user engagement is becoming increasingly evident. This paper aims to advance the field of human–computer interaction by conducting a bibliometric analysis of the user experience associated with VUIs. It proposes a classification framework comprising six research categories to systematically organize the existing literature, analyzes the primary research streams, and identifies future research directions within each category. This systematic literature review provides a comprehensive analysis of the development and effectiveness of VUIs in facilitating natural human–machine interaction. It offers critical insights into the user experience of VUIs, contributing to the refinement of VUI design to optimize overall user interaction and satisfaction.

1. Introduction

User Interface (UI) defines the way humans interact with the information systems. Currently, different types of UIs are being used depending on the context, including Graphical User Interfaces (GUI), Command-Line Interfaces (CLI), Voice User Interfaces (VUI), Touchscreen User Interfaces, Augmented Reality (AR) User Interfaces, and Virtual Reality (VR) User Interfaces.

Voice User Interfaces are artificial intelligence-enabled conversational user interfaces that enable interaction through spoken commands and responses [1]. They are increasingly being used by individuals in their day-to-day lives to fulfil diverse needs (e.g., utility, hedonic, and social).

Graphical User Interfaces (GUIs), Command-Line Interfaces (CLIs), Touchscreen Interfaces, and other traditional UIs are increasingly being supplanted by Voice User Interfaces (VUIs) across various applications, owing to their ability to substitute text-based or written input with speech-based interaction [2,3]. The unique benefits of mobile and hands-free engagement are provided by VUIs [4].

However, VUIs are frequently associated with usability problems that result in bad user experiences [5,6]. Usability is defined by the ISO as an “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. Usability can be considered as a part of user experience (UX), which is defined by ISO as “user’s perceptions (including the users’ emotions, beliefs, preferences, perceptions, comfort, behaviors, and accomplishments that occur before, during and after use) and responses that result from the use and/or anticipated use of a system, product or service”.

In order to evaluate the efficiency and user-friendliness of VUIs, the study of user experiences is essential [7]; and to provide a seamless and gratifying user experience, it is crucial to assess VUIs’ usability [8].

A variety of procedures can be used to evaluate user experience and usability of VUIs [9,10]. Examining variables like learnability, effectiveness, satisfaction, and error prevention in VUIs is a part of usability evaluation [11]. Therefore, to fully comprehend how successfully VUIs satisfy user wants and expectations, it is essential to grasp these mechanisms.

However, two main research gaps can be identified in this field. First, the literature remains dispersed, making it difficult to keep track of the various findings. Therefore, it is necessary to synthesize the main findings of these studies, as they offer valuable knowledge for academics and practitioners [12]. Second, it is necessary to discern important underexamined areas in this field [13], providing a strategic platform for future scholarship. Therefore, academics and practitioners can develop novel and interesting research questions, ideas, theories, and empirical studies [14]

To address the aforementioned research gaps, the study proposes three research objectives:

• Identification of the top contributing countries, authors, institutions, and sources in the area of user experience, usability, and voice user interfaces;
• Development of a classification framework to classify user experience, usability, and voice user interface research papers on the basis of relevant commonalities;
• Development of a future research agenda in the field.

The remainder of this paper is organized as follows: Section 2 outlines the research methods and tools employed for the systematic literature review and category identification. Section 3 details the findings of the bibliometric analysis. Section 4 introduces the proposed classification framework. Section 5 delineates the future research agenda. Section 6 discusses the implications of the findings. Finally, Section 7 concludes the paper, addressing research limitations.

2. Research Methodology

The research was conducted following the criteria of preferred reporting items for systematic reviews and meta-analysis (PRISMA) [15]. These include the following steps: (1) eligibility criteria; (2) information sources; (3) search terms; (4) study selection; and (5) data collection process and synthesis (Figure 1).

Figure 1. Research methodology steps.

2.1. Eligibility Criteria

Studies were eligible for inclusion if they were research papers; review papers and conference/congress papers, since they are regarded as true knowledge [16]; directly relevant to sustainable customer relationship management; written in English; or published in peer-reviewed journals.

We excluded studies if they were not written in English; books, thesis, and conference proceedings; or papers that focused more on the direction of speech technologies and acoustics.

2.2. Information Sources

We performed a structured, systematic, and comprehensive search across two prominent online databases: Web of Science and Scopus. These databases were chosen for their rigorous selection criteria and extensive interdisciplinary coverage. Consequently, they are the primary sources of bibliographic citations employed in bibliometric analyses [17].

2.3. Search Terms

The collection of papers was carried out by selecting those that contained specific keywords pertinent to our research aims and questions in the title, abstract, or keywords section (Table 1). The keywords used included voice user interfaces (VUIs), usability evaluation, natural language, evaluation methods, VUI development, and user experience optimization.

Table 1. Search strategy.

Database	Keywords	Content	Period	Document	Language
Scopus and Web of Science	(“voice user” OR “voice assistants” OR “VUI”) AND (“User Experience” OR “Usability” OR “UX”)	TITLE ABS KEY	Until July 2024	Research and reviews Journal and conference papers	English

Logical operators were connected with different sets of keywords and designed as follows: (“voice user” OR “voice assistants” OR “VUI”) AND (“User Experience” OR “Usability” OR “UX”).

2.4. Study Selection

The study selection process aimed to analyze, evaluate, and identify relevant articles in alignment with the goals of our systematic review. This process was conducted independently by the two co-authors of this study to ensure objectivity and reduce potential bias.

First, records were identified through various information sources, specifically online databases, using predefined keywords. This initial search yielded a comprehensive list of potential studies.

Second, the retrieved records were scrutinized to remove duplicates, following the method outlined by [14]. This step ensured that each study was unique and avoided redundant analysis.

Third, the remaining records were screened based on their titles, abstracts, and keywords. This screening process aimed to exclude studies that did not meet the predefined eligibility criteria.

Finally, a full-text review of the remaining studies was performed to ensure a thorough evaluation. During this stage, the co-authors convened to discuss and reach a consensus on the final set of studies to be included in the systematic review. This collaborative discussion was essential to ensure that the included studies were rigorously selected and relevant to our research objectives.

2.5. Data Collection Process and Synthesis

The search focused on research and review papers published up to 10 July 2024. Initially, we identified 359 papers (138 from Web of Science and 221 from Scopus). After removing 110 duplicate papers, we applied eligibility criteria by screening titles, abstracts, and keywords. Papers where the acronym UX was associated with interpretations such as universal exchange, ultimate experience, utility exchange, unified execution, etc., were excluded, as were those where VUI could be interpreted as virtual user interface, vector unit instruction, etc. Subsequently, papers were further screened based on a full-text review. As a result, the final dataset comprised 125 papers (67 from Web of Science and 58 from Scopus).

The analysis of the data collection allowed us to develop a bibliometric analysis, identifying the top contributing countries, authors, institutions, and sources in the area of user experience and voice user interfaces to establish a classification framework composed of research categories, and to identify a future research agenda.

The bibliometric analysis followed established methodologies as demonstrated in several studies [18]. Initially, relevant papers were extracted from Scopus and Web of Science in BIB format. These files were merged into a consolidated dataset using the Bibliometrix (version 4.1) package in R [18,19]. Duplicates were systematically removed, and the cleaned dataset was exported to Excel (Version 2406 Build 16.0.17726.20206) for further analysis. Titles and abstracts underwent rigorous screening to eliminate papers unrelated to the study’s focus.

The refined dataset was imported back into Bibliometrix to compute key metrics such as top authors, journals, and sources [20]. Data merging, cleaning, and analysis stages were conducted using RStudio alongside the Bibliometrix package. For visualizing trends and patterns, we utilized the Biblioshiny package. Through these steps, a comprehensive bibliometric analysis was conducted, and the final results were exported for subsequent analysis.

Basic publication data such as publication date, title, authors, publisher, DOI, URL, pages, volume, issues, and keywords were collected using Microsoft Excel. The analysis of this data allowed us to identify the top contributing countries, authors, institutions, and sources in user experience and voice user interface (VUI) research.

For the research categories identification, the comparative method proposed by [21] was used. This method enables the identification of common points shared by the papers through a content analysis, so that the categories emerge. Content analysis is “an effective tool for analysing a sample of research documents in a systematic and rule-governed way” [22]. It allows an objective identification of the content in a data set, such as selected articles [23].

A first categories classification was performed, taking into account the aim of the paper and its contribution to the state of the art. Then, the capacity of the categories classification to arrange all the papers was checked paper by paper. If a paper did not fit into any research category, the classification was redesigned to integrate the incompatible paper. The categories classification was reconsidered several times until all the papers in the sample were properly distributed.

As a result, six main research categories, as well as a future research agenda in the field, were identified. We also created a qualitative and quantitative evidential narrative summary for each research category.

Any disagreements between the co-authors of this study were settled through consensus.

3. Bibliometric Analysis

3.1. Trend in Annual Scientific Publications

Since the inception of the systematic literature review in 2009, the evolution of publications has been notable, as illustrated in Figure 2. The trend shows significant growth over time, culminating in a peak of approximately 14 papers in 2022. As of the current year, the publication rate has shown a slight decrease, with fewer than 10 papers published to date. This trend highlights a dynamic landscape of research activity in the field, reflecting periods of intensified scholarly output followed by potential fluctuations in publication rates.

Figure 2. Trend in annual scientific publication.

3.2. Most Influential Authors

In the bibliometric analysis of authors within the research domain, several key contributors have been identified based on their publications and citation metrics. Author Klein A. has contributed four publications with a total of seven citations. Similarly, author Munteanu C. also has four publications but with a higher citation count of 16. Notably, author Myers C. has made significant contributions, with three publications that have amassed a total of 150 citations. These findings underscore the varying levels of impact and productivity among authors within the field, reflecting their respective influence and scholarly contributions (Table 2).

Table 2. Top contributing authors.

Author	No. of Publications	Total Citations	h_Index	g_Index	m_Index	Start of Publication Year
Klein A.	4	7	2	2	0.4	2020
Munteanu C.	4	16	2	4	0.285714286	2018
Myers C.	3	150	3	3	0.428571429	2018
Pal D.	3	30	2	3	0.333333333	2019
Arpnikanondt C.	2	29	2	2	0.333333333	2019
Furqan A.	2	145	2	2	0.285714286	2018
Li J.	2	5	2	2	1	2023
Moore R.	2	19	2	2	0.285714286	2018
Thomaschewski J.	2	5	2	2	0.4	2020

The top 10 authors in this analysis, ranked by their citation impact and productivity, include authors such as Myers C., Munteanu C., and Klein A., as well as others with comparable or higher metrics. Each author’s h-index, g-index, m-index, total citations, number of publications, and start of publication year were considered to assess their scholarly impact comprehensively. These metrics provide insights into the research influence and contribution of each author, highlighting the diverse contributions and impact levels within the research domain.

The h-index reflects a researcher’s productivity and impact, with a h-index of h meaning h papers have each been cited at least h times, which is widely used to assess research influence [24]. The g-index identifies highly cited papers, defined as the largest number of top papers that together accumulate g² citations, complementing the h-index [25]). The m-index evaluates individual contributions in collaborative research by considering co-authorship, aiming to quantify unique scholarly output [26].

3.3. Most Influential Countries

The table below depicts the research output of the top 10 contributing countries in the field. The United States leads with 55 publications, followed by Germany with 24, and Canada with 21. China and India each contributed 16 and 13 publications, respectively, while Spain and South Korea also recorded 13 publications each. Australia, Japan, and Thailand each contributed seven publications. This distribution underscores the global engagement and varying levels of research activity across nations, illustrating their significant roles in shaping scholarly discourse within this research domain (Table 3).

Table 3. Top 10 contributing countries.

Country	No. of Publication
USA	55
Germany	24
Canada	21
China	16
India	13
Spain	13
South Korea	12
Australia	7
Japan	7
Thailand	7

3.4. Top Contributing Institutions

The table below presents the top 10 universities contributing to research within the specified topic. Leading the list is the University of Toronto with 10 publications, followed by King Mongkut’s University of Technology Thonburi and the University of Seville, each with seven publications. Drexel University ranks next with six publications, while the University of Waterloo, Samsung R&D Institute, Seoul National University, Tokyo Institute of Technology, University of Applied Sciences Emden Leer, and Kookmin University each have four to three publications. This distribution underscores the active participation of these academic institutions in advancing scholarly discourse and research advancements within the field (Table 4).

Table 4. Top contributing institutions.

Institutes	Articles
University of Toronto	10
King Mongkuts Univ Technol Thonburi	7
Universidad de Sevilla	7
Drexel University	6
University of Waterloo	5
Samsung R&D Inst	4
Seoul National University	4
Tokyo Inst Technol	4
University of Applied Sciences Emden/Leer	4
Kookmin Univ	3

3.5. Most-Cited Papers

Regarding the citation analysis of the papers, the most cited paper has 113 citations, which stands out significantly from the rest of the citations. Table 5 shows the ten most cited papers and their main contribution.

Table 5. Ten most-cited papers.

References	Contribution	Total Citations	Total Citations Per Year
[27]	The main contribution of the paper is identifying obstacle categories and user tactics in interacting with the VUI calendar system DiscoverCal, revealing that NLP errors are common but less frustrating than other obstacles, with users often resorting to “guessing” over using visual aids or knowledge recall.	113	16.1
[28]	The paper demonstrates significant security risks of smart devices with voice assistants, amplified by operating systems and IoT connectivity. It analyzes how these attacks can be launched and their real-world impacts.	95	11.9
[29]	This paper reports on research aimed at improving the visibility and learnability of voice commands in an M-VUI application on Android. The study confirmed longstanding challenges with voice interactions and explored methods to enhance onboarding and learning experiences. Based on these findings, they propose design implications for M-VUIs.	89	9.9
[30]	This paper encourages research on voice assistants (VAs) for older adults (65+), outlining key issues such as perceptions, barriers to use, conversational user experience, and anthropomorphic design, while raising provocative research questions to stimulate debate and discussion.	52	8.7
[31]	This paper reveals that user characteristics like assimilation bias and technical confidence significantly impact interactions with the VUI-based calendar DiscoverCal, affecting performance and approach.	32	5.3
[32]	This study examines the moderating effect of user experience on user resistance in the voice interface of in-vehicle infotainment (IVI) systems. User experience positively moderates the relationship of uncertainty costs with user resistance.	31	3.5
[33]	This paper applies HCI and the parasocial relationship theory to study trust and user behaviour towards VCEDs, finding that users treat these devices as social objects influenced by performance, effort expectancy, presence, and cognition, with privacy concerns showing minimal impact on trust.	26	4.3
[34]	This workshop invites contributions on advancing voice interaction research, focusing on methodologies, social implications, and design insights for improving user experience and addressing challenges in voice UI design.	13	1.9
[33]	This panel examines the challenges in studying the daily use of voice assistants (VAs), highlighting privacy concerns, VA personalization, user value, and UX design considerations.	12	1.7
[35]	This paper addresses the challenge of quantifying user experience (UX) in voice-based systems, identifying gaps, and proposing frameworks to guide future research and standardize measurement methods.	8	2

3.6. Source Analysis through Bradford’s Law

Understanding citation patterns through Bradford’s law provides valuable insights into the distribution of citations across publications. Bradford’s law, proposed by Samuel C. Bradford in 1934 [36], posits that scholarly journals can be categorized into core, secondary, and tertiary groups based on their citation frequencies. This law suggests that a small number of journals (or papers) receive a disproportionately large number of citations, while the majority of papers receive fewer citations, following an exponential distribution. By applying Bradford’s law to bibliometric analyses, researchers can identify seminal papers that have significantly influenced the field, elucidating trends in research impact and scholarly communication [36]. This approach facilitates a deeper understanding of the intellectual structure and impact dynamics within specific research domains. Figure 3 shows the Source Analysis through Bradford’s Law.

Figure 3. Core sources analysis using Bradford’s law.

4. Classification Framework

A content analysis of the 125 articles was carried out to identify (1) a classification framework composed of six research categories that organizes papers according to common issues, and (2) a future research agenda in user experience and voice user interfaces.

The research categories obtained and their main contributions are shown in Table 6. The number of papers in each category is showed in brackets in the first column.

Table 6. Research categories.

Research Categories	Description
User Experience and usability Measurement and Evaluation (26)	This category focuses on assessing and enhancing interactions with voice-based systems, such as virtual assistants and voice user interfaces (VUIs). Studies emphasize the need for improved evaluation metrics and methods for VUIs, and reveal a diverse array of approaches to evaluating UX. For example, the system usability scale (SUS), to better understand and enhance usability [11]. Another example is the UEQ Plus framework has been extended to incorporate voice communication scales to measure user experience quality, specifically for voice assistants [37]. Additionally, an importance-performance matrix analysis (IPMA) has been utilized to strategically evaluate user experience with intelligent voice assistants, highlighting key factors, such as performance expectancy and barriers that influence user adoption [38]. Research also delves into factors influencing the adoption of AI-based voice assistants, identifying crucial elements, like user satisfaction and the role of AI in enhancing UX [39]. The context of use and context-dependent UX quality testing are also emphasized, addressing the challenges of evaluating non-visual interfaces in varied environments [1]. Moreover, the effects of infotainment auditory–vocal systems’ design on workload and usability have been examined, underscoring the importance of system design elements in influencing user satisfaction [40]. First impressions and the perceived human likeness of voice-activated virtual assistants across different modalities and tasks have been explored, showing how these factors impact user emotions and the overall UX [41]. Additionally, studies have analyzed parameters of the user’s voice as indicators of their experience with intelligent voice assistants, revealing the deep connection between voice control and emotional responses [42]. The research also addresses the usability evaluation of voice interfaces through qualitative methods, providing protocols for assessing voice assistants in specific contexts, such as healthcare [43]. In addition, other studies focus on creating reliable scales for evaluating VA usability, such as the voice usability scale developed to measure user experience, specifically with voice assistants [44] Finally, there is a focus on enhancing voice technologies through innovative methodologies and evaluation frameworks, such as examining the role of English language proficiency in using VUIs [43] and developing adaptable utterances to improve learnability [45]. The collective goal across these studies is to create more effective, user-friendly voice technologies by understanding and improving the UX through comprehensive and context-aware research.
Usability Engineering and Human–Computer Interaction (28)	This research category delves into the engineering aspects related to usability and human–computer interaction of voice assistants (VAs) and voice user interfaces (VUIs). As these technologies gain popularity, their usability becomes crucial for widespread adoption. Usability with multi-agent versus single-agent systems are examined to understand the complexities involved in managing multiple voice assistants [46]. The engineering implications in developing verbal consent mechanisms are also explored, emphasizing the importance of legal obligations and ethical best practices [47]. The integration of VUIs in virtual environments (VEs) is also explored, comparing the effectiveness of voice interfaces with graphical user interfaces in virtual reality settings [48]. Industry-focused research underscores the barriers and opportunities in supporting VUI designers on the job, pointing to gaps in current design practices and offering insights for improvement [49]. The role of proactive dialogue strategies in enhancing user acceptance is studied through comparisons of explicit and implicit dialogue strategies for conversational recommendation systems [50]. The impact of energy-saving applications incorporating VUIs is also examined, as seen in the development of a cloud-based smart home energy application aimed at multi-unit residential buildings [51,52]. Collectively, these studies contribute to the development of more intuitive, accessible, and user-friendly VUIs, addressing the diverse needs of users across different contexts and environments.
Voice Assistant Design and Personalization (31)	This category focuses on the evolving landscape of voice assistants (VAs), particularly how these systems are designed, personalized, and received by users. The field examines various aspects such as the user-centered design of VAs, intercultural user studies, the impact of personality and anthropomorphism on user experience, and the challenges of implementing effective voice user interfaces (VUIs). This category can be further streamlined into three subcategories: User-centered design and personalization: Research in voice assistant design emphasizes creating natural, human-like interactions through user-centered approaches. For example, designing natural voice assistants for mobile platforms through a user-centered design approach has been highlighted as a crucial element in enhancing user interaction and satisfaction [53]. Studies also explore how adaptive suggestions can increase the learnability of VUIs, improving user engagement over time [54]. Furthermore, the importance of designing emotionally engaging avatars and optimizing dialogue to enhance user experience and acceptance is evident in research focusing on the implementation of emotion recognition in VAs [55]. Cultural and psychological factors: Studies in this subcategory highlight that cultural and psychological factors need to bear in mind on VUIs design. For instance, intercultural comparisons, such as those between Germany and Spain, reveal similar usage patterns despite privacy concerns, underscoring the need to consider cultural nuances in VUI design [56]. Additionally, psychological models exploring users’ mental models of anthropomorphized voice assistants demonstrate how users perceive these technologies, ranging from empathetic, human-like designs to mere machines [57]. Another study on user perceptions of extraversion in chatbot design further supports the notion that personality traits significantly affect user experience after repeated use [58]. Moreover, the importance of cultural sensitivity in VUI design is evident in research from Japan, demonstrating the need to adapt interactions to fit different cultural contexts [59]. Improvement in voice user interfaces design: There is a strong need for tools and guidelines for improvement and enhance the design of voice interfaces. Improving UX is crucial for VUIs. Research identifies common user frustrations, including poor response quality and conversational design issues. For example, a survey-based study focusing on German users found that frequent annoyances included the misinterpretation of commands and inadequate responses, which significantly impacted user satisfaction [60]. On the other hand, Meta-analyses of existing VUI guidelines suggest a move towards unified standards, which can help streamline the design process and ensure a consistent user experience across different platforms [61]. Moreover, the development of tools like the user experience tool selector for VUIs provides valuable resources for researchers and practitioners aiming to improve VUI design effectiveness. Additionally, research on addressing VUIs design challenges for VUIs highlights the importance of accessibility and adaptability in these systems [62].
Privacy, Security, and Ethical issues (10)	This research category delves into the critical aspects of privacy, security, and ethical design in voice user interfaces (VUIs) and related technologies. The studies in this category explore various facets of these themes, emphasizing the need for careful consideration in the design and implementation of VUIs to ensure they meet the diverse needs and expectations of users. This category can be further split into two subcategories:Privacy concerns in voice user interfaces: Privacy is a major concern for VUI users. Studies reveal risks like unwanted activations and data security vulnerabilities. For instance, the StealthyIMU attack demonstrates how motion sensors can covertly steal sensitive information from VUIs, highlighting significant privacy threats that need to be addressed [63]. Additionally, security issues, such as those explored by [64], emphasize the importance of securing data to maintain user trust. Ethical design and inclusion: Ethical design and inclusivity are vital for VUIs. Research on older adults reveals specific challenges and opportunities for creating more accessible and effective interfaces, as discussed by [65]. Similarly, studies focusing on non-native English speakers highlight usability issues and propose solutions to make VUIs more user-friendly, inclusive, and equitable for diverse populations [66]. The inclusion of ethical considerations in the design process ensures that VUIs are not only effective but also respect the diverse needs of their users.
Cross-Cultural Usability and Demographic Studies (6)	This category explores how voice assistant technologies and related applications interact with diverse user groups and cultural contexts. Research in this area investigates various aspects of voice assistants, including personalization, naturalness, and human-like traits, to enhance user experience. One study focuses on defining optimal voice assistant characteristics considering different cultural issus, such as communication style, personality, dialogue, and appearance, to create guidelines for more natural interactions, contributing to the development of more human-like VUIs [67]. Another study compares the efficiency of voice assistants like Google Home with traditional technologies in educational settings, revealing that voice assistants can significantly speed up information retrieval and improve user satisfaction, though challenges such as privacy concerns and limited databases remain [68]. Research has also examined the impact of cultural differences on the UX of voice assistants, showing that localization and cultural adaptation are crucial for ensuring a positive user experience across different regions [69]. Other research investigates differences in usability and satisfaction between native and non-native English speakers, shedding light on the challenges non-native speakers face with intelligent personal assistants [70]. Additionally, the reasoning capabilities of dialogue systems are analyzed, focusing on large-scale interpretable knowledge graph reasoning [71]. An investigation into emotional responses to voice user interfaces (VUIs) in India emphasizes the importance of speech modulation in conveying emotions, demonstrating how user-affective responses can be influenced by voice interface stimuli [72]. The role of accessibility in voice interfaces has become a significant research focus, especially for users with disabilities. The category further explores the challenges of designing VUIs for different user demographics, such as age groups, by conducting usability studies to tailor VUIs accordingly [73]. The social dynamics of family interactions with VUIs are highlighted in studies that investigate how families interact with and perceive VUIs in group settings [74]. In addition, studies have explored how different demographic groups, including older adults and individuals with disabilities, experience and interact with VUIs, highlighting the need for inclusive design practices that cater to a diverse user base [75]. The introduction of the Sonos Voice Control Bias Assessment Dataset offers valuable insights into demographic biases in voice assistant performance, providing new methods for assessing and improving inclusivity in these technologies [76]. Lastly, a bibliometric analysis of the impact of artificial intelligence on branding reveals the evolving role of AI, including voice assistants, in shaping brand strategies and consumer perceptions over recent decades [77]. Overall, these studies contribute to a deeper understanding of how voice assistants perform across different cultural and demographic groups, aiming to improve user experience and address technological challenges.
Technological challenges and applications (13)	This category explores various facets of voice-based technologies, including conversational voice assistants (CVAs) and intelligent voice assistants (IVAs). The research encompasses a wide range of topics, from usability and user experience to the impact of these technologies on specific applications. This category can be further refined into two subcategories:Technical challenges: Research into technical design of CVAs and IVAs addresses significant challenges, including response accuracy, natural language understanding, and user interaction patterns, which are critical for creating usable and effective voice technologies [78]. Additionally, frameworks like VORI have been developed to test the interactability of VUIs, providing a structured approach to evaluate these systems’ performance [8]. Application contexts: User experience with voice technologies is influenced by factors beyond mere task success, including how system responses align with user expectations and cultural contexts [67]. For instance, studies examining the use of voice technology in caregiving show potential in supporting complex home care tasks, yet they also reveal concerns about the usability and impact of these systems on caregiver experiences [79]. Similarly, research into the application of voice assistants in reminiscence therapy for older adults demonstrates the potential of these technologies to enhance user engagement, though issues related to system usability and user perceptions remain [80]. Further research examines the application of voice technology in specific contexts, such as a mobile health app for rheumatoid arthritis patients. This study shows moderate adherence and mixed satisfaction with the voice features, highlighting the potential and limitations of voice-enabled health technologies [81]. Furthermore, the mediating effects of IVAs on service quality, satisfaction, and loyalty in various service contexts further highlight the importance of user experience in the successful deployment of voice technologies [42]. Additionally, efforts to improve conversational agent responses when they lack sufficient information underline the need for better user interaction strategies to maintain trust and satisfaction [82]. Overall, these studies contribute to a deeper understanding of how voice assistants perform across different contexts, aiming to improve user experience, address usability challenges, and enhance the applicability of these technologies across diverse user groups.

5. Future Research Agenda

Voice user interfaces (VUIs) are becoming increasingly prevalent, yet the study of the user experience of VUIs present several limitations and challenges. Addressing these issues is crucial for advancing VUI technology and enhancing user experiences.

This systematic literature review has allowed us to define a future research agenda in the field (Table 7).

Table 7. Future research agenda.

Category Name	Research Challenges
User Experience and usability Measurement and Evaluation	Integration of multimodal interactions: developing comprehensive evaluation models that account for the integration of voice with other modalities (e.g., touch, gesture) and how these interactions impact overall UX Context-aware UX measurement: designing tools and methods to measure UX that accurately reflect the context in which VUIs are used (e.g., home vs. car, with or without disabilities) Standardization of UX metrics: establishing consensus on standardized metrics and scales for evaluating voice-based user experiences, given the current lack of agreement and the overreliance on unvalidated measures
Usability Engineering and Human–Computer Interaction (HCI)	Designing multi-agent voice assistant systems: the traditional single-agent voice assistant model may limit user experience, particularly in complex, multi-task environments like smart homes Improving dialogue systems with knowledge reasoning capabilities: current dialogue systems often require users to phrase requests in a specific manner, which limits user experience due to the lack of reasoning capabilities Platform-agnostic application modeling for voice assistants: Developing a systematic approach for creating, distributing, and managing applications across diverse platforms without requiring extensive client-side intervention is challenging
Voice Assistant Design and Personalization	Addressing mental models and psychological factors: understanding and designing for the diverse mental models users have when interacting with voice assistants, particularly regarding personification and anthropomorphism Integration of emotional intelligence in voice assistants: embedding emotional intelligence into voice assistants to enable more empathetic and supportive interactions, especially in contexts requiring emotional disclosure Cultural sensitivity and cross-cultural design: addressing cultural differences in the adoption and use of voice assistants, particularly regarding privacy concerns and interaction styles
Privacy, Security, and Ethical issues	Enhancing privacy and security against unauthorized access and eavesdropping: VUIs are vulnerable to various security threats, including unauthorized access through voice commands, motion sensor eavesdropping, and unintended activations that may compromise user privacy Incorporating cultural sensitivity and localization in VUI design: existing VUI designs often lack the cultural sensitivity required for effective and respectful interactions across different languages and social contexts, particularly outside of English-speaking regions Balancing discreetness and functionality in public VUI interactions: users often feel uncomfortable using VUIs in public due to privacy concerns and social awkwardness, which limits the practicality and adoption of VUIs in these environments
Cross-Cultural Usability and Demographic Studies	Personalization in voice assistants across cultures: understanding how cultural differences influence user expectations for personalization in voice assistants, particularly in communication style, personality, and non-verbal cues Enhancing voice assistant performance for diverse demographics: reducing performance disparities in voice assistants across different demographic groups, including variations in age, ethnicity, and dialect Emotion recognition and cultural sensitivity in voice user interfaces (VUIs): addressing the cultural variations in emotional recognition and response within voice user interfaces, particularly in contexts where emotion plays a significant role in user experience
Technological challenges and applications	Impact of task success on user experience with speech assistants: understanding why task success does not significantly correlate with overall user experience (UX) in speech assistants. Identifying factors other than task success that contribute to the UX in interactions with speech assistants. Evaluation of voice-based interactive systems: the lack of standardized evaluation tools tailored for voice user interfaces (VUIs), which are currently in their infancy

6. Discussion

This study has applied the PRISMA systematic literature review approach, which categorized the existing literature in a systematic and valid manner, but also identified the main potential areas for future research. PRISMA allows a replicable, scientific, and transparent process to minimize bias and provides an audit trail of the reviewer’s decisions, procedures, and conclusions, which is a necessary requirement in systematic reviews [83].

The findings shown in this paper contribute to the theory on user experience and voice user interface theory as follows:

(1)

According to [12,84] a systematic literature review should be written when there is a substantial body of work in the domain (at least 40 articles for review) and no systematic literature review has been conducted in the field in recent years (within the last 5 years). Therefore, this paper covers a gap in the domain of user experience and voice user interfaces, because this is the first systematic review in the field. Other systematic reviews related to user experience and voice user interfaces focused on different subjects, such as the identification of the scales used for measuring UX of voice assistants, as well as assessing the rigor of operationalization during the development of these scales [44], the synthesis of current knowledge on how proactive behavior has been implemented in voice assistants and under what conditions proactivity has been found more or less suitable [85], or the identification of the usability measures currently used for voice assistants [86];

(2)

According to [14], descriptive statistics (e.g., frequency tables) should be used to summarize the basic information on the topic gathered over time in systematic reviews. This paper uses bibliometric statistical analysis techniques to show significant information in the user experience and voice user interface theory domain, such as the top contributing countries, authors, institutions, and sources;

(3)

According to [14,87] to make a theoretical contribution, it is not enough to merely report on the previous literature. Systematic literature reviews should focus on identifying new frameworks, promoting the objective discovery of knowledge clusters, or identifying major research streams. Through a content analysis, this paper proposes a classification framework composed of six research categories that shows different ways of contributing to the current state of knowledge on the topic: user experience and usability measurement and evaluation; usability engineering and human–computer interaction; voice assistant design and personalization; privacy, security, and ethical issues; cross-cultural usability and demographic studies; and technological challenges and applications;

(4)

According to [12], to make a theoretical contribution, systematic literature reviews can focus on identifying a research agenda. However, this research agenda should follow and accompany another form of synthesis, such as a taxonomy or framework. This paper synthesizes the future research challenges in each research category of the proposed classification framework.

7. Conclusions

To advance in the state of the art in user experience and voice user interface theory, in this paper, a systematic literature review in this field has been carried out. A sample of 125 papers were analyzed to assess the trend of the number of papers published and the number of citations of these papers; to identify the top contributing countries, authors, institutions, and sources; to reveal the findings of the ten most cited papers; and to establish research categories and future research challenges in the area.

This in-depth review focused on the user experience study on VUIs. The review covered the creation of VUIs, the identification of research categories related to user experience in the literature, and considerations of restrictions, difficulties, and research gaps. The results highlight the constant need for VUI design, evaluation, and improvement to enable seamless and positive user experiences.

This review did note a number of gaps and difficulties in the area that offer chances for further research. The lack of standardized evaluation techniques, the scant attention paid to user variety and context, the absence of guidelines for user-centered design, the accuracy of speech recognition, the handling and recovery of errors, the cross-cultural usability evaluation, and the inclusivity and accessibility considerations are a few of them.

Researchers and practitioners can improve the design and evaluation of VUIs and produce voice interactions that are more efficient and user-friendly by bridging these research gaps. In the end, this enhances the human–machine connection and helps VUI technology blend seamlessly into our daily lives. In summary, this review emphasizes the importance of usability evaluation for VUIs and offers helpful tips for the field’s researchers, designers, and developers. We can realize the full potential of VUIs and produce logical and interesting voice interactions that improve user experiences by constantly assessing and enhancing VUIs.

Finally, it is important to highlight the limitations of this work: (1) Only two bibliographical databases have been studied, Scopus and Web of Science. Other databases could be analyzed to extend and contrast the findings; (2) there is a language bias, due to the fact that the search was carried out only in English; (3) other keywords could have been used and might have produced other findings; (4) the comparative method proposed by [21] was used. Other methods, such as network analysis or latent dirichlet allocation (LDA), might be used for research categories identification and may result in other classifications.