Combined Framework of Multicriteria Methods to Identify Quality Attributes in Augmented Reality Applications

Abstract

This study proposes a combined framework of multicriteria decision methods to describe, prioritize, and group the quality attributes related to the user experience of augmented reality applications. The attributes were identified based on studies of high-impact repositories. A hierarchy of the identified attributes was built through the multicriteria decision methods Fuzzy Cognitive Maps and DEMATEL. Additionally, a statistical analysis of clusters was developed to determine the most relevant attributes and apply these results in academic and industrial contexts. The main contribution of this study was the categorization of user-experience quality attributes in augmented reality applications, as well as the grouping proposal. Usability, Satisfaction, Stimulation, Engagement, and Aesthetics were found to be among the most relevant attributes. After carrying out the multivariate analysis, two clusters were found with the largest grouping of attributes, oriented to security, representation, social interaction, aesthetics, ergonomics of the application, and its relationship with the user’s emotions. In conclusion, the combination of the three methods helped to identify the importance of the attributes in training processes. The holistic and detailed vision of the causal, impact, and similarity relationships between the 87 attributes analyzed were also considered. This framework will allow the generation of a baseline for the use of multicriteria methods in research into relevant aspects of Augmented Reality.

1. Introduction

Augmented reality is a variant of virtual environments (VE), or virtual reality as VE technologies are more commonly called, which completely immerse the user within a synthetic environment. While immersed, the users cannot see the real world around them. However, AR allows virtual objects to be superimposed on the real world [1].

There are studies in the educational field that evidence the increase in the use of AR in teaching and learning processes. In the report 2020 EDUCAUSE Horizon Report™ Teaching and Learning Edition [2], AR is included in the category of “Emerging Technologies and Practices”. In this classification, the technologies and practices that will have a significant impact on teaching and learning in higher education are found. The report indicates that higher education is experimenting with AR as a vehicle for collaborating in the learning process. There is evidence of an increase in the creation of laboratories and centers focused on the development of experiences with Extended Reality—XR (augmented reality AR, virtual reality VR, mixed reality MR, HAPTIC).

In the last decade, the implementation of AR-based solutions has grown substantially. In the study carried out by [3] the authors identify nine AR application areas: Perception, Medical Education, Entertainment and Gaming, Industrial Application, Navigation and Driving, Tourism and Exploration, Collaboration, and Interaction. Similarly, in the research developed by [4] seven areas are identified: Medical, Assembly, Education, Shopping, Entertainment, Militaries, and Furniture design. In addition, in the work by [5] four areas are highlighted: Training and Education, Entertainment and Commerce, Navigation and Tourism, and Medical and Construction. These works demonstrate the growing increase in the development of AR applications in different areas such as health, education, games, tourism, and industry, among others.

Taking into account the relevance of AR in multiple areas, this article seeks to make a contribution to knowledge in the academic-business context, proposing a ranking of user-experience quality attributes in AR applications, which contribute to the training evaluation process in industrial environments mediated by augmented reality from the user experience approach.

Since the human factor is a crucial aspect for the success in the adoption of this technology, the adequate preparation of workers must be considered. Practical training is important for many disciplines and professions, such as medical workers, mechanics, technicians, electricians, engineers, sailors, pilots, and firefighters, among others. Compared to traditional training for the consulting firm ABI Research, AR/VR training can save between USD 2000 and USD 2500 per employee, improve the candidate/employee experience, and build a competitive employer brand [6].

The article is focused on the analysis of quality attributes from the perspective of user experience, mixing technology, quality, and training processes in a set that allows companies to generate added value. This analysis will be the input to subsequently proposed AR applications with an objective approach to the attributes.

Therefore, it is important to explore which attributes are the ones that affect the industrial training process within the context of every worker and its environment. In this way, the following research question arises: What user experience quality attributes could be prioritized in applications mediated by augmented reality in training processes?

The Section 2 describes the importance of multicriteria decision methods in various disciplines and the validity of their use. The entire methodological process of ap-plying the framework to identify quality attributes in augmented reality applications is presented in the Section 3. Section 4 describes the most relevant findings after ap-plying each of the multicriteria methods. The following section analyzes the results obtained in light of the validity and applicability in scenarios of use. Finally, the conclusions consolidate the contributions of this study.

2. Literature Review

2.1. UX Attributes in AR Applications

In the literature, there is evidence of studies around the evaluation topic of the user experience using augmented reality applications. These works address aspects such as user experience measurement through methods and models, devices evaluation, and use description of specific applications.

In Ref. [7], the fundamental areas for the development of AR applications using the DELPHI-AHP method through five first-level indicators and twenty second-level indicators are identified. This method is tested and verified with six model visualization applications which found that the most important first-level indicators that affect the user experience are the functionality of a system and its visualization.

Ref. [8] defines a Uses and Gratifications theory-based model for the adoption of AR mobile games. The model is composed of eight constructs: enjoyment, fantasy, escapism, social interaction, social presence, achievement, self-presentation, and continuance intention. The case study is Pokémon Go. The results confirmed the positive influence of the aforementioned constructs on the intention to continue using the game.

Ref. [9] proposes the User Experience Measurement Index (UXMI). This indicator is calculated through different evaluation methods when testing the MAR Madness application. The tests were carried out before, during, and after using the application. In the first stage, a questionnaire with questions about age, gender, familiarity with AR applications, and tendency to suffer motion sickness was completed. In the second stage, the user’s physiological data were collected with the EMOTIV EPOC + EEG headset. In the last stage of the process the UEQ questionnaire was used.

Ref. [10] presents the results obtained from the analysis of the user experience (UX) of a digital educational material (DEM) based on augmented reality. The AttrakDiff model for evaluation is used.

Ref. [11] describe the behavior of a test group that uses an HMD application to perform an industrial task. Quantitative measures such as accuracy, task completion, consistency and time taken are evaluated.

Other measurement models are presented in [12,13].

In works such as [14,15,16,17,18] specific applications in the industrial area, tourism, and education are assessed, among others.

The evaluation of devices used in AR was also studied in the works by [11,19,20,21,22,23].

After the literature review, several research opportunities can be identified that may be addressed:

▪

AR research focus has been oriented toward the definition and measurement of quality indicators of user experience [24]. However, there is no general consensus on what elements should be analyzed when fully evaluating an AR solution;

▪

There are studies on evaluations of the use of AR applications where they use tools such as surveys, interviews, evaluation by experts, and biometric measurements. However, they are mechanisms external to the application and have a high degree of subjectivity;

▪

No tools were identified that evaluate augmented reality applications in industrial contexts with quantitative techniques or methods integrated into the industrial prototype;

▪

Interest in the industrial sector in conducting training through AR applications. In the works by [25,26,27], training with AR applications is carried out, resulting in improvements in the time for completing a task (task completion time), and in the reduction in the number of errors (error counts).

2.2. Importance of Multicriteria Decision Methods

Multicriteria decision-making methods (MCDM), used in different areas, are an important tool to weigh factors where the experience and knowledge of experts are taken into account. Previous research shows the usability and advantages of integrating multicriteria decision methods in application areas such as engineering, medicine, finance, education, and even in the social sciences [28,29,30,31,32,33,34,35]. Here are some of the most commonly used methods.

2.2.1. Analytic Hierarchy Process (AHP)

The Analytic Hierarchy Process was postulated by Tomas Saaty in 1980. The fundamental scale of the AHP is an absolute number scale used to answer a basic question in each of the paired comparisons: How many times is one element more dominant than another with respect to a certain criterion or attribute? It models relationships of interdependence and alternatives [36].

2.2.2. Analytic Network Process (ANP)

The Analytical Hierarchy Process was postulated by Tomas Saaty in 1996 as a generalization of the AHP method. The essential characteristic of ANP, unlike AHP, is to allow the inclusion of interdependence and feedback relationships between elements of the system [37].

2.2.3. Clustering

Clustering also known as Cluster Analysis, Numerical Taxonomy or Pattern Recognition, is a multivariate statistical technique whose purpose is to divide a set of objects into groups, so that the profiles of the objects in the same cluster are very similar among themselves (internal cohesion of the group) and the distinct cluster objects are different (external isolation of the group) [38]. Three main groups can be distinguished:

• Hierarchical Methods: These types of algorithms do not require the user to specify the cluster number in advance (agglomerative clustering, divisive clustering);
• Non-Hierarchical Method: These types of algorithms require the user to specify in advance the number of clusters to be created (K-means, K-medoids, CLARA);
• Other methods: Methods that combine or modify the previous ones (hierarchical K means, fuzzy clustering, model-based clustering, and density-based clustering).

2.2.4. DEMATEL

The DEMATEL (Decision Making Trial and Evaluation Laboratory) technique was first introduced by Gabus and Fontela in 1972 to solve real-world problems. The DEMATEL technique is an effective way to analyze direct and indirect relationships between system components according to their type and intensity [39]. Analyzing the general relationship between the components with the DEMATEL technique allows a better understanding of structural relationships, as well as an ideal way to solve complex system problems [40].

The DEMATEL method can be summarized as the following steps: build the direct influence matrix, normalize the direct influence matrix, obtain the total relationships matrix, develop a causal diagram, establish the threshold value, and obtain the impact-digraph map.

2.2.5. Fuzzy Cognitive Map (FCM)

A Fuzzy Cognitive Map (FCM) is a graphical representation consisting of nodes indicating the most relevant factors of a decisional environment, and links between these nodes representing the relationships among those factors [41].

They are a tool that can be used in different situations or problems to identify, define and validate the constructs or items of a system and identify the cause–effect relationships that exist among them, in order to propose strategies and help the decision-making process [31].

Previously, only precise causal relationships had been discussed, the same cause always causing the same effect. However on a day-to-day basis, this does not usually happen in this way, there are causal relationships that are imprecise, and the same cause does not always cause the same effect [31,42].

The methodology uses four matrices to represent the results that the methodology provides in each one of its stages. These are Initial Matrix of Success (IMS), Fuzzified Matrix of Success (FZMS), Strength of Relationships Matrix of Success (SRMS) and Final Matrix of Success (FMS). The result of these computations is a new matrix called Strength of Relationships Matrix of Success (SRMS) [41].

2.2.6. Multi-Attributive Border Approximation Area Comparison (MABAC)

Multi-Attributive Border Approximation Area Comparison (MABAC), developed by Pamucar and Cirovic [43], is one of the recent approaches to multi-attribute decision making (MCDM/MADM) problems. The basic principle used in MABAC is that it divides the performances of each criteria/attribute function into an Upper Approximation Area (UAA) containing ideal alternatives and Lower Approximation Area (LAA) containing anti-ideal alternatives. In other words, this method provides a direct observation of the relative strength and weakness of an alternative to others according to each criterion. In the classical MABAC method, the performance rating and criteria weights were represented by crisp/deterministic numerical values [44].

2.2.7. TODIM

An acronym in Portuguese for Interactive Multi-criteria Decision Making. The idea of TODIM is to compare the alternatives with respect to each criterion in a pairwise fashion in terms of gains and losses. The gains and losses are then passed to the prospect function to obtain the partial dominances and then the partial dominances are aggregated to form the final dominance. The rank order of the alternatives is basically based on this final dominance. In the standard formulation, the TODIM method only deals with crisp numbers [45].

2.2.8. Technique for Order Performance by Similarity to Ideal Solution (TOPSIS)

The TOPSIS method was proposed by Chen and Hwang in 1992. It is based on an aggregation function representing “closeness to ideal”, which originated from the commitment programming method. It determines a solution with the shortest distance to the ideal solution and the greatest distance from the negative ideal solution [46].

2.2.9. VIKOR

The VIKOR method was proposed by Serafín Opricovic in 1990. This method focuses on categorizing a series of alternatives aimed at solving a decision-making problem in a scenario in which there are conflicting criteria through the introduction of a multicriteria classification index based on the mean of the “closeness” to the ideal solution to that problem [47].

2.3. Use of Multicriteria Decision Methods in AR

As mentioned in the Section 2.2, the MCDM are useful in many disciplines. Within the framework in this research, it is important to highlight their use in AR environments.

In [42] a methodology to capture IT project requirements applying FCM and BPM (Business Process Model), through the use of an AR mobile application is described.

Reference [48] presents the evaluation of adopting augmented reality systems for maintenance activities. The identification of advantages was carried out using Fuzzy TOPSIS. The four evaluated alternatives were HoloLens, Tablet, Smartphone and Vuzix.

Continuing with the evaluation of AR devices in maintenance operations, it is found that in [49] AHP and Fuzzy-TOPSIS Model are used.

Reference [35] proposes a new recovery approach based on a multicriteria decision method to generate a compact descriptor that represents a signature for each 3D model. The aim is to try to extract the measure by obtaining a combined score using the data envelopment analysis method (DEA).

Evaluating the effectiveness of the use of AR applications in museums is also one of the environments where multicriteria decision methods were used. Data analysis was carried out via multiple correspondence analysis and hierarchical agglomerative clustering analysis [34].

Reference [40] proposes a model to select the most appropriate 4.0 technologies in the supply chain. In this work, DEMATEL was used, and augmented reality was among the first level technologies.

3. Methodology

Due to the nature of this research, the high number of variables to be analyzed and the conditioning of the extensive work of the experts linked to the study, the work is organized into four phases, as shown in Figure 1. The scope of the first phase was to identify the quality attributes that affected a user’s experience with augmented reality applications, resulting in 87 attributes.

In the second phase, the objective was to select the most appropriate multicriteria decision methods for the investigation, build the instruments for data collection and analysis and, finally, organize the panel of experts.

The third phase was the densest of the investigation because it had two stages. In the first stage, the intention was to find the direct and indirect causal relationships between the attributes, degrees of importance and polarity of the attributes. In the second stage, the aim was to group the 87 attributes by degree of similarity.

The fourth and final phase aimed to reduce the dimensionality of the attributes, as well as build the ranking and interpret the influence relationships among the 19 attributes.

The following sections will describe the steps of the proposed methodology.

3.1. Phase 1. Selection of Quality Attributes to Include in the Study

A systematic literature review was carried out to identify the quality attributes that affect the user experience using augmented reality applications. This review took the protocol guidelines proposed by Kitchenham et al. [50] as a reference.

The research questions that guided this search were:

• Q1: How is the user experience measured in AR applications?
• Q2: What are the user experience quality metrics used in augmented reality applications?

The review protocol begins with the selection of search repositories. This selection was made based on the work by [51]. The repositories used in this study were IEEE Xplore, SCOPUS and ACM Digital Library. The search string that was established is presented in Figure 2.

Exclusion filters were applied such as (1) duplicated documents and (2) observation window posterior to 2009.

For the pre-selection of the studies, the metadata verification process was carried out: title, abstract and keywords. The pre-selection criteria applied included studies that addressed the subject of AR, defining concepts and variables that affected the user experience.

As (Kitchenham, 2007 [50]) mentions, after having the initial sample of papers available, the relevance of the study for research must be verified. For this reason, quality criteria were established to decide if a study was a candidate to enter the base of primary studies of this research.

The quality criteria applied to select the works that are part of this study were: (1) the study presents evaluations of AR applications, (2) the study describes criteria associated with user experience, and (3) the study describes the results of research and/or application.

3.2. Phase 2. Selection of Multi-Criteria Methods for the Construction of the Ranking

3.2.1. MCDM Analysis

As it was mentioned in the Section 2.2, multi-criteria decision methods are a useful tool in decision-making when problems are complex and involve a large number of factors, variables and/or characteristics. The nature of this research, having a high number of attributes, implied a rigorous and exhaustive analysis of the influence of each attribute in the industrial training processes with AR applications.

In this research, the AHP method is not used since this method assumes linear dependence relationships between the criteria and the alternatives, which may not be suitable for complex problems with direct and indirect relationships. This limits the ability of the AHP method to analyze the complexity of the system studied in this research. On the other hand, in AHP, marginal changes in the comparison matrices can have a significant impact on the results. The ANP method is also not used, because it can be more complex and require more effort compared to other methods, such as DEMATEL. In general, it may take a longer time to complete the analysis and obtain results. The interpretation of the comparison matrices can be more complicated than with the DEMATEL method. This could limit its usefulness in some contexts. In this research, methods such as TOPSIS, VIKOR, MABAC or TODIM were not applied, since it was not necessary to select or weigh between different alternatives or choices. Moreover, this research focuses on the weighting and selection of the most relevant attributes. The main objective was to assign weights or importance to the different attributes that comprises the system. This weighting of attributes allows for a more complete vision of the augmented reality attributes, without the need to make a comparison between different alternatives.

Therefore, the criterion for selecting FCM was the functionality of the method to achieve consensus among the interested parties, since they actively contribute to the development of the solution. It is able to represent not only the direct relationships between the attributes but also the indirect ones and, if they exist, the inverse relationships can be identified [52,53,54]. Fuzzy cognitive maps make it possible to represent and manage vague or ambiguous knowledge that cannot easily be expressed in exact or precise terms. In addition, they can be used for making decisions in systems with uncertainty. Fuzzy cognitive maps provide an intuitive and visual graphical representation of knowledge and the relationships between concepts.

In the case of clustering, the reasons were compelling because this method allows grouping the elements that have common characteristics and separating those that are more dispersed among them. Thus, this method was essential to generate the attribute subgroups and find interest subgroups depending on the context of use. Clustering makes it possible to identify patterns associated with a set of observations. In addition, it permits the discovery of natural groups, which can represent observations with similar characteristics, facilitating their interpretation. It provides an overview of the structure and distribution of the data, and allows the identification of trends, anomalies, outliers and relationships that are not evident in the data. Finally, another advantage of clustering is that it allows a reduction in the data dimensionality by grouping similar objects and summarizing complex information in a set of representative groups.

In the case of DEMATEL, it is a widely used method among the scientific community, as evidenced by a large number of studies [40,55,56,57,58,59,60,61]. It has demonstrated effectiveness in various areas of knowledge. In addition to being easy to implement, the results obtained are significant when prioritizing the attributes, as well as in identifying the impact between the cause and effect attributes. One advantage of DEMATEL is its ability to identify causal relationships among the criteria, determining whether it is a cause or an effect. In addition, it permits obtaining its weighting or influence within the study context. Compared to ANP, DEMATEL is simpler in terms of its application according to the experts.

3.2.2. Construction of the Instruments to Be Evaluated by Experts

The objective of the evaluation was to identify the attributes that affect the user experience using applications with augmented reality (AR). The concept of user used in this research is presented below.

User: a person who must operate a machine/equipment after receiving a training process using an application with AR.

For the FCM method the type of assessment used is the Likert scale [62,63]. For the case study of this research, the extremes of the valuation are defined as follows:

• Very Low: The attribute does not contribute significantly to the user experience using augmented reality applications.
• Very High: The contribution of the attribute is essential to improve the user experience using augmented reality applications.

The scale defined in

Table 1. Comparative scale used by the experts applying the FCM method.

Scale	Abbreviation	Value
Very high	VH	9
High	H	7
Medium	M	5
Low	L	3
Very low	VL	1

was used to evaluate the attributes to find the influence relationship of the attributes. This scale is the one generally used in fuzzy method applications [32]. In this stage, the instrument “Qualitative Level of Influence instrument” (QLIi) is built.

For the DEMATEL method, the selected assessment scale is the one described in

Table 2. Comparative scale used by experts applying the DEMATEL method.

Scale	Value
No influence	0
Low influence	1
Medium influence	2
High influence	3
Very high influence	4

[33]. This scale was used to evaluate the attributes and their influence among them. In this stage, the instrument is designed to identify the level of influence of the attributes.

3.2.3. Expert Panel Selection Process

The process to select the experts was carried out by taking into account a bank of profiles that was assembled based on information about people who are linked to technology centers, development companies and recognized universities. The steps to link the experts were the following:

• Review of profiles in academic and business platforms. The selected profiles were those who met: (1) two-year experience or more in software development with AR, (2) two-year experience or more in the productive sector projects with AR, and (3) one-year experience or more in the academic environment;
• Sending an email where the investigation is presented, inquiring if the person was interested in joining the panel of experts;
• Review of the people who gave an affirmative answer by the researchers;
• Sending an email with the Qualitative Level of Influence instrument;
• In the second phase of the investigation, the DEMATEL assessment instrument is sent.

3.3. Phase 3. Identification of Causality and Clustering between Attributes

3.3.1. Stage 1: FCM Method

The steps for the implementation of the fuzzy cognitive maps method in this research are described below.

• Step 1: Generation of the initial matrix of success IMS.

This initial matrix is a linguistic assessment carried out by the experts, using the scale of Table 1. The data collected in the QLIi instrument are transformed into numerical vectors associated with each one of the attributes identified. Each vector element represents the value that the corresponding attribute has for each one of the experts interviewed. Each element $O_{ij}$ of the matrix represents the importance that an expert $“j”$ gives to attribute $“i”$, as the results are later transformed into a fuzzy set with values between 0 and 1. The elements $O_{i1} ,O_{i2},\dots ,O_{im}$ are the components of the vector $V_i$ associated with the Attribute belonging to row $“i”$ of the matrix. Thresholds are applied to the IMS in order to adapt the information contained to the real world and maintain the logical integrity of data [41]. Then, IMS is a matrix of $n \times m$, where $n$ is the number of attributes, and $m$ is the number of experts.

• Step 2: Obtaining the fuzzified matrix of success FZMS.

After having the initial matrix of concepts, it was converted into a fuzzy matrix called We removed the heading style, please confirm all FZMS. The FZMS containing the grades of membership of each component of a vector to a fuzzy set. The numerical vectors are converted into fuzzy sets with values within the interval [0, 1] based on Equations (1)–(3).

Find the maximum value in $V_i$, and assign $X_i=1$ to it, that is:

$$MAX\left(O_{iq}\right)\Rightarrow X_i\left(O_{iq}\right)=1$$

(1)

Find the minimum value in $V_i$, and assign $X_i=0$ to it, that is:

$$MIN\left(O_{ip}\right)\Rightarrow X_i\left(O_{ip}\right)=0$$

(2)

In this phase, the minimum and maximum by rows of the matrix must be found to then apply Equation (3), which is the equation used to return the dataset to its fuzzy form.

$$X_{i }\left(O_{ij}\right)=\frac{O_{ij}-Min\left(O_{ip}\right)}{Max\left(O_{iq}\right)-Min\left(O_{ip}\right)}$$

(3)

where $X_i\left(O_{ij}\right)$ is the degree of membership of the element $O_{ij}$ to the vector $V_i$.

• Step 3: Obtaining the strength of relationships matrix of success SRMS.

The importance of each factor is calculated with the FZMS fuzzy matrix in order to generate the Strength of Relationships Matrix of Success SRMS using fuzzy cognitive maps. The SRMS matrix is an $n\times n$ matrix.

$S_{ij}$ can accept values in the interval [−1, 1]. Each attribute is represented as a numerical vector $S_i$ containing $n$ components, one for each concept to be represented in the map.

• Step 4: Indicators assessment.

With the results of the SRMS matrix, the indicators called outdegree, indegree, and centrality [31] are obtained. Additionally, the maximum value per row and the standard deviation are identified, and a dispersion graph between the outdegree and the deviation can be represented.

• The outdegree indicator is the sum of the values in the adjacent array associated with the connectors leaving a node or variable. A transmitter variable presents a high outdegree.
• The indegree indicator is the sum of the values of the adjacent matrix associated with the connections entering a node. This indicator shows the degree of dependence of the variable. A receiving variable has a high indegree.
• The centrality indicator is the sum of the outdegree and indegree indicators. This indicator shows the degree of participation or importance of the variable in the system.

3.3.2. Stage 2: Clustering Method

Due to the large number of attributes to work on in this research, it was not only necessary to prioritize but also to group them to identify common characteristics among them. The type of clustering carried out was non-hierarchical using the K-Mean measure.

The input variables for the clustering implementation were those identified in the SRMS matrix, found through the method of fuzzy cognitive maps. Therefore, the matrix that enters the clustering is $n\times n$. The steps of this procedure are detailed below.

• Step 1: Defining the number of clusters.

Among the most used methods to define the number of clusters is the Elbow method, Equation (4). Where $k$ is the number of clusters, $n_r$ is the number of points in the $r$ group, and $D_r$ is the sum of the distances between all the points in the group.

$$W_{k }={\displaystyle \sum }_{r=1}^k\frac{1}{n_r}crease of{D}_r$$

(4)

In this investigation the number of points is high, so the cost of applying this method is not justified. For this reason, the definition of the number of clusters was based on a previous study [64].

• Step 2: SRMS matrix processing.

The processing of the SRMS matrix to identify the clusters was carried out using the SPSS software [65], which handles algorithms such as intergroup linkage, intragroup linkage, nearest neighbor (simple chaining), farthest neighbor (full chaining), Hierarchical, K-Medoids and Ward.

3.4. Phase 4. Attribute Prioritization and Cause-Effect Analysis by DEMATEL Method

In order to reduce the number of quality attributes to be analyzed through the DEMATEL method, the criterion of literary warrant was used. According to the ANSI/NISO Z-39.19 standard [66], literary warrant is defined as the justification for the representation of a concept in an indexing language or for the selection of a preferred term due to its frequent presence in literature. Based on the literary warrant criterion, the attributes with the highest number of references were selected.

The steps of the DEMATEL method applied in this investigation will be presented below.

• Step 1: Generation of the direct relationship matrix A.

The evaluation of the direct relationships of the attributes was carried out by experts in the subject of software development with AR, using the assessment scale described in Table 2. The direct relationships matrix $\mathit{A}$ is described in Equation (5), where $a_{\mathrm{ij}}$ represents the degree of influence of attribute $i$ on attribute $j$, and sets its corresponding diagonal effect to 0.

$$\mathit{A}=\left[\begin{array}{ccccc}{a}_{11}& \cdots & a_{1j}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{i1}& \cdots & a_{ij}& \cdots & a_{in}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nj}& \cdots & a_{nn}\end{array}\right]$$

(5)

• Step 2: Normalization of the direct relationship matrix.

The normalized matrix $\mathit{M}$ is generated using Equations (6) and (7). The objective of the transformation is to have a matrix with norm less than 1.

$$k=\min \left(\left(\frac{1}{\underset{1\le i\le n}{\max }{{\displaystyle \sum }}_{j=1}^n\left|a_{ij}\right|}\right),\left(\frac{1}{\underset{1\le j\le n}{\max }{{\displaystyle \sum }}_{i=1}^n\left|a_{ij}\right|}\right)\right) i, j \in \left\{1, 2, 3, \dots , n\right\}$$

(6)

$$\mathit{M}=k\ast \mathit{A}$$

(7)

• Step 3: Obtaining the total relationship matrix.

Subsequently, the total relationship matrix $\mathit{S}$ was generated using Equation (8). The $\mathit{S}$ matrix contains the direct and indirect relationships among the attributes.

$$\mathit{S}=M+M^2+M^3+\dots ={{\displaystyle \sum }}_{i=1}^{\infty }{M}^i=M{\left(1-M\right)}^{-1}$$

(8)

• Step 4: Determine the cause group and effect group.

Based on Equations (9)–(11) a vector was generated with the sum of the elements by rows of matrix $\mathit{S}$, called $D$; then a vector was created with the sum of the elements by columns of matrix $\mathit{S}$ and it was called $R$.

$$\mathit{S}={\left|{\mathit{S}}_{ij}\right|}_{n\times n} i,j \in \left\{1, 2, \dots , n\right\}$$

(9)

$$D={\displaystyle \sum }_{j=1}^n{\mathit{S}}_{ij}$$

(10)

$$R={\displaystyle \sum }_{i=1}^n{\mathit{S}}_{ij}$$

(11)

In this step the values $D+R$, $D-R$ and the group are calculated. The “$D+R$” column represents importance and the “$D-R$” column represents correlation [58].

The higher the $D+R$ value, the more important the attribute is for the study. However, if the correlation ($D-R$) is a positive value, the attribute belongs to the “Cause” group, because it served as the origin for other attributes, while if the correlation value is negative, it belongs to the “Effect” group, which means that this attribute is a consequence of others. In the DEMATEL method, an indicator is considered a cause when it has a significant impact or influence on other indicators/factors within the system under study and is regarded as the sources or roots of the study. On the other hand, an indicator is considered an effect when it is influenced by other indicators in the system and is subject to changes or impacts due to those influences.

• Step 5: Weighting of the attributes.

In this step, the attributes that have the greatest weight in augmented reality applications are identified. For the calculation of the weighting coefficient, Equation (12) [55] is used. The standardized coefficients are obtained by applying Equation (13) [55].

$$W_i=\sqrt{{\left(D_i+R_i\right)}^{2 }+{\left(D_i-R_i\right)}^{2 }}$$

(12)

$$W_i=\frac{W_i}{{\sum }_{i=1}^n{W}_i}$$

(13)

4. Case Study

4.1. Selected Quality Attributes

After applying the review protocol described in the Section 3.1, 101 studies passed the quality evaluation phase of this research. The summary of the execution of the systematic literature review protocol is presented in Figure 3. Relevant information on basic AR concepts, UX quality criteria, trends and challenges of AR applications was collected from these studies.

From the analysis of the studies in this investigation, four types of AR were identified: HMD, Mobile, PAR (Pervasive Augmented Reality) and SAR (Interactive Spatial Augmented Reality). Most works use mobile AR apps. Figure 4 presents the distribution. It can be seen that the most used type of AR is the mobile AR with 54%.

Six AR application areas were identified: Education, Entertainment and Gaming, Industrial Application, Medical, Navigation and Driving, and Tourism and Exploration. The most widespread application area among the analyzed studies is Industrial Application. The studies focused on Industrial Application are presented in

Table 3. Studies of the industrial application area.

Study	Identified Attributes	Type of AR
AR-based interaction for human-robot collaborative manufacturing [67]	Safety, Information Processing, Ergonomics, Autonomy, Competence, Relatedness	HMD
End-Users’ augmented reality utilization for architectural design review [68]	Satisfaction, Physical Demand, Perceived usefulness, Perceived ease of use
Combining Embedded Computation and Image Tracking for Composing Tangible Augmented Reality [69]	Usability
The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos [70]	Camera height, Actor behavior, Viewer position
Augmented reality user interface evaluation performance measurement of HoloLens, Moverio and mouse input [20]	Performance, Satisfaction, Cognitive demand
Optical-Reflection Type 3D Augmented Reality Mirrors [71]	Space perception
Ethnographic study of a commercially available augmented reality HMD app for industry work instruction [11]	Task Performance: Accuracy, Task completion, Consistency, Time taken
DMove: Directional Motion-based Interaction for Augmented Reality Head-Mounted Displays [19]	Correctness
Comparing HMD-Based and Paper-Based Training [15]	Performance, Task completion, Cognitive demand
Comparative Reality: Measuring User Experience and Emotion in Immersive Virtual Environments [12]	Mental effort, Engagement, Task Difficulty, Comfortability, Biofeedback
Overcoming Issues of 3D Software Visualization through Immersive Augmented Reality [22]	Navigability, Task completion, Correctness, Space Perception, Engagement
A study on user experience evaluation of glasses-type wearable device with built-in bone conduction speaker: Focus on the Zungle Panther [23]	Usability, Aesthetic
Visual User Experience Difference: Image compression impacts on the quality of experience in augmented binocular vision [72]	Image quality metrics
Study on assessing user experience of augmented reality applications [7]	Performance	Mobile
Holistic User eXperience in Mobile Augmented Reality Using User eXperience Measurement Index [9]	Attractiveness, Efficiency, Perspicuity Dependability, Stimulation, Novelty
Emotions detection of user experience (UX) for mobile augmented reality (mar) applications [73]	Engagement
Attractiveness of augmented reality to consumers [74]	Engagement
CryptoAR Wallet: A Blockchain Cryptocurrency Wallet Application that Uses Augmented Reality for On-chain User Data Display [75]	Usability
Fire in Your Hands: Understanding Thermal Behavior of Smartphones [76]	Coefficient of thermal spreading
User experience problems in immersive virtual environments [77]	Immersion, Satisfaction, Credibility, Naturalness
User experience and user acceptance of an augmented reality based knowledge-sharing solution in industrial maintenance work [78]	Satisfaction, Usability, Usefulness
Measuring user experience of mobile augmented reality systems through non-instrumental quality attributes [13]	Aesthetics
User experience in mobile augmented reality: Emotions, challenges, opportunities and best practices [79]	Engagement, Cognitive demand
UX design principles for mobile augmented reality applications [80]	Space Perception
Multi-layered mobile augmented reality framework for positive user experience [81]	Ergonomics, Usability
Enhancing user experience through physical interaction in handheld Augmented Reality [82]	Task completion
Augmented Reality Interfaces [83]	Usability
A case study on user experience (UX) evaluation of mobile augmented reality prototypes [84]	Ease of use, Naturalness, Novelty
MARTI: Mobile Augmented Reality Tool for Industry [85]	Space Perception, Aesthetics, Ergonomics
AVIKOM: towards a mobile audiovisual cognitive assistance system for modern manufacturing and logistics [86]	Usability, Ergonomics
Mixed prototypes for the evaluation of usability and user experience: simulating an interactive electronic device [87]	Usability, Performance, Task completion, Satisfaction, Ergonomics	PAR
Kinetic AR: A Framework for Robotic Motion Systems in Spatial Computing [88]	Usability	SAR
Interactive Spatial Augmented Reality in Collaborative Robot Programming: User Experience Evaluation [89]	Space Perception
Enhancing user experiences of mobile-based augmented reality via spatial augmented reality: Designs and architectures of projector-camera devices [90]	Novelty, Practicality

.

Overall, the 101 reviewed studies present information related to how UX-related features are evaluated in AR applications. From these works, 87 quality attributes were extracted, which are presented in Supplementary Material Table S1.

4.2. Selected Multicriteria Methods

4.2.1. Analysis of the Selected Multicriteria Methods

Based on the analysis performed in Section 3.2.1, three multicriteria methods were selected for this research. These methods contribute to this research because the FCM method was used to identify how much an attribute influenced the user experience using augmented reality applications. The expert evaluated it with a linguistic scale from very low to very high (Table 1). The second method was Clustering, which helped to group the 87 attributes and identify similarities among them. In the last phase of the investigation, the DEMATEL method was applied to prioritize and find causality analysis among the attributes.

Due to all of the above, in this research these three methods were chosen. The strength of the methodological process is the integration of the methods to obtain the most appropriate solution.

4.2.2. Applied Instruments and Experts Profiles

After applying the protocol described in Section 3.2.3, the experts completed and sent their contributions to the investigation. The evaluations of the experts were tabulated and contrasted by the researchers in order to apply the corresponding statistical methods. In this way, the relationships evaluation matrix among the attributes was carried out by the expert panel.

Table 4. Experts panel profiles.

Academic Degrees			Software Development with AR	Experience in the Productive Sector with AR	Academic Experience
Eng.	Master	PhD	Average	Average	Average
20%	20%	60%	13.4 years	12.2 years	13.4 years

describes the profiles of the experts.

4.3. FCM Method Application

4.3.1. Indicator Analysis of the IMS and SRMS Matrices

After applying “Step 1: Generation of the initial matrix of success IMS”, the IMS matrix is obtained.

Table 5. Analysis indicators of the 87 attributes resulting from the IMS and SRMS matrix.

ID	Attribute	IMS			SRMS
		Average	Score Max	Score Min	Outdegree	Indegree	Centrality
A01	Usability [14,16,23,69,75,78,81,83,87,91,92,93,94,95,96,97,98,99]	9.0	9	9	53.40	53.40	106.80
A02	Usefulness [78,100]	7.0	9	3	67.52	67.52	135.04
A03	Efficiency [9,101,102,103,104,105,106,107,108]	7.8	9	5	52.46	52.46	104.92
A04	Effectiveness [103,107,109]	8.6	9	7	44.96	44.96	89.92
A05	Navigability [16,21,102,107,110,111,112,113,114,115,116,117]	8.2	9	7	39.64	39.64	79.28
A06	Accessibility [118]	6.2	9	1	66.72	66.72	133.44
A07	Perspicuity [9,101,102,104,105,106]	6.6	9	1	67.64	67.64	135.28
A08	Dependability [9,101,102,104,105,106]	7.4	9	5	46.12	46.12	92.24
A09	Coefficient of thermal spreading [76]	3.4	5	1	66.40	66.40	132.80
A10	Interactivity [118]	7.8	9	5	63.10	63.10	126.20
A11	Safety [67]	7.0	9	3	67.52	67.52	135.04
A12	Practicality [90]	5.0	9	1	64.88	64.88	129.76
A13	Performance [7,11,15,16,20,87,107,119,120,121]	7.4	9	5	49.20	49.20	98.40
A14	Accuracy [11,122]	7.8	9	7	56.56	56.56	113.12
A15	Task completion [11,12,15,16,22,82,87,107,116,119,120]	5.8	9	1	68.14	68.14	136.28
A16	Consistency [11]	6.6	9	1	67.64	67.64	135.28
A17	Time taken/task completion time [11,120]	6.6	9	3	56.22	56.22	112.44
A18	Accomplish the task [93,120]	6.6	9	1	67.82	67.82	135.64
A19	Correctness [16,19,21,22,103,111,123,124]	7.4	9	3	67.22	67.22	134.44
A20	Reliability [108,118,125]	6.2	9	1	51.54	51.54	103.08
A21	Space Perception [12,14,22,71,80,85,89,126,127,128]	5.8	9	1	66.46	66.46	132.92
A22	Image quality [72,129,130,131]	7.0	9	3	52.56	52.56	105.12
A23	Visual clarity [118]	7.4	9	5	66.40	66.40	132.80
A24	Camera height [70]	7.8	9	7	50.76	50.76	101.52
A25	Actor behavior [70]	8.2	9	7	65.00	65.00	130.00
A26	Viewer position [70]	6.6	9	1	67.64	67.64	135.28
A27	Identifiability [17,118]	7.4	9	7	45.84	45.84	91.68
A28	Representation [100]	5.4	9	1	64.08	64.08	128.16
A29	Transparency [132]	4.2	7	1	65.90	65.90	131.80
A30	Comprehensibility [108]	8.6	9	7	53.88	53.88	107.76
A31	Comprehensivity [118]	7.4	9	5	58.48	58.48	116.96
A32	Memorability [118]	5.8	9	3	59.46	59.46	118.92
A33	Information Processing [67]	5.8	9	1	68.14	68.14	136.28
A34	Engagement [12,22,73,74,79,95,103,119,127,133,134,135,136,137]	7.0	9	1	66.72	66.72	133.44
A35	Satisfaction [20,68,77,78,87,95,100,103,107,109,119,133,134]	7.0	9	1	66.72	66.72	133.44
A36	Perceived satisfaction [138]	6.6	9	1	67.64	67.64	135.28
A37	Enjoyment [8,138,139]	7.4	9	1	64.52	64.52	129.04
A38	Enjoy [140]	7.0	9	1	66.72	66.72	133.44
A39	Stimulation [9,17,101,102,104,105,106,108]	5.8	9	1	64.56	64.56	129.12
A40	Pleasant [98]	8.6	9	7	64.52	64.52	129.04
A41	Frustration [107,121,139]	6.2	9	1	43.38	43.38	86.76
A42	Valence [135,140]	5.4	9	1	62.40	62.40	124.80
A43	Importance [132,140]	6.2	9	1	63.46	63.46	126.92
A44	Arousal [135,140]	5.4	7	1	60.94	60.94	121.88
A45	Impressed [140]	6.6	9	1	64.52	64.52	129.04
A46	Connectedness [125]	5.0	9	1	64.88	64.88	129.76
A47	Comfortability [12]	7.8	9	5	59.90	59.90	119.80
A48	Biofeedback, [12,18,119,127]	5.0	9	1	64.88	64.88	129.76
A49	Competence [67]	5.4	9	1	65.48	65.48	130.96
A50	Continuance intention [8]	4.6	9	1	62.54	62.54	125.08
A51	Aesthetics [13,141]	7.4	9	3	67.22	67.22	134.44
A52	Attractiveness [9,101,102,104,105,106,108]	5.0	9	1	65.00	65.00	130.00
A53	Ergonomics [67,81,85,87]	7.4	9	1	64.52	64.52	129.04
A54	Ease-of-Interaction [142]	7.8	9	5	67.82	67.82	135.64
A55	Cognitive demand [12,15,20,79,107,121,124,139]	6.6	9	1	67.64	67.64	135.28
A56	Cognitive load [143]	6.6	9	1	67.64	67.64	135.28
A57	Physical Demand [68,107,121]	7.8	9	7	56.56	56.56	113.12
A58	Temporal demand [107]	6.2	9	5	52.86	52.86	105.72
A59	Effort [107,121]	5.8	9	1	66.46	66.46	132.92
A60	Learning performance [92,138,144]	7.8	9	7	56.56	56.56	113.12
A61	Novelty [9,90,101,102,104,105,106]	7.0	9	5	55.22	55.22	110.44
A62	Innovation [84,108]	7.8	9	7	54.28	54.28	108.56
A63	Perceived usefulness [68,145]	8.2	9	7	45.44	45.44	90.88
A64	Perceived ease of use [68,145]	9.0	9	9	53.40	53.40	106.80
A65	Easy to use [84,100]	8.2	9	7	65.00	65.00	130.00
A66	Willingness to Use [95]	6.2	9	1	63.46	63.46	126.92
A67	Affordance [146]	7.4	9	5	58.48	58.48	116.96
A68	Price Value [16,147]	5.0	9	1	64.88	64.88	129.76
A69	Social interaction [95]	5.0	9	1	64.88	64.88	129.76
A70	Social presence [8,95]	5.4	9	1	65.48	65.48	130.96
A71	Social behavior [138]	5.4	9	1	65.48	65.48	130.96
A72	Social Richness [95]	6.2	9	3	61.42	61.42	122.84
A73	Social Realism [95]	5.8	9	1	64.38	64.38	128.76
A74	Achievement [8]	6.6	9	5	45.36	45.36	90.72
A75	Self-presentation [8]	5.4	9	1	65.48	65.48	130.96
A76	Captivation [125]	7.8	9	5	58.18	58.18	116.36
A77	Immersion [77]	6.2	9	1	58.42	58.42	116.84
A78	Immersion aspect [96]	5.8	9	1	60.62	60.62	121.24
A79	Fantasy [8]	5.4	9	1	65.86	65.86	131.72
A80	Escapism [8]	5.8	9	1	66.46	66.46	132.92
A81	Attention [140,143]	6.6	9	5	53.04	53.04	106.08
A82	Intuitiveness [121]	8.2	9	7	50.28	50.28	100.56
A83	Naturalness [77,84]	5.8	7	1	64.52	64.52	129.04
A84	Sense of realism [122]	7.8	9	5	58.18	58.18	116.36
A85	Autonomy [67]	6.2	7	3	42.68	42.68	85.36
A86	Relatedness [67]	6.6	9	1	67.64	67.64	135.28
A87	Credibility [77]	8.2	9	7	65.00	65.00	130.00

describes the analysis of the IMS matrix, identifying the average, the minimum score, and the maximum score obtained for each attribute.

4.3.2. Analysis of the FZMS Matrix Importance

As described in “Step 2: Obtaining the fuzzified matrix of success FZMS”, three equations are used to obtain the FZMS matrix. From the FZMS matrix, it is important to analyze the level of importance of the attributes.

Table 6. Importance values obtained from the FZMS.

Importance: 1.0	Importance: 0.8	Importance: 0.7	Importance: 0.6	Importance: 0.5	Importance: 0.4	Importance: 0.3	Importance: 0.2
$\left[\begin{array}{c}{A}_{01}\\ A_{64}\end{array}\right]$	$\left[\begin{array}{c}{A}_{04}\\ A_{30}\\ A_{34}\\ A_{35}\\ A_{37}\\ A_{38}\\ A_{40}\\ A_{53}\\ A_{83}\\ A_{85}\end{array}\right]$	$\left[\begin{array}{c}{A}_{02}\\ A_{03}\\ A_{06}\\ A_{07}\\ A_{10}\\ A_{11}\\ A_{16}\\ A_{18}\\ A_{19}\\ A_{20}\\ A_{22}\\ A_{26}\\ A_{36}\\ A_{41}\\ A_{43}\\ A_{44}\\ A_{45}\\ A_{47}\\ A_{51}\\ A_{54}\\ A_{55}\\ A_{56}\\ A_{66}\\ A_{76}\\ A_{77}\\ A_{84}\\ A_{86}\end{array}\right]$	$\left[\begin{array}{c}{A}_{05}\\ A_{08}\\ A_{09}\\ A_{13}\\ A_{15}\\ A_{17}\\ A_{21}\\ A_{23}\\ A_{25}\\ A_{28}\\ A_{31}\\ A_{33}\\ A_{39}\\ A_{42}\\ A_{49}\\ A_{59}\\ A_{63}\\ A_{65}\\ A_{67}\\ A_{70}\\ A_{71}\\ A_{73}\\ A_{75}\\ A_{78}\\ A_{79}\\ A_{80}\\ A_{82}\\ A_{87}\end{array}\right]$	$\left[\begin{array}{c}{A}_{12}\\ A_{29}\\ A_{32}\\ A_{46}\\ A_{48}\\ A_{50}\\ A_{52}\\ A_{61}\\ A_{68}\\ A_{69}\\ A_{72}\end{array}\right]$	$\left[\begin{array}{c}{A}_{14}\\ A_{24}\\ A_{57}\\ A_{60}\\ A_{62}\\ A_{74}\\ A_{81}\end{array}\right]$	$\left[A_{58}\right]$	$\left[A_{27}\right]$

presents the attributes grouped by level of importance. The importance is a vector that oscillates between [0, 1], where 1 represents the highest value that can be given to an attribute. The attributes with the highest value were Usability and Perceived ease of use.

4.3.3. Analysis of the SRMS Matrix Results

The calculation to obtain the SRMS matrix described in the “Step 3: Obtaining the strength of relationships matrix of success SRMS” was implemented in MATLAB (see Algorithm 1). The complexity of the algorithm is $\mathsf{\Theta }\left(n^2·m\right)$. We consider $m$ as a constant, then,

$$\mathsf{\Theta }\left(n^2·m\right)=\mathsf{\Theta }\left(n^2\right);n\gg m$$

Algorithm 1 Implementation of matrix SRMS
1:	Enter the FZMS matrix.
2:	The size of the FZMS matrix is calculated.
3:	k = 1
4:	while (k $\le$ n)
5:	for (j = 1, m)
6:	for (i = 1, n)
7:	if (FZMS ((k, j) − FZMS (i, j)) < 0)
8:	B(i, j) = FZMS (i, j) − FZMS ((k, j)
9:	else
10:	B(i, j) = FZMS ((k, j) − FZMS (i, j)
11:	end if
12:	end for
13:	end for
14:	for (i = 1, n)
15:	accumulated = 0
16:	for (j = 1, m)
17:	accumulated = accumulated + B(i, j)
18:	end for
19:	Average = accumulated/m
20:	P(i) = Average
21:	D(i) = 1 − P(i)
22:	end for
23:	for (j = 1, n)
24:	S(k, j) = D(j)
25:	if (k = j)
26:	S(k, j) = 0
27:	end if
28:	end for
29:	k = k + 1
30:	end while

The SRMS matrix is not presented in this article due to its large size. It can be seen at Supplementary Material Table S2. Next, the structural indicators of the SRMS matrix will be analyzed.

4.3.4. Indicators Analysis

As described in the “Step 4: Indicators assessment”, Outdegree and Indegree indicators are necessary to obtain Centrality. Therefore, these two indicators affect the degree of importance of each of the variables presented in Table 5.

As it can be seen in Table 5, the values of outdegree and indegree are the same. These two indicators have the same value, since they were calculated using the SRMS matrix. The SRMS matrix is symmetric, therefore the sum of the rows equals the sum of the columns.

Figure 5a presents the relationship between the Outdegree indicator vs. the deviation. The attributes with the highest standard deviation value are Enjoyment, Pleasant, Ergonomics, and Naturalness, and correspond to the Outdegree of quartile 2 or median. The attribute with the lowest deviation value is Effectiveness located between the minimum and quartile 1.

The diagram in Figure 5b presents the distribution of the centrality indicator values in Table 5. The minimum value is 79.28, quartile 1 is 112.78, quartile 2 is 129.04, quartile 3 is 132.86 and the maximum value is 136.28. As it can be seen in Figure 5b the centrality indicator does not have outliers.

4.3.5. Weighting of Attributes Based on the Centrality Criterium

According to the degree of importance of each variable, 13 attributes with the highest weighting were identified. The three highest values were selected, finding attributes with the same value. As a result, the attributes described in Figure 6 were selected.

The thirteen (13) attributes identified with the highest value of the centrality indicator represent the most important attributes for the user experience of a worker who uses augmented reality applications in their industrial training process. Although Figure 6 represents the attributes selection with the highest weighting; the prioritization of the first two groups from 1 to 2 and from 3 to 4 demarcate only the aspirational, compliance, and familiarity factors which workers will undergo during a training with AR. Thus, the user occupation will be focused on learning, organizing, and intuitively addressing each of the requirements, with the aim of fulfilling certain types of tasks. In the next group, as factors 5 to 11 are more related to aspects of load and cognitive processing mediated by technologies, the motivational learning process will be focused on stimulating a perception of individual or collaborative achievement in the user. The last group, from 12 to 13, will stimulate the sensation of a safe environment where the learning will be seen as a useful, necessary, reliable, and effective factor.

This concentric order reveals how the AR importance in industrial training environments must respond to an order of attributes and priorities, which in deterministic terms will affirmatively condition the experiences that users will live to guarantee the training of any process.

4.4. Clustering Method Application

4.4.1. Distances and Cases by Clusters

As mentioned in Section 3.3.2, in [64] two categories and nine subcategories of quality attributes in augmented reality applications are identified; based on this proposal, in this work $k=2$ and $k=9$ are used.

The data matrix that enters SPPS is the SRMS matrix. After its processing, the distances between clusters and the number of cases per cluster for the two $k$ selected are obtained.

Figure 7 and Figure 8 present the distances between clusters and the number of cases per cluster.

4.4.2. Clustering Method Results

As mentioned in “Step 1: Defining the number of clusters” of Section 3.3.2, two values for $k$ were used. It was proceeded to calculate the initial cluster centers, membership clusters, final cluster centers, distances among final cluster centers, and the number of cases in each cluster. After analyzing the results of the resulting clusters with $k=2$ and $k=9$, it was concluded that with $k=9$ a better analysis of relationships among attributes can be performed. Therefore, only the results for the 9 clusters will be presented in this section. Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 describe the clusters identified when $k=9$.

Figure 9 presents cluster 1 with four attributes that are related to the use of applications in its two dimensions: real and perceived. The connection that emerges from these attributes is based on technological determinism, where a factor such as camera height conditionally influences the cognitive dimension of users. The consumption of content assisted with augmented reality leads to translating these reactions into collective evaluations, where the degrees of usability, the sensation of ease of use, and intuitiveness, although they are attributes anticipated objectively by the developers, are perceived by the users as subjective evaluations, where its effectiveness is represented by the quality of the lived experience.

Cluster 2, described in Figure 10, has 27 attributes. This cluster is the second largest in this iteration. However, the connection of the attributes that appear in this cluster seems to be predominantly focused on aesthetics and human factors. It places its accent on emotional stimuli by subordinating cognitive aspects, with the aim of generating experiences in the user that evoke sensations of wonder, naturalness, spontaneity, pleasure, and happiness.

Figure 11 presents cluster 3 with 12 attributes. In this cluster features related to accuracy, time to complete the task, reliability, and understanding the application content are found. The connection among these attributes is justified by the fact that the user can meet his needs for consumption and compression of digital content from instances and/or processes that imply the least amount of cognitive effort to achieve it. The attributes of this cluster point to precision and synthesis of the information, in order to achieve understanding, internalization, and learning, among other important aspects.

Figure 12 groups two attributes with the same distance, one focused on assistance and the other on autonomy. The relationship between them is based on the coherence that must exist between a sensation of choice derived from or dependent on assistance and support, by offering users different ways or options to complete the same process easily and precisely.

Cluster 5 groups the attributes related to the guidance and user interaction within an application, as well as the perceptions of that use (Figure 13). In this cluster, those attributes exclusively linked to avoiding user frustration due to inadequate user flow planning are prioritized. In the relationship among these four attributes, the ease of navigation use prevails, the passage and transitions from one wireframe to another, and some aspects leading to the practical utility perception of the content offered by the digital product with AR.

Cluster 6 in Figure 14 has four attributes that are responsible for attention and innovation, and it highly stimulates the feeling of novelty and wonder from innovative aspects in the interface and in the disruptive way in which the digital content with augmented reality is displayed.

Cluster 7 presented in Figure 15 is the most numerous with 31 attributes. It focuses on security, representation, and social interaction. These attributes are based on trust; hence, they are present in digital environments and products that offer security, practicality, transparency, ease of use, desires, group approval, and aspirational expectations, among other aspects that are subordinated to dominant contexts of social interaction. These cluster attributes are consistent with the weighting results of the FCM method presented in Figure 6. This fact emphasizes the importance of task fulfillment and adaptation to the work environment that the worker must develop.

Cluster 8 has two attributes with the same distance (Figure 16). When the algorithm finds attributes that have the same distance, it groups them and does not include another attribute with a close value. Provided that performance is subject to the prevention of bottlenecks to which users are exposed when interacting with an AR digital product, the Identifiability attribute is also related to the same principle, but differently. While the reduction in waiting times arises from simple and efficient algorithms composed of the least number of actions to process, if the graphic and chromatic unit that comes from the Identifiability attribute is assertively designed, it will be understandable from the UI and the sensation of performance and efficiency that the user will experience is considered as affirmative.

Cluster 9 has only one attribute with a distance value of zero (Figure 17). Although this solitary attribute is subject to the desire to progress and compete with others, unlike Cluster 7, it does not take its reference point in social interaction, but rather in the achievements, scopes or growth that reflect an individual progressive improvement.

4.5. DEMATEL Method Application

4.5.1. Results Matrices

Applying the criterion of literary warrant, the dimensionality of the attributes is reduced from 87 to 19 (see Section 3.4). For this reason, the analysis of the experts for the DEMATEL method was built from a 19 × 19 attributes matrix.

The attributes that entered into this phase of the project are those described in

Table 7. Attributes that entered phase 4 with DEMATEL.

ID	Attribute
A01	Usability
A03	Efficiency
A05	Navigability
A07	Perspicuity
A08	Dependability
A13	Performance
A15	Task completion
A19	Correctness
A21	Space Perception
A22	Image quality
A34	Engagement
A35	Satisfaction
A39	Stimulation
A48	Biofeedback
A51	Aesthetics
A52	Attractiveness
A55	Cognitive demand
A53	Ergonomics
A61	Novelty

. It is important to clarify that throughout the article the IDs of each attribute are those defined in Table 5.

According to “Step 1: Generation of the direct relationship matrix A” the direct relationships matrix among attributes $\mathit{A}$ with dimensions 19 × 19 was generated, based on the information recorded by the experts. This matrix was found as the average of the evaluations by the experts.

Table 8. Initial direct relationships matrix resulting from the evaluation average by the panel of experts.

	A01	A03	A05	A07	A08	A13	A15	A19	A21	A22	A34	A35	A39	A48	A51	A52	A53	A55	A61
A01	0.0	3.6	3.8	3.0	3.2	2.8	3.2	3.6	2.6	1.6	3.4	3.2	3.6	2.4	3.2	3.0	3.0	3.6	2.0
A03	3.6	0.0	2.6	3.2	3.4	3.0	3.2	2.8	1.8	1.4	2.6	3.0	2.6	3.2	2.4	1.8	2.2	4.0	1.6
A05	3.8	3.6	0.0	3.2	2.4	3.2	3.0	3.4	2.2	2.6	3.6	3.4	3.4	2.0	3.2	2.8	3.2	3.6	1.6
A07	3.6	3.0	3.2	0.0	3.0	1.6	3.2	3.2	2.2	1.2	2.8	3.4	3.2	2.4	2.8	2.8	2.6	3.2	2.0
A08	3.8	3.2	2.6	3.2	0.0	2.4	2.8	3.0	2.0	1.0	3.2	3.2	2.6	2.0	2.8	2.8	2.4	2.6	2.2
A13	3.2	4.0	3.0	1.8	3.0	0.0	2.6	3.0	1.6	2.0	3.6	3.4	2.6	2.0	1.8	1.8	2.2	3.2	1.4
A15	3.0	3.0	2.6	2.0	2.4	2.4	0.0	2.2	1.6	1.0	2.2	2.8	3.0	2.6	1.2	1.6	2.6	2.8	1.4
A19	3.6	3.2	2.8	3.2	3.2	2.0	2.8	0.0	1.8	1.2	3.4	3.4	3.0	2.6	3.4	3.2	2.4	3.0	1.8
A21	3.0	2.6	2.8	1.8	2.6	1.6	1.8	2.6	0.0	1.2	2.4	2.4	3.2	2.4	3.2	2.4	2.2	3.0	2.0
A22	3.4	2.4	2.6	2.6	1.6	2.6	1.6	2.8	1.4	0.0	4.0	2.8	3.8	2.6	4.0	4.0	1.6	3.4	1.4
A34	3.0	1.8	1.8	2.2	3.0	2.0	2.4	3.2	1.6	2.4	0.0	3.6	3.2	2.4	2.6	3.4	2.8	3.0	2.2
A35	3.4	3.0	3.0	2.4	2.8	1.8	2.8	2.8	2.2	1.8	4.0	0.0	3.8	2.8	3.0	2.2	2.8	3.2	2.0
A39	3.6	2.0	2.2	2.4	2.6	1.4	2.8	2.8	2.2	2.4	3.4	3.4	0.0	3.0	2.4	2.4	2.6	3.8	2.2
A48	2.4	2.0	1.2	2.2	1.4	1.4	2.2	2.2	1.8	1.6	1.8	2.6	2.6	0.0	1.6	2.4	2.0	2.6	2.0
A51	3.2	2.6	1.8	3.0	2.4	2.0	2.2	2.8	3.4	3.2	3.8	3.2	4.0	2.4	0.0	4.0	3.0	2.6	1.6
A52	3.0	2.4	2.2	3.6	2.8	1.8	3.0	2.2	3.0	2.4	4.0	4.0	4.0	2.8	4.0	0.0	3.0	2.8	2.0
A53	3.2	2.8	3.2	2.6	2.0	3.2	2.6	2.6	2.6	1.6	3.0	3.6	3.4	3.2	2.4	2.2	0.0	3.0	1.6
A55	2.8	2.8	2.0	2.6	1.6	1.6	2.2	2.8	2.2	1.4	2.6	2.6	2.8	2.6	3.2	2.4	2.6	0.0	2.2
A61	2.4	2.2	2.4	1.8	3.4	1.6	1.8	2.0	2.0	2.4	3.4	2.8	3.0	2.6	2.6	2.6	2.0	2.6	0.0

presents matrix $\mathit{A}$ with the obtained averages.

Applying the procedure of “Step 2: Normalization of the direct relationship matrix”, the matrix $\mathit{M}$ is obtained. A fragment of the results of $\mathit{M}$ is presented in

Table 9. Normalized direct relationships matrix fragment resulting from the transformation of matrix $\mathit{A}$.

	A01	A15	A34	A35	A39	A51	A53
A01	0.00	0.06	0.06	0.06	0.07	0.06	0.05
A15	0.05	0.00	0.04	0.05	0.05	0.02	0.05
A34	0.05	0.04	0.00	0.07	0.06	0.05	0.05
A35	0.06	0.05	0.07	0.00	0.07	0.05	0.05
A39	0.07	0.05	0.06	0.06	0.00	0.04	0.05
A51	0.06	0.04	0.07	0.06	0.07	0.00	0.05
A53	0.06	0.05	0.05	0.07	0.06	0.04	0.00

(full data can be found in Table S3: Matrices generated with DEMATEL).

Applying Equation (8) presented in “Step 3: Obtaining the total relationship matrix”, the $\mathit{S}$ matrix is obtained.

Table 10. Total relationships matrix fragment resulting from the transformation of matrix $\mathit{M}$.

	A01	A15	A34	A35	A39	A51	A53
A01	0.43	0.40	0.48	0.47	0.49	0.42	0.39
A15	0.37	0.26	0.35	0.36	0.37	0.29	0.30
A34	0.42	0.34	0.36	0.42	0.42	0.36	0.34
A35	0.45	0.37	0.45	0.38	0.45	0.39	0.36
A39	0.43	0.35	0.42	0.42	0.37	0.36	0.34
A51	0.46	0.36	0.46	0.45	0.47	0.35	0.37
A53	0.44	0.36	0.43	0.44	0.44	0.37	0.30

presents a fragment of the data in matrix $\mathit{S}$.

Applying Equations (10) and (11) described in “Step 4: Determine the group of causes and the group of effects”, the vectors $D$ and $R$ are obtained.

Table 11. Attributes classified by cause or effect according to DEMATEL.

ID	Attribute	D + R	D − R	Group
A22	Image quality	11.48	2.26	CAUSE
A61	Novelty	10.90	1.37
A05	Navigability	14.01	1.18
A13	Performance	11.92	1.10
A52	Attractiveness	14.08	0.71
A21	Space Perception	11.55	0.66
A53	Cognitive demand	13.26	0.39
A07	Perspicuity	13.56	0.32
A51	Aesthetics	14.13	0.23
A08	Dependability	13.32	0.16
A19	Correctness	14.07	−0.01	EFFECT
A03	Efficiency	13.76	−0.25
A01	Usability	15.72	−0.41
A15	Task completion	12.26	−0.88
A35	Satisfaction	14.89	−0.95
A34	Engagement	14.49	−1.37
A39	Stimulation	14.68	−1.37
A48	Biofeedback	11.57	−1.39
A55	Ergonomics	13.87	−1.76

presents the calculation values of D + R and D − R and the group. This table is organized according to the column “D − R” in descending order. Ten attributes were identified in the Cause group and nine in the Effect group.

From the analysis of these indicators, it can be inferred that the “Usability” attribute is the one with the highest value of importance, so it should be considered a priority to improve this criterion. It is followed by Satisfaction, Stimulation, Engagement and Aesthetics, with values higher than 14.13. An interesting finding is that four of the five most important attributes are located in the “Effect” group.

The attributes located in the Cause group such as Image quality, Novelty, Navigability, Performance, Attractiveness, Space Perception, Cognitive demand, Perspicuity, Aesthetics and Dependability are responsible for originating the effect attributes.

As described in “Step 5: Weighting of the attributes”, the weighted coefficients and standardized coefficients of the 19 attributes were calculated.

Table 12. Weighting coefficient of the 19 attributes.

A01	A03	A05	A07	A08	A13	A15	A19	A21	A22	A34	A35	A39	A48	A51	A52	A53	A55	A61
15.73	13.76	14.06	13.56	13.32	11.97	12.30	14.07	11.57	11.70	14.55	14.92	14.75	11.65	14.14	14.09	13.26	13.98	10.99

presents the weighting coefficients obtained for the 19 attributes. The standardized coefficients are described in

Table 13. Standardized coefficient of the 19 attributes.

A01	A03	A05	A07	A08	A13	A15	A19	A21	A22	A34	A35	A39	A48	A51	A52	A53	A55	A61
0.0618	0.0541	0.0553	0.0533	0.0524	0.0471	0.0483	0.0553	0.0455	0.0460	0.0572	0.0587	0.0580	0.0458	0.0556	0.0554	0.0521	0.0550	0.0432

.

4.5.2. Ranking by DEMATEL

After applying the weighting (Table 12) and standardized coefficients (Table 13), the attributes with the greatest weight in the augmented reality applications were identified and presented in Figure 18.

At the top of the ranking is the “Usability” attribute, corresponding to the measure of how well a specific user in a specific context can use a product/design to achieve a defined goal effectively, efficiently, and satisfactorily. Designers usually measure a design’s usability throughout the development process from wireframes to the final deliverable to ensure maximum usability [14,16,23,69,75,78,81,83,87,91,92,93,94,95,96,97,98,99].

In the second position is “Satisfaction”, which can be defined as that affective, emotional, and cognitive appeal that digital products and services arouse in the users for whom they are developed [20,68,77,78,87,95,100,103,107,109,119,133,134].

In the third position is “Stimulation”, defined as satisfying inexplicit desire by stimulating personal development; by communicating a sense of belonging and by in-spiring/evoking memories [9,17,101,102,104,105,106,108].

4.5.3. Influence Relationships among Attributes

According to the DEMATEL results, Figure 19 describes how much one attribute affects another and how much it is affected by another. The attribute “A05—Navigability” greatly influences the attribute “A01—Usability” which would be an expected result because if an application does not handle adequate navigability, it will be proportionally reflected in the usability. The coherence of a product with augmented reality is defined by the intuitiveness that highlights its navigability; this is perceived by the user as a sensation of ease of use.

Nine attributes affect the “A01—Usability” attribute, which indicates that this is the one that receives the greatest influence from the other factors. The attribute that most affects the other problems is “A01”, with an occurrence of 11 out of 19. This bidirectional relationship of influence denotes the great importance of the Usability attribute in the processes of creating and using augmented reality applications over all the other attributes.

Although the DEMATEL Method in Figure 18 reveals to what extent an attribute causes a deterministic effect or influences another, the attribute with the greatest weight or value will continue to be Usability. The interactive, dependency, affectation or subordination relationship that emerges among the 19 attributes, also shown in Figure 19, is graphed in a single or two-way relationship that the DEMATEL method manages to visualize. This method quantifies how the ease of use of a product with AR, in dynamic relation by weight with its 19 preponderant and hierarchical attributes, can guarantee the motivations for use in any type of user as one of the most important contributions of this research.

Figure 20 shows the influence relationship between attribute A01—Usability and attributes A07—Perspicuity, A19—Correctness and A53—Ergonomics. The graphed data correspond to the values of D + R and D − R of each attribute (see Table 11). The grouping was carried out based on the fact that the affected attributes were located in cluster 2 as a result of applying the clustering method of Section 3.3.2 of this investigation. The bidirectional relationship between A01 and A19 confirms the consistency that exists between design and use since the interface elements must be distinguishable and predictable so that the task workflow can be performed effectively, efficiently, and satisfactorily. A01 directly affects A07, because if the task can be carried out properly, in the end the process will be clear and easy to understand. Between A01 and A53, a two-way relationship is also revealed, because the interactions between the worker and the elements of the system proportionally affect the task development in the context of use.

Figure 21 shows the influence relationship between attribute A01—Usability and attributes A05—Navigability and A08—Dependability. The graphed data correspond to the values of D + R and D − R of each attribute (see Table 11). The grouping was carried out based on the fact that the affected attributes were located in cluster 5 as a result of applying the clustering method of Phase 3 Stage 2 of this investigation. The bidirectional relationship between A01 and A05 is due to the fact that the navigation concept is a basic aspect that configures usability and, in turn, usability affects the fulfillment of the objectives, establishing the best routes for the development of the task. Between A01 and A08 the relationship occurs because the interaction with the system/product must be predictable and safe in order to meet expectations, which leads to the system being usable.

In summary, the attribute A01—Usability represents the most important attribute identified with the DEMATEL method, since it has the highest D + R value and obtained the greatest weight in the attributes vector.

5. Discussion

The results of this investigation are based on a processual methodology that having been organized by four instances of inquiry was administered by iterative dynamics. The preliminary selection of such attributes was made from sources referenced by significant citation indices. Hence, 1165 works from IEEE Xplore, Scopus, and ACM Digital Library repositories, that appear as the case studies addressed, were part of the state of the art. The iterations that emerged between describing, grouping, and prioritizing the quality attributes that must prevail in all AR-supported digital products led to revealing subgroups of attributes that were connecting systematically. These connections found their justification from hierarchies identified through the implementation of Multicriteria Decision Methods such as Fuzzy Connective Maps, Clustering and DEMATEL, and a revealing matrix of cohesion and distancing indices thanks to a contrasting analysis of experts in the Frontend and Backend areas. The training particularities of these experts were based on the knowledge and trajectory in the topics of Digital Interface Design and User Experience Design, only and exclusively in applications linked to AR that pursue the formation of skills from semi-immersive experiences.

In any case, although this work concludes that there is coherence amongst the literature analyzed with the results that the matrix produced, its most important contribution is based on the binding conditions between the concepts UX, UI and AR in the field of digital products. Although other works have already shown that this UI + UX + AR triad raises skill formation rates much more than the products that only implement the UX + UI, current works are lacking more standardized and precise advice. Hence, the central contribution of this study consisted in such a categorization, consolidated in a grouping proposal that can be used by designers and developers of digital products when the objective is to encourage the development of skills from applications assisted with AR. It is also clarified that although in the theoretical framework of this work the exhaustive differences between AR and VR are defined, from the work of Azuma [1] and other authors the focus is only on products assisted with AR and not with VR. In addition, the previous works that were taken into account are Nielsen, Budiu, and Riders [148], Krug [149], and Gutiérrez et al. [64], among others.

Although any digital product assisted with Augmented Reality responds to a typology of various functions, the “practical functionalities” [150], which relate the perceived reality and the design of contents that are superimposed on it, in order to stimulate cognitive aspects [1]. Nonetheless, although AR can be classified as a subproduct of software, the aesthetic effects that it generates for users account for another type of relationship which implies a determinism that focuses on the emotional dimensions of people. Hence, content design with AR is causally linked to UI Interface Design, and thus to UX User Experience Design [148]. The triadic dynamics of these three aspects that operate systematically, depending on a partial immersion, seeks, through the mediation of one or several technological devices (hardware), the transfer of knowledge and/or particular skills to those who consume them [149]. Therefore, all emerging content resulting from AR is deterministic and operates with the UI and UX in an aligned manner in the search to optimize experiences.

The methods and instruments that support the UX are intended not only qualitatively and phenomenologically to describe the effects that digital products have on users, but also to measure their level of usability in quantitative terms. When the contents with AR are systematically conceived together with the UI and the UX [1], improvements are presented in aesthetic, practical and operational aspects that can determine cognitive and emotional reactions in the users thanks to their semi-immersive effects.

Concerning the application areas where the design, development, and consumption of content with AR ensure success in improving the user experience, there are HUDs (Head Up Displays), multisensory devices, and mobile devices, among others. The first two tend to be found on the dashboards of fighter jets and commercial flights [151], on ships, submarines [152], and also on any type of automobile or high end utility vehicle [153]. These offer data that are essential for navigation, whether by air, water or land. The information provided by AR combined with reality takes into account human factors that, in anthropometric and ergonomic terms, avoid distractions towards traditional meters and reduce cognitive processing in the consumption of information. Various investigations have demonstrated that multisensory devices [154] and “fused interfaces” [155] prevent accidents and reduce risks [156]. In the third category, mobile devices, cell phones, tablets or smart glasses, and “Spatial AR” [157], markers are used that display enveloping information that facilitates user tasks. These are tested and implemented in the contexts of the market economy [158], and in education, both in elementary schools [159,160], in universities [161], and also in the industrial sector [162].

A possible scenario where the attributes identified in the ranking of this study can be used is in the implementation of information technology architectures.

These architectures are used in academic and industrial contexts [163,164], evidencing the need to improve or innovate processes based on the implementation of business logic. Through information technology architectures, a broad understanding of specific domains is achieved, promoting the reuse of design experience and facilitating the development, standardization, and evolution of software systems [165].

Figure 22 represents a reference model of information technology architectures that is product of a review of works described in [163,166,167,168,169]. Five layers can be seen: the quality layer, the training processes layer, the technological layer, governance, and AR. The attributes identified in this study feed the quality layer of this type of architecture. The quality layer is in charge of managing the UX quality metrics and the respective indicators in order to measure and evaluate the use of the AR training system.

The results obtained by combining the three methods favor the identification of the attributes importance in training processes, because the holistic and detailed vision of the causal, impact and similarity relationships among the 87 attributes was taken into account. The proposed framework will allow the generation of a baseline for the use of these multicriteria methods in the identification of relevant aspects in the area of AR.

6. Conclusions, Limitations, and Future Work

6.1. Conclusions

This article presents an analysis of the quality attributes of user experience in augmented reality applications in four phases. The first phase of the study sought to identify the attributes present in this dynamic. This process of recognizing the problem was carried out through a systematic review of literature in high-impact repositories. In the second phase, the multicriteria methods to be used were selected, the instruments were built, and the panel of experts was created. The third phase, made up of two stages, was the hierarchy of the attributes employing the FCM method. It made it possible to identify the most relevant attributes in the process of training the workers with augmented reality applications and the level of relationship among these attributes. Stage two of the second phase developed by means of the Clustering method was the formation of the conglomerates or clusters. The fourth phase consisted of reducing the dimensionality of the 87 attributes to 19 in order to apply the DEMATEL method and thus obtain a vector of weighted attributes.

The contribution of this research can be considered in three aspects: (1) Ranking of attributes with multicriteria decision methods in the context of augmented reality applications. (2) Grouping of UX attributes in AR by similarity of characteristics among them, and (3) Methodological framework for the use of the FCM, Clustering and DEMATEL multicriteria methods.

The conclusions of this work are related below according to the phases of the investigation.

From the systematic literature review process carried out, it was possible to obtain information about the attributes, such as concepts, uses, and metrics. These results evidence the interest of the academic community in presenting alternatives to implement strategies for improving this process.

According to the results obtained by applying the multicriteria techniques, the attributes “Information processing” and “Task completion” were identified in the FCM method as transmitters and the weighting value was high. These results may indicate the importance of information processing in training processes and the need not to omit steps in an industrial process.

Based on the results of the Clustering method, it is found that the cluster with the most attributes is cluster 7. These attributes are highly cohesive and are oriented toward security, representation, and social interaction. For this reason, the role of the designer becomes very important in developing these types of applications. It is noteworthy that the best weighted attributes with FCM were all located in cluster 7, which reinforces the importance of this cluster.

The ranking attributes differ when comparing the results of the two multicriteria decision methods FCM and DEMATEL. With DEMATEL, the five best weighted attributes are Usability, Satisfaction, Stimulation, Engagement, and Aesthetics. This difference in the results is due to the fact that the FCM worked with the relationships of 87 attributes, which has an impact on different values of causality.

These results can be used depending on the context. If it is wanted to build applications oriented to great handling of information and repetitive tasks, the attributes “Information processing” and “Task completion” would be ideal. However, if it is wanted to build augmented reality applications oriented to industrial training, Usability, Satisfaction, Stimulation, Engagement, and Aesthetics are fundamental requirements to generate a positive user experience.

It was evidenced that there is coherence between the literary warrant that supports the quality attributes presented in this study and the approaches that the experts in the development area highlight as relevant when it comes to identifying weaknesses in the process.

6.2. Limitations and Future Work

As demonstrated in the evaluations results, the attributes have different impacts according to the application scenario. In the present research, the study scenario corresponds to the training processes in industrial environments with AR. Therefore, one of the objectives of the application of decision methods was to identify which were the attributes that had more or less weight within the scope. In other words, the weighting process was a tangible result from applying the multicriteria decision methods. This allowed prioritizing, organizing and in some cases eliminating by the obtained degree of importance, being this, one of the objectives of the research presented. It is important to take into account that, if the methodology proposed in this study is applied in other contexts, it is expected that the weighting will change.

The metrics identified in the review will be the material of a subsequent study. In the current investigation, they were not analyzed from the perspective of multicriteria decision methods.

In this phase of the investigation, there was no participation of workers/operators. This is a limitation because it is the employees who are directly involved in the training processes. In future studies, the participation of employees could be counted on to continue validating the proposed model.

As a future line of work, it is proposed to implement the best ranked attributes in an AR application that allows measuring and comparing the results during the three moments of an industrial training (before-during-after) to determine the impact of using AR as training support technology, all within the framework of building an information technology architecture.