The Comparison of Evaluation on User Experience and Usability of Mobile Banking Applications Using User Experience Questionnaire and System Usability Scale

: The digitalization in the banking sector is improving rapidly as shown in the 46.72% increase in the number of digital banking system users in Indonesia, with signiﬁcant increase in the number of mobile banking users. This enthusiasm should be followed by proper development of user experience (UX) and usability of the applications. User experience is a factor that affects the continuity intention to use an application which later also affects customer satisfaction. Usability also positively affects customer satisfaction. In this study, the UX and usability of four mobile banking applications in Indonesia were measured using the User Experience Questionnaire (UEQ) and System Usability Scale (SUS). The outcomes of the measurement of each application were then presented and compared to analyse the UX and usability of the application. Of four applications, the minimum rating was NEUTRAL, and the usability was regarded as GOOD. These results indicate that users have accepted the usability of the mobile banking applications that they used.


Introduction
The development of digital banking systems in Indonesia is rapidly increasing as both the COVID-19 pandemic and Regulation No.12/POJK.03/2018issued by the Financial Services Authority on the Implementation of Digital Banking Services by Commercial Banks accelerate it.The regulation encourages banking service efficiency using advancements of IT, allowing banks to provide faster and easier quality service that fits customers' needs [1].The number of digital bank service users in Indonesia has increased by 46.72% in 2021, with the highest increase found in mobile banking service [2][3][4].The increase in the number of users is then grouped into the increases in transaction volume using mobile banking and total downloads of mobile banking applications (acquisition of new users).
This positive trend should be followed by good user experience and usability of banking applications.User experience (UX) deals with user-related factors (users' tendencies, expectations, needs, motivations, and moods), system-design-related factors (complexity, purpose, usability, and functionality) and the context or environment where the interaction takes place [5].UX covers the overall acceptance of a product or service that includes various aspects, including the pragmatic and hedonic of a product [6] that deals with the ease of use of an application that fits users' feelings and emotions when using the application [7].Usability is a quality attribute that assesses the ease of the system interface of an application [8].In a broader perspective, usability is the extent to which a product or system can be used by certain users to achieve certain goals in effective and efficient ways that it makes users satisfied in certain contexts of use [8,9].Several studies have Proceedings 2022, 82, 87 2 of 7 been carried out to examine how user experience and the usability of a product affect customer satisfaction and/or continuous intention.Lei and Lee (2020) verified that user experience is an important factor influencing users' intention to use mobile game applications continuously [10].Similarly, Mantala and Firdaus (2016) also found that better user experience was positively associated with higher customer satisfaction [11].Febrian et al. (2021) also emphasized that customer experience had a positive and significant effect on customer satisfaction [12].Those studies illustrated the importance of user experience in relation to customer satisfaction and/or continuous intention.The relationship between user experience, customer satisfaction, continuous intention, and customer loyalty is shown in Figure 1.
Proceedings 2022, 69, x FOR PEER REVIEW 2 of 7 interface of an application [8].In a broader perspective, usability is the extent to which a product or system can be used by certain users to achieve certain goals in effective and efficient ways that it makes users satisfied in certain contexts of use [8,9].Several studies have been carried out to examine how user experience and the usability of a product affect customer satisfaction and/or continuous intention.Lei and Lee (2020) verified that user experience is an important factor influencing users' intention to use mobile game applications continuously [10].Similarly, Mantala and Firdaus (2016) also found that better user experience was positively associated with higher customer satisfaction [11].Febrian et al.
(2021) also emphasized that customer experience had a positive and significant effect on customer satisfaction [12].Those studies illustrated the importance of user experience in relation to customer satisfaction and/or continuous intention.The relationship between user experience, customer satisfaction, continuous intention, and customer loyalty is shown in Figure 1.Usability relates to the ease of use, learning, and user satisfaction which is one of the crucial attributes in designing a product.Alshira'H (2020) explains that usability has a positive role in user satisfaction of e-government websites in Jordan [13].Besides having a positive effect on user satisfaction, usability also affects user experience in accessing a system [14].Aspects of usability such as simplicity and ease of use make the human-computer interaction more efficient for the gap between the human and the computer, and the software becomes closer with good interface, making the use of an application more effective In this study, we evaluated the UX and usability of several mobile banking applications in Indonesia based on UEQ and SUS metrics.Concerning the massive growth of mobile banking usage and increasing bank transactions, it is necessary to identify users' perceptions of usability and UX flow in mobile banking applications.Users' perceptions can be used as a basis for the development and maintenance of mobile banking applications.In addition, it is important to obtain data on UX and usability can support the projection and direction of customer satisfaction and/or continuous intention.

Method
The flow of this study is presented in Figure 2. UEQ and SUS questionnaires were distributed to mobile banking users.There were four mobile banking applications set as the objects of this study.At least 80 users of each banking application participated in this study.Respondents' demographic data include: (1) respondents are users of one of the four mobile banking services; (2) male and female; (3) aged between 20-40 years.The aspects measured in UEQ and SUS questionnaires are shown in Figures 3 and 4. Usability relates to the ease of use, learning, and user satisfaction which is one of the crucial attributes in designing a product.Alshira'H (2020) explains that usability has a positive role in user satisfaction of e-government websites in Jordan [13].Besides having a positive effect on user satisfaction, usability also affects user experience in accessing a system [14].Aspects of usability such as simplicity and ease of use make the humancomputer interaction more efficient for the gap between the human and the computer, and the software becomes closer with good interface, making the use of an application more effective.
In this study, we evaluated the UX and usability of several mobile banking applications in Indonesia based on UEQ and SUS metrics.Concerning the massive growth of mobile banking usage and increasing bank transactions, it is necessary to identify users' perceptions of usability and UX flow in mobile banking applications.Users' perceptions can be used as a basis for the development and maintenance of mobile banking applications.In addition, it is important to obtain data on UX and usability can support the projection and direction of customer satisfaction and/or continuous intention.

Method
The flow of this study is presented in Figure 2. UEQ and SUS questionnaires were distributed to mobile banking users.There were four mobile banking applications set as the objects of this study.At least 80 users of each banking application participated in this study.Respondents' demographic data include: (1) respondents are users of one of the four mobile banking services; (2) male and female; (3) aged between 20-40 years.The aspects measured in UEQ and SUS questionnaires are shown in Figures 3 and 4.
interface of an application [8].In a broader perspective, usability is the extent to which a product or system can be used by certain users to achieve certain goals in effective and efficient ways that it makes users satisfied in certain contexts of use [8,9].Several studies have been carried out to examine how user experience and the usability of a product affect customer satisfaction and/or continuous intention.Lei and Lee (2020) verified that user experience is an important factor influencing users' intention to use mobile game applications continuously [10].Similarly, Mantala and Firdaus (2016) also found that better user experience was positively associated with higher customer satisfaction [11].Febrian et al. (2021) also emphasized that customer experience had a positive and significant effect on customer satisfaction [12].Those studies illustrated the importance of user experience in relation to customer satisfaction and/or continuous intention.The relationship between user experience, customer satisfaction, continuous intention, and customer loyalty is shown in Figure 1.Usability relates to the ease of use, learning, and user satisfaction which is one of the crucial attributes in designing a product.Alshira'H (2020) explains that usability has a positive role in user satisfaction of e-government websites in Jordan [13].Besides having a positive effect on user satisfaction, usability also affects user experience in accessing a system [14].Aspects of usability such as simplicity and ease of use make the human-computer interaction more efficient for the gap between the human and the computer, and the software becomes closer with good interface, making the use of an application more effective In this study, we evaluated the UX and usability of several mobile banking applications in Indonesia based on UEQ and SUS metrics.Concerning the massive growth of mobile banking usage and increasing bank transactions, it is necessary to identify users' perceptions of usability and UX flow in mobile banking applications.Users' perceptions can be used as a basis for the development and maintenance of mobile banking applications.In addition, it is important to obtain data on UX and usability can support the projection and direction of customer satisfaction and/or continuous intention.

Method
The flow of this study is presented in Figure 2. UEQ and SUS questionnaires were distributed to mobile banking users.There were four mobile banking applications set as the objects of this study.At least 80 users of each banking application participated in this study.Respondents' demographic data include: (1) respondents are users of one of the four mobile banking services; (2) male and female; (3) aged between 20-40 years.The aspects measured in UEQ and SUS questionnaires are shown in Figures 3 and 4  The User Experience Questionnaire (UEQ) consisted of 26 items measuring user experience, including [15,16]: Attractiveness (users' general perception); 2.
Perspicuity which relates to how easy an application is to be used; 3.
Efficiency (the amount of time required in completing a task); 4.
Dependability (whether users have control over the application); 5.
Stimulation (the extent to which users are motivated to use the product; 6. Novelty (innovation in a system, service and product).
Of 6 aspects of UEQ, attractiveness is a pure valence dimension.Meanwhile, Perspicuity, Efficiency and Dependability are pragmatic quality (goal-directed) aspects, and Stimulation and Novelty are aspects of hedonic quality (not goal-directed) [17].
The System Usability Scale (SUS) was employed to measure the usability attributes in mobile banking applications, including effectiveness, efficiency, satisfaction, easiness to learn, easiness to remember and few errors.The SUS questionnaire consisted of 10 items [18,19] containing positive and negative tone items.
In the data-gathering process, the questionnaires were distributed through Google forms, which results were then analysed based on UEQ and SUS metrics.The score of the answers for each UEQ question ranged from 1 to 7 according to the indicators of each aspect.Each SUS question is expressed in a 5-point Likert scale.The User Experience Questionnaire (UEQ) consisted of 26 items measuring user experience, including [15,16]: 1. Attractiveness (users' general perception); 2. Perspicuity which relates to how easy an application is to be used; 3. Efficiency (the amount of time required in completing a task); 4. Dependability (whether users have control over the application); 5. Stimulation (the extent to which users are motivated to use the product; 6. Novelty (innovation in a system, service and product).
Of 6 aspects of UEQ, attractiveness is a pure valence dimension.Meanwhile, Perspicuity, Efficiency and Dependability are pragmatic quality (goal-directed) aspects, and Stimulation and Novelty are aspects of hedonic quality (not goal-directed) [17].
The System Usability Scale (SUS) was employed to measure the usability attributes in mobile banking applications, including effectiveness, efficiency, satisfaction, easiness to learn, easiness to remember and few errors.The SUS questionnaire consisted of 10 items [18,19] containing positive and negative tone items.
In the data-gathering process, the questionnaires were distributed through Google forms, which results were then analysed based on UEQ and SUS metrics.The score of the answers for each UEQ question ranged from 1 to 7 according to the indicators of each aspect.Each SUS question is expressed in a 5-point Likert scale.

Results and Discussions
Questionnaires were filled out by respondents, and results were processed using UEQ and SUS metrics.Table 1 presents the number of respondents of each mobile banking application.Figure 5 shows that all mobile banking applications have slightly different values in functionality and interaction concepts.t-Test calculation at Alpha level 0.05 indicates "Significant Differences" in the Perspicuity and Efficiency aspects of BNI Mobile, Livin and Octo Mobile.BNI Mobile's Perspicuity compared with the one of BCA Mobile also shows a "Significant Difference", whereas the results of other comparisons show "No Significant Difference".Table 2 presents the interpretation of the UEQ scale for each mobile banking application, with an average value of −0.8 to 0.8 (neutral), value > 0.8 (positive), and an average value < −0.8 (negative) [17,21,22].

Results and Discussions
Questionnaires were filled out by respondents, and results were processed using UEQ and SUS metrics.Table 1 presents the number of respondents of each mobile banking application.Figure 5 shows that all mobile banking applications have slightly different values in functionality and interaction concepts.t-Test calculation at Alpha level 0.05 indicates "Significant Differences" in the Perspicuity and Efficiency aspects of BNI Mobile, Livin and Octo Mobile.BNI Mobile's Perspicuity compared with the one of BCA Mobile also shows a "Significant Difference", whereas the results of other comparisons show "No Significant Difference".Table 2 presents the interpretation of the UEQ scale for each mobile banking application, with an average value of −0.8 to 0.8 (neutral), value > 0.8 (positive), and an average value < −0.8 (negative) [17,21,22].

Results and Discussions
Questionnaires were filled out by respondents, and results were processed using UEQ and SUS metrics.Table 1 presents the number of respondents of each mobile banking application.Figure 5 shows that all mobile banking applications have slightly different values in functionality and interaction concepts.t-Test calculation at Alpha level 0.05 indicates "Significant Differences" in the Perspicuity and Efficiency aspects of BNI Mobile, Livin and Octo Mobile.BNI Mobile's Perspicuity compared with the one of BCA Mobile also shows a "Significant Difference", whereas the results of other comparisons show "No Significant Difference".Table 2 presents the interpretation of the UEQ scale for each mobile banking application, with an average value of −0.8 to 0.8 (neutral), value > 0.8 (positive), and an average value < −0.8 (negative) [17,21,22].Raw SUS scores are converted into percentiles which indicate the extent to which mobile banking applications included in this study compare with other sets of data in the database.The average score (at the 50th percentile) is 68, implying that scores greater than 68 are above average and scores lower than 68 are below the average.SUS scores can be interpreted based on several measures, as shown in Figure 7 [23].Adjective ratings describe the usability of a system using a phrase to make the usability value more meaningful.Acceptability ranges are another indicator as to whether a system is accepted.Conversely, the grade scale ranges from grade A for superior performance to F for failing performance.In the interpretation using the adjective ratings as shown in Figure 7 shows that: (a) the adjective ratings of all mobile applications are GOOD; (b) the grade of the mobile banking applications is C+, except BCA Mobile which obtained B-grade; (*c) all applications are acceptable.Based on the UX and usability evaluation of mobile banking applications, all applications have a neutral and positive UX design, as well as good usability.Users also accept the usability of the four mobile banking applications.In the interpretation using the adjective ratings as shown in Figure 7 shows that: (a) the adjective ratings of all mobile applications are GOOD; (b) the grade of the mobile banking applications is C+, except BCA Mobile which obtained B-grade; (*c) all applications are acceptable.
Based on the UX and usability evaluation of mobile banking applications, all applications have a neutral and positive UX design, as well as good usability.Users also accept the usability of the four mobile banking applications.

Conclusions
This study measured the UX and usability of four mobile banking applications in Indonesia using UEQ and SUS metrics.The minimum category for every UX aspect of the four mobile banking applications is NEUTRAL with GOOD usability.These results show that the general impression of users on the mobile banking application is positive.In addition, users have "accepted" the usability of the mobile banking applications that they used.BNI Mobile and Livin were rated NEUTRAL in terms of Novelty.Therefore, the Novelty aspect of BNI Mobile and Livin needs to be reviewed and improved for more positive evaluation.The T-Test showed that the Perspicuity and Efficiency aspects of BNI Mobile with Livin and Octo Mobile with Livin have a "Significant Difference".Likewise, BNI Mobile's Perspicuity when compared with that of BCA Mobile is significantly different which can be further improved.

Figure 1 .
Figure 1.The relationship among user experience, customer loyalty, and continuous intention.

Figure 1 .
Figure 1.The relationship among user experience, customer loyalty, and continuous intention.

Figure 1 .
Figure 1.The relationship among user experience, customer loyalty, and continuous intention. .

Figure 2 .
Figure 2. Method of the study.

Figure 3 .
Figure 3.The indicators of user experience questionnaire.Figure 3. The indicators of user experience questionnaire.

Figure 3 .
Figure 3.The indicators of user experience questionnaire.Figure 3. The indicators of user experience questionnaire.

Figure 5 .
Figure 5. UEQ comparison on the mobile banking application.

Figure 5 .
Figure 5. UEQ comparison on the mobile banking application.Figure 5. UEQ comparison on the mobile banking application.

Figure 5 .
Figure 5. UEQ comparison on the mobile banking application.Figure 5. UEQ comparison on the mobile banking application.

Figure 6
Figure6presents the usability of mobile banking applications measured using SUS.Raw SUS scores are converted into percentiles which indicate the extent to which mobile banking applications included in this study compare with other sets of data in the database.The average score (at the 50th percentile) is 68, implying that scores greater than 68 are above average and scores lower than 68 are below the average.SUS scores can be interpreted based on several measures, as shown in Figure7[23].Adjective ratings describe the usability of a system using a phrase to make the usability value more meaningful.Acceptability ranges are another indicator as to whether a system is accepted.Conversely, the grade scale ranges from grade A for superior performance to F for failing performance.

Figure 6 .
Figure 6.Results of usability measurement using SUS.

Table 1 .
The number of respondents of each mobile banking application.

Table 1 .
The number of respondents of each mobile banking application.

Table 1 .
The number of respondents of each mobile banking application.