Next Article in Journal
Printed Modular Approach on Selected High School Learners in Infanta Pangasinan for Successful Learning Delivery Amidst Disruption
Previous Article in Journal
Maritime Security Policy for Increasing National Economic Growth in Archipelagic Country
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Proceedings
Volume 82
Issue 1
10.3390/proceedings2022082087
proceedings-logo
Submit to this Journal
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Proceeding Paper

The Comparison of Evaluation on User Experience and Usability of Mobile Banking Applications Using User Experience Questionnaire and System Usability Scale †

by
Nina Setiyawati
1 and
Dwi Hosanna Bangkalang
2,*
1
Department of Informatics Engineering, Universitas Kristen Satya Wacana, Salatiga 50715, Central Java, Indonesia
2
Department of Information System, Universitas Kristen Satya Wacana, Salatiga 50715, Central Java, Indonesia
*
Author to whom correspondence should be addressed.
†
Presented at the International Academic Symposium of Social Science 2022, Kota Bharu, Malaysia, 3 July 2022.
Proceedings 2022, 82(1), 87; https://doi.org/10.3390/proceedings2022082087
Published: 28 September 2022
(This article belongs to the Proceedings of International Academic Symposium of Social Science 2022)
Browse Figures
Versions Notes

Abstract

:
The digitalization in the banking sector is improving rapidly as shown in the 46.72% increase in the number of digital banking system users in Indonesia, with significant increase in the number of mobile banking users. This enthusiasm should be followed by proper development of user experience (UX) and usability of the applications. User experience is a factor that affects the continuity intention to use an application which later also affects customer satisfaction. Usability also positively affects customer satisfaction. In this study, the UX and usability of four mobile banking applications in Indonesia were measured using the User Experience Questionnaire (UEQ) and System Usability Scale (SUS). The outcomes of the measurement of each application were then presented and compared to analyse the UX and usability of the application. Of four applications, the minimum rating was NEUTRAL, and the usability was regarded as GOOD. These results indicate that users have accepted the usability of the mobile banking applications that they used.

1. Introduction

The development of digital banking systems in Indonesia is rapidly increasing as both the COVID-19 pandemic and Regulation No.12/POJK.03/2018 issued by the Financial Services Authority on the Implementation of Digital Banking Services by Commercial Banks accelerate it. The regulation encourages banking service efficiency using advancements of IT, allowing banks to provide faster and easier quality service that fits customers’ needs [1]. The number of digital bank service users in Indonesia has increased by 46.72% in 2021, with the highest increase found in mobile banking service [2,3,4]. The increase in the number of users is then grouped into the increases in transaction volume using mobile banking and total downloads of mobile banking applications (acquisition of new users).
This positive trend should be followed by good user experience and usability of banking applications. User experience (UX) deals with user-related factors (users’ tendencies, expectations, needs, motivations, and moods), system-design-related factors (complexity, purpose, usability, and functionality) and the context or environment where the interaction takes place [5]. UX covers the overall acceptance of a product or service that includes various aspects, including the pragmatic and hedonic of a product [6] that deals with the ease of use of an application that fits users’ feelings and emotions when using the application [7]. Usability is a quality attribute that assesses the ease of the system interface of an application [8]. In a broader perspective, usability is the extent to which a product or system can be used by certain users to achieve certain goals in effective and efficient ways that it makes users satisfied in certain contexts of use [8,9]. Several studies have been carried out to examine how user experience and the usability of a product affect customer satisfaction and/or continuous intention. Lei and Lee (2020) verified that user experience is an important factor influencing users’ intention to use mobile game applications continuously [10]. Similarly, Mantala and Firdaus (2016) also found that better user experience was positively associated with higher customer satisfaction [11]. Febrian et al. (2021) also emphasized that customer experience had a positive and significant effect on customer satisfaction [12]. Those studies illustrated the importance of user experience in relation to customer satisfaction and/or continuous intention. The relationship between user experience, customer satisfaction, continuous intention, and customer loyalty is shown in Figure 1.
Usability relates to the ease of use, learning, and user satisfaction which is one of the crucial attributes in designing a product. Alshira’H (2020) explains that usability has a positive role in user satisfaction of e-government websites in Jordan [13]. Besides having a positive effect on user satisfaction, usability also affects user experience in accessing a system [14]. Aspects of usability such as simplicity and ease of use make the human–computer interaction more efficient for the gap between the human and the computer, and the software becomes closer with good interface, making the use of an application more effective.
In this study, we evaluated the UX and usability of several mobile banking applications in Indonesia based on UEQ and SUS metrics. Concerning the massive growth of mobile banking usage and increasing bank transactions, it is necessary to identify users’ perceptions of usability and UX flow in mobile banking applications. Users’ perceptions can be used as a basis for the development and maintenance of mobile banking applications. In addition, it is important to obtain data on UX and usability can support the projection and direction of customer satisfaction and/or continuous intention.

2. Method

The flow of this study is presented in Figure 2. UEQ and SUS questionnaires were distributed to mobile banking users. There were four mobile banking applications set as the objects of this study. At least 80 users of each banking application participated in this study. Respondents’ demographic data include: (1) respondents are users of one of the four mobile banking services; (2) male and female; (3) aged between 20–40 years. The aspects measured in UEQ and SUS questionnaires are shown in Figure 3 and Figure 4.
The User Experience Questionnaire (UEQ) consisted of 26 items measuring user experience, including [15,16]:
  • Attractiveness (users’ general perception);
  • Perspicuity which relates to how easy an application is to be used;
  • Efficiency (the amount of time required in completing a task);
  • Dependability (whether users have control over the application);
  • Stimulation (the extent to which users are motivated to use the product;
  • Novelty (innovation in a system, service and product).
Of 6 aspects of UEQ, attractiveness is a pure valence dimension. Meanwhile, Perspicuity, Efficiency and Dependability are pragmatic quality (goal-directed) aspects, and Stimulation and Novelty are aspects of hedonic quality (not goal-directed) [17].
The System Usability Scale (SUS) was employed to measure the usability attributes in mobile banking applications, including effectiveness, efficiency, satisfaction, easiness to learn, easiness to remember and few errors. The SUS questionnaire consisted of 10 items [18,19] containing positive and negative tone items.
In the data-gathering process, the questionnaires were distributed through Google forms, which results were then analysed based on UEQ and SUS metrics. The score of the answers for each UEQ question ranged from 1 to 7 according to the indicators of each aspect. Each SUS question is expressed in a 5-point Likert scale.
Proceedings 82 00087 g004 550
Figure 4. Items of the system usability scale. Adapted from [20].
Figure 4. Items of the system usability scale. Adapted from [20].
Proceedings 82 00087 g004

3. Results and Discussions

Questionnaires were filled out by respondents, and results were processed using UEQ and SUS metrics. Table 1 presents the number of respondents of each mobile banking application.
Figure 5 shows that all mobile banking applications have slightly different values in functionality and interaction concepts. t-Test calculation at Alpha level 0.05 indicates “Significant Differences” in the Perspicuity and Efficiency aspects of BNI Mobile, Livin and Octo Mobile. BNI Mobile’s Perspicuity compared with the one of BCA Mobile also shows a “Significant Difference”, whereas the results of other comparisons show “No Significant Difference”. Table 2 presents the interpretation of the UEQ scale for each mobile banking application, with an average value of −0.8 to 0.8 (neutral), value > 0.8 (positive), and an average value < −0.8 (negative) [17,21,22].
Figure 6 presents the usability of mobile banking applications measured using SUS. Raw SUS scores are converted into percentiles which indicate the extent to which mobile banking applications included in this study compare with other sets of data in the database. The average score (at the 50th percentile) is 68, implying that scores greater than 68 are above average and scores lower than 68 are below the average. SUS scores can be interpreted based on several measures, as shown in Figure 7 [23]. Adjective ratings describe the usability of a system using a phrase to make the usability value more meaningful. Acceptability ranges are another indicator as to whether a system is accepted. Conversely, the grade scale ranges from grade A for superior performance to F for failing performance.
In the interpretation using the adjective ratings as shown in Figure 7 shows that: (a) the adjective ratings of all mobile applications are GOOD; (b) the grade of the mobile banking applications is C+, except BCA Mobile which obtained B- grade; (*c) all applications are acceptable.
Based on the UX and usability evaluation of mobile banking applications, all applications have a neutral and positive UX design, as well as good usability. Users also accept the usability of the four mobile banking applications.

4. Conclusions

This study measured the UX and usability of four mobile banking applications in Indonesia using UEQ and SUS metrics. The minimum category for every UX aspect of the four mobile banking applications is NEUTRAL with GOOD usability. These results show that the general impression of users on the mobile banking application is positive. In addition, users have “accepted” the usability of the mobile banking applications that they used. BNI Mobile and Livin were rated NEUTRAL in terms of Novelty. Therefore, the Novelty aspect of BNI Mobile and Livin needs to be reviewed and improved for more positive evaluation. The T-Test showed that the Perspicuity and Efficiency aspects of BNI Mobile with Livin and Octo Mobile with Livin have a “Significant Difference”. Likewise, BNI Mobile’s Perspicuity when compared with that of BCA Mobile is significantly different which can be further improved.

Author Contributions

Conceptualization, N.S. and D.H.B.; methodology, N.S. and D.H.B.; software, N.S.; validation, N.S. and D.H.B.; formal analysis, N.S.; investigation, N.S.; resources, N.S.; data curation, N.S. and D.H.B.; writing—original draft preparation, N.S.; writing—review and editing, N.S. and D.H.B.; visualization, N.S. and D.H.B.; supervision, N.S.; project administration, N.S.; funding acquisition, N.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

Paper presented during a panel at the 2022 International Academic Symposium of Social Science (IASSC 2022).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. OJK. Digital Finance Innovation Road Map And Action Plan 2020–2024. Available online: http://www.digifin.org/digital-finance-innovation/ (accessed on 24 April 2022).
  2. Walfajri, M.; Handojo, B.I. Transaksi Digital Banking Meningkat 46,72% yoy Sampai September 2021. NewsSetup. Available online: https://newssetup.kontan.co.id/news/bi-transaksi-digital-banking-meningkat-4672-yoy-sampai-september-2021 (accessed on 24 April 2022).
  3. Ariesta, A. Transaksi Mobile dan Internet Banking Melonjak 300 Persen di 2021. IDX Channel. Available online: https://www.idxchannel.com/banking/transaksi-mobile-dan-internet-banking-melonjak-300-persen-di-2021 (accessed on 24 April 2022).
  4. Bayu, D.J.; Ridhoi, M.A. Layanan Perbankan Digital Makin Sering Digunakan Saat Pandemi. Available online: https://databoks.katadata.co.id/datapublish/2020/11/18/layanan-perbankan-digital-makin-sering-digunakan-saat-pandemi (accessed on 24 April 2022).
  5. Berni, A.; Borgianni, Y. From the Definition of User Experience to a Framework to Classify Its Applications. In Proceedings of the Design Society, Gothenburg, Sweden, 16–20 August 2021; Cambridge University Press: Cambridge, UK; Volume 1, pp. 1627–1636. [Google Scholar] [CrossRef]
  6. Yazid, M.A.; Jantan, A.H. User experience design (UXD) of mobile application: An implementation of a case study. J. Telecommun. Electron. Comput. Eng. 2017, 9, 197–200. [Google Scholar]
  7. Putra, D.O.; Setiawan, A. The Importance of User Experience Analysis in the Design of an Education Information System Application. In Proceedings of the 1st Borobudur International Symposium on Humanities, Economics and Social Sciences (BIS-HESS 2019), Magelang, Indonesia, 30 October 2019; Atlantis Press: Dordrecht, The Netherlands; 2020; Volume 436, pp. 1208–1211. [Google Scholar] [CrossRef]
  8. Weichbroth, P. Usability of Mobile Applications: A Systematic Literature Study. IEEE Access 2020, 8, 55563–55577. [Google Scholar] [CrossRef]
  9. Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. Available online: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en (accessed on 22 April 2022).
  10. Lei, B.; Lee, J. Analysis on continuous usage intention of chinese mobile games from the perspective of experiential marketing and network externality. J. Inf. Technol. Appl. Manag. 2020, 27, 197–224. [Google Scholar] [CrossRef]
  11. Mantala, R.; Firdaus, M.R. Pengaruh customer experience terhadap customer satisfaction pada pengguna smartphone android (studi pada mahasiswa politeknik negeri banjarmasin). J. Wawasan Manaj. 2016, 4, 153–164. [Google Scholar]
  12. Febrian, D.; Simanjuntak, M.; Hasanah, N. The Effect of Benefits Offered and Customer Experience on Re-use Intention of Mobile Banking through Customer Satisfaction and Trust. J. Keuang. Perbank. 2021, 25, 551–569. [Google Scholar] [CrossRef]
  13. Badran, O.; Al-Haddad, S. The impact of software user experience on customer satisfaction. J. Manag. Inf. Decis. Sci. 2018, 21, 1–20. [Google Scholar]
  14. Mohamad Salleh, M.A.; Salman, A.; Ali, M.N.S.; Hashim, H. The Importance of Usability Features in Enhancing Online Communication Satisfaction. Malays. J. Commun. 2016, 32, 1–15. [Google Scholar] [CrossRef]
  15. Hartzani, A.G. Evaluasi User Experience Pada Dompet Digital Ovo Menggunakan User Experience Questionnaire (UEQ). Universitas Islam Negeri Syarif Hidayatullah. Available online: https://repository.uinjkt.ac.id/dspace/bitstream/123456789/57729/1/AL%20GHIFARI%20HARTZANI-FST.pdf (accessed on 11 April 2022).
  16. Izabal, S.V.; Aknuranda, I.; Az-zahra, H.M. Evaluasi dan Perbaikan User Experience Menggunakan User Experience Questionnaire (UEQ) dan Focus Group Discussion (FGD) pada Situs Web FILKOM Apps Mahasiswa Fakultas Ilmu Komputer Universitas Brawijaya. J. Pengemb. Teknol. Inf. Ilmu Komput. Univ. Brawijaya 2018, 2, 3224–3232. [Google Scholar]
  17. Schrepp, M.; Hinderks, A.; Thomaschewski, J. Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In Proceedings of the International Conference of Design User Experience, and Usability, Crete, Greece, 22–27 June 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 383–392. [Google Scholar] [CrossRef]
  18. digital.gov. System Usability Scale (SUS): Improving Products Since 1986. Available online: https://digital.gov/2014/08/29/system-usability-scale-improving-products-since-1986/ (accessed on 16 May 2022).
  19. Sidik, A. Penggunaan system usability scale (SUS) sebagai evaluasi website berita mobile. J. Ilm. Technol. 2018, 9, 83–88. [Google Scholar] [CrossRef]
  20. Lewis, J.R.; Sauro, J. Item benchmarks for the system usability scale. J. User Exp. 2018, 13, 158–167. [Google Scholar]
  21. Rauschenberger, M.; Schrepp, M.; Perez-Cota, M.; Olschner, S.; Thomaschewski, J. Efficient Measurement of the User Experience of Interactive Products. How to use the User Experience Questionnaire (UEQ). Example: Spanish Language Version. Int. J. Interact. Multimed. Artif. Intell. 2013, 2, 39–45. [Google Scholar] [CrossRef]
  22. Hinderks, A.; Winter, D.; Schrepp, M.; Thomaschewski, J. Applicability of user experience and usability questionnaires. J. Univers. Comput. Sci. 2019, 25, 1717–1735. [Google Scholar]
  23. Bangor, A.; Kortum, P.; Miller, J. Determining what individual SUS scores mean: Adding an adjective rating scale. J. User Exp. 2009, 4, 114–123. [Google Scholar]
Proceedings 82 00087 g001 550
Figure 1. The relationship among user experience, customer loyalty, and continuous intention.
Figure 1. The relationship among user experience, customer loyalty, and continuous intention.
Proceedings 82 00087 g001
Proceedings 82 00087 g002 550
Figure 2. Method of the study.
Figure 2. Method of the study.
Proceedings 82 00087 g002
Proceedings 82 00087 g003 550
Figure 3. The indicators of user experience questionnaire.
Figure 3. The indicators of user experience questionnaire.
Proceedings 82 00087 g003
Proceedings 82 00087 g005 550
Figure 5. UEQ comparison on the mobile banking application.
Figure 5. UEQ comparison on the mobile banking application.
Proceedings 82 00087 g005
Proceedings 82 00087 g006 550
Figure 6. Results of usability measurement using SUS.
Figure 6. Results of usability measurement using SUS.
Proceedings 82 00087 g006
Proceedings 82 00087 g007 550
Figure 7. SUS score interpretation.
Figure 7. SUS score interpretation.
Proceedings 82 00087 g007
Table
Table 1. The number of respondents of each mobile banking application.
Table 1. The number of respondents of each mobile banking application.
Mobile Banking ApplicationNumber
BCA Mobile88
Octo Mobile80
BNI Mobile94
Livin100
Table
Table 2. The interpretation of the mobile banking application UX.
Table 2. The interpretation of the mobile banking application UX.
ScaleBCA MobileOcto MobileBNI MobileLivin
AttractivenessPositivePositivePositivePositive
PerspicuityPositivePositivePositivePositive
EfficiencyPositivePositiveNeutralPositive
DependabilityPositivePositivePositivePositive
StimulationPositivePositivePositivePositive
NoveltyPositivePositiveNeutralNeutral
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Setiyawati, N.; Bangkalang, D.H. The Comparison of Evaluation on User Experience and Usability of Mobile Banking Applications Using User Experience Questionnaire and System Usability Scale. Proceedings 2022, 82, 87. https://doi.org/10.3390/proceedings2022082087

AMA Style

Setiyawati N, Bangkalang DH. The Comparison of Evaluation on User Experience and Usability of Mobile Banking Applications Using User Experience Questionnaire and System Usability Scale. Proceedings. 2022; 82(1):87. https://doi.org/10.3390/proceedings2022082087

Chicago/Turabian Style

Setiyawati, Nina, and Dwi Hosanna Bangkalang. 2022. "The Comparison of Evaluation on User Experience and Usability of Mobile Banking Applications Using User Experience Questionnaire and System Usability Scale" Proceedings 82, no. 1: 87. https://doi.org/10.3390/proceedings2022082087

Article Metrics

Back to TopTop