Developing Usability Guidelines for mHealth Applications (UGmHA)

Abstract

Mobile health (mHealth) is a branch of electronic health (eHealth) technology that provides healthcare services using smartphones and wearable devices. However, most mHealth applications were developed without applying mHealth specialized usability guidelines. Although many researchers have used various guidelines to design and evaluate mHealth applications, these guidelines have certain limitations. First, some of them are general guidelines. Second, others are specified for mHealth applications; however, they only cover a few features of mHealth applications. Third, some of them did not consider accessibility needs for the elderly and people with special needs. Therefore, this paper proposes a new set of usability guidelines for mHealth applications (UGmHA) based on Quinones et al.’s formal methodology, which consists of seven stages starting from the Exploratory stage and ending with the Refining stage. What distinguishes these proposed guidelines is that they are easy to follow, consider the feature of accessibility for the elderly and people with special needs and cover different features of mHealth applications. In order to validate UGmHA, an experiment was conducted on two applications in Saudi Arabia using UGmHA versus other well-known usability guidelines to discover usability issues. The experimental results show that the UGmHA discovered more usability issues than did the other guidelines.

1. Introduction

In the recent era, the exponential evolution of technology is accelerating world development by making our life easier, more flexible and more effective. This development has been involved in the healthcare field in presenting electronic health (eHealth) technology. eHealth can enhance healthcare services using the internet and wireless technologies, usually through traditional desktops [1]. All over the world, countries are adopting eHealth services to improve the safety, standards and quality of their healthcare systems.

In relation to eHealth, a specific and essential part must be considered, called mobile health (mHealth) technology [2]. The concept of mHealth was defined and coined by Robert Istepanian in 2004 [3]. mHealth moves healthcare services from traditional desktops into mobile technologies, such as smartphones and wearable devices [4,5]. The key features of mHealth applications are the ability to monitor chronic disease by patients and facilitate communication between patients and doctors [6].

Despite the valuable benefits of mHealth applications, they can harm a patient’s life and health if they do not work as expected, such as erroneous results or wrong actions, whether by patients or clinical staff [7]. To the best of our knowledge, most mHealth applications are developed without using standard and specialized mHealth usability guidelines [8]. A set of usability guidelines is an evaluation method used to determine the extent to which a mHealth application’s interface is safe, efficient and usable by different kinds of people [9,10]. Accordingly, mHealth applications need to be regulated by applicable usability guidelines in order to assess their efficiency and safety for patients.

Many kinds of studies used various existing guidelines to design and evaluate mHealth applications; however, there are three main problems with these guidelines, which are:

• Some of the guidelines are general, which means they are not specified for mHealth applications, such as Nielsen’s principles [11]. Since mHealth applications are intended to affect the health and the body of people, they need to have more rigorous and restricted guidelines.
• Some of the guidelines are specified for mHealth applications; however, they cover only a few features of mHealth applications, such as:

  (a)

  Xcertia [12], which covered access to personal health information and notification features, while other features, such as self-monitoring, were not covered.

  (b)

  HE4EH [6], which covered self-monitoring, health goals and tips and biometric measurement features, while other features, such as consultations, were not covered.

  (c)

  Telemedicine recommendations [13], which covered consultations and booking appointment features, while other features, such as self-monitoring, were not covered.

• Some of the guidelines did not consider accessibility features for the elderly and people with special needs, such as [11,13]. Accessibility is important to ensure that the application is accessible to people of different ages and needs.

In this study, our goal is to develop a comprehensive set of usability guidelines for mHealth applications, titled (UGmHA), that satisfy the following criteria:

• Simple to follow and understandable by designers, developers and evaluators.
• Accessible to all people, including elderly people and people with special needs.
• Specialized in designing and evaluating mHealth applications that can cover different features. The following are examples of features gathered from different studies:
  • Self-monitoring for chronic diseases [14,15].
  • Online consultations [14,16].
  • Sharing data with health care providers (HCP) [14,15].
  • Booking appointments [16].
  • Biometric measurements [14].
  • Health goals and tips [14,15].
  • Notifications and reminders [14].
  • Access to personal health information [15].

This study has two main objectives. The first objective is to develop UGmHA using the systematic and formal methodology of Quinones et al. [17]. The second objective is to validate UGmHA by comparing UGmHA against the Nielsen and Xcertia guidelines, which are well-known and global guidelines. The experiment was conducted in Saudi Arabia by two authors of the paper using two local applications, which are named “Sehhaty” and “Sokry”.

The rest of the paper is organized as follows. Section 2 presents a background and literature review of mHealth-related guidelines as well as common development approaches and validation methods for new usability guidelines. Section 3 illustrates how UGmHA was developed and validated. Section 4 shows the results of the development and validation of UGmHA. Section 5 discusses the results obtained from the development and validation of UGmHA. Finally, our conclusions and future work are provided in Section 6.

2. Literature Review

This section presents the background information related to eHealth, mHealth and usability. We also review the existing guidelines related to mHealth applications, common development approaches and validation methods for new usability guidelines.

2.1. Electronic Health (eHealth) and Mobile Health (mHealth)

The term eHealth was established by Mitchell in 1999 [18] and is a broad term that integrates healthcare and technology to improve healthcare efficiency and reduce costs. The main objectives of eHealth are to improve patient safety and provide accurate diagnostics and appropriate treatment [19]. eHealth offers many different services and functionalities, such as electronic health records (EHR), archiving, electronic prescribing, order-entry systems and computerized decision support systems (CDS) [20]. Therefore, countries worldwide have adopted eHealth services to improve the standards and quality of their healthcare systems.

mHealth is a specific branch of electronic health (eHealth) technology. The “mHealth” terminology was coined and established by Robert Istepanian in 2004 [3]. mHealth can be defined as collecting and processing health data using smartphones, tablets and wearable devices [5]. mHealth applications are crucial since they enable a patient to monitor their own chronic disease and activities, support behavior changes for patients, improve their lifestyle and facilitate communication between patients and doctors [6,21]. Moreover, they provide doctors with greater mobility to help them care for their patients from different areas [5].

2.2. Usability and Guidelines

Usability is a key feature of a successful system because it is essential to create a well-designed and highly usable system [22,23]. According to ISO 9241-11 standard, usability is defined as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [24].

Usability can be implemented and evaluated effectively using usability guidelines. The usability guidelines have the advantage that they can be applied early in the design and used to evaluate and discover usability issues before application release [25]. Nielsen’s principles are the first and best-known usability guidelines, which were introduced by Nielsen and Molich in 1990 [26]. These guidelines are considered traditional and general for most user interfaces [27,28].

However, there is evidence in the literature confirming that general guidelines are not appropriate to design and evaluate domain-specific applications and may overlook essential elements that need to be considered in specific applications [27,28,29,30]. Moreover, Hermawati et al. [28] indicated that any set of usability guidelines should depend on the specific requirements of the user, task and environment of use. This means that any change in these requirements may introduce new features and issues of usability that are not considered by general guidelines. Therefore, it is crucial to develop mHelath-specific guidelines to ensure that usability features related to mHealth applications are identified.

Usability guidelines may include checklist items that work as a guide to evaluate domain-specific applications [17]. The checklist items can add more details and specifications for usability guidelines to make them easily tailored to the specific features of an application [31]. Without using checklist items, there is the potential for the wrong association of usability issues to the corresponding guideline, missed domain-specific usability issues and unreliable findings due to the lack of evaluator consensus [32].

Several guidelines have been used by studies for the design and evaluation of mHealth applications.

Nielsen’s principles [11] comprises ten usability guidelines, which are:

• Visibility of system status.
• Match between system and the real world.
• User control and freedom.
• Consistency and standards.
• Error prevention.
• Recognition rather than recall.
• Flexibility and efficiency of use.
• Aesthetic and minimalist design.
• Help users recognize, diagnose and recover from errors.
• Help and documentation.

Despite Nielsen’s principles being globally well-known for designing and evaluating most user interfaces, they are general and not specialized for distinct features of mHealth applications [27,28]. Furthermore, accessibility features for the elderly and people with special needs are not considered. However, several studies used Nielsen’s principles as a basis for generating new usability guidelines by modifying or adding new guidelines or checklist items of domain-specific applications [27].

Roy et al. [33] discussed the framework of the Xcertia guidelines, which were founded by leaders of the healthcare industry, such as the Association of American Medical Colleges, U.S. Food and Drug Administration (FDA) and Healthcare Information and Management Systems Society (HIMSS). The Xcertia guidelines aim to fulfill two primary purposes, which are assessing how an mHealth application is designed to be safe and optimizing mHealth applications to be used by specified users within a specified environment [33].

As shown in Figure 1, Xcertia includes five key workgroups: privacy, security, content, usability and operability. Each workgroup includes a set of guidelines, and each guideline can be measured by a set of performance requirements or checklists. Nonetheless, this research will focus only on the “usability” workgroup because it is within our scope. The “usability” workgroup includes ten guidelines with checklist items for each guideline. Below are the guidelines of the “usability” workgroup of Xcertia:

Figure 1. Xcertia workgroups.

• Visual design.
• Readability.
• App navigation.
• Onboarding.
• App feedback.
• Notifications, alerts and alarms.
• Help resources and troubleshooting.
• Historical data.
• Accessibility.
• Ongoing app evaluation.

Xcertia guidelines are developed mainly for mHealth applications and the “usability” in Xcertia considers some of the accessibility features for people with special needs. However, despite Xcertia guidelines being developed mainly for mHealth applications, most of the usability guidelines are still general and do not evaluate specific features of mHealth applications, such as self-monitoring, consultations, health advice and booking appointments.

Khowaja et al. [6] presented a modified set of specialized guidelines for mHealth applications called (HE4EH). HE4EH includes 25 guidelines with checklist items. The guidelines are:

• Visibility of system status.
• User control and freedom.
• Match between system and real world.
• Consistency and standards.
• Error prevention.
• Help users recognize, diagnose and recover from errors.
• Recognition rather than recall.
• Flexibility and efficiency of use.
• Aesthetic and minimalist design.
• Help and documentation.
• Privacy.
• Skills.
• Pleasurable interaction.
• Accessibility.
• Compatibility between different platforms.
• Minimized human–device interaction.
• Physical interaction and ergonomics.
• Readability and layout.
• Non-interrupting app information visualization.
• Content.
• Display.
• Navigation.
• Interactivity.
• Behavior change.
• Self-monitoring.

However, the study only focused on self-monitoring of diabetes and blood glucose levels without considering hypertension, obesity, etc. [6]. Moreover, since the scope of the study was about self-monitoring and behavioral-change features, the other features related to mHealth applications, such as consultations and booking appointments, were not covered. Furthermore, the study included more than 16 guidelines, while it is recommended to keep the number of usability guidelines between 10 and 16 [17].

Aldekhyyel et al. [13] evaluated the usability of mHealth applications with telemedicine features deployed during the COVID-19 pandemic in Saudi Arabia. The mHealth applications were “Sehha”, “Cura” and “Dr. Sulaiman Alhabib”, which were evaluated using Nielsen’s ten principles with a five-point severity rating scale (SRS). Then, the study provided usability design recommendations for each application based on the discovered usability issues. However, since the scope of the study was about telemedicine applications, it focused only on consultations, appointments, etc., without considering other mHealth-specific features, such as self-monitoring and health advice. Moreover, accessibility features for the elderly and people with special needs were not covered.

Al-Razgan et al. [34] converted and adapted some of the mobile phone usability guidelines and recommendations to develop modified usability guidelines for mobile launchers designed for elderly people. The study included 13 guidelines, which are classified into three sections as shown below:

• Look and Feel
  • Make elements on the page easy to read.
  • Easy recognition and accessibility.
  • Make clickable items easy to target and hit.
  • Use language and culture appropriate for the elderly; minimize technical terms.
• Interaction
  • Provide clear feedback on actions.
  • Provide preferable gestures for the elderly.
  • Provide the elderly with information on launcher/elderly status.
  • Use conventional interaction items.
  • Ergonomic design.
• Functionality
  • Provide functions that reduce elderly memory load.
  • Elderly does not feel lost or stuck (elderly control and freedom).
  • Prevent errors from occurring.
  • Provide necessary information and settings.

However, despite the study being mainly about mobile launchers’ usability for elderly people, we can make use of the guidelines to add the accessibility features for elderly people.

Arachchi et al. [35] integrated different learning theories and usability guidelines to develop an eLearning module for people with intellectual disability needs. The guidelines considered the specific abilities and context of learners, such as attention, mobility, cognitive capacity, learning ability and ability to read and write. The study incorporated nine guidelines with checklist items. The guidelines are:

• Minimised distractions.
• Easy to operate.
• Readability and visualization.
• Consistency.
• Feedback.
• Navigability.
• Personalizing.
• Easy to understand and relevant.
• Learner-friendly.

However, despite the study being mainly about eLearning usability for intellectual disabilities, which is a part of special needs, we can make use of its guidelines to add accessibility features to support people with special needs.

2.3. Common Approaches for Developing New Usability Guidelines

Some common approaches are used to develop domain-specific usability guidelines, such as methodologies, literature review, usability problems, existing guidelines, interviews, theories and mixing processes [27]. However, the methodologies approach attracts the interest of most studies since it is the only formal approach that follows a systematic and clearly defined process. On the other hand, all other approaches are considered informal since their activities and processes are performed in a non-systematic way [27].

Different methodologies can be followed to generate a new set of usability guidelines, such as the Rusu methodology [10], which consists of six stages. This methodology is the most popular one [27,28] and is used by several studies, such as [36,37,38,39]. These studies emphasized the importance of the methodology to facilitate the generation of domain-specific guidelines. Nevertheless, Rusu et al.’s methodology [10] has been modified and adapted by the same authors with the support of usability experts and researchers to generate a new and examined methodology, here called Quinones et al.’s methodology [17]. Quinones et al.’s methodology consists of seven main stages, which are [17]:

• Exploratory stage: to collect existing studies related to the main topics of the research, such as general or related usability guidelines, principles, application features and attributes.
• Descriptive stage: to highlight the most important information of the previously collected studies to formalize the main concept of the research.
• Correlational stage: to determine the characteristics that the usability guidelines of the specific domain should have based on traditional guidelines.
• Selection stage: to keep, adapt or discard the existing sets of usability guidelines that were selected in the descriptive stage.
• Specification stage: to formally specify the set of proposed guidelines using a standard template.
• Validation (experimental) stage: to test the proposed guidelines against traditional ones through experiments.
• Refinement stage: to refine the proposed guidelines based on the validation-stage results.

The advantage of employing Quinones et al.’s methodology is that it provides a standard template, which is used to explain more details about an individual guideline, including the ID, priority, name, definition, explanation, application features, examples, benefits, checklists and usability attributes. Moreover, it is possible to perform as many iterations as needed for all or some stages to improve the usability guidelines or the performance of experiments used for the validation stage [17].

2.4. Common Methods for Validating New Usability Guidelines

Three types of validation methods are proposed in Quinones et al.’s methodology. The first type is recommended by the methodology, and the other two types are for obtaining additional feedback as shown below: [17]:

• Guideline evaluation: to check the proposed guidelines against traditional or specialized guidelines in terms of the number of discovered usability issues (The recommended method).
• Expert judgment: using a questionnaire that assesses expert and evaluator perceptions of the proposed usability guidelines (To receive additional feedback).
• User testing: to obtain users’ opinions about the proposed set of usability guidelines (To receive additional feedback).

However, the first type offers more opportunities to provide in-depth information about the effectiveness of the usability guidelines [17,28]. The comparison can be achieved by using a between-subject or within-subject study [40]:

• Between-subject: each participant is exposed to only one condition of testing.
• Within-subject: each participant is exposed to more than one of the conditions of testing.

The obtained results can be analyzed using some of the following criteria [17]:

• Number of the discovered usability issues.
• Number of discovered specific usability issues.
• Number of severe usability issues.
• Number of critical usability issues.

The majority of studies have applied the comparison method to validate their guidelines. However, the number of evaluators, compared guidelines and applications vary between the studies as shown in Table 1.

Table 1. Review of the validation process in different studies.

Ref	Validation Process	Validation Result
[41]	Four evaluators evaluated an application by comparing Nielsen guidelines against the authors’ guidelines.	The authors’ guidelines were more effective than Nielsen’s since they discovered more usability issues (19 issues, 6 of them were severe) than Nielsen’s guidelines (15 issues, 5 of them were severe).
[42]	Six evaluators evaluated two applications by comparing Nielsen guidelines against the authors’ guidelines.	The author’s guidelines worked better than Nielsen’s since they discovered more usability issues (App1: 43, App2: 45) than Nielsen’s guidelines (App1: 28, App2: 24)
[43]	Two evaluators evaluated four animated agents through new guidelines.	The results showed which agent’s design was better.
[44]	Eighteen evaluators evaluated two Arabic applications by comparing the authors’ guidelines against Nielsen’s guidelines and game usability principles (two applications and three guidelines).	The author’s guidelines worked better than the others since they discovered more usability issues (Game1: 13, Game2: 12) than Nielsen (Game1: 6, Game2: 5) and the game usability principles (Game1: 10, Game2: 8).

3. Materials and Methods

This section presents the methodology used to develop and generate UGmHA guidelines as well as the methods used to validate the resulting UGmHA guidelines.

3.1. Developing UGmHA

In the previous section, the literature review showed different guidelines used to evaluate mHealth applications. Some of these guidelines are general but used by various studies to evaluate mHealth applications, such as Nielsen’s principles [11]. On the other hand, some of the guidelines are used mainly to evaluate mHealth applications, such as Xcertia [12], HE4EH [6] and telemedicine mobile applications [13].

Each related work provides a diverse wealth of information and ideas around designing and evaluating mHealth applications. However, the guidelines and principles have limitations as described in Section 2. Inspired by previous work, we developed a set of comprehensive usability guidelines for mHealth applications (UGmHA), consisting of guidelines and checklist items generated from various studies and existing guidelines.

In order to develop our proposed guidelines effectively and efficiently, we employed the formal methodology of Quinones et al. [17], which was explained in Section 2. This methodology involves seven main stages, which were applied to fit within the context of our situation and research goals. The first five stages (exploratory, descriptive, correlational, selection and specification) were implemented by the first author of this paper and supervised by the other authors, while the sixth stage (validation) was performed by the first and third authors of the paper.

3.1.1. Stage 1: Exploratory Stage

In this stage, a systematic literature review was performed to collect information related to mHealth applications, consisting of application features, usability attributes, guidelines and recommendations. Since our study aims to make the applications accessible to different kinds of people, the reviewed literature is also related to elderly people and people with special needs.

In order to perform the systematic review, the method presented by Kitchenham and Charters [45] was employed using the following keywords:

• For features: (“mHealth applications” OR “mobile health applications”) AND “features”.
• For attributes: (“mHealth applications” OR “mobile health applications”) AND “attributes”.
• For guidelines and recommendations: (“mHealth applications” OR “mobile health applications”) AND (“usability guidelines” OR “usability principles” OR “usability heuristics” OR “design recommendations”).
• For accessibility guidelines: (“mHealth applications” OR “mobile health applications”) AND “accessibility guidelines”.

All keywords were used to search for relevant information using multiple databases, including MDPI, IEEE, Springer Link, JAMIA and JMIR.

The inclusion criteria for this study were: (1) studies related to mHealth applications; (2) studies related to the accessibility guidelines; (3) studies with a clear description of features, attributes, guidelines or recommendations; (4) studies that are publicly available; and (4) studies written in English. The exclusion criteria were (1) studies that were unavailable as a full-text; (2) studies not specified for mHealth applications or accessibility guidelines; (3) studies that did not provide enough information about targeted topics; and (4) survey papers.

As shown in Figure 2, a total of 54 studies were reviewed from databases, including features, attributes and guidelines related to mHealth applications and accessibility. We excluded eight studies due to the duplication of information. Then, we applied the inclusion and exclusion criteria based on the title, abstract and full text; a total of 34 studies additional were excluded. Finally, 12 studies were included for the goal of our study, consisting of 3 studies for features, 3 for attributes and 6 for guidelines.

Figure 2. Flowchart for the systematic literature review of UGmHA.

The obtained information from this stage is shown below:

• Features: these features are related to mHealth applications: [14,15,16], which will be discussed in the next stage.
• Usability attributes, which include:

  –

  ISO attributes (effective, efficiency and satisfaction) [46].

  –

  Nielsen’s attributes (learnability, efficiency, memorability, errors and satisfaction) [47].

  –

  PACMAD attributes, which combine both attributes of ISO and Nielsen in addition to the cognitive load attribute (effective, efficiency, learnability, memorability, errors, satisfaction and cognitive load) [48].

• Usability guidelines and design recommendations, which include:

  –

  Usability guidelines related to mHealth applications, which are Nielsen [11], Xcertia [12], HE4EH [6] and telemedicine mobile application recommendations [13].

  –

  Usability guidelines related to elderly people [34].

  –

  Usability guidelines related to people with special needs [35].

Section 2 presents more details regarding guidelines and design recommendations relevant to our study.

3.1.2. Stage 2: Descriptive Stage

In this stage, we highlighted and selected the essential information from the collected literature review in the previous stage.

• For features: we gathered all features from [14,15,16] and summarized them to obtain a list of integrated features. Additionally, we added two more features related to our research, which were accessibility to elderly people and accessibility to people with special needs. Below are the selected features:
  • Booking appointments [16].
  • Consultation with HCP via text, voice messages and video calls [14,16].
  • Sharing data with HCP [14,15].
  • Access to personal health information [15].
  • Self-monitoring of chronic disease [14,15].
  • Allowing uploading and viewing of biometric measurements [14].
  • Graphic display of patient’s information [14].
  • Set health goals and treatment plan [14,15].
  • Track health progress [15].
  • Reminders and notifications [14].
  • Health tips and motivation [14,15].
  • Sharing health data with friends [15].
  • Interactive prompts [14].
  • Earn rewards [15].
  • Bluetooth technology connection [14].
  • Accessibility to elderly people.
  • Accessibility to people with special needs.
• For usability attributes: we selected only the PACMAD [48] attributes since they combine attributes of both ISO [46] and Nielsen [47]. Below are the selected attributes and their definition based on PACMAD [48].

  –

  Effectiveness: the ability to produce the desired outputs.

  –

  Efficiency: the reduction of wasted materials, such as time, money and effort.

  –

  Satisfaction: the fulfillment of the user’s needs and desires.

  –

  Learnability: a user can learn how to use the application easily.

  –

  Memorability: a user can remember how to use the application after a while.

  –

  Errors: minimizing the user’s error rate while using the application.

  –

  Cognitive Load: the ability to use a mobile application while doing daily activities.

• For usability guidelines: we classified the studies into the main guidelines, which will be used as basic guidelines for the next stages, and checklist items, which help to make the main guidelines more specific to the mHealth applications:

  –

  Main guidelines: we selected all of Nielsen’s principles [11] as well as the “usability” workgroup of Xcertia guidelines [12]. The reason for selecting these two guidelines is that Nielsen’s principles are well-known guidelines that are used to evaluate most user interfaces. Moreover, Xcertia is specified to evaluate mHealth applications in general and developed by authorized organizations in the US.

  –

  Checklist items: we selected [6,12,13], which are related to mHealth applications. Additionally, we selected the guidelines related to elderly people [34] and people with special needs [35].

On the other hand, we discarded all other workgroups of Xcertia [12], which were “privacy”, “security”, “content” and “operability” since they are out of the scope of this study. Moreover, we discarded some checklist items in [6] that are very specific to diabetes and cannot be generalized to all chronic diseases, and we also discarded checklist items in [34,35] that cannot be used for mHealth applications. The results of this stage are shown in Table 2.

Table 2. The information highlighted in the descriptive stage.

Topic	Collected Information	Selected Information
Features	Features provided by [14]	Features of these studies are combined and
	Features provided by [15]	summarized in Table 3.
	Features provided by [16]
	Accessibility to elderly people	Selected.
	Accessibility by people with special needs	Selected.
Attributes	Usability attributes proposed by ISO standard [46] (three attributes): effective, efficiency and satisfaction	Unselected.
	Usability attributes proposed by Nielsen [47] (five attributes): learnability, efficiency, memorability, errors and satisfaction	Unselected.
	Usability attributes proposed by Harrison [48] (seven attributes): effective, efficiency, learnability, memorability, errors, satisfaction and cognitive load	Selected because it combines both attributes of ISO and Nielsen in addition to the cognitive load.
Existing guidelines	Nielsen’s guidelines (10 guidelines) [11]	The set of Nielsen’s principles are selected.
	Xcertia guidelines (five workgroups: privacy, security, content, usability and operability) [12]	The only selected workgroup is “usability” and discards all others since our focus is only on usability.
	HE4EH (25 guidelines) [6]	Select some checklists that cover the feature of self-monitoring in general and discard checklists that cannot be generalized to all chronic diseases.
	Telemedicine mobile application design recommendations [13]	All recommendations are selected.
	Elderly people guidelines (13 guidelines) [34]	Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the mobile launchers.
	Intellectual disabilities guidelines (31 guidelines) [35]	Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the eLearning systems.

3.1.3. Stage 3: Correlational Stage

In this stage, the selected features, attributes and the main usability guidelines were matched as shown in Table 3. The purpose of this stage is to identify the specific features and attributes of mHealth applications that need to be evaluated by the main guidelines [17]. The process of matching should be controlled by features, which means that we chose to match the attributes that can measure a specific feature. Then, we chose the main guidelines (from Xcertia and Nielsen) that evaluate attributes with a feature as shown in Figure 3.

Table 3. Matching among features, attributes and guidelines.

No.	Features	Attributes	Nielsen	Xcertia
F1	Booking appointment [16]	Efficiency and effectiveness	NH1	XU5, XU6, UX8
F2	Consultation with HCP via text, voice messages and video calls [14,16]	Efficiency and effectiveness	NH1	XU5
F3	Sharing data with HCP [14,15]	Memorability and efficiency	NH1, NH6	XU5
F4	Access to personal health information [15]	Memorability, error, efficiency and effectiveness	NH1, NH5, NH6, NH9	XU4, XU5, XU8
F5	Self-monitoring of chronic disease [14,15]	Learnability, cognitive load, efficiency and effectiveness	NH1, NH6	XU6, XU8
F6	Allowing uploading and viewing of biometric measurements [14]	Memorability, error, efficiency and effectiveness	NH1, NH3, NH6	XU3, XU6, XU8
F7	Graphic display of patient’s information [14]	Memorability, satisfaction, effectiveness and efficiency	NH2, NH4, NH6, NH8	XU1, XU2, XU8
F8	Set health goals and treatment plan [14,15]	Satisfaction, learnability, cognitive load, effectiveness	NH1, NH2	XU2
F9	Track health progress [15]	Memorability, efficiency and cognitive load	NH1	XU5, XU6, XU8
F10	Reminders and notifications [14]	Memory and cognitive load	NH6	XU6
F11	Health tips and motivation [14,15]	Satisfaction and effectiveness	NH2	XU2
F12	Sharing health data with friends [15]	Satisfaction and learnability	-	-
F13	Interactive prompts [14]	Learnability	NH10	XU6, XU7
F14	Earn rewards [15]	Satisfaction	-	-
F15	Bluetooth technology connection [14]	Efficiency and effectiveness	NH1	XU5
F16	Accessibility to elderly people	Satisfaction, efficiency, effectiveness and memorability	-	XU9
F17	Accessibility by people with pecial needs	Satisfaction, efficiency, effectiveness and memorability	-	XU9

Figure 3. Matching process for features, attributes and guidelines.

Let us take the booking appointments feature as an example: the most suitable attributes and guidelines can be matched with:

• Attributes:

  –

  Effectiveness: to measure if the appointment is booked at the right date and time selected by the user.

  –

  Efficiency: to measure if this feature saves users’ time and effort without the need to go to the clinic.

• Guidelines:

  –

  Visibility of system status in Nielsen and App feedback in Xcertia: to evaluate booking appointments with the effectiveness attribute if the application provides feedback to the user regarding the right date and time of the appointment.

  –

  Notifications and alarms in Xcertia: to evaluate booking appointments with the efficiency attribute if the application notifies a user after an appointment is booked. Thus, the users’ time and effort to remember the appointment are minimized.

  –

  Historical data in Xcertia: to evaluate booking appointments with the efficiency attribute if the application saves previous and upcoming appointments. Thus, the users’ time and effort to call the clinic to ask for their appointments are minimized.

Another example, is the earn rewards feature, which can be measured by the satisfaction attribute because, when the user earns rewards for a good action or habit, they will be more satisfied with the application. However, there is no matched guideline from Nielsen and Xcertia to evaluate this feature (which will be discussed in the next stage).

3.1.4. Stage 4: Selection Stage

In this stage, an evaluation and determination were made for each main guideline identified in the previous stage [17]:

• Keep: without any change if the guideline is clear and correctly matched to the specific feature of mHealth applications.
• Adapt: if the guideline needs a change to be more related to the mHealth applications.
• Eliminate: if the guideline is redundant or not relevant to the features of mHealth applications.
• Create: if a new guideline is required to evaluate the specific feature of the mHealth applications that the main guidelines cannot evaluate.

In this sense, we adapted all of Nielsen’s guidelines as well as some of the Xcertia guidelines, including onboarding, notifications, historical data and accessibility by adding more checklist items from different studies to be more specific to mHealth applications. Furthermore, we kept the ongoing app evaluation guideline in Xcertia. On the other hand, we eliminated the other Xcertia guidelines, which were visual design, readability, app navigation, app feedback and help resource and troubleshooting because we considered them and their checklist items as parts of Nielsen’s usability guidelines.

Therefore, we merged the checklists of each eliminated guideline in Xcertia with similar guidelines in Nielsen. For example, we found that the checklist items of visual design guideline in Xcertia satisfy the definition of Nielsen’s consistency and standards, recognition rather than recall and error prevention guidelines. Hence, the visual design guideline was eliminated, and its checklist items were moved to the corresponding guideline in Nielsen. This process was applied to all eliminated guidelines in Xcertia. However, F12 and F14 features were not covered by any of the selected main guidelines.

That is why we needed to create a new guideline (Interactive and motivations) that evaluates these features. After that, we determined the applicability of each guideline to identify its importance, whether (1) useful, (2) important or (3) critical based on the guideline definition and checklists. The results of this stage are described in detail in Table 4, Table 5 and Table 6.

Table 4. Guideline selection process (Nielsen) [11].

ID	Guideline Name	Action	Covered Features	Applicability
NH1	Visibility of system status	Adapt *	F1–F6, F8, F9, F15	(3) Critical
NH2	Match between system and the real world	Adapt *	F2, F7, F8, F11	(2) Important
NH3	User control and freedom	Adapt *	F1, F4, F5, F6	(2) Important
NH4	Consistency and standards	Adapt *	F7, F11	(2) Important
NH5	Error prevention	Adapt *	F5, F6	(3) Critical
NH6	Recognition rather than recall	Adapt *	F4, F5	(2) Important
NH7	Flexibility and efficiency of use	Adapt *	F3, F4, F5, F8	(1) Useful
NH8	Aesthetic and minimalist design	Adapt *	F4, F7	(1) Useful
NH9	Help users recognize, diagnose and recover from errors	Adapt *	F1, F4	(3) Critical
NH10	Help and documentation	Adapt *	F13	(2) Important

* Add more checklist items related to mHealth applications from different studies.

Table 5. Guideline selection process (Xcertia) [12].

ID	Guideline Name	Action	Covered Features	Applicability
XU1	Visual design	Eliminate (the checklists were moved to Nielsen’s NH4, NH5 and NH6 guidelines)	F7	-
XU2	Readability	Eliminate (the checklists were moved to Nielsen’s NH8 guideline)	F7, F8, F11	-
XU3	App navigation	Eliminate (the checklists were moved to Nielsen’s NH3 guideline)	F6	-
XU4	Onboarding	Adapt *	F4	(2) Important
XU5	App feedback	Eliminate (the checklists were moved to Nielsen’s NH1 guideline)	F1–F4, F9, F15	-
XU6	Notifications, alerts and alarms	Adapt *	F1, F5, F6, F9, F10, F13	(3) Critical
XU7	Help resource and troubleshooting	Eliminate (the checklists were moved to Nielsen’s NH10 guideline)	F13	-
XU8	Historical data	Adapt *	F1, F2, F4–F7, F9	(2) Important
XU9	Accessibility	Adapt *	F16, F17	(3) Critical
XU10	Ongoing app evaluation	Keep (no change for this guideline)	F1–F17	(2) Important

* Add more checklist items related to mHealth applications from different studies.

Table 6. Guideline selection process (new created guidelines).

ID	Guideline Name	Action	Covered Features	Applicability
N1	Interactivity and Motivations	Create (to cover F12 and F14 features since they are covered neither in Nielsen’s nor in the Xcertia guidelines.)	F12, F14	(1) Useful

3.1.5. Stage 5: Specification Stage

In this stage, the new set of usability guidelines was formally defined. Quinones et al. [17] recommended keeping the number of usability guidelines between 10 and 16 because it is difficult to employ many guidelines in practice and recommended using checklists to add more details to the guidelines. Therefore, we integrated the guidelines of Nielsen [11] and Xcertia [12] to form the main guidelines of UGmHA and keep them within the recommended range.

However, since Nielsen’s guidelines are general and most of Xcertia guidelines and checklist items are not specific to the features of mHealth applications, we adapted and extended them by adding more checklist items from different studies, which were [6,13,34,35]. The reasons for adding these checklist items are to make the main guidelines more specific to the features of mHealth applications and support the involvement of accessibility features for the elderly and people with special needs. The obtained results of this stage are 16 usability guidelines as shown in Table 7 in addition to 154 checklist items classified by the main guidelines.

Table 7. The set of proposed usability guidelines for mHealth applications (UGmHA).

ID	Guideline Name
MHP1	Visibility of system status
MHP2	Match between system and the real world
MHP3	User control and freedom
MHP4	Consistency and standards
MHP5	Error prevention
MHP6	Recognition rather than recall
MHP7	Flexibility and efficiency of use
MHP8	Aesthetic and minimalist design
MHP9	Error diagnosis and recovery
MHP10	Help and documentation
MHP11	Notifications, alerts and alarms
MHP12	Onboarding
MHP13	Historical data
MHP14	Accessibility
MHP15	Ongoing app evaluation
MHP16	Interactivity and motivations

The resulting template of this stage will be discussed in Section 4.

3.2. Validating UGmHA

3.2.1. Stage 6: Validation Stage

The validation stage was divided into three phases as suggested by Quinones et al.’s methodology [17]:

• Guideline evaluation phase: check the new set of UGmHA against Nielsen and Xcertia guidelines regarding the number of identified usability issues and severity.
• Expert judgment phase: evaluate the usefulness, efficiency and effectiveness of UGmHA guidelines by questionnaire or interview. Then, the results will be used in the seventh stage (refining stage) of Quinones et al.’s methodology [17].
• User testing phase: enhance the usability of an existing application using refined guidelines of UGmHA. Then, perform user testing for the enhanced application.

However, the first phase of validation (guideline evaluation) will be presented in this paper, while the second and third phases (expert judgment and user testing) are ongoing work that will be published as soon as they are finished. Below are the details of the first phase of the validation.

• Experiment Design:

  –

  Participants: The evaluators who participated in this experiment are two of the authors of this research. The first evaluator was Mrs. Eman Nasr, a master’s student in the Faculty of Computing and Information Technology (FCIT) at King Abdulaziz University with a medium knowledge of usability evaluation. The second evaluator was Dr. Duaa Sinnari, an assistant professor in FCIT at King Abdulaziz University with high experience in usability evaluation and human–computer interaction (HCI).

  –

  Guidelines: UGmHA was tested against Nielsen’s principles and the Xceria guidelines. The reason for selecting these two guidelines is that UGmHA is based mostly on Nielsen’s and Xcertia guidelines and because Nielsen’s principles are well-known guidelines and Xcertia is specified to evaluate mHealth applications.

  –

  Applications: We selected the “Sehhaty” and “Sokry” applications. Figure 4 shows the home screen of (a) the first application (Sehhaty) and (b) the second application (Sokry) on the Android operating system in the Arabic language. The reasons for selecting these two applications are because they are simple to use and to achieve diversity between the selected applications based on governmental/ private, functionality and popularity as shown in Table 8.

  

  Figure 4. (a) The home screen of the first application. (b) The home screen of the second application.

  Table 8. Details of the “Sehhaty” and “Sokry” applications.

  	Sehhaty	Sokry
  Government/ Private	A government mHealth application that belongs to the Ministry of Health (MOH) in Saudi Arabia.	A private mHealth application.
  Functionality	Provides different services, such as booking appointments, children’s vaccines, immediate consultations to make video and audio calls with doctors, COVID-19 services and reviewing health data, including insurance and registered information in MOH.	Helps users manage their diabetes by recording blood sugar readings, meals, exercises and medications. In addition, provides health advice related to diabetes.
  Popularity	Popular with the people since it is connected with health records of MOH and considered an essential application for people in Saudi Arabia (+10 million downloads) [49].	Less popular to the people (+50,000 downloads) [50].

• Experiment Procedure: To assess the performance, each of the evaluators evaluated both applications individually in a within-subject study that compared the UGmHA, Xcertia and Nielsen guidelines. First, the evaluators evaluated the “Sehhaty” and “Sokry” applications individually using Nielsen’s guidelines and wrote down the discovered usability issues for each application. Second, the assessments of each evaluator using Nielsen’s guidelines were grouped together to generate a single list of usability issues for each application.
  Third, the evaluators worked together to rate each usability issue in the generated list based on the Severity Rating Scale (SRS). The SRS has five points (0–4) that can be used to prioritize and estimate which usability issues are important to be fixed as shown in Table 9 [51]. Then, the same aforementioned processes were repeated using Xcertia followed by the UGmHA guidelines. From this experiment, we produced six lists of usability issues, which are:
  Table 9. Severity Ranking Scale (SRS) [51].

  Rating	Description
  0	I do not agree that this is a problem at all.
  1	Cosmetic problem only. Need not be fixed unless extra time is available in the project.
  2	Minor usability problem. Fixing this should be given low priority.
  3	Major usability problem. Important to fix so it should be given high priority.
  4	Usability catastrophes. Imperative to fix this before the product can be released.

  • Using Nielsen’s guidelines: list of usability issues of the “Sehhaty” application.
  • Using Nielsen’s guidelines: list of usability issues of the “Sokry” application.
  • Using Xcertia guidelines: list of usability issues of the “Sehhaty” application.
  • Using Xcertia guidelines: list of usability issues of the “Sokry” application.
  • Using UGmHA guidelines: list of usability issues of the “Sehhaty“ application.
  • Using UGmHA guidelines: list of usability issues of the “Sokry” application.

3.2.2. Stage 7: Refining Stage

In this stage, the expert and user feedback obtained from stage 6 (validation stage) are used for the refining [17]. The process of refining is described as follows:

• Document the changes that should be made to the guidelines.
• Define the guidelines to be created, refined or deleted.
• Iterate and repeat some stages again, if necessary.

However, since the experts’ judgment and user testing in the validation stage are future work. Accordingly, the refining stage is future work for this study.

4. Results

This section presents the results of developing and validating the UGmHA guidelines.

4.1. Results of UGmHA Development

The standard template of Quinones et al. [17] can contain different elements. Still, it depends on the researcher to determine whether to use the complete template or choose the needed elements [52]. Therefore, we decided to select the ID, priority, guideline name, guideline definition, application features and checklist items as described in Table 10. Appendix A shows a detailed description of each guideline in UGmHA.

Table 10. Descriptive elements for the UGmHA guidelines.

ID	Guideline ID
Priority	(3) Critical, (2) Important or (1) Useful.
Name	Name of the guideline that resulted from the integration of Nielsen [11] and Xcertia [12].
Definition	Identify the guideline and its purpose.
Features	The selected features covered by the guideline.
Checklists	The checklist items selected from [6,12,13,34,35] to add more details to the guideline and to make it more related to the features of mHealth applications and accessible to the elderly and people with special needs.

4.2. Results of the UGmHA Validation

After receiving the usability results from the sixth stage (validation stage) described in Section 3, we defined the following findings:

4.2.1. Number of Usability Issues among the Three Guidelines

As depicted in Table 11, the numbers of usability issues discovered in the “Sehhaty” application were 73 using UGmHA, 22 using the Xcertia guidelines and 17 using Nielsen’s guidelines. On the other hand, the numbers of usability issues discovered in the “Sokry” application were 95 using UGmHA, 28 using the Xcertia guidelines and 25 using Nielsen’s guidelines.

Table 11. Numbers of usability issues of both applications based on the three guidelines.

	Sehhaty	Sokry	Total
UGmHA	73	95	168
Xcertia	22	28	50
Nielsen	17	25	42

4.2.2. Severity of Usability Issues among the Three Guidelines

As shown in Table 12 and Figure 5, the number of issues with a “Catastrophic” rating was the highest using all guidelines for both applications. Except for Nielsen’s guidelines, on “Sehhaty”, the highest number of issues were rated with a “Major” rating. However, there were no issues with the “no problem” rating for all guidelines on both applications.

Table 12. Severity of usability issues of the “Sehhaty” and “Sokry” applications.

	Sehhaty			Sokry
Rating	UGmHA	Xcertia	Nielsen	UGmHA	Xcertia	Nielsen
No Problem	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Cosmetic	7 (10%)	2 (9%)	0 (0%)	6 (6%)	2 (7%)	0 (0%)
Minor	10 (14%)	1 (5%)	0 (0%)	17 (18%)	4 (14%)	2 (8%)
Major	17 (23%)	6 (27%)	12 (71%)	26 (27%)	10 (36%)	9 (36%)
Catastrophic	39 (53%)	13 (59%)	5 (29%)	46 (48%)	12 (43%)	14 (56%)
Total	73	22	17	95	28	25

Figure 5. Severity of issues in the “Sehhaty” and “Sokry” applications among the three guidelines.

For the UGmHA guidelines, as shown in Figure 6, the most unapplied guidelines in “Sehhaty” were accessibility with 11 issues and then user control and freedom and aesthetic and minimalist design with 9 issues, while the most unapplied guidelines in “Sokry” were the visibility of system status and accessibility with 13 issues and then user control and freedom with 11 issues.

Figure 6. Number of usability problems found by UGmHA.

However, the least unapplied guidelines in “Sehhaty” were error diagnosis and recovery with one issue, while the least unapplied guideline in “Sokry” was match between system and real world with one issue.

5. Discussion

This section presents the discussion of results obtained from developing and validating the UGmHA guidelines.

5.1. UGmHA Development Discussion

If we compare the resulting guidelines of UGmHA with the Nielsen and Xcertia guidelines, we find that the number of guidelines in UGmHA (16 guidelines) is greater than the number of guidelines in Nielsen (10 guidelines) and Xcertia (10 guidelines). The reason is that UGmHA integrated Nielsen and Xcertia by adapting and eliminating their guidelines through Quinones et al.’s methodology [17] as well as created a new guideline that covers features that Nielsen and Xcertia did not cover. Table 13 shows the UGmHA guidelines with the corresponding guidelines in Nielsen and Xcertia.

Table 13. Relation of the UGmHA guidelines with Nielsen and Xcertia.

ID	UGmHA	Corresponding Guideline (Nielsen)	Corresponding Guideline (Xcertia)
MHP1	Visibility of system status	Visibility of system status	App feedback
MHP2	Match between system and the real world	Match between system and the real world	-
MHP3	User control and freedom	User control and freedom	App navigation
MHP4	Consistency and standards	Consistency and standards	Visual design
MHP5	Error prevention	Error prevention	Visual design
MHP6	Recognition rather than recall	Recognition rather than recall	Visual design
MHP7	Flexibility and efficiency of use	Flexibility and efficiency of use	-
MHP8	Aesthetic and minimalist design	Aesthetic and minimalist design	Readability
MHP9	Error diagnosis and recovery	Error diagnosis and recovery	-
MHP10	Help and documentation	Help and documentation	Help resource and troubleshooting
MHP11	Notifications, alerts and alarms	-	Notifications, alerts and alarms
MHP12	Onboarding	-	Onboarding
MHP13	Historical data	-	Historical data
MHP14	Accessibility	-	Accessibility
MHP15	Ongoing app evaluation	-	Ongoing app evaluation
MHP16	Interactivity and Motivations	-	-

Furthermore, the number of checklist items in UGmHA (154 items) is greater than the number of checklist items in Xcertia (60 items), while there are no checklist items in Nielsen. As mentioned before, Nielsen’s guidelines are general, and a small number of checklist items in Xcertia guidelines are specific to the features of mHealth applications.

Therefore, we adapted and extended the guidelines by adding more checklist items that cover specific features of mHealth applications and features of accessibility for the elderly and people with special needs. The checklist items were extracted from the following studies:

• HE4EH [6], which covers self-monitoring, biometric measurements, etc.
• Telemedicine of mHealth applications [13], which covers consultation, booking appointments, etc.
• Elderly people [34], which covers the feature of accessibility for elderly people.
• People with special needs [35], which covers the feature of accessibility for people with special needs.

This leads to the main contribution of UGmHA by including more guidelines, checklist items and mHealth-specific checklist items than Nielsen and Xcertia as shown in Table 14.

Table 14. Guidelines and checklist items of UGmHA, Xcertia and Nielsen.

	UGmHA	Xcertia	Nielsen
Number of guidelines	16	10	10
Number of checklist items	154	60	0
Number of mHealth-specific checklist items	44	6	0
Number of accessibility checklist items	18	7	0

5.2. UGmHA Validation Discussion

From the first experiment of UGmHA validation, we conclude that UGmHA discovered more usability issues than Xcertia and Nielsen for both applications. This is because UGmHA includes more guidelines and checklist items than the others. Moreover, the usability issues discovered in the “Sokry” application are more than those in the “Sehhaty” application for the three guidelines, which means that UGmHA can perform like Nielsen and Xcertia to determine which application is more usable.

For the severity of usability issues, we found that UGmHA found more severe usability issues than Xcertia and Nielsen. Again, this is because UGmHA includes more checklist items and more mHealth-specific checklist items.

For the UGmHA guidelines, we found that the highest number of issues were with the accessibility guideline in both applications. The reason is that the specifications of special needs and elderly people were not considered in the design, such as color preferences, font sizes and the number of words and sentences. This highlights the importance of the accessibility guideline and its checklist items that determine to what extent the application is accessible. For the newly created guideline Interactivity and motivations, we found several usability issues in both applications. This also determines to what extent the applications motivate users to take action and achieve a specific goal.

However, we could not use the Ongoing app evaluation guideline in the experiment since it is up to the application team. Therefore, we will leave it up to the expert judgment phase to acquire their opinion and suggestion about it. In the near future, we plan to provide design recommendations for “Sehhaty” or “Sokry” based on the discovered usability issues and, then, to provide these recommendations to the application team to take their opinion.

One of the challenges we faced was that we could not perform phases 2 and 3 of the validation stage. The reason is that we contacted the user experience (UX) team of the “Sehhaty” application to provide their opinion regarding UGmHA and discovered usability issues, they were willing to cooperate with us; however, right now, they are busy with government projects. Therefore, we postponed the implementation of phases 2 and 3 for a few months in future work.

6. Conclusions and Future Work

With the evolution of mobile applications that have become a supportive technology for different fields, mobile health (mHealth) applications are important to improve the efficiency of healthcare delivery using mobile devices, such as smartphones and wearable devices.

mHealth allows patients to monitor and track their health data and allows communication between patients and their physicians without meeting face to face. However, mHealth applications can harm users if they are not working as intended. Although some studies have discussed general and specific mHealth guidelines, there are certain limitations to them.

In this research, we proposed comprehensive usability guidelines for mHealth applications (UGmHA) that address some of the limitations of previous studies and cover various features related to mHealth applications. The proposed guidelines consist of sixteen (16) guidelines with 154 checklist items built from global and well-known guidelines, such as Nielsen and Xcertia.

In order to validate the effectiveness of the UGmHA guidelines, an experiment was conducted by measuring the performance of the proposed guidelines through comparing the outcomes of the UGmHAto worldwide guidelines. This comparison was applied to two mHealth applications in Saudi Arabia, “Sehhaty” and “Sokry”, to determine which guidelines could identify more usability issues. The results showed that the proposed guidelines discovered more usability issues compared with the others. Thus, UGmHA can assist expert evaluators, designers and developers in designing or evaluating mHealth applications by measuring the usability issues that can influence the user experience of mHealth applications.

For future work, further steps will be conducted to validate and refine UGmHA, which include:

• Generating design recommendations based on the discovered usability issues for “Sehhaty” or “Sokry” and obtaining the application team’s opinion.
• Expert judgment by reviewing guidelines by user experience (UX) experts and acquiring their advice and comments to perform the Refining stage.
• User testing by enhancing the usability of an existing application using refined guidelines of UGmHA and, then, performing user testing for the enhanced application.