A Framework for Designing Usability: Usability Redesign of a Mobile Government Application

Abstract

The existing usability models have been used primarily for evaluation, not for usability engineering. The models were found to be general for specific mobile applications. They also lack appropriate guidelines to apply the usability models to m-government applications. Earthquake information is an example of critical information delivered to citizens via m-government applications. Usability design is considered a very important key factor to the success of such applications. This research addresses the challenges in finding the usability factors important to m-government applications and choosing appropriate factors for specific m-government applications. A questionnaire was administered to 49 citizens. The results include six usability factors which are learnability, simplicity, satisfaction, security, privacy, and memorability. Descriptions of the usability factors were later added to provide a clearer definition for each factor. This paper proposes the usability design framework for m-government applications. The use of the framework was illustrated based on the user interface redesign of the EarthquakeTMD application. The main aim was to demonstrate the applicability of the framework. The quality of the original UI design of the application in the case study was assessed with a questionnaire which was administered to 57 Thai citizens who lived in the areas affected by the disasters. Four designers participated in UI redesigning and produced four different UI designs. The new UI designs were evaluated via two usability tests on two sample groups of representative users. The first usability test was conducted with 24 participants. Twenty-four test cases were used. The second usability test was conducted with 351 representative users. After the tests, both sample groups were given a questionnaire based on the SUS (System Usability Scale). The same two UI designs by experienced and inexperienced designers who used the framework received the highest scores: 89.58 and 87.60 on the first usability test. They also received the highest score on the second usability test: 89.10 and 90.88. The results reveal that the citizens preferred the new user interfaces designed using the framework. It was found that the scores of the UI designed by inexperienced designers who used the framework were as high as the scores of the UI designed by experienced designers, whereas the UI designs from the designers who did not use the framework received the lowest scores: 63.23 and 54.27 on the first usability testing and 59.34 and 46.53 on the second usability testing.

1. Introduction

Mobile government applications have been increasingly used by federal and local governments to deliver public services to citizens and involve them in decision-making processes [1]. The use of mobile government applications helps promote communication and interaction between government departments, agencies, and citizens. Services offered via the applications range from news, information, contracts, surveys, health, education, finances, etc. Some types of information, e.g., safety and warning messages, are more critical than others. Such messages are critical and should be presented to the intended audience in a way that they can be quickly located, learned, understood, and used so that the risks for damage, e.g., the number of deaths and economic losses can be lowered [2]. The interfaces designed with appropriate usability factors are highly likely to have high usability value and a positive effect on intention toward continually using the application.

Mobile government applications are perceived as the fastest and the most convenient way for citizens to receive information broadcasted. As mobile technologies become cheaper, mobile phones are affordable and easy to purchase even in remote areas where other means of communication are limited or unavailable. The growth of mobile devices also gives rise to mobile government applications which are used by government agencies to disseminate information and communicate with citizens.

There are challenges in developing mobile applications: first, the applications must be designed to function on a variety of small screen sizes; second, the usability design of mobile government applications; and third, designing for effective presentation of information for citizens who represent a wide range of recipients. The challenges can be addressed on many levels ranging from devices’ form factors, mobile application development approaches, information design process, usability design, and so on.

Studies on usability frameworks and guidelines for user interface and usability design [3] revealed important factors for the usability of mobile government applications. The factors include learnability, privacy, simplicity, governance, simultaneousness, organization, effectiveness, security, responsiveness, memorability, and notification [3].

The designers of mobile government applications are expected to have knowledge on the usability characteristics of the applications [4]. However, the important usability characteristics are not well addressed by the existing approaches. This suggests that the usability design approach for mobile government applications should recommend the usability design factors to the designers and provide a guideline on how they can be used. This paper presents the usability design framework for mobile government applications that includes the usability factors and the usability design guideline. The guideline is created to aid in the transformation of the interface requirements to the design of information according to the usability factors. Section 2 of this paper discusses existing usability models and frameworks. Section 3 introduces the framework. Section 4 describes an application of the framework. Section 5 presents the evaluation. The last section concludes the paper.

2. Usability-Related Models and Framework

Usability Model

The models that reflect the usability of mobile applications include Mobile Goal Question Metric (mGQM) [5,6] and People At the Centre of Mobile Application Development (PACMAD) [7]. mGQM is a model used for usability evaluation. It is based on ISO 9241-11. There are six performance goals namely simplicity, accuracy, time taken, features, safety, and attractiveness. Although mGQM was originally designed for mobile application development, the model does not provide sufficient explanation on how to select appropriate usability characteristics for a specific mobile application. The framework outlines the order in which they accomplish their objectives using questions, and the results are measured using the metrics. PACMAD, on the other hand, is a usability model that combines attributes from Nielsen’s model and the ISO model. The seven attributes of PACMAD are effectiveness, efficiency, satisfaction, learnability, memorability, errors, and cognitive load. Although the model is useful for designing the usability of the user interface of mobile applications, the designers find no guideline in achieving the usability of the user interface [7]. 7C is another framework that highlights seven factors that can be used to design user interfaces. The factors are context, content, community, customization, communication, connection, and commerce. The framework was later extended to include two factors in the mobile environment: mobile device constraints and mobile settings. Thus, it can be used to evaluate the information quality of m-government’s interface [8]. However, the factors are not directly translated to usability factors. Therefore, additional efforts are required.

Usability framework

Azeez and Lakulu [9] proposed a framework to evaluate M-Government Service Success (M-GSEF) for mobile government services from the citizen’s perspective. The framework identified eight dimensions and various factors that affect m-government’s success. Dimensions are system quality, information quality, service quality, citizens’ use, citizens’ satisfaction, citizens’ trust, perceived m-government service quality, and perceived effectiveness of m-government services. Some attributes are defined in more than one dimension. In addition, this framework provides a broad evaluation guideline that does not consider specific details such as information display and does not provide any example of how the framework is used. However, it does describe information quality, service quality, and system quality, all of which are lacking in guidelines or procedures.

Isagah and Wimmer [10] presented MGOV, a framework supporting organizations and designers responsible for designing mobile public services in developing countries. MGOV contains four iterative phases, which are (1) initiation, (2) design, (3) implementation and deployment, and (4) maintenance and governance. The framework identified ten components that support designing m-government services. The framework includes preconditions, guidelines for implementing the framework, design principles, guidelines for designing m-government services, MGOV method, requirement management, viewpoints, metamodel, stakeholder management, and framework repository. The set of stepwise approaches for government agencies to follow when providing services was established. The steps focus on service design. They are not restricted to mobile application development or usability.

Poor user interface was identified as a technological challenge in both developing and developed countries [11]. Also identified were the lack of skills in designing mobile government applications (e.g., process framework and guideline) and the lack of proper guidelines (e.g., proper content presentation) for content development. Under the sociocultural challenge category, the lack of user-friendly application and poor user readiness for m-government service were identified.

Zamzami and Mahmud [8] presented a mobile interface of m-government services for any mobile government applications using the relation between 7C’s design and information quality to determine the level of citizen satisfaction. This framework focuses on the information quality of the application based on perceived usefulness, perceived ease of use, and perceived accessibility. 7C’s design focuses on interface design elements including context, content, communication, and customization. The framework does not provide any guidelines or steps to follow. Another study by Ahmad, Rextin, and Kulsoom [12] presented a systematic literature review of usability guidelines for mobile application. The guidelines cover recommendations for information presentation and human–computer interaction. The recommendations do not address the connection between usability design goals and the outcomes of the design.

Amores, Vasardani, and Tanin [13] presented map design and memory load as part of usability design for emergency applications. Appropriate maps were recommended to support the presentation of information. Weichbroth [14] studied 75 attributes and found frequently used usability factors for mobile application from 790 documents during 2001 to 2018 which are (1) efficiency (70%), (2) satisfaction (66%), (3) effectiveness (58%), (4) learnability (45%), (5) memorability (23%), (6) cognitive load (19%), (7) errors (17%), (8) simplicity (13%), and (9) ease of use (9%).

Al-nuiam and Al-Harigy [15] found 14 user interface guidelines that were appropriate for mobile applications. The guidelines cover content selection, content organization, layout, information presentation, human–computer interaction, and mobile context. A similar guideline was found in [16]. It emphasizes that things to focus on in designing are layout, information presentation, organization, human–computer interaction, and language.

Challenges of the existing usability models and frameworks

Unique characteristics of mobile applications [17] pose challenges in designing usability for the interfaces. There are several usability guidelines developed for mobile applications [5]. However, the models are general and not suitable for specific mobile applications [5,18]. The usability factors defined are often described at a high level of abstraction, and thus, they provide very little or no implementation detail. The ISO standards, though they come with guidelines and metrics, are general [5].

While m-government applications share common attributes with other types of mobile applications, they have specific usability characteristics that require specific measurements and guidelines [18]. The existing usability models such as in [6,7,9] do not provide the guidelines to choose appropriate usability characteristics for specific applications. Thus, the need for appropriate guidelines on how to choose suitable usability attributes is recognized. With appropriate guidelines, suitable factors can be chosen for a particular type of m-government application. The usability characteristics can be shaped during the design phase. For example, understanding and learnability are very important usability characteristics of an earthquake report system, a life-critical mobile application. If the most important usability characteristics can be identified early in the development process, the appropriate user interface elements can be chosen to promote the usability of the user interface.

The existing usability models lack guidelines on how to determine appropriate usability characteristics for m-government applications, due to their unique nature of information, diversity of applications, and users. M-government applications range from simple information systems to life-critical systems. They serve millions of users, regardless of application type [19]. The diversity of users is exceptional. Thus, usability is considered very important in allowing users to get the most out of the m-government applications they use.

Hence, m-government applications should be developed with a focus on usability engineering which can lead to user interfaces that contain suitable usability characteristics. The challenges for providing support for the usability design process for m-government applications are as follows.

• The guidelines are, therefore, needed to allow practitioners to choose suitable usability attributes in engineering the usability of the applications.
• Usable software can increase technology acceptance as well as achieve the goals of m-government applications.
• The usability attributes should be adequately described to ensure the proper selection of user interface elements.
• The description of usability attributes can be linked to the measurement of specific mobile applications.

Thus, usability should be considered early in the development process, especially in requirements engineering and design to achieve usable software via usability design of the user interfaces.

Usability for people with disabilities

Usability and accessibility were found to be important factors for designing the UI for disabled people [20]. Screen design and multimedia content should be considered. For visually impaired users, color-conscious design is important [21].

Accessibility, usability evaluation, information architecture, user research, user interface, web analytics, and content strategy were also found to be the focus when developing UIs for mobile applications for people with disabilities [22]. The questionnaire, cognitive walkthrough, heuristic evaluation, think aloud, and SUS were the most frequently used methods for usability evaluation.

Usability design, information science, and computing

Usability design is significant to the acceptance of the interfaces, task completion, and success of businesses. UIs redesigned based on Nielsen’s usability heuristics were found to be more efficient than the original screen [23]. Placing the important information items on the first page allows them to be easily accessible [24]. The screens that were designed with usability factors were improved in performance, efficiency, and satisfaction [25].

The results of the previous studies suggest that usability is important to how information is seen, used, and contributed to the success of user tasks and businesses. Therefore, usability design is vital to information science in the way that it helps users to easily locate and retrieve information. It is also essential to the success of information dissemination.

Computing techniques and algorithms can contribute to the usability design process. For example, an automatic analysis of software requirements can be performed with the help of NLP. It can allow the usability characteristics to be determined and the evaluation metrics to be suggested.

Table 1 addresses the limitations of the usability design of the models and frameworks and challenges. The PIN framework proposed in the next section describes how some of the challenges are addressed.

Table 1. Limitations of the existing models and how they can be addressed.

mGQM
Usability factors	Limitations	Challenge
Simplicity, Accuracy, Time taken, Features, Safety, Attractiveness	- mGQM includes usability characteristics identified as an evaluation model for mobile applications. However, it does not include a usability design model.	- Although most usability models are created for evaluation purposes, the usability design model is also needed to guide usability requirements engineering that can lead to usable interfaces.
	- The model does not contribute to the design of the system under evaluation. Thus, the evaluation is performed based on the system behavior rather than system specification. In addition, the metrics derived may not be appropriate for verifying the system based on specification.	- The evaluation model should support an evaluation based on the system specification, especially usability requirements, as well as an evaluation based on system behavior.
	- While the model is for mobile applications, it may not be suitable for mobile government applications where additional usability characteristics are required.	- Additional usability characteristics should be identified for mobile government applications.
PACMAD
Usability factors,	Limitations	Challenges
Effectiveness, Efficiency, Satisfaction, Learnability, Memorability, Errors, and Cognitive load	- The original PACMAD usability model for evaluating mobile applications lacks guidelines and metrics. Later, the model is extended to include the usability metrics using GQM, but the evaluation guideline is still absent.	- There should be guidelines for applying the model for the usability evaluation of a particular mobile application as well as for selecting appropriate usability factors and related dimensions.
	- Usability factors identified are user, task, and context of use. How these factors are used to evaluate mobile applications is unclear.	- A guideline for identifying measures for the usability factors can increase the model’s applicability.
	- The model does not provide suitable descriptions of the usability attributes. Thus, the attributes are described in general rather than specific to mobile applications.	- Each usability attribute should be described in enough detail that can be linked to the measurement of specific mobile applications.
MGOV
Usability factors	Limitations	Challenges
Unspecified	- The framework focuses on designing m-government services by providing the conceptual development guideline from initiation to maintenance and governance. The usability model of the electronic services is not included.	- The guideline can be refined to include detailed activities of each phase.
M-GSEF
Usability factors	Limitations	Challenges
Unspecified	- Usability is described as one of the success factors of m-government services based on the information systems success model. However, the description of usability is very vague, incomplete, and, thus, not a practical measure. Moreover, there is no guideline to select suitable factors for specific types of m-government services.	- The framework should provide suitable descriptions for usability factors that can be linked to measurement. - A guideline to support choosing appropriate factors (out of 46) can facilitate practitioners in designing and evaluating a particular type of m-government services.
Goal-Directed Design (GDD)
Usability factors	Limitations	Challenges
Unspecified	- The model includes the development guideline for mobile applications which focuses on designing functional models. Usability is tested to find improved areas but is not previously considered in any of the design and development activities.	- Usability should be considered early in the development process, especially in requirements engineering and design to achieve usable software.

The existing usability models, frameworks, and SDLC

The frameworks mentioned previously, MGOV and M-GSEF, and usability models, mGQM and PACMAD, can be described in relation to the Software Development Life Cycle (SDLC) as shown in Figure 1.

Figure 1. Relationships of the existing usability-design-related frameworks and SDLC.

MGOV is related to the SDLC in the defining, designing, developing, testing, and deploying phases of mobile government services. For instance, in the defining phase, service strategies are defined. In the design phase, the architecture of data, services, and applications is designed. For M-GSEF, it is related to the SDLC in the defining and testing phases. In the defining phase, the service quality is defined. In the testing phase, information quality is evaluated based on customers’ points of view. The two usability models, mGQM and PACMAD, are related to the SDLC in the testing phase as they provide measures and metrics for the usability evaluation of mobile applications. The next section introduces PIN, a usability design framework proposed to address the challenges in the usability designing of m-government applications.

3. Introduction to the PIN Usability Design Framework

The study of the related literature on m-government application and usability described in Section 2 raises the following research questions (RQs).

RQ 1:

What are the usability factors that are important to the user interface design of m-government applications?

RQ 2:

How can usability factors be used to support designing to incorporate usability characteristics into the user interface?

RQ 3:

Do the existing usability frameworks or models offer usability design guidelines to the designers of mobile government applications?

RQ 4:

How are suitable usability factors identified for the user interface?

Methodology

RQ 1: The survey was administered to 49 Thai citizens to investigate citizens’ points of view on the usability characteristics of m-government applications. The result gives an answer to RQ 1. It identified six usability factors consisting of learnability, simplicity, satisfaction, security, privacy, and memorability.

RQ 2: Kureerung and Ramingwong [26] studied the related literature and found that the existing research on usability design is unable to provide specific guidelines to help the designers of m-government applications in using the usability factors to design the user interfaces. The literature [5] suggested that the link between the usability characteristics and the supporting metrics must be defined to use the usability model with a specific application. Thus, a divide-and-conquer approach was used to define the usability characteristics to provide more specific descriptions and at the same time make it simpler to measure. The descriptions are identified as sub-usability characteristics which also help to provide links to the corresponding metrics.

RQ 3: A literature review was conducted. The results are explained in Section 2 and Table 1. It identified that most of the existing usability models lack guidelines for evaluating particular applications as well as for designing the usability of them.

RQ 4: In any software development process, requirements and design are closely related. The outputs of requirement activities determine how good the design is. The analysis of requirements can lead to the identification of usability characteristics and the corresponding design characteristics that contribute to the achievement of the usability characteristics. Therefore, the design process was designed to realize the suitable usability characteristics for a specific m-government application. The outcome of the process is a user interface that encompasses the appropriate usability characteristics for the particular application.

The usability factors obtained from the survey in response to RQ 1 represent the quality characteristics of usability from the users’ viewpoint. From the designers’ viewpoint, the factors represent the goals of the user interface design. Hence, the process begins with goal identification based on the requirements. Because the overall usability of a user interface can be determined based on the usability of each information entity located on the user interface, the entities must be identified first. Then, appropriate usability factors for each entity can be identified. An information entity of a user interface can refer to an abstract entity that may consist of several UI components.

After the UI components are identified, the corresponding usability factors can be determined for each component. As mentioned earlier in response to RQ 2, the usability factors identified should be described. Thus, the description of each UI component is created by identifying sub-factors. The sub-factors, though useful, still need to be given tangible descriptions of the components. The tangible descriptions are referred to as Criteria. This allows the detailed design to be specified directly for the corresponding UI components. During the design process, Criteria can be used to measure whether the design outcomes satisfy the goals of user interface design. The measurement may be repeated until the desired level of usability is reached. The sections below describe the PIN framework and the core component, usability factors.

3.1. PIN: The Usability Design Framework

This paper describes a new usability design framework, PIN, which consists of two processes: Preparation of Interaction (PI) and Necessary quality element identification (N). Figure 2 shows the architecture of PIN, which represents the guidelines to achieve the user interface with usability characteristics. The two processes, PI and N, are described as follows.

Figure 2. PIN usability design framework’s structure.

3.1.1. Preparation of Interaction (PI)

This is a semi-automatic process to prepare information entities for the user interface under design. It involves analyzing requirement sentences to recognize the information entities. During the analysis, the technical terms may need to be clarified if the analyst does not have knowledge of the terms. After that, NLP is used to perform the extraction of information entities from the requirement sentences. Figure 3 shows the process flow which contains requirement elicitation, entity extraction, and transformation.

Figure 3. The Preparation of Interaction (PI) process.

• Requirement elicitation

Once requirement information is retrieved, the requirements are examined to ensure nothing is lost before beginning extraction. In this activity, additional information may be added to the original requirements, if necessary.

The requirement elicitation activity is encouraged to ensure that the requirements are sufficient for the next stage. The requirements may be insufficient if the original requirements lack usability information or they may require an update due to environmental changes, e.g., emerging user requirements, new interface design standards, new technologies, etc.

B.

Entity extraction

This step involves extracting information entities from the requirements obtained from the previous step. The Natural Language Processing (NLP) unit is used for extracting words. The automatic analysis of words leads to a more accurate result than a manual analysis which could result in bias or prejudgment of the words.

First, the requirement sentences stored as unstructured text are forwarded to the NLP unit to separate words. Then, punctuation and stop words are removed to cut down on unnecessary words. Next, the part-of-speech (POS) tagging procedure is carried out. The results are forwarded to the chunking process which initiates the search for tokens (word phrases from unstructured requirement sentences) or information entities from the requirement sentences.

C.

Transformation

The purpose of this step is to analyze information entities to identify the corresponding UI components. At the beginning of the process, the list of UI components is created as an empty list. An information entity will be added to the list and identified as a new UI component or merged with the existing component. The UI component list contains groups of information entities. Some entities may belong together or be expressed by the same UI component.

3.1.2. Necessary Quality Element Identification (N)

In this process, the Main Usability Factors (MUFs) are identified for each UI component to describe the usability characteristics of the components. MUFs are split into Sub Usability Factors (SUFs) to provide detailed descriptions to the designers. Then, Criteria are specified for SUFs. Criteria direct the designers to obtain an explicit design guideline for each MUF and provide a way for usability evaluation. Figure 4 shows the Necessary quality element identification (or N) process.

Figure 4. The Necessary (N) quality element identification process.

3.2. Usability Factors

Usability factors are the core element of the PIN framework. These factors are referred to as the Main Usability Factors (MUFs) which can be broken down into smaller elements, namely Sub Usability Factors (SUFs). Each SUF is then described further as Criteria. MUF, SUF, and Criteria form the three elements of usability design. The first element, MUF, indicates the main usability goals of a UI component. The second element, SUF, indicates the MUF’s components, and Criteria indicate the usability measure.

Main Usability Factors (MUFs) reflect the extent of usability of the user interface under design. It consists of Learnability, Simplicity, Satisfaction, Security, Privacy, and Memorability.

• Learnability (L): Rapid understanding and use of information.
• Simplicity (Si): Easy to understand and use.
• Satisfaction (Sa): Extent of pleasure in using the system.
• Security (Se): Protection and safety of data and program.
• Privacy (P): Protection of users’ personal information.
• Memorability (M): Easy to remember and recall.

A Sub Usability Factor (SUF) serves as a description of the Main Usability Factors (MUFs). SUFs inform the designers of the components of usability. For example, learnability is identified with the following SUFs: readability, understandability, and actionability. This means that the designers should make sure the design of the specific UI follows the SUFs identified. Here is the list of MUFs and their corresponding SUFs:

• Learnability (L): understandability (L1), readability (L2), and actionability (L3);
• Simplicity (Si): usefulness (Si1), intuitiveness (Si2), operability (Si3), and aesthetics (Si4);
• Satisfaction (Sa): error guidance (Sa1), likeability (Sa2), and accessibility (Sa3);
• Security (Se): confidentiality (Se1), and trustfulness (Se2);
• Privacy (P): protective (P1);
• Memorability (M): minimal memory load (M1).

Similarly, SUFs are broken down into Criteria, the smallest element in the PIN framework. A UI component may be described by one or more MUFs, SUFs, and Criteria. The next section presents the case study used in this paper to demonstrate how the framework was applied.

4. The User Interface Redesign Using PIN Usability Design Framework

This section illustrates how the framework was used to assist the designers in redesigning the user interface of the mobile government application. The framework contains the usability factors for mobile government applications as well as the guideline for incorporating them into the usability design process and the product. In this paper, the two-stage framework was used to redesign the user interface of the earthquake report screen to improve the usability of the original interface. The user interface in this case study is from the mobile government application called EarthquakeTMD from the Earthquake Observation Division, Thailand. This mobile application was created on August 6, 2018, in response to the need to provide quick access to current and timely earthquake information to Thai citizens. This allows people to quickly access and learn about earthquakes nearby. When information on earthquake strikes or any type of information is available on the application, the ability of the application to communicate the messages to the recipients and allow them to quickly use it directly depends on the usability of the application.

Shown below in Figure 5 is the original design of the earthquake information report screen of EarthquakeTMD, the mobile government application. The interface consists of two main parts: earthquake information and a map to show the earthquake’s location. The PIN framework was used to redesign the original user interface to improve usability. The questionnaires were administered to a group of citizens to ask questions regarding the quality of usability of the original design including the three usability factors: learnability, simplicity, and satisfaction. Fifty-seven respondents were Thai citizens living in northern, northeastern, and central Thailand. The areas were impacted by flood calamity and PM 2.5 in the past. The questionnaire used a 5-point Likert scale: Excellent, Good, Neutral, Fair, and Poor. The survey’s result is shown in Table 2.

Figure 5. The original design of EarthquakeTMD’s earthquake report screen.

Table 2. The result of the questionnaire represents the citizens’ perspectives on the three usability factors or factor in use: learnability, simplicity, and satisfaction.

Factor in Use	Factor Requirement	Questions	Mean	S.D	Meaning
Learnability	Understandability	The user interface is clear and concise. Users can easily comprehend it.	3.35	1.03	Neutral
	Readability	The texts are simple, meaningful, organized and well arranged.	3.51	1.00	Good
	Actionability	Access to information does not exceed three tasks.	3.30	1.09	Neutral
Simplicity	Usefulness	The screens are comprehensive and reliable.	3.56	0.95	Good
	Intuitiveness	All message signs are clear and correctly designed.	3.47	1.09	Good
	Operability	Citizens can meet their objectives through the interface.	3.47	1.04	Good
	Aesthetics	The screens and components are attractive to use.	3.21	1.10	Neutral
Satisfaction	Likeability	The design of the Information is aesthetically pleasing. It meets users’ objectives and is easy to use.	3.77	0.85	Good
	Accessibility	Information is easily accessible by people of various abilities and supports specific disabilities.	3.46	1.00	Good

The result of the questionnaire for the original user interface design reveals the degree of usability, strengths, and weaknesses. Likeability was ranked with the highest score (mean value of 3.77), while aesthetics was ranked with the lowest score (mean value of 3.21). Understandability, actionability, and aesthetics were areas that could be improved.

The user interface was redesigned using the PIN usability design framework. The framework emphasizes requirement elicitation which gives the designers a clear view of the requirements and provides the basis for the analysis of the requirements. The activity prepares the ground for traceability and assessment of the design.

The framework involves elicitation and analysis of the requirements (shown in Table 3). In the elicitation activity, the requirements were refined or extended using a set of questions created to elicit additional requirements or refine them. This is to help the designers to get a more complete understanding of the user interface. Once the elicitation was completed, the words were extracted from the new set of requirements to look for information items. These steps facilitate the generation of requirements and identification of information entities. The design detail report was created as a result of the elicitation and analysis activities. Table 3 shows the first part of the report which indicates the requirements that are prepared for elicitation. It includes interface name, original requirements, input data (resulted from the requirement analysis), design recommendation, and discussion questions. The input data were identified by analyzing the original requirements to prepare input data for the user interface design process. The design recommendation sets the focus of the requirement, whereas the discussion questions were generated based on the original requirements. The purpose of the questions was to clarify some parts of the requirements. Table 4 shows part two of the report where the result of using NLP to extract the keywords is displayed. These information entities denote the elements of information that should appear on the new user interface.

Table 3. Part one of the design detail report.

Design Detail Report
Interface Name:	Individual Earthquake Report
Interface Requirement:	Citizens need to be able to understand technical information easily and information have to be displayed on the map to show where the coordinates of the earthquake are. Displays the position of the earthquake on the map with details of the earthquake.
Requirement Analysis:
Input Data	Region, Date(UTC), Time(UTC), Magnitude, Epicenter, Depth, Direction
Design Recommendation	Citizens need to be able to understand technical information easily
Discussion:	Q1: How the position is represent on the map? (Answer: Marker with detail on click) Q2: What details are required? (Answer: Location, Magnitude, Date, Time) Q3: How to represent them? (Answer: Marker/Text, Graphic/Text, Text, Text)

Table 4. The outcome of POS tagging on structured requirement sentences.

Structured Sentence

POS Tag *

Citizens need to be able to understand technical information easily and information have to be displayed on the map to show where the coordinates of the earthquake are. Display the position of the earthquake on the map with details of the earthquake. How the position is represented on the map? (Answer: Marker with details on click) What details are required? (Answer: Location, Magnitude, Date, Time) How to represent them? (Answer: Marker/Text, Graphic/Text, Text, Text)

citizens(N), need(V), able(Adj.), understand(Adj.), technical(Adj.), information(N), easily(Adv.), information(N), displayed(V), map(N), show(N), coordinates(N), earthquake(N), display(N), position(N), earthquake(N), map(N), details(N), earthquake(N), position(N), represented(N), map(Adj.), answer(N), marker(N), details(N), click(V), details(N), required(V), answer(Adj.), location(N), magnitude(N), date(N), time(N), represent(Adj.), answer(N), marker(N), text(N), graphic(Adj.), text(N), text(N), text(N)

(* “N”: noun, “V”: verb, “Adj.”: adjective, “Adv.”: adverb).

The requirement sentences and additional requirements obtained from the answers to the discussion part (as in Table 3) were forwarded to the NLP unit for separation of words. Then, punctuation marks and stop words were removed to identify the information entities. Next, part-of-speech (POS) tagging was performed. Table 4 presents the outcome of POS tagging consisting of requirement sentences and associated POS tags. In the end, a group of information entities was identified. Next, noun phrase (NP) chunking was performed to identify meaningful text. The rule used in this step is {<JJ.?>*<NN.?>} which means that a chunk should be formed whenever zero or more adjectives (JJ) are followed by zero or more singular nouns (NN).

The outcome of POS tagging is the list of information entities. The list contains information, map, earthquake, position, represent, marker, location, magnitude, date, time, and text. This list was used to identify the UI components. Table 5 presents the list. The identification of UI components started by checking the existence of each entity, symbolized as E. If the entity was not already in any group, symbolized as C, then a new group of components was created. The entity was added to the group. For example, when E₁ was considered and not found in any existing groups, a new group, C₁, was created, and E₁ was added to C₁. When another entity was considered to have the same meaning or purpose as the entity in any of the existing groups, it was added to that group. The steps were repeated until all the UI components were identified. In this case study, the UI components identified were information, map, position, magnitude, date, time, and text.

Table 5. The UI components identified from the list of information entities.

Entity		Group	Component
E1	information	information	C1
E2	map	map	C2
E3	earthquake	information, earthquake	C1
E4	position	position	C3
E5	marker	position, marker	C3
E6	magnitude	magnitude	C4
E7	date	date	C5
E8	time	time	C6
E9	text	text	C7

The next step is identifying the usability factors relevant to the UI components found earlier. Figure 6 presents the usability factor identification model showing the UI components as inputs for usability factor identification. Three elements were identified for each component: Main Usability Factor (MUF), Sub Usability Factor (SUF), and Criteria.

Figure 6. The usability factor identification model.

The flow shown in Figure 6 leads to the creation of the design checklist as shown in Table 6. It shows the usability factors for each UI component. This table was used as a design guideline. For example, information should be designed to be understandable, readable, useful, intuitive, and aesthetically pleasing.

Table 6. The design checklist for UI components.

Component	Learnability (L)			Simplicity (Si)
	Understandability	Readability	Actionability	Usefulness	Intuitiveness	Operability	Aesthetics
information	✓	✓	-	✓	✓	-	✓
map	✓	✓	-	-	✓	-	✓
position	✓	✓	-	✓	-	-	-
magnitude	✓	✓	-	✓	-	-	✓
date	✓	✓	-	✓	-	-	-
time	✓	✓	-	✓	-	-	-
text	✓	✓	-	-	✓	-	✓
Component	Learnability (L)
	Error Guidance	Likeability	Accessibility
information	-	✓	-
map	-	✓	✓
position	-	-	✓
magnitude	-	-	✓
date	-	-	✓
time	-	-	✓
text	-	-	-

Some usability design factors, including Security (Se) and Privacy (P), are not shown in Table 6 as the EarthquakeTMD application did not include a security function. It did not have a login feature and did not involve storing personal information. In an application where personal data must be collected and displayed on the screen, the Security (Se) factor must be considered. Table 7 presents the usability design checklist obtained for the UI components from Figure 6. This checklist serves as a guideline and criteria to be used in choosing design options and evaluating the finished product.

Table 7. The design checklist.

Component	Sub Usability Factor (SUF)
C1 information	L1	L2	Si1	Si2	Si4	Sa2
C2 map	L1	L2	Si2	Si4	Sa2	Sa3
C3 position	L1	L2	Si1	Sa3
C4 magnitude	L1	L2	Si1	Si4	Sa3
C5 date	L1	L2	Si1	Sa3
C6 time	L1	L2	Si1	Sa3
C7 text	L1	L2	Si2	Si4

L1: Understandability, L2: Readability, Si1: Usefulness, Si2: Intuitiveness, Si4: Aesthetics, Sa2: Likeability, Sa3: Accessibility.

Table 8 shows the usability design guideline for a UI component of EarthquakeTMD. The guideline depicts the abstract goals (MUFs and SUFs) and the concrete goals (Criteria) recommended for the UI components. Once the design is implemented on the user interface, Criteria can be used to evaluate whether the final design satisfies SUFs and MUFs. The user interface that is designed to meet the desired usability characteristics (factors) is highly likely to reach the target audiences sooner and allow them to effectively use the intended message.

Table 8. (a) The usability design guideline for designing component “texts” and (b) the corresponding design.

(a) The Usability Design Guideline for Component “Texts”.
Design Component: Texts
MUF	SUF	Criteria
Learnability	Understandability	L1_1	Completeness of information
		L1_2	The application functions are evident to the user
		L1_3	Users are able to understand correctly
		L1_4	Easy to understand
	Readability	L2_1	Elements are well-arranged and simple
		L2_2	Graphics are meaningful
Simplicity	Intuitiveness	Si2_1	All signs are designed clearly of meaning
	Aesthetics	Si4_1	Screens and components must be arranged properly with balance, order, and visual appeal.
(b) The corresponding design resulted from MUFs and SUFs listed in (a).
N-F
E-F

To evaluate the usefulness of the usability design guideline, the guideline was given to inexperienced and experienced designers who were tasked to redesign the user interface of EarthquakeTMD. Table 9 presents the profiles of the four designers who participated in this study. The next section presents the evaluation of the redesigned user interface of EarthquakeTMD’s earthquake report screen.

Table 9. The designers’ profiles.

Designer	Profile	Background	Tools	UI Name *
Designer 1 (Novice/ Inexperienced)	- 4th-year undergraduate student in computer science, University A.	- None	- Usability design framework (PIN) - Google Maps - Prott	N-F
Designer 2 (Expert/ Experienced)	- 3 years experience in web development. - 2 years experience in user interface design of government agency’s mobile application.	- Graphic design, lettering design, illustration, and HCI	- Usability design framework (PIN) - MockFlow - Google API	E-F
Designer 3 (Expert/ Experienced)	- 10 years experience in website design and development. - 5 years experience in mobile application development.	- Graphic design	- FluidUI - Google API	E-nF
Designer 4 (Novice/ Inexperienced)	- 4th-year undergraduate student in computer science, University A	- None	- Pencil - Google Maps	N-nF

* Note: N = Novice, E = Expert, F = Framework, nF = no Framework.

The four redesigned screens of the earthquake report interface are shown in Figure 7.

Figure 7. The redesigned user interfaces: (a) N-F by designer 1; (b) E-F by designer 2; (c) E-nF by designer 3; (d) N-nF by designer 4.

5. Evaluation: Results and Discussion

The evaluation was divided into three parts: process, usefulness, and outcome. Table 10 shows the questions and answers for process evaluation. Table 11 shows time taken to complete redesign activities when PIN framework was used. The framework was used by the four designers to redesign the earthquake report screen of EarthquakeTMD. Two of them (one inexperienced and one experienced) used the framework to guide the redesign process. The other two designers (one inexperienced and one experienced) received no treatment and redesigned the user interface without using the framework. For outcome evaluation, the four redesigned user interfaces were tested for usability. Then, users completed a 10-item SUS questionnaire at the end of the test.

Table 10. The questions for the designers who used PIN framework.

Questions	Inexperienced	Experienced
1. PIN framework suggests a sequence of steps suitable for designing the user interfaces for the mobile government application.	Strongly agree (5)	Strongly agree (5)
2. PIN framework leads to a usable user interface of the mobile government application.	Strongly agree (5)	Strongly agree (5)
3. The usability design factors from PIN framework help to design a usable user interface of the mobile government application.	Extremely helpful (5)	Extremely helpful (5)
4. The usability factors enable inexperienced designers to think about the characteristics of the user interface of the mobile government application and what to do to achieve them. (This question for inexperienced designer.)	Strongly agree (5)	-
5. The usability design factors lead to a usable user interface of the mobile government application even though the designers are inexperienced. (This question for inexperienced designers.)	Strongly agree (5)
6. The usability design factors from PIN framework help experienced designers to create a better and usable design. (This question for experienced designer.)	-	Strongly helpful (5)
7. The usability design factors should be used in user interface designing of mobile government applications in the future.	Strongly require (5)	Strongly require (5)
8. The Preparation of Interaction (PI) process is not complicated and can be easily followed.	The NLP method used in the framework is not complicated but should be understood prior to use. The transformation of elements requires practices and skills to use with confidence.	NLP is good for supporting requirement analysis and eliciting missing or additional requirements.
9. The process of identifying the appropriate usability design factors is easy to follow.	There is a step-by-step focus on the process, making it easy to understand.	The framework comes with a workflow that provide a step-by-step guide. The design goals can be traced. The factor index is easy to understand.
10. The Necessary quality element identification process is essential to designing high-quality user interface elements.	The process is easy to follow, but comprehensiveness could be improved. It could be better with help from other team members. Due to lack of experience, it is difficult to foresee the importance of each interface element and the related design that should be used. This step helps a lot.	The design framework serves as a guideline that makes it easy to understand. It helps setting a clear focus on what the design should be for the interface elements.

Table 11. Time taken to complete redesign activities when PIN framework was used.

Designer	Process	Activity	Time Spent	Design Methods Used
Inexperienced	Necessary quality element identification process Design	Understand the design details and design factors. Design and revise user interface	5 h 8 h	- -
Experienced	Necessary quality element identification process Design	Understand the design details and design factors. Design and revise user interface	2 h 4 h	- Consistency and standards - Aesthetic and Minimalist Design - Theory of Colors

5.1. Process Evaluation

The process evaluation was performed by interviewing the designers who used the PIN framework. The questions were based on the design sequence suggested by the framework. Both closed-ended and open-ended questions were used. For closed-ended questions, a 5-point Likert scale was used. The interview was conducted via an online meeting application to follow Thailand’s COVID-19 government measures. The result is shown in Table 10. Overall, the framework provided both experienced and inexperienced designers with several benefits: first, the step-by-step usability design guideline, and second, desirable outcomes. Though there is a learning curve in some steps, the designers agree that the framework helps a lot.

Table 11 shows the durations of activities performed as part of the redesign process guided by the framework. Also shown are the design methods/principles used. The inexperienced designer spent more time, 200% on the element identification activity and 250% on the design activity, than the experienced designer. This could suggest that the framework, though it provides the guideline for the usability design process, does not necessarily speed up the design process of inexperienced designers. Experience is still an important factor that affects the duration of the process. Undeniably, the framework makes it easier for them to move forward with the usability design process while keeping track of what to do and what is done.

5.2. Usefulness Evaluation of PIN Usability Design Framework

A set of questions was given to both designers, experienced and inexperienced, who used the PIN usability design framework. The responses were based on a Likert scale that ranges from 1 to 5. The questions were asked via an online meeting application due to Thailand’s COVID-19 travel restrictions. The result of the evaluation is shown in Table 12. Both inexperienced and experienced designers found the PIN framework useful in supporting their design processes. When compared with other similar usability frameworks, the designers agree that the PIN framework offered a higher level of facilitation and overall support. The time spent on designing the interface was much shorter, unlike other similar frameworks. Table 12 shows the questions and answers for usefulness evaluation.

Table 12. The result of the evaluation to validate the usefulness of PIN framework.

Questions	Novice	Expert
1. PIN framework suggests a step-by-step process that supports a systematic design of usability.	Strongly agree (5)	Strongly agree (5)
2. PIN framework supports the design process even for designers with no prior design experience. (This question for inexperienced designer.)	Extremely helpful (5)	-
3. The level of facilitation the following usability frameworks/models provide for a user interface design process of the mobile government applications. Rate 1–5 (strongly facilitates, facilitate, somewhat facilitate, inadequate, strongly inadequate)
(1) Usability design framework (PIN)	(1) Strongly facilitate (5)	(1) Strongly facilitate (5)
(2) mGQM	(2) Somewhat facilitate (3)	(2) Somewhat facilitate (3)
(3) PACMAD	(3) Somewhat facilitate (3)	(3) Somewhat facilitate (3)
4. Reducing the time it takes to perform the process from the requirement analysis to the enactment of design using the following usability frameworks/models. Rate 1–5 (extremely, very, moderately, slightly, not at all)
(1) Usability design framework (PIN)	(1) Extremely (5)	(1) Extremely (5)
(2) mGQM	(2) Moderately (3)	(2) Moderately (3)
(3) PACMAD	(3) Moderately (3)	(3) Moderately (3)
5. The level of the overall support the following usability frameworks/models provide to user interface designing of mobile government application. Rate 1–5 (strongly support, support, moderately support, little support, no support)
(1) Usability design framework (PIN)	(1) Strongly support (5)	(1) Strongly support (5)
(2) mGQM	(2) Moderately support (3)	(2) support (4)
(3) PACMAD	(3) Moderately support (3)	(3) Moderately support (3)

5.3. Usability Evaluation of the Four Redesigned User Interfaces

The four redesigned user interfaces were evaluated by two sample groups. The first group, 24 participants, took part in a usability test. Twenty-four test cases were designed to cover all possible sequences of using the four user interfaces. The participants were separated into four groups. Each group was divided into three subgroups (student, working age, and retired), with two participants in each group. They used the interface via mobile phones.

The second group, 400 participants (participants with a maximum allowable error of 0.05), viewed the mockup interface via the website. A questionnaire based on SUS questions was administered to both groups. Example questions include “I think that I would like to use this system frequently” and “I thought the system was easy to use”. The vertically aligned scores indicated that some records were eliminated due to inconsistent scoring. After cleaning, some of the records were removed, and data of 351 participants were left.

• Usability testing via mobile phone: Every possible order of the redesigned user interfaces was used by one participant. Thus, twenty-four test cases were derived. Twenty-four participants were recruited. As there were four interfaces, the test cases were divided into four groups. There were six test cases in each group. Considering that age influences user preferences, each group was divided further into three subgroups (student, adult, and elderly) with two test cases per group. The participants completed the usability test of the four user interfaces in different orders. Table 13 shows participants’ demographic information and SUS scores. Table 14 shows the summary of SUS scores.
  Table 13. Demographic information of 24 participants and SUS scores (ordered by age).

  User	Occupation	Gender	Age	Education	M-App Exp. (Years)	M-Gov Exp. (Years)	SUS Score *				Favorite UI
  							N-F	E-F	E-nF	N-nF
  U4	Student	M	19	B	≥6	1–3	62.5	82.5	62.5	47.5	E-F
  U7	Student	F	19	B	≥6	1–3	100.0	100.0	72.5	72.5	E-F
  U14	Student	M	19	B	≥6	1–3	67.5	80.0	62.5	62.5	E-F
  U18	Student	F	19	B	≥6	1–3	72.5	80.0	62.5	52.5	E-F
  U3	Student	M	20	B	≥6	1–3	65.0	82.5	62.5	52.5	E-F
  U11	Student	M	20	B	≥6	1–3	90.0	92.5	60.0	45.0	E-F
  U20	Student	F	21	B	≥6	1–3	97.5	97.5	62.5	85.0	N-nF
  U1	Employee	F	22	B	≥6	1–3	100.0	100.0	75.0	67.5	E-F
  U23	Student	M	22	B	≥6	1–3	90.0	92.5	62.5	55.0	E-F
  U13	Caterer	F	33	SS	4–5	No	92.5	92.5	57.5	57.5	E-F
  U2	Lecturer	M	35	M	≥6	1–3	100.0	100.0	70.0	72.5	E-F
  U9	Lecturer	F	37	M	6	6	57.5	80.0	62.5	52.5	E-F
  U8	Construction Management	M	38	M	≥6	4–5	100.0	92.5	95.0	60.0	N-F
  U15	Bank manager	F	40	B	≥6	1–3	92.5	100.0	57.5	25.0	E-F
  U21	Coffee shop owner	F	40	M	≥6	1–3	100.0	100.0	100.0	75.0	E-nF
  U19	Lecturer	F	46	M	≥6	1–3	100.0	100.0	70.0	72.5	N-F
  U10	Housewife	F	51	HVC	≥6	4–5	100.0	100.0	75.0	65.0	E-nF
  U6	Housewife	F	65	B	4–5	No	95.0	82.5	42.5	32.5	N-F
  U12	Retired teacher	F	65	B	≥6	1–3	70.0	80.0	57.5	45.0	E-F
  U24	Retired teacher	M	65	B	4–5	1–3	80.0	80.0	55.0	45.0	E-F
  U5	Retired teacher	F	66	B	≥6	1–3	95.0	82.5	47.5	45.0	N-F
  U16	Retired teacher	F	67	B	≥6	1–3	85.0	87.5	50.0	35.0	E-F
  U17	SAO Management	M	70	M	≥6	1–3	95.0	82.5	47.5	45.0	N-F
  U22	Retired teacher	F	71	B	≥6	No	95.0	82.5	47.5	45.0	N-F

  * N-F: No experience, Use framework; E-F: Experience, Use framework; E-nF: Experience, No framework; N-nF: No experience, No framework; B: Bachelor’s; M: Master’s; SS: Secondary School; HVC: High Voc. Cert.
  Table 14. The SUS scores from the 24 test cases.

  User Interface	SUS Score	Meaning
  N-F	87.60	Excellent
  E-F	89.58	Excellent
  E-nF	63.23	Poor
  N-nF	54.27	Poor

Table 14 shows that the most user-friendly and usable design is the E-F interface, chosen by 15 participants. The N-F interface follows closely, chosen by six participants. The third, E-nF, and fourth, N-nF, were chosen by two and one participant, respectively.

• Usability testing via the website: A total of 351 participants took part in the usability testing with random 10 test cases. Table 15, Table 16, Table 17 and Table 18 present users’ demographic information. The sizes of male and female populations are similar. Most of the population was aged 21–50, while 20% of the population were teenagers and elderly. Table 19 shows the SUS scores. Almost everyone had 6 years or more of experience in using mobile applications, while 76.64% of the population had 1–3 years of experience in using mobile government applications.
  Table 15. Gender.

  Gender	% (Number of Participants)
  Male (M)	48.43% (170)
  Female (F)	51.57% (181)

  Table 16. Age range.

  Age Range	% (Number of Participants)
  15–20	13.11% (46)
  21–30	23.93% (84)
  31–40	27.35% (96)
  41–50	29.06% (102)
  51 and more	6.55% (23)

  Table 17. Experience with mobile application information.

  Experience (Year)	% (Number of Participants)
  Never	-
  1–3	0.28% (1)
  4–6	1.71% (6)
  6 or more	98% (344)

  Table 18. Experience with mobile government application information.

  Experience (Year)	% (Number of Participants)
  Never	6.84% (24)
  1–3	76.64% (269)
  4–6	14.25% (50)
  6 or more	2.28% (8)

  Table 19. SUS scores after completing the ten random test cases.

  User Interface	SUS Score	Meaning
  N-F	89.10	Excellent
  E-F	90.88	Excellent
  E-nF	59.34	OK
  N-nF	46.53	Poor

The result from Table 19 shows that the most user-friendly interface was E-F, chosen by 281 participants. In second place was N-F, chosen by 59 participants. In third place, E-nF was chosen by 11 participants. There was no fourth place as N-nF was not selected.

5.4. Discussion

Table 13 reveals interesting findings. The average values of SUS scores of the interfaces redesigned with the framework are higher than the interfaces redesigned without the framework. From Table 14, the average SUS scores of inexperienced and experienced designers who used the framework are slightly different. The scores of inexperienced and experienced designers when there was no framework are quite different. Furthermore, the SUS scores of the interface redesigned by the inexperienced designer who used the framework were higher than the one redesigned by the experienced designer who did not use the framework in almost every case. Based on the findings, it can be implied that the framework resulted in more usable interfaces. In addition, the framework lessened the difference between inexperienced and experienced designers. Both groups could produce desirable outcomes with the framework. When the framework was not present, the designer’s skills and experience played an important role in producing a better outcome.

For the cases involving the framework, most participants favored the design by the experienced designer. There were four cases that did not follow. Three cases were from the elderly group. The reason could be that the elderly group preferred a simpler design of the interfaces redesigned by the inexperienced designer. The experienced designer might add too much detail to the redesigned interface that the elderly group did not prefer.

The outcomes of this research were analyzed and used to answer the following research questions (RQs) mentioned in Section 3.

RQ 1:

What are the usability factors that are important to the user interface design of mobile government applications?

Answer: The usability design factors found in the previous study are learnability, simplicity, satisfaction, security, privacy, and memorability. These are defined as the Main Usability Factors (MUFs). Each can be elaborated on further as Sub Usability Factors (SUFs). The structure of the usability factors described in Section 3.2 and Section 3.1.2 can be used by experienced and inexperienced designers as a design guideline to achieve usable user interface designs.

RQ 2:

How can usability factors support designing a usable user interface?

Answer: It helps set the design focus. After associating usability factors to user interface components, Criteria to achieve such factors were identified. Criteria signify specific characteristics of the UI components, thus allowing the designers to effectively design the UI components. This reflects a bottom-up design approach that supports detailed design which leads to a usable user interface. When all suitable factors required to engineer usability are designed into the user interface via MUFs, SUFs, and Criteria, the overall usability of the application can be realized. The mobile government application, EarthquakeTMD, in this paper, shows the improvement in usability scores after the framework was used to support the design process (see Section 5.3).

RQ 3:

Do the existing usability frameworks/models offer usability design guidelines to the designers of mobile government applications?

Answer: The PIN usability design framework focuses on designing the usability of the user interface of mobile government applications in which specific usability factors must be achieved for the user interface to hold the expected usability characteristics. The framework contains the Preparing for Interaction (PI) process for identification of the user interface components. The other existing models and frameworks did not include a similar process (see Section 3.1.1) that deals with the identification of the UI components and their detailed characteristics. The PIN usability design framework also assists the designers in the identification of the usability design factors which act as a design guideline. (See Section 5.1 and Section 5.2.) Table 20 compares PIN and other related frameworks/models.

Table 20. A comparison of usability models for mobile government application.

Model	Usability Design Factor	Mobile Application	Design	Guideline
PIN	Learnability, Simplicity, Satisfaction, Security, Privacy, Memorability	Government	UI and Usability	Yes
mGQM	Simplicity, Accuracy, Time taken, Features, Safety, Attractiveness	General	General	Yes
PACMAD	Effectiveness, Efficiency, Satisfaction, Learnability, Memorability, Errors, and Cognitive load	General	General	Yes
MGOV	Unspecified	Government	General	Yes
M-GSEF	Unspecified	Government	General	Yes

RQ 4:

How to identify suitable usability factors for the user interface components?

Answer: NLP is used to identify information entities from the requirements. The entities are used to identify the user interface components during the PI process. Once done, the associated usability factors can be identified for each component. Criteria, which represent the specific detail of SUFs (Sub Usability Factors) establishes the detailed design of the UI components (see Section 3.2).

The support for usability design provided by the PIN usability design framework is shown in Figure 8 along with other usability-design-related frameworks and their relations to the SDLC. It can be seen that PIN focuses on designing the user interface screens starting from defining, designing, and developing to testing phases.

Figure 8. PIN usability design framework, other usability-design-related framework, and SDLC.

6. Conclusions and Future Work

This paper presents the case study of using the PIN usability design framework to redesign the earthquake report screen of EarthquakeTMD, the mobile government application to report earthquake information from the Thai government agency. The original interface was redesigned by four designers: two (experienced and inexperienced) designers used the framework, and the other two (experienced and inexperienced) did not use the framework. The four redesigned user interfaces were evaluated by 400 citizens. The results reveal that the citizens favored the user interfaces which were redesigned using the framework. The framework minimized the differences in the experience of the designers who participated in the study. This finding supports the idea of the framework to help with the usability design process by both inexperienced and experienced designers. Without the framework, it was found that the usability score was lower than the ones produced with the PIN framework, even by the experienced designer.

The designers were asked about the use of usability factors in the design process and the outcomes. Both experienced and inexperienced designers agreed that the framework provides a step-by-step process that positively affects the work done during the design and the outcomes of the designs.

The analysis of the research outcomes against the purpose of the framework indicates room for improvement. In addition, the research methodology and the research questions, were reviewed. Several issues were recognized and investigated. Accordingly, improvement areas were proposed. First, identification of the usability factors is an essential step to provide a constructive foundation for usability design. This inception step can provide valuable information that can be used as input for subsequent activities. One way to accomplish this is to have the associated requirements processed. Once the desired outcomes are obtained, the designers can approve or enhance them. Second, transforming usability factors into implementation criteria is the step that attempts to describe characteristics of the usability factors. To ensure the consistency of the transformation, the criteria data and their relations to user interface components should be collected and kept in a database that can be enhanced over time. Thus, promoting the reliability of the transformation. Third, more designers with differences in background, experience, development platform, application domain, and design methodology should be recruited to use the framework. Moreover, the research methodology should be modified to incorporate different types of m-government applications and mobile applications to demonstrate the applicability of the framework. While, in this paper, the framework was used in redesigning an existing m-government application’s user interface, it could also be used to design the user interface of a new m-government application. It would be interesting to know if the framework can provide equivalent benefits to the new interface’s design process.

Furthermore, automation of activities in the framework can ease the design process and increase efficiency. For example, automation of requirement analysis and UI component identification can speed up the design process and provide better support to the designers, especially inexperienced designers. Figure 9 shows the mockup screens for the automatic analysis of the requirements. The potential areas that can be automated include clustering of UI components, factors analysis, and a detailed design report.

Figure 9. (a) A mockup screen for automatic analysis of the requirements. (b) A mockup screen of automatic identification of UI components.