Effects of Asymmetry between Design Models and User Models on Subjective Comprehension of User Interface

Abstract

This study attempted to discuss the design principles for enhancing usability in terms of asymmetry of mental models between users and designers. If the user model is asymmetrical to the design model, i.e., the degree of agreement between models is low and the user’s mental model is not compatible with the actual system, the user cannot operate the system properly, which may cause errors. The aim of this study was to investigate the characteristics of users who have asymmetrical mental models and identify what such users did not understand. Twenty-four subjects participated in an experiment that involved a digital camera operation task and mental model tests that measure the level of the agreement of the mental model construction in terms of functional and structural models. Initially, the participants were grouped based on mental model test scores: symmetrical mental model group (n = 17) and asymmetrical mental model group (n = 7). Then, the groups were compared in terms of performance and subjective comprehension. The comparison indicated that the symmetrical mental model group performed more quickly and accurately than the asymmetrical group. The results also confirmed that the asymmetrical mental model group had a lower level of comprehension in terms of understanding the device status, detecting and responding to device status changes, and understanding the hierarchical structure of the screen.

1. Introduction

In considering usability improvement and error prevention of user interfaces (UIs), it is essential to consider the user’s mental model. Although many definitions of mental models have been proposed [1], in this study, a mental model is defined as a mental representation that a user forms to facilitate operating a user interface [2,3]. This study is based on the notion that mental models predict knowledge and explain each user interface object’s behavior and relationships between the user’s operation and the object’s behavior. This indicates that mental models do not necessarily include the meaning of any working principle.

Mental models are mental representations in an individual user’s mind; thus, they can be said to be arbitrary and subjective models that include user assumptions. According to Kaiho et al. [4], Norman [5], and Carley and Palmquest [6], mental models have the following characteristics. All users have their own mental models. Mental models are unstable, i.e., they can change constantly. For example, considering an appliance’s operation, users may misunderstand how the appliance functions by forming inappropriate models when they initially use the appliance. However, users can modify their mental models to match an existing system by learning the system’s operation through iterative trials. Moreover, it is said that mental models have universality [4,6], and many users have a standard universal mental model. For example, the operating method of popular devices (e.g., a mouse and/or a keyboard attached to a personal computer) and/or widely accepted conventional wisdom could be considered universal models.

The model proposed by Norman [7] can be used as a reference to consider usability and error prevention in device operation from the perspective of mental models. That model comprises three aspects, i.e., user model, design model, and system image. The user model is formed by a user through interaction with a system. The design model is developed by the system designer, and the system image is the working principle of an actual system. The interface between a system and a user is based on the design model; thus, the symmetry between the user model and the design model is important to improve usability and prevent errors. When there is symmetry between the user model and the design model, i.e., the degree of agreement between models is high, users can operate the system following their individual user model and achieve high performance. However, if the user model is asymmetrical to the design model, the user cannot operate the system correctly, which may cause errors. The relationship between the design model and the user model is summarized in Figure 1.

Considering the usability of a product from the perspective of mental models as described in the above paragraph, it is important to consider the asymmetry between the user model and the design model. This study focuses on the asymmetry of mental models as an insight to improve usability in user interface operations. This study aimed to investigate the characteristics of users who build mental models that are asymmetric to the design model. In this study, the symmetrical mental model is defined as a user model that is the same or similar to the design model (a small gap between a design model and a user model). The asymmetrical mental model is defined as a user model that is less similar to the design model (a large gap between a design model and a user model). In order to examine the similarity or gap between mental models, the author used the concepts of functional model and structural model [8,9]. The functional model is a model that represents how to use a system, such as operating procedures and functions, and the structural model is a model that represents how a system works, such as operating principles and screen hierarchy. From the perspective of these two models, this study examined the asymmetry between the design model and the user model by considering how the user model matches the actual system’s operation procedures and hierarchical screen structure.

As Johnson-Laird [10] stated, mental models describe and understand the human cognitive process for solving deductive reasoning problems, mental models are an internal model for explaining external reality. Mental models are necessary for users to understand the behavior of a system properly [11,12]. By having an appropriate mental model for users, the users can facilitate their understanding of the system and perform more accurate operations [13]. However, there are various ways of representing mental models. Jonassen et al. [14] and Rasmussen [15] have examined the types of knowledge explained by mental models, respectively. Mayer et al. [16,17,18] discussed the effects of visual and verbal information on users’ construction of mental models. Mayer et al. [16,17,18] conducted a comparative experiment using different ways of giving visual and verbal information and reported that presenting visual and verbal information in succession was effective. Although there are various ways to represent mental models, mental models are considered important for users’ problem-solving in all of the above studies.

An appropriate mental model for equipment operation will help the user understand the system and operate it correctly. Kieras and Bovair [19] identified the role of mental models in learning a new system. They showed that participants who have formed an appropriate mental model of an experimental system benefited from a shorter learning period and were able to manipulate the system more efficiently. Kellogg et al. [20] also studied software usability from the perspective of the degree of agreement between the developer’s system model and the user’s mental model. They asked expert and novice users to sort and group cards with system commands on them and compared the difference between the grouped cards and groups of actual system commands. The experts’ mental model was considered closer to the system model than the novice’s mental model. Furthermore, they reported that system usability could be improved by matching the system model to the novices’ mental model. Although the target systems of Kieras and Bovair [19] and Kellogg [20] are different, both studies suggested that having mental models that fit the actual system (symmetric mental models) leads to improved operational performance. There have been studies of mental models for user interfaces of various devices (e.g., digital camera [2,21,22], multi-functional controller [3], calculator [11], online catalog [12], fictional electrical device [19,23], software [20]). Besides, since cognitive processes during user interface operations can be explained typically to some extent, such as the seven stages of action [24], the importance of mental models can be generalized to some extent to user interface operations in general.

Moreover, the mental model perspective focuses on the design and usability evaluation of various systems and products. Lei et al. [25] conducted a usability evaluation of a cell phone and designed its UI. In this study, they investigated the effects of the information layout and the type of icons on mental models from the reaction time. The results showed that the layout of information influences mental models and UI design based on users’ mental models is necessary to improve usability. Yamashita et al. [26] studied the usability gap when users changed from a cell phone familiar with the operation to a cell phone that is not. The users of Sony and Nokia cell phones were asked to perform an operation task for each cell phone. The results showed that users familiar with the operation of the Nokia phone understood the new phone more quickly than those who were not. Utilizing hierarchical task analysis, they reported that one of the reasons was their mental models. Masood and Thigambaram [27] reported a usability test of a mobile educational application for children. The evaluated UI was based on adult mental models and did not match the mental models of the actual users, children. Forster et al. [28] conducted a user test for user education for an automated driving system and reported the importance of building appropriate mental models for the automated driving system. They also proposed a method of education that uses both manuals and tutorials to support users’ mental models.

As described above, the importance of a mental model adapted to the device has been shown; however, few previous studies have focused on general design principles about mental models. In many studies, usability is evaluated and improved by conducting user studies from the viewpoint of mental models, but these studies aim to improve the usability of specific systems. These studies focused on identifying usability problems that did not match users’ mental models and did not aim to examine general design principles from the viewpoint of mental models. In examining design principles, it is important to consider both the fitness of a system image with the existing mental models of general users and the cognitive characteristics of users when they construct new mental models in the initial use of a device. Especially, the cognitive characteristics of users should be considered when they construct new mental models via the interaction. Although there are individual differences in the ability of users to construct mental models [5,29], the variation among users has not been explored thoroughly. It is also important to consider how to support users who are not good at constructing mental models.

To clarify this point, it is necessary to identify what users who construct mental models that are asymmetrical to the design model do not understand. Doi [21] and Yamaoka et al. [30] compared user utterances during device operation with different degrees of mental model construction using the protocol analysis. Doi et al. [21] discussed the differences in the formation of mental models among participants with high and low comprehension about the function of a digital camera was investigated. The participants were divided into two groups based on their measured comprehension regarding the level of functional model formation of the digital camera. They compared the verbal protocols obtained during the digital camera operation tasks between both groups. The results reveal differences among participants with different levels of mental models. In particular, concept formation based on an understanding of the situation might be an important factor for enhancing the formation of mental models. Moreover, the users that could form mental models appeared to have better planning ability.

Yamaoka et al. [30] proposed nine factors related to mental model building based on the differences in utterances during device operation between users with different degrees of mental model building and examined the design guidelines related to these factors. The results of these studies are important for applying the perspective of mental models to UI design. However, these studies all involve researchers’ subjective categorization of user utterances during operation; the users’ self-evaluations are unknown. To support and strengthen the findings of these previous studies, investigating the factors that prevent the construction of mental models from the viewpoint of users’ subjective understanding is important. By considering these factors when designing UIs, the gap between users and designers could be minimized to enhance mental model building. These findings could be used as a design guideline to enhance usability.

This study aimed to compare users who were able to construct a mental model symmetrical to the design model and users who constructed an asymmetrical mental model in the initial use of a device. In addition, this study investigated the characteristics of asymmetrical mental model users in terms of their understanding of the equipment operation. In other words, the author aimed to investigate the reasons that lead to asymmetrical mental models, i.e., what aspects of the target system are difficult to understand. Thus, the author investigated the differences in participants’ subjective comprehension with different levels of mental model construction. The 17-item questionnaire proposed by Doi et al. [23] was used to investigate the subjective level of comprehension. By investigating whether each item’s evaluation values differed between groups, the author tried to reveal what users with asymmetrical mental models did not understand.

This manuscript is composed of the following six chapters. In Section 1, the importance of mental models in user interface operation and related research are presented. Additionally, it was described in this chapter to investigate the subjective comprehension of UI operations of users who construct asymmetrical mental models as one of the findings to examine how to support users who are not good at constructing mental models. For this purpose, the experiment was conducted to compare users who built symmetrical mental models and users who built asymmetrical mental models for digital camera operations. Section 2 describes the method of this experiment, including the tasks, the method of grouping participants, and the procedure of analysis. In Section 3, the experiment results are presented, including the results of the grouping participants and the statistical analysis to compare the performance and the subjective comprehension of the digital camera operation task between the groups. Section 4 discusses the results presented in Section 3 and describes the characteristics of the asymmetrical mental model group. Moreover, the implications of design for the findings of this study are described. Section 5 shows the limitations of this study and what should be done in future work. Finally, Section 6 summarizes the findings of this study.

2. Methods

2.1. Participants

The participants were 24 students (16 males, 8 females, average age 23.2 years, SD = 1.1). All participants had 20/20 vision, either naturally or with vision correction. All participants provided informed consent after receiving a brief explanation of the experiment’s aim and content. Participants were orally asked about the frequency of use and experience with a digital camera. The results were almost similar for all participants. This study was not gender-balanced because there are no significant differences between participants’ gender.

2.2. Task

2.2.1. Digital Camera Operation

The participants were required to perform four tasks using a digital camera (Ricoh CX3). Because this study aims to contribute to the GUI design of typical personal appliances and investigate the characteristics of asymmetrical mental model users in the initial use of a device, a typical digital camera that none of the participants had previously encountered was used in the experiment. Each task consisted of five to seven operational steps (e.g., “Change the ISO setting to 100”). The steps of each task were presented to the participant in a written document. To complete the task, the participant was required to operate the buttons on the digital camera. The task was considered complete when the participant judged and reported verbally that all the steps had been completed. The task performance was evaluated by measuring the task completion time and the accuracy of the operation. The task completion time was defined as the time from task presentation to task completion. For the accuracy of operation, the participant’s behavior in each step of the operation task (5 to 7 steps per task) was scored from zero to four as follows.

• 4: Correct operation
• 3: Slightly unsmooth operation (hands stop for a moment, momentary confusion, trivial operation error)
• 2: Unsmooth operation (hands stop for a long time)
• 1: Operation error (perform the same operation multiple times, operate blindly without understanding)
• 0: Not accomplished

2.2.2. Mental Model Tests

After completing the digital camera operation, two tests to confirm the degree of agreement between the actual system image based on the designer’s mental model and the participant’s mental model were performed. In this study, the degree of agreement was measured based on functional and structural mental models. These models have different characteristics [8,9]. The functional model relates to the comprehension of a context, function, and procedure to assume “how to use a device?”. The structural model relates to the comprehension of a structure and a working principle to assume “how a device works?”. Mental model tests from the perspective of functional and structural models have been used in previous studies and found to be valid [21,22,23]. Therefore, the author adopted these tests to measure the level of mental model construction. This study adopted the following two tests conducted by Doi et al. [23] and Ishihara et al. [22] to predict the level of both functional and structural models: (1) a test to measure the comprehension of the task procedure (functional model), and (2) a test to measure the comprehension of the hierarchical structure of the screen (structural model).

For the functional model test (1), participants are required to remember the procedure of each task by verbally recalling the names of the buttons and menu items to be pressed in sequence. The task performance was scored as follows: two points were deducted for each incorrect button or menu item name, and one point was deducted for each incorrect term or ambiguous explanation (a perfect score is 36 points for the four tasks). For the structural model test (2), the experimenter presented cards with each button’s name and asked the participants to recall the hierarchical structure by positioning the cards in parallel or lower/upper hierarchical locations to show the hierarchical structure of the screen. The participants were asked to create a hierarchical structure related to each task of four tasks. Here performance was scored as follows: two points were deducted for each error in the hierarchical structure, and one point was deducted for each error in an item in the hierarchy (a perfect score is 68 points for the four tasks).

Note that the upper limits of the scores (perfect scores) of the two tests (the functional model test: 36 points, the structural model test: 68 points) are different because the perfect score of the functional model test is determined by the total number of the operating procedures and functions used in the digital camera operation task and the perfect score of the structural model test is determined by the total number of the items of all screen hierarchies used in the digital camera operation task. Because each score is an independent measure, and the two measures are not directly compared or integrated, the difference of the perfect scores does not affect the validity and the results of this study.

2.3. Questionnaire

To evaluate users’ subjective comprehension of the digital camera operation, the 17-item questionnaire proposed by Doi et al. [23] was administered (

Table 1. Questionnaire to measure subjective comprehension of a user interface.

Q1	Do you understand the terms displayed on the device during its operation?
Q2	Do you understand the means of contents indicated on the device and its screen?
Q3	Do you understand the status of the device during operation?
Q4	Do you understand what you should do to achieve the operational goal?
Q5	Do you understand the operation procedure during operation?
Q6	Do you use your existing knowledge to operate the device?
Q7	Do you understand the functions of the device from the labels and/or names of the parts of the device?
Q8	Do you understand how to use the functions of the device?
Q9	Do you predict the state of the device after the operation before operating it?
Q10	Do you assume how the device will behave?
Q11	Do you operate the device based on your assumption about its functions?
Q12	Do you understand the hierarchical structure of the device screen?
Q13	Do you understand the relationship among the parts of the device?
Q14	Do you understand the feedback from the device?
Q15	Do you detect any changes in the status of the device during operation?
Q16	Do you cope with any changes in the status of the device appropriately?
Q17	Do you confirm whether the operation was correct after the operation?

). A five-point Likert scale was used (5: Strongly agree, 4: Agree, 3: Neither agree nor disagree, 2: Disagree, 1: Not at all agree). This questionnaire, which was developed based on factors related to building mental models through interaction between users and a user interface [23], was designed to measure the level of mental model construction subjectively. According to previous studies [3,31], the items in the questionnaire are associated with mental model construction; therefore, the author judged that the questionnaire was appropriate to provide data about the subjective comprehension of digital camera operation that was sufficient to analyze the relationship between subjective comprehension and the level of the mental model construction.

2.4. Procedure

First, each participant was required to adjust his or her seat so that they could operate the camera comfortably. The experiment began only after the task was explained and the experimenter confirmed that the participants understood the tasks. For the digital camera operation task, the participants were instructed to perform the tasks as quickly and accurately as possible. The order of the four tasks was randomized for each participant. The questionnaire to measure the subjective comprehension was administered after all experimental tasks were completed. Then, the mental model tests were administered. For the functional model test, the participants were required to explain orally the name of the buttons and/or menu items used to achieve each task in sequence. Subsequently, the experimenter provided an example to explain the card sorting process, and the participants were required to do the card sorting, i.e., the structural model test). The experiment took approximately 60 min in total. There was no interval between each task and the subsequent question unless the participant asked the experimenter to give them a break.

2.5. Analysis

The author analyzed the characteristics and factors of the participants who were able to construct a mental model compatible with the existing system by comparing them to those who could not construct a mental model. First, the participants were divided into two groups based on the results of the mental model tests from the perspective of functional and structural models. Cluster analysis was used to group the participants. Since there is no objective standard for the mental model tests, a t-test was used to confirm whether there was a significant difference in the functional and structural model test scores between the two groups.

Then, to verify whether the mental model adapted to the existing system affects operational performance, the task completion times and accuracy scores of the two groups were compared using a t-test. In addition, to clarify what users who could not construct an appropriate mental model did not understand, the subjective comprehension scores of the two groups were compared using a t-test.

3. Results

3.1. Grouping Participants Based on Mental Model Tests Scores

The participants were divided into two groups based on the results of the mental model tests, and Ward’s cluster analysis was performed using the functional and structural model scores. The grouping results are listed in

Table 2. Results of grouping participants by mental model test scores.

Group	n	Functional Model Score		Structural Model Score
		Mean	SD	Mean	SD
S-MM group	17	28.3	4.1	43.9	11.9
A-MM group	7	22.7	3.6	16.0	9.3

. To confirm the difference in the degree of mental model construction between the two groups, the functional model and structural model scores were compared using a t-test, and significant differences were found in both cases (functional model score: |t(22)| = 3.14, p = 0.005, structural model score: |t(22)| = 5.54, p < 0.001). In this study, the groups with high and low degrees of mental model construction are referred to as symmetrical mental model (S-MM) group and asymmetrical mental model (A-MM) group, respectively because a high score in the mental model tests means that the participant can fit their mental model to the actual system model.

3.2. Digital Camera Operation Performance

The task completion time and accuracy scores of each group are shown in Figure 2. t-test results confirmed the differences in task completion time and accuracy scores between the two groups. The results show that there was a significant difference in both task completion time and accuracy score of the operation (task completion time: |t(22)| = 2.81, p = 0.01, accuracy score: |t(22)| = 2.66, p = 0.02). The S-MM group showed a shorter task completion time and higher accuracy scores than the A-MM group.

3.3. Subjective Comprehension

The t-test results for subjective comprehension are shown in

Table 3. The t-test results for comparative subjective comprehension between S-MM and A-MM groups.

	t		p	Difference in Mean	SE of Difference
Q1	−0.81		0.43	−0.34	0.41
Q2	−2.04		0.05	−0.71	0.35
Q3	−2.79	*	0.01	−0.92	0.33
Q4	−1.57		0.13	−0.40	0.26
Q5	−1.13		0.27	−0.45	0.40
Q6	0.67		0.51	0.23	0.34
Q7	−0.36		0.72	−0.13	0.37
Q8	−1.95		0.06	−0.68	0.35
Q9	−1.37		0.18	−0.56	0.41
Q10	−1.03		0.31	−0.36	0.35
Q11	−1.51		0.15	−0.51	0.34
Q12	−2.46	*	0.02	−1.00	0.41
Q13	−1.45		0.16	−0.56	0.39
Q14	−1.60		0.12	−0.60	0.37
Q15	−4.36	**	0.00	−1.43	0.33
Q16	−2.21	*	0.04	−0.76	0.35
Q17	−1.33		0.20	−0.49	0.37

**: p < 0.01, *: p < 0.05.

. The results of the t-test showed significant differences in the following four items: Q3, Q12, Q15, and Q16. In each of these four items, the S-MM group scored significantly higher than the A-MM group.

4. Discussion

In this study, the participants were divided into two groups, S-MM and A-MM, based on the functional and structural model scores. To confirm the validity of the grouping, the functional and structural model scores between the two groups were compared. t-test results showed that both the functional and structural model scores were significantly higher in the S-MM group compared to the A-MM group. A previous study [21] had classified users in terms of functional and structural models and reported differences in cognitive processes and operational performance. Therefore, in the present study, grouping participants in terms of significant differences in the functional and structural model scores, i.e., a group in which the mental model adapted to the system (S-MM) and a group in which the mental model was not constructed (A-MM), was considered reasonable.

Next, performance on digital camera operation tasks was examined to determine if there was any difference between the two groups. Here, the t-test results showed that the task completion time was significantly shorter in the S-MM group than the A-MM group. In addition, the accuracy score was significantly higher in the S-MM group compared to the A-MM group. This indicates that, in terms of both speed and accuracy, the performance on an operational task is superior when a mental model that is compatible with the system is constructed. This finding is consistent with the results of previous studies on mental models in usability and cognitive engineering fields. Previous studies [19,20,22] reported that operation performance improved when users construct appropriate mental models and that the usability of products for which mental models are easy to construct is good. In other words, to improve the usability of a product and increase the user’s operating performance, it is important that the user is able to construct a mental model that fits the system.

To realize a UI design that allows users to construct an appropriate mental model, it is necessary to clarify what the group that has not been able to construct a mental model (A-MM group) does not understand. In this study, subjective comprehension was investigated using a 17-item questionnaire to determine what the A-MM group did not understand about the operation of the device.

Comparison between the symmetrical and asymmetrical mental model groups showed that the asymmetrical group had significantly lower scores on four items (Q3, Q12, Q15, and Q16). Of these, Q3, Q15, and Q16 were items related to understanding the state of the device during operation. In particular, Q15 and Q16 were related to the ability to understand and respond to changes in the state of the device. This is consistent with the results reported by Doi [21]. Doi [21] investigated the mental model formation process of users with different levels of comprehension of digital camera operation by analyzing their utterances during the operation. That study found that users with a high level of comprehension of the device operation were able to conceptualize the operation based on their understanding of the device status. The present study found that there was a difference in the degree of comprehension of the device status, suggesting that it is always important to be able to understand the device status. In addition, analysis of the responses to Q15 and Q16 indicate that to build an S-MM, it is important to detect and understand changes in the device status.

Q12 probed comprehension of the hierarchical structure of the device screen. The A-MM group’s scores on the structural model task were low, suggesting that they were not able to construct an appropriate structural model. To construct a structural model, it is necessary not only to be able to operate the device but also to understand its structure. According to Doi [32] and Yamaoka [33], a structural model is gradually constructed through repeated manipulation of devices, and whether a structural model can be constructed is an important point to consider determining whether a suitable mental model for the system can be constructed. In other words, to improve usability, it is important that users are able to thoroughly understand the structure of the device and build a structural model at an early stage.

To this point, the design implications of the findings of this study have been discussed. In this study, the difference in subjective understanding between the S-MM and A-MM groups was investigated. The difference in subjective understanding provides insight into the particular lack of understanding of users who are not able to build a mental model that fits the system. The A-MM group is considered to be particularly poor at detecting and understanding the state of the device under operation as well as changes in device status. In addition, the A-MM group was not able to understand the hierarchical structure of the screen. This may be one of the reasons why they have difficulty building mental models. In other words, if designers take care to support these points during design development, usability will be improved. Thus, the findings of this study can serve as a design guideline when considering the ease of mental model building.

5. Limitations

The findings of this study are based on the results of an experiment conducted on a limited number of participants, 24 young Japanese students. Since there are individual differences in mental models, there is room for users with other attributes. Moreover, experiments with many participants and participants with different attributes will lead to the generalization of the findings. Besides, if the characteristics of the participants are different, the distribution of the participant groups in this study may be different. There is room to study the effect of the distribution of the participants on the experimental results. In this study, the experiment was conducted using only a digital camera. It is thought that the findings of this study can be applied to graphical user interface operations that have the same features as those of digital cameras, but it is necessary to study the applicable range of the findings. In particular, careful consideration should be given to user interfaces that have different characteristics from digital cameras, such as webs that consist only of screens or systems operated in conjunction with other systems.

Moreover, although the findings of this study are expected to be applied to product development, the possibility of using the findings in practice has not been sufficiently verified in the scope of this study. It is thought that verifying the usefulness of the results in actual product development projects will lead to design principles.

6. Conclusions

This study compared users who construct a mental model that is symmetrical with the design model of a device and users who construct an asymmetrical mental model. The characteristics of the asymmetrical mental model users in terms of their understanding of device operation were investigated. Initially, the users were grouped in terms of structural and functional models, and it was confirmed that there was a significant difference in the degree of mental model construction between the two groups. Then, performance on a digital camera task and subjective comprehension were compared between the two groups. The comparison demonstrated that the symmetrical mental model group was able to operate more quickly and accurately than the asymmetrical mental model group. It was confirmed that the asymmetrical mental model group had a lower level of comprehension relative to understanding device status, detecting and responding to changes in device status, and understanding the hierarchical structure of the screen.