User-Friendly Locations of Error Messages in Web Forms: An Eye Tracking Study

Abstract

Error messages presented to users are one of the most important elements of Web forms. Error messages are embedded in different parts of the forms available on the Internet and presented in various formats. One of the measures of a user-friendly error message design is the ability to easily capture users’ attention and facilitate fast error correction. In this empirical study, I tested four different locations of error messages frequently used in Web forms on 32 participants. In addition, I analysed the participants’ interactions with error messages through their eye movements. The results of the study showed that the participants spotted the error message fastest when it was displayed on the right side of the erroneous input field. When error messages displayed further the input field users have less saccades to and fixations on error messages compared to those located near to this field, suggesting that less effort has been spent to understand the given message. However, group mean dif-ferences were not statistically significant for form completion time, error recognition time, the number of saccades, and error correction time.

Introduction

Forms usually have complex requirements (Harms, 2013; Nielsen, 2005; Thompson & Torabi, 2007) and present usability issues (Nielsen, 2005). However, beyond this complexity, the logic underlying forms is very clear and straightforward considering that users must enter the data in pre-defined fields presented to them (Harms, 2013). Significant tasks in creating forms include designing userfriendly form fields (Bargas-Avila, Orsini, Piosczyk, Urwyler & Opwis, 2011), increasing the efficiency of users through effective form designs (Katsanos, Karousos, Tselios, Xenos & Avouris, 2013), and facilitating the collection of data from users (Bargas-Avila, Brenzikofer, Roth, Tuch, Orsini & Opwis, 2010).

When designed in accordance with interface guidelines, forms possess a higher level of usability than the previous versions designed before (Harms, 2013).

A well-designed form should prevent users from making mistakes during data entry (Bargas-Avila et al., 2011; Pauwels, Hübscher, Leuthold, Bargas-Avila & Opwis, 2009). Form fields are an active part of daily life and users encounter them in almost every facet of their lives from education to work. Therefore, it is important for them to fully understand and effectively use these elements (Alton, Rinn, Summers & Straub, 2014). In addition, considering that users will make mistakes during data entry, the form design should allow users and developers to prevent, recognize, and correct errors in a rapid and simple manner (Jarrett & Gaffney, 2008; Mockovak, 2007).

In the literature, only a limited number of studies have focused on the efficiency of information presented to users from the perspective of their behavior (Alton et al., 2014). Error messages presented to users are one of the most important elements of Web forms. As well, error message design is critical for Web forms in terms of efficient information presentation considering design guidelines. Furthermore, despite the availability of form design guidelines, there are few empirical studies that have tested these guidelines (Alton et al., 2014). Therefore, the aim of this study is to perform an experimental analysis of most frequently used methods for displaying error messages in Web forms reported by Seckler, Tuch, Opwis, and BargasAvila (2012) in accordance with design guidelines presented by Seckler, Heinz, Bargas-Avila, Opwis, and Tuch (2014). The study uses an eye-tracking device to conduct a behavioral analysis of the users’ interaction with form completion tasks and error messages presented in these forms.

Theoretical Background

Web Forms

Nowadays, many companies and institutions choose to use Web forms to obtain important information about their customers and users (Wroblewski, 2008). Forms are considered to be a bridge between users and applications (Seckler et al., 2014). Users interact with various form elements such as form content, form template, data input types, and error messages (Bargas-Avila, Brenzikofer, Tuch, Roth & Opwis, 2011).

Designing user-friendly form fields is one of the most critical aspects of overall user experience with Web forms (Karousos, Katsanos, Tselios & Xenos, 2013). Web forms lacking usability result in a significant level of user dissatisfaction thus increasing the drop-out rate, preventing the collection of the required data (Seckler, Heinz, Bargas-Avila, Opwis & Tuch, 2013), and decreasing the number of logins to a Web application (Thompson & Torabi, 2007). The usability of form fields vary greatly depending on how they are designed, but minor changes to the form design make it possible to increase the interaction speed, reduce user errors, and improve user satisfaction (Seckler et al., 2014). Two main issues related to form design are (1) identifying the appropriate input for the data to be entered and (2) determining the correct way of entering data (Pauwels et al., 2009). According to Wroblewski (2008), the labels and data input fields used in Web forms should inform users about the content and type of data to be entered.

It is also important to clearly show to users, before they start filling the form, which fields must be completed and which are optional (Linderman & Fried, 2004). Although user errors cannot be avoided, they can be reduced by displaying accurate information and placing certain restrictions within the forms (Bargas-Avila et al., 2011; Seckler et al., 2012) such as the designation of the required and optional input fields (Tullis & Pons, 1997). However, despite all these design guidelines, user errors are inevitable (Thompson & Torabi, 2007). Thus, the design and presentation of error messages during data entry is of great importance.

Error Handling in Web Forms

In the literature, there are only a limited number of studies that have experimentally investigated how to create effective and understandable error messages (Bargas-Avila, Oberholzer, Schmutz, de Vito & Opwis, 2007) and present them in Web forms (Seckler et al., 2012). As a result of two empirical studies conducted with 77 and 90 participants, Bargas-Avila et al. (2007) reported that the most effective way of presenting error messages is after users have completed the whole forms. In another study, Pauwels et al. (2009) studied different solutions for presenting the required and optional input fields with 24 participants. They found that colour-coding the fields resulted in fewer errors, faster data entry, and higher user satisfaction. Seckler et al. (2012) conducted a study with 303 participants to investigate the efficiency of displaying the error messages in the different locations: right, left, above, and below the erroneous input field and at the top and bottom of the form. They found that the most effective, productive, and satisfactory result was achieved when the error messages were presented next to the erroneous input field. A further result was that the participants found the messages most satisfying and intuitive when they were placed on the right side of the field. However, as explained in the next section, they did not study the users’ behaviour when facing these error messages.

Several studies in the literature have reported on and provided guidelines for the effective design of error messages in Web forms. These studies mostly focus on the content, duration, design, and location of these messages (Seckler et al., 2012). For example, Nielsen (2001) suggested that error messages should be located in a way that is highly visible to users and, thus, assist users in correcting the error. According to Brown (1983), error messages should also be informatory. In addition, information about the errors should be presented using a respectful and polite language (Seckler et al., 2012). Similarly, Linderman and Fried (2004) suggested that a simple language should be used to inform the users about errors and give them instructions to correct it. According to Bargas-Avila et al. (2011), if users are expected to use a specific input format, they should be informed before they start filling in the field. Another suggestion to easily capture users’ attention is to use colors, icons, or text to highlight errors (Seckler et al., 2012; Wroblewski, 2008). Once the form has been completed, as much of the data entered by users as possible should be preserved in the event of an erroneous entry in a field. The users should not be expected to fill in the whole form again, only being required to correct or complete the specific field in question (Nielsen, 2001).

Eye-tracking Research in Web Forms

Eye-tracking research dominates the small number of available empirical studies concerning the usability of Web forms. This approach tracks and records the users’ eye movements to monitor their attention (Duchowski, 2007; Sharafi, Soh & Gueheneuc, 2015). In addition, using these devices allows the collection of significant data about the cognitive processes of the participants (Sharafi et al., 2015). Understanding the visual attention of users when performing a task and making inferences about their eye movements can provide valuable information about the usability of Web forms (Strohl, Gonzalez, Sauser, Montazeri & Griepentrog, 2015).

Das, McEwan, and Douglas (2008) used an eyetracking system to examine the usability of label positions in Web forms and found that right-aligned labels provided the shortest completion time. In another study conducted with 23 participants, optimized Web forms were found to produce faster data entry, fewer errors, less eye fixation, and higher user satisfaction (Seckler et al., 2013). In their usability study, Strohl et al. (2015) optimized the design of existing forms according to the following facets of user experience: usefulness, usability, accessibility, credibility, findability, and desirability. The authors reported that the original forms led to longer eye fixation and saccade whereas the newly designed forms were more productive and placed less cognitive load on users. Similarly, Seckler et al. (2014) conducted an empirical eye-tracking study with 65 participants to analyze user behavior towards forms optimized according to the design guidelines developed by Bargas-Avila et al. (2010). The authors reported faster form completion time, fewer input errors, and less eye movements using the optimized Web forms.

Problem and Methodology

Seckler et al. (2012) and Seckler et al. (2014) offered the most comprehensive analysis of locations of error messages and error message design. However, very little is known about empirical analysis of user behavior towards locations of error messages in these studies. Thus, the current study attempts to analyze user behaviors thoroughly by utilizing eye-tracking device.

In this study, I designed and implemented a Web form consisting of the following four steps; “Membership Information”, “Personal Information”, “Educational Information”, and “Preferences”. Because the length of Web forms have an impact on user performance and preference (Wang & Yueh, 2004), to minimize this effect in the study all the form fields were designed to be of the same length for all the participant groups (Figure 1).

In the literature, there is no previous experimental research focusing on locations of error messages in Web forms using an eye tracking device. The main aim of this study is therefore to evaluate locations of error messages in Web forms using an eye-tracking device and to conduct in-depth behavioral analysis of the users’ interaction with form completion tasks and error messages contained in these forms.

The data was collected using a demographic survey, eye tracking, a think-aloud protocol, and retrospective interviews. Eye-tracking data based on particular design guidelines provides valuable information about the language used in Web forms as well as a deep insight into user performance (Strohl et al., 2015). In the current study, Users’ eye movements were obtained during their completion of the tasks presented in the Web forms to analyze, in detail, the pattern of their interactions with error messages.

I presented error messages at four different locations; (1) on the right side of the erroneous input field and (2) below the erroneous input field, (3) at the top, and (4) bottom of the form. To explore the impact of being exposed to different error locations on form completion time, error recognition time, the number of saccades, and error correction time, a series of between-subject one-way analysis of variance (ANOVA) was conducted utilizing SPSS 20.0. In the analyses, the statistical effects were tested at the .05 significant level. For all analyses, the assumption of equality of variance was not violated as indicated by non-significant Levene’s test for homogeneity.

Participants

A total of 32 participants, 12 women and 20 men, participated in the study voluntarily. Participants were recruited using a convenient sampling method. All participants were computer engineering students and left-to-right readers and writers. Mean age of participants was 26.88 (SD=1.72). The educational levels of the participants were bachelor’s degree (n= 20) and master’s degree (n= 12). Of the participants, 62% connected to the Internet using their computers (62%), 19% on their smartphones, 13% on both computers and smartphones, 3% on their tablets, and 3% used all three devices. In terms of the daily use of the Internet, 63% used it for 5 hours or more, 28% 3 to 5 hours, and 9% 1 to 3 hours. The majority of the participants stated that they used the Internet mostly for communication and accessing information.

Participants rated their level of skills concerning the use of computers and the Internet on Likert type scale ranging from 1 (lowest level) to 7 (highest level). Regarding computer use, of the participants, 69% reported to have expert skills on computer. Of the participants, 63% reported their internet use skill level as expert.

Procedure

I divided the participants into four study groups, each containing 8 participants (3 female and 5 male). All the participants were informed about the study at the beginning of the experiments. Then, they filled out a questionnaire form consisting of questions about demographic information such as age, gender, education level, and computer, and Internet skills.

For the experiment, participants completed a Web form consisting of the following four steps: Membership Information, Personal Information, Educational Information, and Preferences. All groups were exposed the same form fields but error messages were positioned differently for each group. I designed and manipulated the form fields in such a way that users would receive the same error message at each step. The participants moved on to the following screen after they corrected the error(s) in the input upon receiving the error message(s) (Figure 2).

An eye-tracking device was used to record the participants’ eye movements. The participants were asked to think aloud during the experiments and their feedbacks were recorded in the form of notes. The researcher did not intervene in the form-filling process, but, at the end of the experiment, asked participants to discuss about the recording of their behavior during the experiment. Each retrospective interview lasted approximately 20 minutes about locations of error messages in Web forms.

Apparatus

Tobii TX300 device was used to record the participants’ eye movements during the display of error messages and eye movement patterns were analysed using the Tobii studio software (version 3.2.1). The participants were positioned 60 cm from the device and, to ensure the accuracy and reliability of the results, were asked to only look at the screen and not to move their heads. At the beginning of the study, a calibration test was performed with each participant, in which they were asked to follow points displayed on the screen. Among participants, two failed the calibration test and were replaced by the new ones. The researcher performed the experiment only with the participants that passed this test.

Results

Form Completion Time

Form completion time refers to the time taken from the opening of the form in the browser to completing the form successfully. The average form completion time was 2.42 minutes for Group 1, 2.41 minutes in Group 2, 2.36 minutes in Group 3, and 2.31 minutes in Group 4. As shown in

Table 1. Performance of the participants in completing the forms.

, the participants who were presented the error messages on the right side of or below the erroneous input field took longer to complete the form than those participants who saw these messages further from the erroneous input field (at the top or bottom of the form).

When the error messages were displayed further from the input field, participants spent more time to complete the form compared to those who had the error messages closer to the input field. However, the ANOVA results yielded that differences between group means were not significant for form completion time, F(3, 28) = 0.27, p=0.85.

Error Recognition Time

Error recognition time is the time taken by the user completing the form and clicking the ‘Continue’ button to the moment that s/he identifies the displayed error message. It is the time difference between clicking the continue button and the first fixation on the error message. The average error recognition time was 0.30 seconds for messages displayed on the right side of the erroneous text field (Group 1), 0.33 seconds for messages presented below the erroneous text field (Group 2), 0.34 seconds for those presented at the top of the form (Group 3), and 0.44 seconds for those displayed at the bottom of the form (Group 4) (see Table 1).

The participants who were presented the error messages at the bottom of the form took longer to recognize these messages. According to the observation of the users’ behavior, after clicking the ‘Continue’ button at the bottom of the form, the participants moved their eyes to the top of the form to be ready for the following page, which may be the reason why Group 4 took longer to see the error messages. The shortest error recognition time was obtained from the messages displayed on the right of the erroneous input field. However, the ANOVA results yielded that differences between group means were not significant for error recognition time, F(3, 28) = 2.74, p=0.06.

Number of Saccades to and Eye Fixations on the Error Message

The number of saccades to and fixations on the error messages was analysed from the appearance of the message on the screen to the moment that the participants corrected the error and clicked the ‘Continue’ button. The total number of saccades and fixations was directly related to the participants’ need to read and reread the message until they could correct the entry, thus representing the effort and time spent in this process.

According to the results, the total number of saccades was 244 for both the error messages displayed on the right of and below the erroneous input field. However, this number was significantly lower for the messages located at the top (217) and bottom (206) of the form. This observation indicates that, when trying to correct an error, the participants tended to look at the message more frequently when it was located closer to the erroneous input field. When the messages were displayed further from the related field, the participants tended to look at the message less frequently and exerted less effort to understand the error message. Thus, the number of saccades and fixations was lower for the error messages located at the bottom of the form. However, the ANOVA results yielded that differences between group means were not significant for the number of saccades to and eye fixations on the error message, F(3, 28) = 0.26, p=0.85.

The analysis of the heat maps of the participants’ saccades to error messages provided interesting results. The participants mostly fixated and moved their eyes to the information about the input format and the examples contained in the error messages (Figure 3). They mainly focused on the error message elements that they considered useful in correcting the error, without paying much attention to the remaining content. This observation shows the importance of providing examples in error messages to instruct the users on how to correct the error.

Error Correction Time

Error correction time refers to the time taken by the participants from receiving the error message related to a form field to entering the correct data and moving on to the next step. This data showed whether the location of the error messages had an effect on the time and effort the participants spent on correcting the error.

The participants, who were presented the error messages on the right side of and below the erroneous input field, corrected the error in an average of 12.55 seconds and 12.68 seconds, respectively. However, when the error messages were displayed at the top and bottom of the form, the participants corrected them in shorter time, on average of 12.07 seconds and 11.98 seconds, respectively. These observations indicate that the participants took more time to correct an error when the error message was located closer to the erroneous input field, possibly because the participants’ looked at the message more frequently when it was closer to the erroneous input field. In contrast, in forms where the error messages were displayed further from the input field, the participants fixated the error messages less, thus their error correction time was shorter. Since the error message is located further from the peripheral vision, participants may prefer to focus on input field, rather than error message, thereby spending less time with occasional saccades. The ANOVA results revealed that that differences between group means were not significant for error correction time, F(3, 28) = 0.14, p=0.94.

Participants’ Views on Error Message Design

According to the feedback from the think-aloud procedure and the retrospective interviews, a significant number of the participants preferred a Web form design that places restrictions on data input related to the required format and contains tooltips or placeholders to inform the users about the correct format of the input. They further explained that such a design not only prevents erroneous input but also saves time in completing the form. Furthermore, the participants wished to be warned by the system upon entering erroneous data. They stated that it takes more time to correct an error if they are informed about it after they have filled in the input fields on a form page and clicked ‘Continue’ to move on to the next step. Rather, they suggested that the system should give them feedback about the correct and incorrect input immediately after entering data in each field.

Primarily, the participants of this study preferred error messages that contained examples to inform the user how to correct the error. They commented that such messages help them to more easily understand the content of the error message and, thus, reduces the time for them to correct the error. This feedback highlights the significance of including examples in the instructions in the error message. Furthermore, the participants mentioned that they felt uncomfortable when no examples were provided.

The participants who were presented the error messages on the right side of and below the erroneous input field reported that they were satisfied with the location of these messages whereas almost all the participants who were offered the messages at the top or bottom of the form complained about the distance between the error message and the erroneous input field. Most participants stated that both required and erroneous input fields should be marked with a color to better capture their attention. Thus, users spot the erroneous fields easily and quickly and spend less time for error correction.

Discussion and Conclusions

Forms are one of the basic elements that allow interaction with users in Web applications (Seckler et al., 2012). Poorly designed Web forms result in a loss of data, lower conversion rate, and higher number of dropouts (Strohl et al., 2015). Therefore, Web forms should be designed in a way that is effective, efficient, and satisfactory for users (Bargas-Avila et al., 2011). In this study, eye-tracking data was used to conduct an empirical analysis of the effectiveness of the most frequently used methods for presenting error messages in Web forms according to the design guidelines.

The current study analysed the effect of error message location on the user’s form completion time. Although error messages located closer to the erroneous input field resulted in users taking longer to amend the entry compared to those messages displayed further from the related field, differences between group means were not significant for form completion time. It might be expected to observe higher performance when the error messages were presented closer to the erroneous input field as evidenced by Seckler et al. (2012). However, the present study fell short on statistical significance.

The participants’ error recognition time depends on the location of error messages. There are different findings in the literature about effective and efficient location of error messages to capture users’ attention easily. In the current study, eye tracking data showed that the participants noticed the error messages in a shorter time when they were displayed on the right of the erroneous input field. Eye tracking results of this study are in agreement with those reported by Seckler et al. (2012), who concluded that efficiency of error correction is influenced by the distance between the erroneous input field and the error message and error messages located closer to the input field capture the users’ attention more easily in Web forms. Although, there was no significant difference between group means for error recognition time, the findings of the current study stand contradictory in a way that the close location of error messages to the related field was found to be associated with faster reaction time, however; users spent more time to complete the form when error messages are located closer to the erroneous field. The longer completion time may be explained by users’ tendency for continuous tracking of error message when it is located nearby.

Eye tracking data showed that the number of saccades to and fixations on the error messages to be higher for messages closer to the input field. The participants tended to look at these error messages more frequently during the error correction process. When the message was displayed at the top or bottom of the form, the participants moved their eyes to these messages less frequently and exerted less effort to understand the content. However, the eye tracking findings of the current study do not support the previous research by Alton et al. (2014). They reported that presenting instructions closer to the input field would prevent excess effort on the users’ part in terms of eye fixation. These inconsistent results may due to the differences in Web form complexities used in these studies.

Seckler et al. (2012) reported that when the messages were placed near the input field, the participants tended to make fewer mistakes. In the current study, it was found that close location of errors to the input field is related to more saccades and fixations, which may result in fewer errors. Although current study did not aim to evaluate correctness of error messages, future studies may highlight the relationship between location of error messages and number of mistakes.

According to the heat maps of the error message content, it was found that the participants tended to fixate on content providing information about the correct format and offered examples. They paid more attention to direct instructions on how to correct the error, mostly disregarding the remaining content of the error message.

In the interviews, the participants of the current study stated that their performance improved when the form fields contained inline validation. In addition, they wanted to see tooltips or placeholders that would prevent them from making a mistake. They also wanted to be informed about an error immediately after it occurred and before they completed the whole form. Applications that allow users to check the validity of their input have also been reported to increase the quality of the data (Fox, Mockovak, Fisher & Rho, 2004).

Another common method for reducing errors is to place format-related restrictions on the input fields and only allowing entries that conform to the given rules (Seckler et al., 2012). This method was mostly preferred by the participants of the current study. These restrictions will not only reduce the number of erroneous inputs but are also an efficient way of instructing the users the way they can enter data accurately (Bargas-Avila et al., 2011).

Design features such as the length of the form and the presentation of messages have a significant impact on users’ performance in form-filling (Wang & Yueh, 2014). For example, users are more satisfied with colored coded fields (Pauwels et al., 2009). The current study also demonstrated that the participants preferred a design using striking features, such as coloring to display both the required fields and error messages. Graphic symbols such as an asterisk are among other common elements used in Web forms to capture the attention of users (Mockovak, 2007).

Interface design is of central importance in humancomputer interaction in general and user experience in particular (Tuch, Roth, Hornbaek, Opwis & Bargas-Avila, 2012). One of the challenges in establishing and maintaining this interaction is determining the appropriate interface design (Girgensohn & Lee, 1997). The results of the current study provide valuable data for analysis of user behavior towards error messages in Web forms, which assists in the process of creating an effective, efficient and satisfactory design. Future empirical studies conducted with larger series and different design guidelines will contribute to the further optimization of these forms.

Threats to Validity

The present study has some validity threats that should be taken into account. When interpreting the study findings, these potential threats should be addressed carefully. In terms of internal validity, first, the generalization of the study findings may be problematic because the participants were all young adults and left-to-right readers and writers. The results may be different with older participants and people from cultures in which reading/writing is performed right-to-left or top-to-bottom. Future studies must address this limitation by focusing on participants from different cultures and age segments. Second, the participants had high levels of computer skills, thereby obstructing to generalize the findings for those who have average computer skills. For example, it is likely that they spend less time to recognize the errors, to correct them, and to complete the whole form. It would be better to replicate the current study with those with average computer skills. Thirdly, it is likely that statistical tests performed in this study have low statistical power to detect the potential effects due to small sample size. More robust findings could be attained in larger samples because the small sample sizes may not allow obtaining adequate statistical power.

In terms of construct validity, the following potential threats should be considered before inferring conclusive results. First, in the current study, single Web form was used to evaluate impact of error message locations. However single form designs may cause underrepresentation of all Web forms. Thus, it would be better to implement multiple Web form versions to capture impact of error message locations for user-friendly designs. Second, to calculate number of saccades and fixations, the time period between appearance of the message on the screen and the moment that the participants corrected the error and clicked the ‘Continue’ button was calculated. Although the current study was designed on the basis of single version of error message, it is unclear whether all the time spend by participants was due to the time to enter the correct information. Future studies should be designed carefully to rule out this limitation.

Despite these limitations, this study is the first to investigate location of error messages focusing on users’ behaviors utilizing eye-tracking device. Despite its exploratory nature, this study offers some insight to locate effective, efficient, and satisfactory design of error messages in Web forms.