Designing Eye-Tracking Experiment to Enhance Reading Comprehension in Online Learning Environment ^†

Abstract

Reading is an essential skill for learners, yet numerous individuals face challenges in understanding text, particularly in digital learning. This study aims to introduce an experimental design plan to improve reading performance in online learning environments. The experimental design incorporates various visual components, including images and colored text, alongside audio elements that may influence reading comprehension. This experimental design utilizes an eye-tracking study methodology. The result of this study can be used for robust experimental designs. The result of the experiment can be used to optimize online learning materials, thereby enhancing students’ reading skills in an online context.

1. Introduction

Reading is an essential skill that students, especially those in university, must acquire in their academic pursuits. However, many students currently require assistance with reading narrative texts. Many researchers emphasize the significance of individual words and text structures over the overall text and show a lack of interest in reading, especially narrative texts, due to difficulties in identifying the main idea and summarizing the content [1]. Previous studies on reading performance have focused mainly on display size and working memory. Research on the impact of screen size on comprehension skills often yields contradictory findings. Multiple studies indicate that the electronic medium did not negatively affect comprehension, and in certain cases enhanced the comprehension outcomes [2]. Nonetheless, presentation factors, including sentence division on pages resulting from reduced display sizes, impede comprehension by overloading the working memory [3].

There is a connection between academic performance and working memory. Research indicates that the characteristics of electronic texts, such as hypertext and written annotations, increase cognitive strain. This heightened cognitive load leads to difficulties in retaining and understanding information, which ultimately affects academic performance. Recall has been negatively affected by electronic texts [4]. Previous research indicated that various e-readers, including those utilizing e-ink technology, did not significantly influence the learning outcomes. The variety of individual learning styles and preferences implies that the impact of electronic texts varies among students. Additionally, the incorporation of multimedia features in digital formats affects the learning experience, either positively or negatively, depending on how they are used [5]. As educational technology continues to evolve, it is crucial to explore these dynamics to improve learning processes in digital environments.

On 31 December 2019, a novel pneumonia virus was detected in Wuhan, China, and designated as the coronavirus COVID-19. Within a few months, it rapidly spread around the globe. On 11 March 2020, the World Health Organization (WHO) classified COVID-19 as a pandemic because of its extensive transmission and severity. This declaration marked a crucial juncture in global health, urging countries to implement emergency protocols to curtail the virus’s transmission. Many countries banned public gatherings and closed educational institutions. The closures aimed to protect public health, while revealing the deficiencies in educational systems that were unprepared for a sudden shift. The COVID-19 epidemic resulted in the shutdown of educational institutions, impacting nearly one billion students globally [6,7,8]. This extraordinary disruption resulted in a substantial rise in remote learning efforts, compelling educators and students to swiftly acclimate to online platforms. The shift revealed inequalities in access to technology and resources, highlighting concerns regarding educational equity during the crisis.

Since then, educational institutions have employed diverse online learning platforms with distinct functionalities and methodologies to enhance learning. These platforms encompass basic learning management systems to more advanced ones that integrate interactive elements and real-time collaboration. Universities employ pedagogical approaches, including synchronous online lectures, recorded audio and video lectures, shared digital resources, and blended learning methodologies [9]. This variety of pedagogical methods provides adaptability in addressing the requirements of diverse learners, accommodating distinct learning styles and improving engagement. They utilized online assessment methods, like quizzes, examinations, and assignments, to uphold academic integrity and deliver prompt responses.

During the COVID-19 pandemic, higher-education institutions adopted online learning abruptly, requiring swift modifications to curricula [10]. The epidemic affected students, teachers, and educational outcomes, resulting in challenges, such as diminished motivation, a sense of isolation, and obstacles to sustaining academic intensity. Numerous educational institutions, educators, and learners were ill-equipped for this novel experience [9,10,11,12]. The absence of readiness underscored the necessity for enhanced training and resources to facilitate effective online instruction and learning, along with the significance of formulating comprehensive contingency plans for future disruptions [12].

Online education has long been regarded as a feasible alternative for students who are older and possess more financial and professional obligations compared with their on-campus counterparts [13,14]. Nonetheless, the COVID-19 epidemic fundamentally transformed the educational landscape and resulted in the most substantial interruption to educational systems in history, as reported by the United Nations and the UNESCO. During the COVID-19 pandemic, in-person classes were halted, leading to the adoption of emergency online learning as the most viable alternative. This intervention was effective in containing disease spread; however, it imposed significant strain on all participants in the educational process [15]. The transition to online learning resulted in a significant learning curve for educators and students, increasing stress and anxiety as they adjusted to new technologies and pedagogical approaches. Instructors and learners had to acquire additional experience with online learning to engage with it with minimal support. Educators encountered difficulties in effectively engaging students in a virtual environment, while students experienced challenges in sustaining motivation and focus due to the absence of a traditional classroom. Additionally, many higher-education systems lacked the necessary resources to effectively implement online learning for all students on a large scale [16]. The digital divide has disproportionately impacted students from lower socioeconomic backgrounds, impacting educational quality and raising equity concerns.

Students engage in everyday activities, such as taking notes and highlighting, to meet these requirements, which often require the use of peripherals [17]. These practices are crucial for enhancing comprehension and retention, as they encourage active engagement with the material. However, we need to perfect these essential features, especially using e-readers such as e-books. Many students find that the note-taking and highlighting functions in e-readers are not as intuitive or effective as those for the traditional paper format, which hinders their study processes. Several technologies do not allow the use of colored texts and images [18]. While color coding often organizes information and highlights key concepts, the inability to utilize color limits the effectiveness of visual learning strategies. Additionally, a lack of tactile feedback in digital formats detracts from the overall learning experience, making it challenging for students to interact with the content in a meaningful way. As technology continues to evolve, addressing these limitations is essential for maximizing the educational potential of e-readers and similar devices.

Visual stimuli in eye-tracking studies provoke human visual behavior and cognitive processes. It is necessary to establish an eye movement parameter to assess eye performance. Fixations and saccades represent eye movements generated by the human visual system. In human–computer interaction and psychological studies, fixation refers to a stable state of eye movement, while a saccade denotes rapid eye movement occurring between successive fixations. The duration of fixation for scene processing differs by task, influenced by the cognitive processes involved. Observers need a minimum of 150 ms to encode stimulus properties [19].

Eye-tracking technology effectively records online cognitive activities as essential in computational cognitive science [20]. This method has been extensively utilized by neuroscientists and psychologists to examine the essential cognitive processes in information processing. Recently, the eye-tracking method has been extensively employed to elucidate complex visual cognition, including scene perception, visual search, and natural tasks. This approach has yielded significant insights into how individuals perceive their environment and engage with real-world objects [21]. Consequently, eye movement patterns acquired from an eye tracker provide data to explore individuals’ attitudes toward various objects or subjects.

The inclusion of images and illustrations in educational materials, or the use of multiple representations for children, yields inconsistent effects on comprehension. Illustrations enhance reading comprehension across all reading levels [22,23]. Young readers require a comprehension of the role of illustrations due to the lack of adequate attention given to them. The eye-tracking methodology enables direct observation of reading behavior through eye tracking.

In prior studies, reading performance was assessed through the use of keywords and visual representations [22,23,24]. There are limited studies that examined the effects of integrating colored keywords and pictorial representations with audio support on reading performance of learners, encouraging educational institutions to enhance their educational materials. The developed experimental design incorporates audio support, colored keywords, and pictorial representations to create effective learning materials aimed at enhancing reading performance.

The impacts of many representations exhibited consistency [23]. Young readers possess a restricted comprehension of the role of images, and hence require increased focus on them [24]. Eye-tracking methodology allows for the testing of these assumptions via direct observation of reading behavior. By analyzing the association between eye movement, the factors affecting reading performance can be identified [24].

2. Methodology

The Delphi method is used to organize a group’s communication process to tackle a complex issue [25,26]. In the traditional Delphi method, an open-ended questionnaire is used, but it is time-intensive to complete, yielding a low response rate. A structured questionnaire was created through focus group interviews or literature reviews [27]. We developed the experimental design using a modified Delphi method tailored to specific requirements. The process involved determining the variables and choosing the most appropriate execution method. We conducted a literature review to determine the variables to include. The list of variables was approved by a panel of experts. Figure 1 outlines the steps of the modified Delphi method to establish a rigorous experimental design for this study.

3. Results and Discussion

3.1. Determination of Variables

The modified Delphi method was used to identify three distinct independent variables: colored text, images, and audio support. Colored text is considered one of the most important visual stimuli for humans [28]. It is crucial in improving memory performance and serves as a strong information channel for the human cognitive system [29]. Roslina indicated that picture storybooks have the potential to enhance students’ reading abilities and foster greater interest in reading [30]. This suggests a potential improvement in students’ reading performance.

We identified four dependent variables: saccade duration (SD), number of fixations (NF), fixation duration (FD), and reading score (RS). SD represents the cumulative saccadic duration within a designated area of interest (AOI). NF is the aggregate count of fixations from the virtual origin to the targeted item. FD refers to the interval between saccades during which the eyes remain stationary on an object among surroundings. The dependent and independent variables were confirmed through a review of other publications and by experts. The expert group included lecturers with experience in eye-tracking research.

To examine the eye-tracking experiment design, various metrics of ocular movement were measured: (1) the total reading time (the aggregate of all fixation durations within the area of interest (AOI)), which reflects the overall complexity and cognitive effort required to comprehend reading materials; (2) mean fixation duration (the average duration of all fixations within the AOI), which indicates the time readers require to process words or illustrations (prolonged mean fixation durations on a word typically signify challenges with word decoding, while extended mean fixation durations on illustrations usually denote difficulties in interpretation); (3) the number of fixations (the cumulative count of fixations within the AOI), which signifies the reader’s focus and cognitive engagement with the text; and (4) fixation duration, specifically AOIs divided by total fixation duration throughout the reading session, which illustrates selective attention given to specific target areas during reading [24]. The developed design was investigated using the degree to which the incorporation of colored text, images, and audio accompaniment optimizes reading proficiency, as illustrated in Figure 2.

3.2. Participants

The participants were enrolled students at a university and selected according to their proficiency with reading academic texts. They possessed English language certifications, such as TOEFL or TOEIC, and demonstrated proficiency in English. The minimum score on their English test ranged from 450 to 500, signifying an intermediate performance in standardized assessments for overseas students. Their visual acuity and color perception were tested before the experiment in this study using the Enchroma color vision assessment. These tests were conducted to minimize data variability. All the participants underwent the Ishihara Colour Vision Test for color blindness [31].

3.3. Experiment

In the experiment, various devices and equipment were used. The eye movement data were collected using a Tobi-Pro Nano eye tracker, with a sampling rate of 60 Hz and an accuracy of 0.04° of visual angle [32]. The distance between the screen and the participant was 80 cm. The eye tracker from Tobii was positioned 60 cm in front of the participant. To maintain the same relative distance between each device and the participant, each device was fixed and marked with adhesive tape. The participant completed the task in a dark room (2.5 × 2.0 × 2.37) m³ with cream-colored curtains to block light and create a high-quality environment. The velocity threshold was set at 30° per second [33]. All the devices were secured with adhesive tape. We experimented in a standard dark room to create an ideal environment and ensure an identical experiment for the participants (Figure 3).

3.4. Procedure

The participants’ color vision was assessed utilizing the Ishihara Color Vision Test alongside the E-Chroma color blind test (https://enchroma.com/pages/test, accessed on 10 October 2024) to confirm normal color vision [34]. Upon completion, the results were calibrated to record precise data. Each participant tracked a dot’s progress across the nine points displayed on the monitor. Data loss did not exceed 5%, thus maintaining high-level precision when recording the eye movements. Subsequently, the participants processed the stimuli featuring colored text, images, and audio. Upon finishing each stimulus, they took a five-minute rest according to the Pomodoro technique to enhance their productivity and reading focus. Consequently, the participants utilized the Tobii Pro Nano to document their RS, FD, NF, and saccade duration, all of which affected their reading performance (Figure 4). A post-test was conducted using reading materials, consisting of four questions, to which each participant answered to evaluate their reading score and assess their understanding of each stimulus.

3.5. Reading Materials for Stimuli

The reading assignment was in a PowerPoint format. In this experiment, we employed the Tahoma 22 PT typeface and configured the page margins to 90% to ensure optimal screen visibility for the participant. The text was printed in black, with the keywords emphasized in yellow. Reading difficulties are markedly resolved after using a yellow filter for three months [35]. Colored filters serve as an effective intervention for delayed readers, and yellow enhances the input to the magnocellular system [36]. We used read-aloud as a factor affecting the dependent variable. The decibel level employed in this investigation ranged from 70 to 80 dB in a confined space. We formulated the material layout as follows:

• Material A: colored text (Figure 5);
• Material B: plain text with picture (Figure 6);
• Material C: plain text with audio support (Figure 7);
• Material D: colored text with picture (Figure 8);
• Material E: colored text with audio backing (Figure 9);
• Material F: picture with audio support (Figure 10);
• Material G: colored text, picture, and audio support (Figure 11).

3.6. Statistical Analyses

The Kolmogorov–Smirnov normality test was conducted to determine whether the data set showed a standard distribution. One-way analysis of variance (ANOVA) was conducted with colored text, images, and audio support as the independent factors. Each dependent variable (SD, RFD, NF, reading score, and RS) was tested for the effect size using the TOEFL reading scores of the two participant groups. We set the significance level as 0.05 [37].

4. Conclusions

Learning materials are essential for student learning, especially in online education. An eye-tracking method was used to evaluate the impact of colored keywords and pictorial representation with audio support on reading performance in online learning. We reviewed the results through a literature review to establish robust support for eye-tracking research. Experts validated the dependent and independent variables; the dependent variables included colored text, images, and audio support, and the independent variables included SD, NF, FD, and RS. A detailed design, encompassing participant selection, experimental layout, procedures, and reading materials for stimuli, was validated through statistical analyses. The results present robust support for eye-tracking experiments to enhance reading comprehension in online learning environments using the modified Delphi method.