AraEyebility: Eye-Tracking Data for Arabic Text Readability

Abstract

Assessing text readability is important for helping language learners and readers select texts that match their proficiency levels. Research in cognitive psychology, which uses behavioral data such as eye-tracking and electroencephalogram signals, has shown its effectiveness in detecting cognitive activities that correlate with text difficulty during reading. However, Arabic, with its distinctive linguistic characteristics, presents unique challenges in readability assessment using cognitive data. While behavioral data have been employed in readability assessments, their full potential, particularly in Arabic contexts, remains underexplored. This paper presents the development of the first Arabic eye-tracking corpus, comprising eye movement data collected from Arabic-speaking participants, with a total of 57,617 words. Subsequently, this corpus can be utilized to evaluate a broad spectrum of text-based and gaze-based features, employing machine learning and deep learning methods to improve Arabic readability assessments by integrating cognitive data into the readability assessment process.

1. Introduction

Reading is essential for acquiring knowledge and communicating, involving processes like word encoding and meaning assignment [1,2]. It is crucial for academic success and accessing information across disciplines [3,4,5]. Readability, as defined by Dale and Chall [6], involves elements affecting a reader’s understanding, speed, and interest. It depends on reader-based features like background knowledge and motivation, as well as text-based features like content and syntax [2,7,8,9]. Predicting readability has shifted the focus to text analysis [8,9]. However, traditional methods such as cloze tests and expert judgment are costly and subjective, highlighting the need for technological tools [8]. Additionally, current models rely on expert judgments rather than readers’ cognitive processing, and most Arabic readability research relies on simple features that do not reflect true readability [10]. Although Arabic readability corpora exist, they focus on Arabic first-language (L1) and second-language (L2) learners, with limited corpora for assessing readability for the general Arabic-speaking public [6,11,12,13,14]. Accordingly, because using physiological data like electroencephalograms (EEGs) and eye tracking can improve readability models by predicting human processing efforts during reading [15,16,17], this paper aims to fill this gap by establishing an Arabic human reading corpus to aid in subsequent modeling of Arabic text readability assessment [18].

The establishment of an open-source Arabic human reading corpus, parallel to that in other languages, such as English [18,19], French [18], and German [12], with graded gold-standard texts annotated by both humans and computers using eye tracking, is expected to open the door for the broader utilization of eye tracking for further research related to the improvement in Arabic readability assessments and many Arabic natural language processing (NLP) tasks. Additionally, collected data might help in the development of theories about natural reading behavior through eye movement analysis [20]. For example, such a corpus will help to enhance machine learning (ML) models for diverse NLP operations, such as information extraction, sentiment analysis, text quality assessment, and text simplification [21,22].

2. Background

This section provides an overview of the Arabic language and outlines the principles of eye-tracking technology, emphasizing its relevance to reading generally and its connection to Arabic specifically.

2.1. Arabic Language

Arabic is a Semitic language with a right-to-left cursive script consisting of twenty-eight consonantal letters without upper- or lower-case distinctions [23]. Its letters change shape depending on their prefix and suffix letters [24,25]. Arabic comprises Classical Arabic (CA), Modern Standard Arabic (MSA), and various dialects. CA, a formal language from the pre-Islamic era to the early eleventh century CE, is the foundation for MSA, which is used in modern media and literature and is easier to read. Dialects are regional spoken varieties used in informal communication [26,27,28]. Arabic script uses diacritics for short vowels and pronunciation guides, such as fat’ha, kasra, damma, shadda, and tanween [27]. The addition of diacritics, called diacritization, clarifies pronunciation and meaning, resolving lexical ambiguity in undiacritized text. For example, the undiacritized word “علم” can mean flag (عَلَم), knowledge (عِلْم), or teach (عَلّم), depending on its pronunciation [29]. Diacritization can be full, partial, or absent. Full diacritization adds diacritics to every letter, enhancing clarity but slowing reading speed due to visual noise. Partial diacritization marks selected letters, improving readability for less experienced readers without much visual noise, but it is subjective and time-consuming. Absent diacritization relies on reader knowledge and context for correct pronunciation and meaning, which is challenging for less experienced readers, especially with ambiguous words and homographs [23,24,28,30,31,32,33].

2.2. Eye Tracking

When surveying a scene, the human eye creates varied movement patterns, commonly referred to as the scan path [34]. The estimation of the eye’s position to know exactly where a person is looking (point of gaze), its motion, and for how long in real time is known as eye tracking [34,35]. Interfacing with computers through eye tracking involves analyzing the specific region of the user’s focus, a concept dating to the 18th century [35]. In recent years, eye tracking has gained significant popularity in assessing the visual and cognitive processes of humans [36]. Providing insights into visual attention and user behavior, this technology is widely used today in research and practical applications in psychology, education, marketing, gaming, and user-interface design [34].

2.2.1. Eye Tracking and Reading

According to [15], there is a significant correlation between readers’ eye movements and their cognitive processing of texts. This is linked to Just and Carpenter’s eye–mind hypothesis, which posits that “there is no appreciable lag between what is fixated and what is processed” [37]. Thus, it allows natural reading observations with minimal intervention, providing insights into language processing and serving as an alternative to traditional assessments like cloze tests and think-aloud protocols [20,36]. Studies measure saccades (rapid movements), fixations (stops), and regressions (backward movements). Longer saccades suggest easier text, shorter saccades indicate more effort, brief fixations imply readability, longer fixations suggest difficulty, and regressions indicate comprehension challenges [13,17,36,38,39]. Fixation and saccade patterns reveal information about the reader and the text [36,40]. Thus, eye motion data elucidate cognitive processes related to reading [17]. Although it is resource-intensive, eye tracking offers a natural way to study reading [13,14].

2.2.2. Eye-Tracking Visualization

Eye-tracking visualizations offer detailed data from large groups, confirming hypotheses, detecting patterns, and identifying attention-grabbing or ignored elements. They align individual feedback with group metrics, providing deep insights into human behavior [41,42]. Figure 1 shows visualizations of the results from an eye tracker as participants read two Arabic texts of varying lengths.

Gaze plots show where and for how long a person’s attention is focused, the order of focus, and the reading direction (forward or backward) [43]. Circles (fixation bubbles) represent eye fixations, with size indicating fixation time, and arrows show saccades, with length indicating the scanning path [44]. Fixations are numbered [44,45]. In Arabic, which is read from right to left, less readable text results in shorter saccades, more fixations, and regressions, indicating reading difficulty [17]. Heatmaps visualize eye fixations by aggregating fixation counts or durations from all participants and highlighting the areas receiving the most attention. A color gradient indicates fixation frequency and duration: red for the most or longest fixations, green for the fewest, and intermediate colors for other levels [41].

2.2.3. Eye Tracking and Arabic Language

Arabic presents unique reading challenges compared to other languages. Research indicates that reading Arabic requires stronger visual–spatial processing abilities due to its distinct cognitive demands [45,46]. Unlike Latin languages with non-cursive writing, few studies have explored eye movement control in Arabic reading [30]. Al-Edaily et al. [47] note that Arabic’s complexity—including orthographic directionality, cursive nature, letter size and form changes, use of dots, orthographic ambiguity, and morphological richness—enhances its visual–spatial processing demands [25]. These nuances necessitate investigating specific Arabic attributes influencing reading via eye tracking. Below are some similarities and differences in eye movements when reading Arabic compared to other languages:

• Arabic reading’s informational density makes it more time-intensive than Latin languages, making word identification more challenging [30].
• The direction of reading influences the perceptual span’s asymmetrical extension. In Arabic, this focus area extends more to the left, while in Latin languages, it extends more to the right, impacting visual processing during reading [30].
• The length and familiarity of words affect eye movement decisions in both Arabic and Latin languages. However, the impact of skipping words is less significant in Arabic, with low differences in skipping rates for low- and high-frequency words [30].
• Studies suggest that words in Semitic languages are best understood when the focus is placed on the middle, unlike Latin languages, for which the focus should be placed on the beginning–middle. This difference is due to the morphological structure indicating core meaning [30].
• Diacritics in Arabic text aid in disambiguation and enhance accuracy and attention but can be perceived as visual noise, leading to longer fixation durations. Experienced readers can rely on context and language knowledge without diacritics [24,28,29,30,31].
• Arabic writing is more complex due to the mandatory dots above or below many letters, unlike Latin languages, for which only two lowercase letters have dots [25].
• Numbers in Arabic are read from left to right, unlike text, which is read from right to left. This can cause inversion errors during reading [48].
• Arabic text is more challenging to read than Latin text due to its cursive nature, context-dependent characters, diverse writing styles, and unique letter positioning, such as in the word “محمد” (Muhammad), for which some letters can be placed above others [49].

3. Literature Review

This section reviews eye-tracking studies in reading across multiple languages. It also examines relevant Arabic corpora, along with cognitive corpora in other languages, emphasizing the usefulness and significance of these data in clarifying the intricate relationship between eye movements and human reading.

3.1. Eye Tracking in Reading Studies

Eye-tracking studies reveal that reading involves sequences of fixations and saccades, with some words fixated on more than once or skipped [50]. Just and Carpenter [37] explored eye movement patterns and cognitive loads in reading scientific papers, assuming the eye lingers on each word for processing time. Frazier and Rayner [51] suggested that sentence misinterpretation is corrected through definable strategies, correlating fixation locations and lengths with text processing. Rayner et al. [52] found that text difficulty increases fixation number and duration. Liversedge et al. [53] added regression path duration and rereading times to key reading time metrics, including first fixation duration, first-pass reading time, and total reading time, in order to better understand eye movements related to textual difficulties. Advanced eye movement recording systems have enabled high-quality recordings, aiding linguistic and psycholinguistic studies [36,38,54,55,56,57,58].

Arabic reading, though crucial for many, has been less studied in terms of cognitive processes [59]. Al-Wabil and Al-Sheaha [45] and Al-Edaily et al. [46,47] have examined eye movements in Arabic readers with learning difficulties like dyslexia. AlJassmi et al. [30] reviewed eye movements in Arabic reading, noting limited understanding compared to European languages and identifying future research questions.

Additionally, visual distinctions between diacritized and undiacritized Arabic texts were investigated by Roman and Pavard [31], Bensoltana and Asselah [60], Hermena et al. [23,29], Maroun and Hanley [61], and Awadh et al. [62]. Hallberg [63] analyzed how case markers are processed in Arabic. Furthermore, some studies have investigated the effect of word spacing. For example, Leung et al. [64] investigated the difference between reading spaced and unspaced Arabic texts. Word characteristics, such as length, were investigated by Paterson et al. [25], predictability by Hermena et al. [29], and morphology by Khateb et al. [65] and Hermena et al. [66]. Additionally, Al-Khalefah and Al-Khalifa [67] studied spelling pattern consistency in elementary students’ recognition of English and Arabic words in Saudi Arabia. Moreover, Hermena and Reichle [68] emphasized re-evaluating reading models designed for European languages to better understand Arabic reading. Lahoud et al. [69] examined factors influencing eye movements in skilled Arabic readers, balancing language-specific and universal factors.

3.2. Corpora

This section reviews the existing Arabic corpora for the readability assessment task and some existing eye-tracking corpora for various languages, which provide a general understanding of the primary characteristics that must be considered when crafting a new Arabic eye-tracking corpus.

3.2.1. Arabic Readability Corpora

For Arabic, there are limited corpora available for readability assessments that target Arabic L1 and L2 readers, and these corpora are mainly in the educational domain [70]. Research into readability evaluation in other fields is scarce. A comparison of Arabic L1 and L2 readability studies revealed that many L2 studies drew from similar sources, such as the Global Language Online Support System (GLOSS) platform. In contrast, L1 studies tended to use unique corpora and, consequently, unique annotation scales, which led to different results from those of the L2 studies. This difference could be attributed to the generally larger size of the corpora used in the L2 studies. Nonetheless, it was noted that having a larger corpus size does not necessarily lead to significantly improved accuracy. Therefore, data are probably not the main reason for the observed differences in the outcomes of Arabic readability studies [59]. In terms of clarity and accessibility, [71] noted that L1 corpora are not as transparently detailed or publicly available as L2 corpora, such as Aljazeera. Arabic readability corpora include those used to formulate Arabic readability formulas, such as those in [72,73]; corpora used to design tools for, and evaluate the readability of, health-related Arabic information [74,75]; and corpora used in data-driven studies, such as [71], which was used in [76]. Details on specific Arabic corpora used for Arabic readability assessment tasks are provided in Appendix A.

3.2.2. Eye-Tracking Corpora

Extensive efforts have been made to model and understand human reading patterns. However, research often overlooks how general text properties affect eye movements in real-world reading, focusing instead on a few linguistic features. Recently, new datasets tracking eye movements across continuous text have been developed, increasing interest in naturalistic eye-tracking data. These datasets are crucial for setting benchmarks for eye movements, testing models like E-Z Reader and SWIFT, and assessing psycholinguistic theories [50,77]. Studies [15,19] have surveyed existing eye-tracking datasets in various languages, mostly in English, but also in Chinese, Dutch, German, Persian, and Russian. Some studies have recorded eye movements during normal reading, while others have documented eye movements during various NLP tasks. While some datasets are publicly accessible, others require permission for use [15].

One pioneering dataset is the Dundee Corpus created by Kennedy et al. [18] in 2003. It examines how peripheral vision influences the time spent focusing on the current word (parafoveal-on-foveal effects). The Dundee Corpus contains 56,212 words and 9776 unique word types for English, as well as 52,173 words and 11,321 unique word types for French, from newspaper articles read by ten individuals in each language [14,58]. This corpus provides substantial insights into reading under naturalistic conditions.

Another significant dataset is the Ghent Eye-Tracking Corpus (GECO) developed by Cop et al. [20] in 2017. It includes eye-tracking data from monolingual and bilingual readers navigating Agatha Christie’s The Mysterious Affair at Styles in English and Dutch. The dataset includes 54,364 words and 5012 unique word types for English, as well as 59,716 words and 5575 unique word types for Dutch. GECO is publicly available, while the Dundee Corpus is accessible only for research purposes. GECO offers forty-six pre-extracted gaze features, emphasizing word-based processing, whereas Dundee provides raw gaze data [77].

In 2017, Luke and Christianson [78] introduced the Provo Corpus, focusing on the impact of word predictability on reading. This publicly available corpus includes fifty-five brief paragraphs with 2689 words and 1197 unique word types, read by eighty-four native-English-speaking participants [50,77,79].

The Zurich Cognitive Language Processing Corpus (ZuCo 1.0), created by Hollenstein et al. [22] in 2018, integrates both eye-tracking and EEG data. Data were collected from twelve native English speakers reading 1107 English sentences, totaling 21,629 words and 7099 unique word types. ZuCo 1.0 includes both natural reading and task-specific reading. ZuCo 2.0 [80], built in 2020, differs from ZuCo 1.0 mainly in its experimental procedures. It encompasses 739 English sentences with 15,138 words and 4849 unique word types read by eighteen participants. ZuCo 2.0 is the first corpus to simultaneously capture eye movements and EEG, merging standard and task-specific reading for detailed analysis and comparison.

Other eye-tracking corpora are available for various languages, including Portuguese [81,82], Chinese [83], and Danish [84], providing insights into human reading behavior and aiding in understanding cognitive processes. This information is invaluable for various research fields focused on understanding human reading patterns.

3.3. Discussion

Readability studies’ success hinges on large, well-graded, gold-standard datasets, which are currently scarce for Arabic. Language resources are essential for computational language analysis in NLP. Arabic readability corpora are mainly designed for educational purposes, aiding in selecting texts for different reading proficiency levels and helping educators design age-appropriate materials [70]. While Arabic readability corpora exist, they primarily focus on L1 and L2 learners, with limited resources for general public readability assessments, leading researchers to create their own corpora. While Arabic L1 readability assessments offer opportunities for resource collection and tool development, their application extends beyond academia to fields like politics, law, healthcare, and marketing, though these efforts are still emerging [59]. Additionally, although making informational texts more accessible is crucial, Arabic L1 readability corpora are not as well documented or available as L2 corpora like GLOSS or Aljazeera Learning [59,71]. Furthermore, most Arabic readability corpora are annotated by experts, which may not fully represent the target audience’s needs. Thus, creating a large, representative corpus annotated by the target audience is essential for improving model performance and accurately reflecting text readability levels. In terms of eye-tracking corpora, there is a significant lack of human reading eye-tracking corpora for Arabic, hindering advances in cognitively driven readability assessment [10]. While various eye-tracking corpora exist, there is a notable gap for right-to-left languages like Arabic. Existing cognitive corpora often balance participant numbers with corpus size, with larger texts involving fewer participants. Some also include linguistic feature annotations alongside cognitive data. Traditionally, eye-tracking corpora have focused on short, isolated sentences. Still, new datasets where participants read extended passages of natural text offer deeper insights into eye movements and cognitive loads during reading [79]. This shift to naturalistic reading enhances the analysis of longer, more complex reading processes. Accordingly, this paper addresses this gap by establishing an Arabic human reading corpus akin to the Dundee Corpus [18].

4. Methodology

This section outlines all the investigative phases conducted to achieve the objectives of this study: the construction of an eye-tracking corpus. Given the detailed nature of the corpus-building steps, they are divided into three subsections: corpus preparation, data collection, and data preparation. Figure 2 provides an overview of these phases and the applied methodology.

4.1. Corpus Preparation

This section highlights the foundational tasks for gathering eye-tracking data: recruiting participants, preparing experimental texts, and defining and testing readability guidelines. This study’s corpus-building approach was user-centric, involving participants throughout the process. Figure 3 summarizes this section’s main components.

4.1.1. Identifying Participants’ Criteria

This paper centers on Arabic L1 readers, aiming to assess text readability using the cognitive signals of native speakers. Participants had to be native Arabic speakers. To ensure consistency, participants had to meet several criteria [25,29]: they had to be male or female, aged 20–50 years, and educated Arabic readers holding or pursuing degrees in Arab countries. They came from various professional backgrounds, and to avoid bias toward expert linguists, they were from diverse Arabic-speaking countries. Additionally, their reading skills were assessed using the Avant Arabic Proficiency Test (Avant APT) (https://avantassessment.com). This approach aimed to reduce variability and ensure relevant data collection for the corpus.

4.1.2. Defining Different Aspects of the Participants’ Tasks

This study’s corpus was built through a user-driven process that involved the participants in most of the phases. This section includes an overview of the settings used in the corpus construction. This included various aspects of the participants’ tasks to ensure consistent and unbiased results. For example, instructions were provided both verbally and in writing to avoid behavioral variations, and texts were organized and randomized to prevent bias and maintain balanced reading experiences. Additionally, multiple participants were used in each task to reduce bias, and each task began with a survey to capture demographics and reading habits. Additionally, quality control was crucial for ensuring work quality and participant engagement. Participants were informed about these techniques beforehand to encourage attention and prevent skipping parts of the texts. The study used two methods for quality assurance [85]: follow-up questions and random texts. Participants answered true/false and multiple-choice questions after reading texts to assess task completion quality, providing clear, unambiguous answers [27,86]. Random texts with specific labeling instructions appeared unpredictably to maintain engagement and prevent skimming, ensuring sustained reader attention without excessive questioning [27].

4.1.3. Collecting and Testing Corpus Texts

This step involved collecting texts suitable for readability assessment [87]. Based on a study of 41 Arabic-speaking participants who favored books over newspapers [26], the corpus was built from Arabic books to reflect diverse styles and knowledge sources. MSA texts were sourced from Hindawi Foundation books [88], and CA texts were sourced from the King Saud University Corpus [89], covering authors from the 8th to the 21st centuries. To capture author style without requiring participants to read entire books, short representative excerpts (600–700 words) were selected, such as introductions or standalone sections. Thirteen topics—including grammar, literature, health, and politics—were chosen to ensure variety and engagement [60]. Texts were selected without predefined readability levels, and the open-source metric for measuring Arabic narratives (OSMAN), the Arabic readability tool (v3.0, 2020), was used to ensure variation [20,80,90]. Despite being published, MSA texts required minimal revision, whereas CA texts required more extensive processing. Using the Qalam tool and support from two linguists, preprocessing steps included the normalization of formatting (e.g., traditional layout and Uthmani script) [87], proofreading for spelling and punctuation without changing vocabulary [87], and applying appropriate diacritics—partial for MSA and CA texts and full for Quranic verses and sayings of Prophet Muhammad [33]. Additional adjustments were made to correct paragraph structure, clarify indentations, and standardize punctuation by referencing reliable sources, such as Al Shamela Library (https://shamela.ws) and Wagfeya Library (https://waqfeya.net).

To minimize selection bias and ensure texts were suitable for eye tracking, 40 participants used Google Forms to evaluate the texts and ensure they were self-contained. Over two rounds, participants assessed 94 texts (62,893 words in total), marking whether each text met the condition and explaining if not. Round 1 included 30 MSA and 32 CA texts, and Round 2 focused on 32 new MSA texts. Based on feedback, adjustments were made, novels were excluded, texts were modified or replaced for clarity, more translated and topic-diverse texts were added, and some were shortened to suit the experiment length. The Round 1 results revealed a sharp contrast in difficulty levels, prompting a further diversification of the MSA texts. The final corpus—comprising both MSA and CA texts—totaled 58,045 words and was used in eye-tracking experiments. Quality control was ensured through random text insertion and engagement checks, following the approach in [86].

Table 1. Corpus statistics.

Book Language	Document Count	Paragraph Count	Sentence Count	Word Count
MSA	60	442	2835	37,486
CA	32	145	1936	20,559
Total	92	587	4771	58,045

displays statistics for the collected texts, adjusted according to the results of the tests in Rounds 1 and 2.

4.1.4. Paragraph Segmentation

Previous Arabic readability studies have mostly focused on either individual sentences or entire documents; paragraphs have often been overlooked. However, assessing single sentences can lack context and may not reflect real-life reading behavior, especially when analyzing eye movements such as regression [50,91]. On the other hand, evaluating full documents requires a large corpus and significantly more reading time. Paragraphs strike a balance: they present complete ideas, provide enough context for readers to judge readability accurately, and are manageable for eye-tracking experiments [19]. For this reason, this study focused on paragraph-level readability. Documents were segmented into meaningful paragraphs, each expressing a single idea [39]. Texts were carefully segmented by two linguists, and any disagreements were resolved by a third expert.

4.1.5. Extracting and Testing Arabic Readability Guidelines

A clear gap in Arabic readability guidelines—particularly for native speakers—was identified through a review of previous studies. Most existing research has focused on specific grade levels or second-language learners [92,93], whereas this study aimed to develop general guidelines for Arabic books based on how texts are perceived by everyday readers. Rather than relying solely on expert opinions, a user-centered approach was adopted to assess readability from the reader’s perspective.

In this phase, 28 MSA texts from Round 1 were annotated by 24 participants at both the document and paragraph levels (212 paragraphs in total). Examples of features for each readability level were provided by a linguist to support the annotation process. Initially, four readability levels were tested, but feedback indicated that three levels were more effective and easier to apply. Uncertainty was frequently expressed by participants when determining readability, highlighting the need for guided annotation. Based on participant input, readability guidelines were developed to capture the key characteristics of each level; this offered a useful resource for future applications. For quality control, five random texts were embedded within the corpus. Participants assessed paragraph readability at four distinct levels: (a) an easy paragraph, (b) a medium-easy paragraph, (c) a medium-difficult paragraph, and (d) a difficult paragraph. In addition, they provided reasons for their choices.

To validate the readability guidelines, eight participants annotated 47 paragraphs from nine randomly selected MSA texts chosen using the OSMAN Arabic readability tool to avoid bias. They completed three tasks—rating readability across three, four, and five levels—using Google Forms during a two-hour session with a break. A think-aloud protocol was applied, allowing participants to explain their reasoning while reading and offering insight into how they interpreted the texts and guidelines [94]. They also read half the texts silently and the other half aloud to determine their preferred mode for future eye tracking.

The findings revealed a trade-off: while the inclusion of more levels provided detail, they also caused confusion. A three-level scale offered a good balance of clarity and usability and was chosen for the eye-tracking phase. Participant feedback led to guideline refinements, including clearer notes on pronunciation difficulties, especially with diacritics and loan words. A linguist reviewed the updated MSA guidelines (Appendix B) [23]. Regarding the reading mode, six participants preferred reading aloud for better comprehension, while others favored silent reading to stay focused. Diacritization was further enhanced for uncommon and heterophonic–homographic words. Two quality control methods were also tested: random texts and comprehension questions. While preferences varied, five participants recommended combining both—a question to prompt focus, followed by a random text to maintain engagement.

4.2. Data Collection

This section outlines the process of collecting eye movement data from Arabic participants, including the design and testing of the eye-tracking experiment, data acquisition, and labeling. Matching the text difficulty to the reader’s level is crucial for learning [76]. Labeling texts with readability levels is essential for readability assessment models [87]. NLP has recently focused on annotating data based on user interactions [86]. By engaging real audiences and using eye-tracking technology, this study identified factors affecting text readability through eye movements. This section details the steps to build the first Arabic cognitive corpus using eye-tracking data, as shown in Figure 4.

4.2.1. Pilot Testing the Eye-Tracking Experiment

Based on preliminary findings on Arabic readers’ preferences and behavior, this stage involved a hands-on eye-tracking test to understand reading behavior and fine-tune the experimental procedure. Previous studies highlighted the importance of pilot experiments and participant feedback for reliable results and participant engagement [42,85,86]. The eye-tracking experiment involved three participants, each in three sessions lasting about three hours. Factors considered included instruction clarity, experiment flow, text design features, calibration, quality control, reading mode, session duration, and breaks [7]. While justifying Arabic text enhances readability by adhering to traditional rules, this study focused on textual components. Additionally, the impact of diacritics on readability was also examined. Based on Hindawi’s suggestion and input from two linguists, partial diacritization was selected. Six Arabic natives read nine texts with full, partial, and no diacritics. The results showed the highest reading duration and fixation with full diacritics, followed by partial, and then none, aligning with participant feedback on visual noise and reading speed [24]. Partial diacritics struck a balance, confirming previous findings [24,30,31,60]. To avoid redundancy, detailed procedures of this pilot step were omitted. Based on these pilot tests, the actual eye-tracking experiment was designed and conducted.

4.2.2. Designing the Eye-Tracking Experiment

Tobii Studio software Version 3.4.8 (https://www.tobii.com) was used to record, display, manipulate, and export the eye-tracking data in a Microsoft Excel (.xlsx) file format for further analysis. The design of the experiment involved deciding on its flow, selecting the reading materials, and setting up the experimental environment.

Apparatus and Setup

The eye movement data were recorded using the Tobii X120 eye tracker (Tobii Technology, Inc., Danderyd, Sweden), which has a 120 Hz sampling rate and 0.5-degree precision. Participants could move their heads freely within a 30 × 15 × 20 cm area at a 60–65 cm viewing distance [41,44,45]. Figure 5 shows the eye-tracker setup within the participants’ viewing range.

The eye tracker was placed below a Samsung T35F monitor (Samsung Electronics, Suwon, South Korea), which has a 75 Hz refresh rate and a resolution of 1920 × 1080. Adjustable chairs ensured participants’ eyes were centered for accurate calibration. Positioning parameters were entered into the X120 Configuration Tool in Tobii Studio for each participant. Adjustments to the eye-tracker angle, participant position, and seat height were made before each session. Experiments took place in a well-lit room without direct sunlight or reflective surfaces to minimize distractions [41,44].

Materials

The process began with the entire dataset, including MSA texts from both rounds in Section 4.1.3 (Collecting and Testing Corpus Texts). The texts were displayed right to left, but Tobii Studio’s issues with Arabic text alignment required using images for the experimental materials. Each text appeared as a sequence of images, with one per paragraph. Paragraphs longer than eight lines were split across multiple screens [19,91]. Each paragraph had a title and unique identifier (e.g., GeoAndTra_T2_P4_p1). Sixteen reading sets were created: The first was a practice set marked as a dummy and excluded from the analysis. The remaining fifteen sets included four texts from different topics in varied and randomized orders to ensure diverse and balanced reading sets for each session [13,57,78,95].

4.2.3. Setting Up the Eye-Tracking Experiment

The study was conducted according to King Saud University’s Institutional Review Board rules and regulations and was approved by the Ethics Committee of King Saud University (no. 21/0892/IRB, dated 19 October 2021).

According to [42], gaze patterns vary by task, so clear communication is crucial for accurate observations. Research in [22,23,39,84] shows that aggregated non-expert assessments can match expert quality [96]. Thus, this study included subjective readability annotation in the eye-tracking experiment using guidelines for three readability levels from the previous section [92]. Participants performed two tasks. In Task 1, they read the paragraphs silently while the eye tracker captured their behavior, and then they clicked to proceed [18,19,20,78,95]. In Task 2, they rated readability on a scale of easy, medium, and difficult after each paragraph and document, following methods similar to [97,98].

Guidelines were available throughout the experiment to ensure accurate results. Combining these annotations with eye-tracking data provided a comprehensive understanding of user behavior and context. Figure 6a–c illustrate the tasks and annotation levels. This study was conducted between August 2022 and February 2023. Participants received a Google Form link in advance to collect demographic information and data on visual impairments, as well as to review the annotation procedure and guidelines. Upon arrival, participants were briefed on the texts and the experiment, with time for questions. Key instructions included displaying each paragraph only once, focusing, and limiting head and body movements for data accuracy. Participants read at their own pace with no time limit. Minor disruptions were acceptable unless calibration was lost, in which case re-recording was necessary. Breaks were provided after each reading set.

Participants provided consent for the anonymous recording and use of their data, with the option to withdraw [92]. They were seated comfortably in front of the screen and eye tracker, and the device was calibrated for accurate gaze capture [42]. Calibration involved following points on the screen in a five-point setup, ensuring both eyes were within the optimal range (60–65 cm) [99]. Satisfactory calibration was indicated by short green lines for both eyes. The experiment began with onscreen instructions, and participants used the mouse for navigation to minimize drift.

4.2.4. Conducting the Eye-Tracking Experiment

To familiarize participants with the task, a practice set with two sample texts was introduced, helping them adjust to the eye tracker [20,44]. Participants reported clear visibility of words and diacritics during this practice [23]. The main session involved sequential reading tasks (Task 1 and Task 2) for each document that was recorded by the eye tracker. An observer was present initially but left quietly to minimize distractions. Each session began with a five-point calibration (sixteen per participant) to ensure gaze accuracy. Data collection spanned two days, with sessions lasting three to four hours, including regular short breaks and a longer break after four sets. Binocular tracking analyzed data from both eyes. Texts were displayed in black traditional Arabic font on a white background of size eighteen with expanded character spacing and clear line separation. Margins of 0.2 cm on both sides ensured accurate eye tracking at the edges [99].

4.2.5. Participants

Data on the eye movements of fifteen healthy adults (seven male and eight female; ages 20–45 years) were collected. The primary cognitive data came from at least ten participants, following the benchmark set by the Dundee Corpus for English. This number was able to increase during the experiment. None of the participants had learning disabilities, neurological issues, or reading problems [24].

Table 2. Details of the eye-tracking experiment participants.

Participant Code	APT Score	Gender	Age Range	Country of Origin	Academic Level	Major
P01	9	F	25–30	Saudi Arabia	Master’s	Biochemistry
P02	9	F	25–30	Sudan	Bachelor’s	Business Administration
P03	9	F	40–45	Egypt	Bachelor’s	Arabic Linguistics
P04	10	M	30–35	Saudi Arabia	Master’s	Information Systems
P05	9	F	35–40	Sudan	Master’s	Biotechnology
P06	9	F	35–40	Saudi Arabia	Doctoral	Arabic Literature
P07	9	M	35–40	Saudi Arabia	Bachelor’s	Information Systems
P08	9	F	30–35	Jordan	Master’s	Mathematics
P09	10	F	30–35	Sudan	Bachelor’s	Computer Engineering
P10	9	M	40–45	Saudi Arabia	Master’s	Internet Communication
P11	9	M	25–30	Palestine	Bachelor’s	General Management
P12	9	M	40–45	Saudi Arabia	Bachelor’s	Computer Engineering
P13	9	M	30–35	Syria	Bachelor’s	Healthcare Management
P14	9	M	30–35	Saudi Arabia	Doctoral	Dentistry
P15	9	F	20–25	Yemen	Bachelor’s	Chemistry

details the participants. No participants were excluded based on the APT, as these tasks were preparatory for the eye-tracking experiment, and there was no clear exclusion threshold. The median APT score of previous participants, including those from Section 4.1.3 (Collecting and Testing Corpus Texts) and Section 4.1.5 (Extracting and Testing Arabic Readability Guidelines), in addition to the two pilot tests, was used to establish a threshold of nine for inclusion in the eye-tracking experiment.

All participants satisfied the conditions outlined in Table 2, achieving a minimum APT score of 9, which indicates an advanced–low level with the potential for improvement to superior. This was expected as the participants were highly educated native Arabic speakers, ensuring they could easily interpret the materials [23]. Most participants had normal vision and did not require corrective lenses, while a few reported slight reductions in visual acuity or mild astigmatism and used eyeglasses or contact lenses accordingly. Additionally, none of the participants reported any reading problems. According to Tobii guidelines, the eye tracker effectively monitors individuals regardless of ethnicity, age, or use of eyeglasses/contact lenses, even under varying light conditions. Participants were advised to avoid wearing reflective eyeglasses and makeup to prevent tracking issues. Individuals with exclusion criteria, such as wet eyes (epiphora), were excluded.

4.2.6. Results

Sessions 1 and 2 for MSA Texts

During the eye-tracking experiment, each reading set consisted of four texts recorded in one session. The quality of these recordings was evaluated based on gaze sample percentage, calculated by dividing the number of valid gaze samples by the total number of samples and multiplying by 100. Factors like excessive blinking, inaccurate configuration, and distractions can reduce this percentage [41]. A 100% gaze sample is rare, and the initial acceptance threshold was set at 70% [44]. Analysis after the first session, which included eighty recordings, showed gaze sample percentages ranging from 84% to 98%, indicating that participants were still adjusting during the practice set. Recordings above 80% showed minimal drift and were considered acceptable. Those below this threshold were discarded [41,67].

Among the 160 recordings from the first and second sessions, three participants had less than 80% gaze sample scores. Specifically, participant P01 had two low recordings at 78% and 51%, P04 had seven recordings ranging from 33% to 76%, and P06 had one recording at 75%. To improve accuracy, these participants were asked to repeat the recordings after a one-month interval [24]. Although the percentages improved, P04 was replaced due to persistent low recordings, indicating potential eye issues. The study initially included twenty-three participants, but technical issues and personal reasons led to the exclusion of several people, leaving eighteen participants with complete recordings, averaging 93% gaze point accuracy. In total, 288 recordings were made, with participants rating each paragraph and document as easy, medium, or difficult, as summarized in

Table 3. Distribution of readability ratings at the document and paragraph levels with MSA texts.

Readability Level	Documents	Paragraphs
Easy	22	297
Medium	38	139
Difficult	0	6
Total	60	442

.

Participant observations and feedback indicated that even with annotation guidelines [92], participants often defaulted to the medium rating when uncertain about a text’s readability. They struggled to differentiate between medium and difficult levels and hesitated to use the difficult rating. Rating entire documents was especially challenging, resulting in imbalanced readability ratings due to varied paragraph difficulties within documents. To determine a common readability level, the median was chosen over the mode for its reliability with categorical ordinal data in small samples [100,101]. The mode risked misclassifying texts as easy when many were perceived as medium or difficult. The analysis of ratings shown in Table 3 suggested that MSA texts were more suitably classified into two readability levels instead of three.

Session 3 for CA Texts

To address the imbalance in text readability, CA texts, known for their challenging language and vocabulary from the pre-Islamic era [26,70], were used to represent a more difficult level. Participants from Round 1 found the CA texts to be tougher than the MSA texts, aiding in clearer readability distinctions. MSA texts were anticipated to represent the first two readability levels, while CA texts would cover the third, ensuring a comprehensive range of difficulty levels. With a linguist’s help, guidelines for CA texts were adapted. The third session, focusing on CA texts, lasted three to four hours with one practice set and eight reading sets. This session involved fifteen participants who completed 135 recordings, with one needing a repeat due to technical issues. The session achieved an average gaze point score of 94%.

After the three sessions, data from all 375 recordings (240 from the first two sessions and 135 from the third) were analyzed. The results, shown in

Table 4. Distribution of readability ratings of ninety-two documents by fifteen participants.

Readability Level	MSA	CA	Total	Percent (%)
Easy	22	4	26	28.26
Medium	38	22	60	65.22
Difficult	0	6	6	6.52
Total	60	32	92	100

and

Table 5. Distribution of readability ratings of 587 paragraphs by fifteen participants.

Readability Level	MSA	CA	Total	Percent (%)
Easy	297	59	356	60.65
Medium	139	60	199	33.90
Difficult	6	26	32	5.45
Total	442	145	587	100

, indicate an under-representation of the difficult level in both documents and paragraphs, with only six documents categorized as difficult. This imbalance persisted as participants, uncertain about the correct level, frequently chose medium and hesitated to select difficult, despite struggling with the content, reflecting a tendency to avoid potential overrating. On the other hand, there was a noticeable tendency to rate paragraphs as easy, and to a lesser extent, entire documents. This could be because evaluating paragraphs in isolation might seem simpler than annotating full documents, which can contain a mix of easy, medium, and difficult passages.

This imbalance, with paragraphs often rated as easy and documents as medium, raises concerns about classification models favoring these predominant categories. Imbalanced datasets are common in real-world scenarios and pose challenges for automatic text readability assessments, as highlighted in [102]. Many Arabic readability studies have encountered issues with data imbalance. This can be addressed through various methods, such as data augmentation and cost-sensitive training. Further analysis of model-level solutions and a comparative evaluation of imbalance-handling techniques will be presented in future work. Figure 7 illustrates the distribution of documents and paragraphs across three readability levels in a bar chart.

4.2.7. Quality Control

To ensure full engagement during silent reading, participants were prompted with comprehension questions [18,19,20] and random texts [27] based on prior findings. They were aware that they would be questioned at intervals and informed about quality control measures. Each reading set included either or both measures to maintain concentration. Some documents, chosen at random, were followed by multiple-choice [19,20] or true/false questions [23,95], with the number of questions adjusted based on text complexity and length; more complex or longer texts had two questions, while shorter texts had three. Random questions accompanied detail-rich texts to avoid overwhelming readers, with their locations randomized. Periodic checks ensured thorough reading. The results are detailed in

Table 6. Participants’ performance regarding the quality control measures.

Participant Code	No. of Correctly Answered Questions (CA)	No. of Correctly Answered Questions (MSA)	Total	Percent (%)
P01	14	28	42	77.78
P02	17	27	44	81.48
P03	15	32	47	87.04
P04	12	25	37	68.52
P05	15	34	49	90.74
P07	11	23	34	62.96
P08	12	25	37	68.52
P09	14	25	39	72.22
P10	16	31	47	87.04
P12	13	29	42	77.78
P15	16	26	42	77.78
P16	14	28	42	77.78
P17	14	26	40	74.07
P18	12	31	43	79.63
P19	14	28	42	77.78

, which includes the outcomes of interactions with fifty-four quality control measures for the MSA and CA texts. The data show that participants correctly answered 77.41% of the questions, closely aligning with the 78.27% reported by GECO [20].

The discussion on using comprehension questions and setting participant thresholds in reading studies lacks clear standards. In this study, a 60% threshold on fifty-four questions was deemed sufficient, based on task complexity, participant engagement, and expert consultations in gaze behavior dataset creation and human annotation [19,39]. Factors influencing this decision included the complexity and duration of tasks potentially affecting recall, as each session lasted three to four hours with twenty-five recordings per participant. Participants’ diverse learning styles meant variability in correct responses; some easily recalled numerical data, while others struggled with names. Controlled reading confirmed the engagement, as cursor movements indicated active reading. An expert advised retaining all gaze data to create a comprehensive dataset, including all gaze behavior and comprehension scores. Notably, annotating texts across varying readability levels also enhanced focus, serving as an additional quality control measure.

4.3. Data Preparation

To effectively use the collected eye-tracking data, thorough preparation was necessary, involving tokenization, extracting eye-tracking and textual features, and preprocessing to remove irrelevant information and filter out data captured during track loss. After cleaning, the data allowed for an analysis of the correlation between eye movements and text readability levels. Tobii Studio software facilitated the display, manipulation, and analysis of the behavioral data. Figure 8 summarizes the key components of this process.

4.3.1. Tokenization

Tokenization divides texts into individual words and was crucial for this study [70]. In Tobii Studio, tokenization requires manually defining areas of interest (AOIs) around each word to capture accurate eye metrics, such as time to first fixation, saccades, and regressions [15,19,41]. Typically carried out by white-space tokenization [26,43,70], this process was manually executed for the 57,617 words (excluding book titles) in the text images. Punctuation was attached to preceding words, and stop words were retained to preserve comprehension. Manual adjustments were essential for poems or irregular text formats. Figure 9 showcases word segmentation, with the rectangles defining individual words, each associated with an ID. Colors are assigned to tokens randomly.

4.3.2. Feature Extraction

This section discusses how text features and reader characteristics influence readability, as supported by readability studies and linguistic reading theories [7]. Features influencing readability are divided into three categories [39]:

1.

Text-based features represent the general characteristics or linguistic complexity of the selected texts.

2.

Gaze-based features, derived from eye-tracking experiments, reflect cognitive processing and comprehension through established eye-tracking metrics.

3.

The readability level feature represents the combined subjective readability ratings of paragraphs and documents, as provided by participants during the eye-tracking experiments.

Both text and gaze features are included in the Arabic cognitive corpus, providing a dual perspective on text readability from both the text’s properties and the Arabic L1 readers’ interactions. This holistic approach improves upon traditional readability corpora, which typically rely solely on expert opinions and may be less practical [103]. Each feature category and the methods for extracting these features are summarized in Figure 10.

Text-Based Features

Text-based features associated with readability vary by extraction methods and the complexity levels measured [104]. As shown in

Table 7. Text-based features.

Features		Subfeatures
General	Descriptive	Book Name and Author, Document Code, Paragraph Code, Book Language, Book Topic, Publication Century, Authorship Type, Translation Type, Author Count, Text Source
	Textual Components	Parenthesis Count, Parenthetical Expression Count, Numerical Content Count, Listing Count, Religious Text Count, Poem Verse Count
Linguistic	Textual Complexity	Character Count, Word Count, Average Word Length, Syllable Count, Average Syllables Per Word, Difficult Word Count, Average Difficult Words Count, Unique Loan Word Count, Total Loan Word Count, Unique Foreign Word Count, Total Foreign Word Count, Foreign-Word-to-Token Ratio, Loan-Word-to-Token Ratio
	Structural Complexity	Sentence Count, Average Sentence Length in Words, Average Sentence Length in Characters, Paragraph Count
	Readability Scores	OSMAN Score, Lasbarhets Index Score, Automated Readability Index Score, Flesch Reading Ease Score, Flesch–Kincaid Score, Gunning Fog Score
	Stylistic	Text Style, Script Style, Linguistic Style, Writing Technique

, these features were categorized into two groups: general features and linguistic features. General features summarize the books used in this study, taking a user-centric approach rather than relying solely on expert judgment. In addition to standard text features like character count and average sentence length, features were selected based on the Round 1 and 2 described in Section 4.1.3 (Collecting and Testing Corpus Texts) and Section 4.1.5 (Extracting and Testing Arabic Readability Guidelines), and their impact on participant-reported readability. Linguistic features are shallow or surface features that are basic textual and structural characteristics widely used to gauge grammatical and lexical complexity. This study includes statistical attributes of characters, words, and sentences due to their simplicity and strong correlation with text readability, as well as ongoing interest in readability research [5,7,74,105,106,107,108,109,110].

To enhance text diversity, texts with various features impacting readability were selected. Appendix C shows the details of each feature in the text-based features. All features were calculated for each document and paragraph, except for paragraph count, which was relevant only at the document level.

We aimed to keep the annotation process as automated and objective as possible. Most of the general and linguistic features were extracted using Python scripts (Python 3.12.3) and Arabic-specific tools like PyArabic and OSMAN, with linguists reviewing the results for accuracy. Only a few features—like loan words, foreign words, paragraph count, and stylistic elements—had to be assessed manually, simply because there are not yet reliable tools for these in Arabic. To reduce any bias, two linguists worked independently, and a third stepped in to resolve any disagreements. Given the current limitations in Arabic NLP, this mix of automation and expert review felt like the most practical and accurate approach.

Gaze-Based Features

Text readability focuses on meeting the target reader’s needs [103], and eye-tracking analysis offers insights into what captures visual attention during reading. This study used the intra-saccadic velocity threshold fixation filter in Tobii Studio to identify fixations and saccades based on eye movement speed (degrees per second, deg/s). Fixations were movements slower than 30 deg/s, and data included (x, y) positions of each fixation and timestamps for each gaze point [19]. Choosing the right eye-tracking metrics is crucial and depends on a study’s goals. This study selected key eye behaviors based on past research and Tobii Studio guidelines [13,17,38,41,43,52,56,58,111]. These metrics, popular for capturing cognitive processes during reading, were evaluated for their effectiveness in assessing text readability, particularly in Arabic.

After segmenting texts into tokens and organizing trials, eye-tracking measures for AOIs were extracted and saved in an Excel (.xlsx) format using Tobii Studio. Real-time gaze-based metrics were categorized into two groups: reading metrics and experimental condition metrics. Reading metrics are widely used eye-tracking metrics reflecting cognitive processing during reading. The primary analyzed measures were fixations, saccades, visits, pupils, and click metrics, as mentioned in other studies.

Table 8. Types of eye-tracking reading metrics used.

Metrics	Submetrics
Fixation	Time to First Fixation, Fixations Before, First Fixation Duration, Single Fixation Duration, Total Fixation Duration, Total Fixation Count, Percentage Fixated, Average Fixation Duration, Average Number of Fixations per Word, Fixation Rate
Visit	Total Visit Count, Single Visit Duration, Total Visit Duration, Average Visit Duration, Average Number of Visits per Word
Saccade	Total Saccade Count, Total Saccade Duration, Saccadic Amplitude, Absolute Saccadic Direction, Relative Saccadic Direction, Average Saccade Duration, Saccade-to-Fixation Ratio
Pupil	Pupil Size

summarizes the eye-tracking metrics used. Some of these eye-tracking metrics were extracted directly from Tobii Studio, while others necessitated manual extraction and further calculation after extraction.

Experimental condition metrics include session duration and rating duration, which could contribute to participant fatigue and influence final readability assessments. These metrics represent aspects within the eye-tracking experiment settings distinct from the reading of the experimental texts. They include the evaluation of rating tables following each paragraph and document, as well as the time taken to complete each recording, which consisted of four texts. Appendix D shows the details of each gaze-based feature.

Readability Level Features

All the aforementioned features serve as independent variables (predictors) of the dependent (target) variable, which is the readability level. This level is the aggregated subjective readability of the paragraphs and documents, collected from the fifteen participants during the eye-tracking experiment, with three possible values: easy, medium, and difficult.

4.3.3. Data Preprocessing

Preprocessing the data points is important to enhance their reliability and ensure that the insights derived from them are precise and meaningful. This is essential in later data modeling, as it will influence the model’s performance.

Encoding of Categorical Features

Features in datasets often come as categorical variables rather than merely continuous values. Many ML algorithms necessitate numeric inputs, meaning that categorical data must be converted into a numerical format [100,112,113]. This necessitates different encoding techniques applied to different features.

Data Formatting

In this phase, inconsistencies in the formatting of some features were adjusted. For example, gaze data such as total recording duration were converted from milliseconds to seconds for consistency with the other duration features required in some calculations.

Data Cleaning

In this phase, errors and missing data were addressed through exploratory analyses of quantitative metrics, observing their distribution characteristics, such as normality and skewness. The data exhibited typical imperfections found in real-world datasets. The main aim was to prepare the data for subsequent modeling. The data cleaning process involved addressing various issues with the dataset to ensure accuracy and reliability. For missing values, event types labeled as “Unclassified” by Tobii Studio due to ambiguity in identifying saccade boundaries were excluded, focusing only on definitive classifications of fixation and saccade features. For incorrect values or noise, features such as OSMAN score, Flesch score, Kincaid score, and Fog score, which exhibited unexpected negative scores, were addressed differently. The Kincaid metric was removed due to a high percentage of negative values, while the other metrics had missing values substituted with the median of non-missing values for similar texts. Duplicates, specifically in total saccade duration, were eliminated as this feature was duplicated for each participant due to timestamp calculations. Lastly, non-informative features like book name and author, document code, and paragraph code were removed as they lacked predictive power for determining readability levels.

4.3.4. Corpus Evaluation

Readability prediction, while similar to ML tasks like topic modeling and sentiment analysis, uses more subjective scoring [2]. Unlike tasks with benchmark annotations, readability levels are not definitively correct or incorrect, reducing the need for comparison with expert annotations. However, quality assurance remains crucial as individual participation quality can vary and affect labels [85]. After cleaning the data, the goal was to determine if there was a correlation between the gold-standard aggregated subjective readability annotation and the cognitive annotation from the eye-tracking experiment. The following subsections describe the adopted corpus evaluation approaches, as relying on a single metric can be problematic.

Visualization of Gaze Plots

The method of collecting eye movements in this study focused on normal reading during the readability annotation task. This approach was similar to the natural reading in ZuCo and the Dundee study, in which participants read texts for comprehension, sometimes revisiting words as needed. Figure 11 illustrates three participants’ fixations on a paragraph, showing continuous reading until the end.

This pattern, consistent across all texts in the corpus, resembles typical reading behaviors in standard reading corpora. Unlike task-specific readings with quick skimming, participants in this study exhibited thorough, natural reading, occasionally revisiting words when encountering difficulties.

Interpersonal Consistency in Reading Times

Consistent with GECO and ZuCo, this study analyzed eye-tracking data by aggregating participants’ eye movements, focusing on reading time metrics. Seven metrics were chosen: time to first fixation, first fixation duration, single fixation duration, total fixation duration, total saccade duration, single visit duration, and total visit duration. The histograms in Figure 12 show the distributions of these reading times.

Most reading time variables were normally or approximately normally distributed, except for total visit and fixation durations, which mainly had short readings with a few longer instances. Pearson’s moment coefficient of skewness (G) values in

Table 9. Values of the normality and skewness statistics.

Reading Time Metrics	Coefficient of Skewness (G)
Time to First Fixation	0.512
First Fixation Duration	0.248
Single Fixation Duration	0.190
Total Fixation Duration	4.796
Total Saccade Duration	0.676
Single Visit Duration	0.231
Total Visit Duration	4.881

further support this. Positive G values indicate skewness, with values near zero implying symmetry around the mean. The highest G values for total visit and fixation durations show the most pronounced positive skewness, while other features are closer to zero.

In eye tracking during reading, factors like reading ability, topic familiarity, and reading strategies influence participants’ behaviors. While participants generally exhibit brief fixations, certain words or segments may require longer fixations or rereadings, affecting total visit duration and total fixation duration. Knowing they would assess the text’s readability might have influenced participants’ behavior, leading to multiple revisits and hesitation between classifications like easy or medium, or medium or difficult [22,80]. This likely contributed to longer revisit durations as participants took extra time to decide. The findings from Figure 12 and Table 9 align with reading times from other eye-tracking corpora, such as GECO and ZuCo. Studies on these corpora have shown that even after log transformation, reading times often deviate from a normal distribution and exhibit a rightward skew, especially for total fixation duration and total visit duration.

Association Between OSMAN and Other Features

We anticipated that texts within specific readability levels would show distinct patterns for each duration metric. Following [26], we aimed to correlate validated OSMAN Scores [72], subjective participant annotations, and objective eye-tracker reading time metrics. Our goal was to determine if texts perceived as easy, medium, and difficult by participants displayed common trends in eye movements and OSMAN readability scores.

Table 10. Reading durations in comparison with readability levels and OSMAN scores.

Reading Time Metrics	Easy	Medium	Difficult	Maximum/Minimum
Time to First Fixation	10.990	18.271	20.535	20.535
First Fixation Duration	0.247	0.249	0.259	0.259
Single Fixation Duration	0.242	0.244	0.252	0.252
Total Fixation Duration	0.025	0.026	0.031	0.031
Total Saccade Duration	47.263	79.051	84.421	84.421
Single Visit Duration	0.295	0.301	0.315	0.315
Total Visit Duration	0.026	0.027	0.032	0.032
OSMAN Score	129.586	127.694	125.482	125.482

reveals a predictable pattern: more complex texts required longer reading times, shown by increased fixation and visit durations.

This trend, consistent with Rayner [114] and others [18,36], indicates that as text complexity increases, so does processing time. OSMAN scores also reflected these readability differences, with lower scores indicating more complex texts. These findings confirm our assumptions that texts can be differentiated by their difficulty based on specific eye-tracking patterns.

Based on the observed consistency between the visualization and aggregated readability levels (Visualization of Gaze Plots Section), the interpersonal consistency in reading time metrics (Interpersonal Consistency in Reading Times Section), and the correlation between aggregated readability levels and OSMAN scores at the paragraph level (Association Between OSMAN and Other Features Section), it can be inferred that the generated readability levels and collected eye movements are trustworthy and acceptable.

5. Conclusions, Limitations, and Future Work

This paper marks a significant step toward a cognitively driven Arabic readability model and lays the groundwork for future studies. It gives detailed preparations for eye-tracking experiments, including text acquisition, cleaning, preprocessing into paragraphs, and creating annotation guidelines for assessing Arabic text readability at multiple levels. This process identified key reading factors and produced guidelines based on public input, providing insights into readability levels through direct engagement. The study used two annotation methods, eye-tracking and human-guided annotations, to build the first Arabic cognitive corpus using cognitive sensors. The methodology aimed to authentically capture reading experiences by integrating eye-tracking and human-guided annotations. Participants classified text readability into three levels, easy, medium, or difficult, aiding significant aggregation of text annotations and elucidating readability ratings.

This paper also described the dataset preparation process for the study, starting with the feature extraction of general characteristics, linguistic features, and real-time metrics essential for readability assessment. This was followed by data preprocessing to clean, correct, and format the data. Various cognitive corpus evaluations were conducted to ensure consistency and usability for readability assessment. With the corpus ready, model development and evaluation using different ML and deep learning (DL) approaches make it possible to compare their results. The Arabic cognitive corpus presented in this paper, which includes the experimental texts, participants’ details, and collected eye movement data for all tokens, is available to the Arabic NLP.

When analyzing the experimental results, several limitations should be noted. The small Arabic dataset posed challenges in gaze analysis due to the complexity and resources required for extensive annotation [115,116]. The study’s fifteen participants, while comparable to benchmarks like Dundee [18] and ZuCo [19], highlighted the need for a larger, more diverse pool for robust assessments. Dataset imbalance could be addressed by adjusting training weights to preserve the diversity of human readability judgments [17,117]. Additional techniques, such as oversampling, could also be explored. Refining feature engineering and readability guidelines is crucial for improving model accuracy, especially in distinguishing between medium and difficult texts. Additionally, the models’ performance varied with different text types, indicating the need to test adaptability across various genres to understand the impact of text variability.

The reviewed literature demonstrates the value of combining linguistic attributes with eye-tracking data for various NLP tasks across multiple languages [13,16,95,97,98,118,119,120,121,122]. This suggests a promising direction for future research on Arabic NLP tasks, such as text readability, text simplification, and text summarization. Future work can leverage the Arabic cognitive corpus, which uniquely integrates reading behavior and perceived readability through combined eye-tracking and annotation data. By utilizing both gaze-based and linguistic features collected from the same participants, the potential of merging cognitive and textual signals to improve Arabic readability assessment can be explored. This approach offers opportunities to develop machine learning models that go beyond traditional handcrafted features, using eye-tracking data to gain deeper insights into reading processes and text complexity. Another possible direction for future research is to expand the Arabic cognitive corpus, exploring the impact of text variability on modeling reading difficulties, and analyzing the interaction and importance of linguistic and cognitive features. In addition, incorporating alternative cognitive signals, such as EEGs, and investigating the potential of large language models as alternatives or aids for human readability judgment are recommended for further research.