A Systematic Review of Eye-Tracking Technology in Second Language Research

Abstract

Eye-tracking has become increasingly popular in second language (L2) research. In this study, we systematically reviewed 111 eye-tracking studies published in 17 L2 journals to explore the application and replicability of eye-tracking technology in L2 research. The results revealed eight areas of application of eye-tracking in L2 research, among which grammar and vocabulary were the most frequently examined lines of inquiry. We also identified three types of cognitive mechanisms investigated in L2 eye-tracking studies: attention, higher cognitive processes, and cognitive load. Attention was predominantly measured via fixation temporal indices, while higher cognitive processes were frequently measured by using fixation count and fixation temporal measures. In addition, the measures adopted to assess cognitive load mainly depended on the task type. Finally, with respect to the replicability of the studies, transparent reporting practices were evaluated based on 33 features of replicable studies. We found that more than 95% of the reviewed studies reported less than 70% of the information essential for future replication studies. We suggest that the reporting of the information critical to conducting replicable L2 eye-tracking research needs improvement in transparency and completeness. The implications of this study are discussed.

1. Introduction

Eye-tracking is a real-time data collection method that monitors participants’ eyes while they read texts or view images displayed on a computer screen (Aryadoust and Ang 2021; Godfroid 2019). Over the past two decades, an increasing number of second language (L2) studies have adopted eye-tracking to explore how learners process L2s in the context of various language tasks (Conklin and Pellicer-Sánchez 2016; Godfroid 2019; Roberts and Siyanova-Chanturia 2013). This tool has been frequently used to capture the underlying mechanisms of L2 processing, such as word recognition (e.g., Marian and Spivey 2003) and grammatical processing (e.g., Cunnings et al. 2017; Roberts et al. 2008), and the acquisition of vocabulary and grammar (e.g., Godfroid et al. 2013; Issa and Morgan-Short 2019; Lee and Révész 2020; Pellicer-Sánchez et al. 2021a). Among L2 skills, reading and listening are the two main areas of investigation in the eye-tracking literature (e.g., Aryadoust 2020; Aryadoust and Foo 2023; Aryadoust et al. 2022; Bax 2013; Kho et al. 2023; Kim and Grüter 2021; Low and Aryadoust 2023; Mitsugi 2017; Traxler et al. 2021; Zhou et al. 2020).

There have been two recent research syntheses (Abdel Latif 2019; Godfroid 2019) exploring important topics of L2 eye-tracking research, both showing that eye-tracking has been applied to multiple L2 domains. Godfroid (2019) also provided a list of eye-tracking measures used in L2 research so far. To further this synthesizing work, in the present study, we conducted a systematic review to determine the types of eye-tracking measures used across research domains. More specifically, the first objective of our study is to identify research areas that L2 scholars have focused on, along with measures of gaze behaviors adopted in each research area. Our study differs from the review of Godfroid (2019) in that we developed a framework to classify eye-tracking measures, informed by eye-tracking research across research fields (e.g., Lai et al. 2013). In particular, with the advancement of knowledge and technology, there are emerging eye-tracking measures that might be useful but not yet being applied in L2 research. The study will thus provide insights into how eye-tracking has been used; enable researchers to be aware of the gaps in each research strand; and importantly may open new research frontiers in L2 eye-tracking studies.

Another gap in understanding is the types of cognitive mechanisms explored by L2 researchers and the corresponding eye-tracking measures collected for analysis. Gaze behaviors, which directly indicate visual attention, can reflect a variety of higher cognitive processes, including comprehension and learning (Alemdag and Cagiltay 2018; Rayner 2009; Son et al. 2021). As discussed below, understanding the utility of eye-tracking measures in representing L2 cognitive processes would be of particular interest and use for future research. To this end, this study aims to synthesize a research-based cognitive framework of gaze dynamics, thereby highlighting the utility of eye-tracking measures for diverse research applications. It will also show how eye-tracking measures have been and may be interpreted in various ways across research designs, thus indicating that caution should be exercised in drawing inferences of cognition based on eye-tracking measures.

Finally, informed by ongoing calls for the replicability of primary research, the third objective of this study is to critically evaluate the current state of replicability of L2 eye-tracking studies with a particular focus on reporting practices. Without transparent reporting and sufficient information on empirical research, the replicability of eye-tracking studies would be threatened. With this in mind, improving the methodological rigor and transparent reporting practices by establishing more robust reporting standards is called for in this study. In particular, with the growing interest in methodological quality in L2 research (Norris et al. 2015), it is important to promote higher methodological standards for the use of eye-tracking in L2 research.

2. Eye-Tracking and Cognitive Mechanisms

Gaze behaviors, from the eye fixation to the pupil size, can be modulated by cognition, suggesting that cognitive processes can influence the movement and direction of the eyes during the processing of visual information (Holmqvist et al. 2011; Rayner et al. 2006). Applied eye-tracking researchers have leveraged this notion to investigate various cognitive operations. For example, visual attention, normally conceptualized as a selective process, can be measured by using eye-tracking technology (Aryadoust and Ang 2021; Son et al. 2021). It is generally believed that the eyes are tightly linked to attention when performing cognitively demanding tasks, such as reading (Rayner 2009). Therefore, studies in L2 and applied linguistics have explored, for instance, the allocation of attention to different linguistic constructions while reading (e.g., Godfroid et al. 2013; Issa and Morgan-Short 2019) and attention distribution in the processing of multimodal materials (e.g., Aryadoust and Foo 2023; Low and Aryadoust 2023; Pellicer-Sánchez et al. 2020).

Higher cognitive processes (Wang et al. 2006), such as comprehension and learning, can also be inferred from gaze behaviors (Lai et al. 2013; Rahal and Fiedler 2019; Rayner et al. 2006). Most eye-tracking researchers in applied linguistics and beyond assume that there is a close relationship between what is fixated on and what is processed in the mind, a concept referred to as the “eye-mind assumption” (Conklin et al. 2018; Godfroid 2019; Just and Carpenter 1980; Rayner 2009). Thus, eye movements can be indicative of the moment-to-moment cognitive processes that occur during language comprehension and/or learning. Taking reading as an example, word frequency has been shown to influence word recognition and elicit variations in eye-movement patterns: fixation durations on high-frequency words are shorter than on low-frequency words (Rayner et al. 2006). In addition to eye movements, recent research has also shown that pupil diameters and blinks are valuable indicators of cognitive processing (Eckstein et al. 2017).

However, it is important to acknowledge the limits of eye-tracking, too. Gaze measures cannot be directly linked to specific higher-level cognitive processes and there are bounds as to how much insight eye-tracking can provide (Conklin et al. 2018). Eye-tracking measures are in fact “simply measures of visual behavior (e.g., gaze position and related movements)” (King et al. 2019, p. 6). Therefore, eye-tracking data alone does not show which cognitive processes are operated in the brain (Holmqvist et al. 2011). Put another way, it is not possible to use the gaze behavior recording itself to determine the precise moment at which a word is recognized or integrated to form a coherent understanding of the text (Conklin et al. 2018). Thus, researchers should determine what theoretical variables are operationalized by the collected eye-tracking metrics and properly link gaze behaviors to their assumed underlying cognitive processes.

3. Eye-Tracking Measures

Eye-tracking systems can provide the location, sequence, and duration of eye movements in the areas of interest (AOIs) as well as information on the changes in the pupil and the blinks of the eyes in real time (Holmqvist et al. 2011). There are multiple ways to conceptualize eye-tracking measures. Lai et al. (2013) summarized eye-movement measures based on the scale of measurement used (temporal, spatial, and count) and the type of eye movement (fixation, saccade, and mixed). Temporal measures quantify gaze behaviors temporally and are believed to provide insights into the points at which and for how long cognitive processing is undertaken, as well as into the processing load (Godfroid 2019; Lai et al. 2013). The spatial scale represents eye movements in a spatial dimension and is therefore concerned with locations, distances, directions, or sequences. Spatial measures can provide information as to where and how cognitive processing is undertaken. The count scale quantifies eye movements in terms of number, proportion, rate and/or frequency. Similarly, in SLA and bilingualism research, Godfroid (2019) grouped eye-tracking measures into three overarching categories: fixation, regression, and eye movement dynamics. According to Godfroid (2019), there are four subtypes of fixation-based measures, as follows: (1) fixation counts, probabilities, and proportions, (2) fixation duration, (3) fixation latency (e.g., the duration of time a participant takes before fixating on a specific area of interest), and (4) fixation location. In addition to fixations and saccadic eye movements, measures such as pupil size and blink rate have also emerged in recent studies to explore visual information processing and examine cognitive mechanisms (Eckstein et al. 2017). Drawing on the aforementioned frameworks, the current study groups the commonly used eye-tracking measures identified from language research and related fields of study into eight main categories: fixation, saccade, dwell (visit), regression, skip, pupillometry, blink, and gaze patterns (

Table 1. Definitions of the Eye-tracking Measures.

Eye-Tracking Measure	Definition
Fixation	The periods of time during which the eyes remain stationary on a region.
Saccade	The rapid movements of the eyes between two consecutive fixations.
Dwell (visit)	The period of time during which a participant’s gaze first enters an AOI until exiting that region.
Regression	The backward eye movements during reading.
Skip	The AOI that is never looked at by the participant.
Pupillometry	The fluctuations in the pupil’s size and orientation.
Blink	The rapid closing and reopening of the eyelid.
Gaze patterns	The visualization of the temporal distribution and duration of eye movements.

). The first four types are classified into the abovementioned three scales (Appendix A presents examples and definitions of the commonly used eye-tracking measures).

4. Replicability and Research Reporting Practices

The replication of empirical studies constitutes a crucial method for verifying research findings, uncovering potential biases, and generalizing research findings to different conditions and populations (Makel and Plucker 2014). Although replication is believed to be essential for advancing scientific knowledge, there is a growing concern over the replicability of scientific research across research fields. Notably, the result of a Nature’s survey on 1567 researchers showed that over 70% of researchers tried and failed to replicate another scientist’s study and that more than half of the researchers even failed to replicate their own experiments (Baker 2016). In response to such a “crisis”, various attempts have been made to improve replicability, one of which relates to transparent reporting practices of scientific research.

Transparency in reporting practices involves providing sufficient information about the research procedure and key variables for others to understand, consume and replicate this study (Derrick 2016). Replication studies can be categorized into three types based on the change made to the methodology of the original study. Researchers can opt to duplicate the experimental procedure of the original study (direct replication), change specific facets of the methodology (partial replication) or adopt different methods (conceptual replication) (Makel and Plucker 2014; Marsden et al. 2018). Thus, transparent reporting of the original research is a prerequisite for successful replications.

In L2 research, multiple methodological syntheses have raised concerns regarding the incomplete reporting of instruments (Crowther et al. 2021; Derrick 2016), with there being “a history of inadequate reporting practices” (Marsden et al. 2018, p. 332). Similarly, there is a lack of transparency in reporting eye-tracking studies with respect to key aspects and features of eye-tracking research (e.g., the visual stimuli size, apparatus, eye-tracking data quality and algorithms) in many research fields, as found in the systematic reviews of eye-tracking studies in decision making research (Fiedler et al. 2020), communication science (King et al. 2019) and mathematics education research (Strohmaier et al. 2020). Specifically, Strohmaier et al. (2020) identified “large inconsistencies in the reporting of these methods” (p. 165), while Fiedler et al. (2020) found that many key elements of the eye-tracking research were omitted in reports regarding empirical studies in decision-making research. To promote transparent reporting practices of eye-tracking research, researchers have developed reporting guidelines for eye-tracking studies in several research fields (Carter and Luke 2020; Fiedler et al. 2020), but none of them are specialized for L2 research. This means that key variables of L2 research are neglected in these guidelines.

To our knowledge, there is no research synthesis that has examined the reporting practices of the currently available eye-tracking studies in L2 research. Despite the aforementioned concerns, there is a lack of empirical investigation into the replicability of L2 eye-tracking studies. It is on this basis that the present study aims to examine the extent to which elements critical to the carrying out of L2 eye-tracking studies have been reported in a transparent manner, and thus other researchers can replicate the original studies using the information provided.

5. The Present Study

Systematic reviews are gaining traction in the field of L2 research, because they have clear objectives and are methodologically sound, which are featured with systematicity, rigor, and transparency (Petticrew and Roberts 2008). The present study employs the systematic review approach to synthesize empirical eye-tracking studies in L2 research. This research synthesis sets out to explore where and how eye-tracking has been used in L2 research, including the areas of application, the cognitive mechanisms that have been investigated using eye-tracking, and the types of eye-tracking measures used in those areas and cognitive mechanisms. Moreover, it will critically evaluate the replicability of L2 eye-tracking studies from the perspective of transparent reporting practices. To guide the review, the research questions (RQs) of this study are as follows:

RQ1: What are the main areas of investigation in the L2 eye-tracking literature? What eye-tracking measures have been used in each research area?

RQ2: What types of cognitive mechanisms have been inferred from the eye-tracking measures collected?

RQ3: How replicable are the eye-tracking studies in L2 research?

6. Method

6.1. Study Identification

The dataset for this study was constructed through a sequential process following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Liberati et al. 2009; Moher et al. 2009) in December 2021. In line with the procedures used in previous research syntheses of L2 studies, this study focused on articles published in top-tier peer-reviewed journals because these journals have been considered as the primary means for disseminating high-quality L2 research (Marsden et al. 2018) (see Appendix B for the list of journals). To retrieve research papers that used the eye-tracking method in the selected journals, the Scopus database was chosen to conduct an electronic document search using the “source title” method because it is the largest available database of published studies (Aryadoust et al. 2021; Schotten et al. 2017). The search terms “eye tracking” and “eye movement” were applied using the “OR” Boolean operator. No limitation was set on the year of publications, but the language was restricted to English (Appendix C provides the Scopus search code). The initial search returned 154 articles.

All articles retrieved were screened with the inclusion and exclusion criteria (

Table 2. Inclusion and Exclusion Criteria.

Inclusion Criteria: The Paper …	Exclusion Criteria: The Paper …
1. was published in the selected peer-reviewed journals;	1. was a book chapter, conference proceeding, or dissertation;
2. collected data from L2 (L3, L4, foreign language, artificial language) learners, educators, or materials;	2. did not include data from L2 learners, educators, or materials;
3. used the eye-tracking method;	3. did not use the eye-tracking method;
4. was a primary study that contained empirical data;	4. was secondary research, review, or commentary;
5. was published in English.	5. was inaccessible.

). A total of 111 journal articles were identified for coding and analysis (see Appendix D for the publications included in the systematic review). The PRISMA flow diagram (Moher et al. 2009) in Figure 1 documented the study selection process.

The final dataset consisted of 111 articles, published between January 2003 and December 2021. This dataset was substantially larger than those used in previous reviews of eye-tracking studies in L2 research (Abdel Latif 2019; Godfroid 2019), and could be considered as representative of the domain of interest. As shown in Figure 2, there has been a general upward trend in the number of papers using eye-tracking in L2 research, peaking at 25 in 2021. Since the publication year was not restricted in selection, it indicated that L2 studies that used the eye-tracking method emerged in the selected journals since 2003. This suggests that eye-tracking has become an established research method in L2 research and has gained more attention in recent years.

6.2. Coding

A coding scheme (Appendix E) was developed to derive the information necessary from each study in the sample to address the research questions. In designing the coding scheme, the categories and variables were informed by three main resources: (a) research synthesis guidelines (e.g., Cooper et al. 2019), (b) previous research syntheses of L2 studies and eye-tracking studies (e.g., Crowther et al. 2021; Riazi et al. 2020; Strohmaier et al. 2020), and (c) the eye-tracking literature (e.g., Fiedler et al. 2020; Godfroid 2019; Holmqvist et al. 2011). The initial coding scheme was developed by the authors. Two independent reviewers, with expertise in eye-tracking, subsequently provided suggestions to revise and enhance the categories, variables, definitions, and values within the coding parameters. Thus, the final coding scheme was established through an iterative process in which the authors revised and piloted coding categories and variables, resolved disagreements and amended unclear codes and coding parameters.

Each study was subsequently coded for all the 45 items based on the coding scheme, and the 111 papers were coded by three coders. Firstly, the first author coded 30 papers, and another two independent reviewers each coded half of the coded papers (i.e., 15 papers each). Inter-coder reliability was examined for the coding of these 30 studies.

Table 3. Inter-Coder Agreement Rate for Each Variable.

The Variable	Inter-Coder Agreement Rate
Area of application	83.33%
Cognitive mechanism(s) inferred	80.00%
Eye-tracking measure	93.33%
Sample size	96.67%
Gender distribution	100.00%
Age	96.67%
L1	100.00%
Target L2	100.00%
L2 proficiency	100.00%
Neurological condition	93.33%
Visual condition	100.00%
Hearing condition	93.33%
Research site	93.33%
Visual sitimuli type	100.00%
Font type	100.00%
Font size	100.00%
Text spacing	100.00%
Image size	100.00%
Area of interests	100.00%
Commercial/non-commercial	100.00%
Type	86.67%
Brand/manufacturer	93.33%
Model	100.00%
Data sampling frequency	100.00%
Number of eyes tracked	90.00%
Head movement condition	100.00%
Display monitor	100.00%
Type of software used	96.67%
Name (and version)	96.67%
Eye data source	96.67%
Data quality	100.00%
Data interpolation	100.00%
Noise reduction	100.00%
Techniques for parsing eye movements	100.00%
Fixation threshold	100.00%
Velocity threshold	100.00%

presents the inter-coder agreement rate for each variable in the coding scheme. The inter-coder agreement rate was 96.94%. Subsequently, the authors and reviewers discussed and resolved disagreements. Next, the first author coded 51 papers, and the two independent reviewers each coded 15 papers.

6.3. Data Analysis

The analysis of the codes drew on descriptive statistics. The first research question was addressed by summarizing the research areas identified from the sample. After identifying the research areas, the frequencies and percentages of eye-tracking measure types used across research areas were calculated and summarized. Similarly, the second research question was answered by calculating the frequencies and percentages of eye-tracking measure types applied across the identified cognitive mechanisms. The third research question was intended to examine the reporting practices of the sample by analyzing the number of studies that reported each item. We also created a replicability index which was the function of A/B, where A = the amount of information provided and B = the amount of information needed for an exact replication. It should also be noted that in computing the denominator of the replicability index, we did not count in the sampling frequency and commercial/non-commercial type of the eye tracker, since these pieces of information can be found on the website of the products.

7. Results

7.1. Research Question 1: Research Areas and Eye-Tracking Measures Applied

The first research question was posed to illustrate the eye-tracking measures used across research areas. This was followed by an in-depth analysis of the distribution of eye-tracking research and the prevalence of eye-tracking measures used in each research subfield.

7.2. Span of L2 Eye-Tracking Studies

Figure 3 demonstrates the distribution of empirical L2 eye-tracking studies by research area. The categorization of studies was based on the primary area of research interest expressed by the researchers in the title, abstract, research aims, and research questions addressed by eye-tracking. Overall, eight main research areas and one mixed area emerged from the sample. Grammar (n = 27; 24.3%) is the most researched component, followed by vocabulary (n = 26; 23.4%), and reading (n = 17; 15.3%), while considerably less research attention has been directed to speaking (n = 2; 1.8%) and phonology (n = 2; 1.8%). In addition, as demonstrated in

Table 4. Topics Investigated in L2 Eye-tracking Studies.

Research Area	Topic
1. Grammar	Grammar acquisition and instruction, grammatical processing
2. Vocabulary	Vocabulary acquisition and instruction, bilingual word recognition, formulaic language processing, conceptual transfer, and strategy use
3. Reading	Reading behavior, multimodal reading, and reading test
4. Listening	Predictive language processing, listening test, and prosody
5. Writing	Composing process, computer-mediated communication, feedback, and writing assessment
6. Validity	The validity of eye-tracking method, construct validity, and task validity
7. Speaking	Speaking test and event description
8. Phonology	Visual sonority and phoneme learning
9. Mixed areas

, several research areas consisted of two or more categories, indicating the span of the knowledge base in these research areas. For example, grammar studies tend to focus on grammar acquisition and instruction (e.g., Indrarathne and Kormos 2017; Wong and Ito 2018) and grammatical processing (e.g., Fujita and Cunnings 2021; Keating 2009), while listening research mainly focuses on predicting language processing (e.g., Kim and Grüter 2021; Mitsugi 2017), listening tests (e.g., Aryadoust et al. 2022; Suvorov 2015), and prosody (e.g., Connell et al. 2018; Wiener et al. 2021).

7.3. Eye-Tracking Measures Used across L2 Research Areas

Table 5. Breakdown of Eye-tracking Measures and Number of Studies.

Eye-Tracking Measure		Number of Studies	%
Fixation
	Temporal	71	64.0%
	Count	54	48.6%
	Spatial	2	1.8%
Saccade
	Temporal	0	0.0%
	Count	2	1.8%
	Spatial	3	2.7%
Dwell
	Temporal	17	15.3%
	Count	11	9.9%
	Spatial	0	0.0%
Regression
	Temporal	8	7.2%
	Count	8	7.2%
	Spatial	3	2.7%
Skip		8	7.2%
Blink		0	0.0%
Pupil dilation		0	0.0%
Gaze pattern		6	5.4%
Others		6	5.4%

presents the breakdown of the eye-tracking measures used in the L2 eye-tracking studies reviewed. The most frequently used measure was fixation temporal (n = 71; 64.0%), followed by fixation count (n = 54; 48.6%), dwell temporal (n = 17; 15.3%), and dwell count (n = 11; 9.9%). By contrast, fixation spatial (n = 2; 1.8%) and saccade count (n = 2; 1.8%) were the least frequently applied eye-tracking measures in the sample, while pupil dilation and blink were not employed in any of the studies.

We further investigated the types of eye-tracking measures applied within each research area. Details of the eye-tracking measure types used across the eight research areas and/or subareas are illustrated in Appendix F. Overall, the majority of the research areas and/or the subcategories showed a tendency in the field to use fixation temporal measures. Only a minority of research topics used other measure types more frequently than fixation temporal measures, such as those in bilingual word recognition (in vocabulary) and listening research. Although 12 measure types were found to be employed in the dataset, each research area only utilized a subset of them. The most comprehensive coverage of measures occurred in reading behavior, validity, and mixed areas, wherein seven of the identified measure types were applied.

7.4. Research Question 2: Cognitive Mechanisms

The second research question focused on how L2 researchers used eye-tracking and gaze behaviors to understand different cognitive mechanisms. As previously discussed, eye-tracking provides a means to examine cognition. This capacity was used to study three main types of cognitive mechanisms in the reviewed studies: (1) attention, (2) higher cognitive processes, and (3) cognitive load. We found that 94 studies employed the collected eye-tracking measures to make inferences concerning participants’ cognitive mechanisms, while the other 17 studies used eye-tracking to measure constructs such as viewing time and fluency that were not directly linked to any specific cognitive mechanisms by the authors of the studies (Figure 4).

Attention emerged as the most studied cognitive mechanism in the sample. A total of 51 studies (54.3%) employed eye-tracking to measure, for example, the amount of attention, attentional distribution, and the allocation of attention. The measures used for analyzing attention spanned across eight eye-tracking measure types (Figure 5). The majority of these studies (n = 42; 82.4%) used fixation temporal measures, while 43.1% of the studies (n = 22) employed fixation count and 23.5% of the studies (n = 12) used dwell temporal measures. The other measure types included dwell count (n = 7; 13.7%), gaze pattern (n = 4; 7.8%), skip (n = 3; 5.9%), regression count (n = 3; 5.9%), and saccade spatial (n = 1; 2.0%), which were used less frequently.

The category of “higher cognitive processes” included 46 studies (48.9%), which used eye-tracking measures to infer the moment-to-moment cognitive processes underlying visual information processing, such as the language comprehension processes of lexical access, syntactic parsing, and predictive language processing. The reported measure types in the studies focused on higher cognitive processes spanned across 11 out of 12 measure types identified from the reviewed studies (Figure 6). Among these, fixation count measures (n = 27; 58.7%) were most frequently adopted, followed by fixation temporal (n = 22; 47.8%), regression temporal (n = 7; 15.2%), and regression count (n = 7; 15.2%) measures. The least applied measure types were fixation spatial (n = 1; 2.2%), saccade spatial (n = 1; 2.2%) and dwell count (n = 1; 2.2%) measures, each being reported in only one study. The types of eye-tracking measures used to infer cognitive load are presented in Appendix G.

7.5. Research Question 3: Replicability of L2 Eye-Tracking Studies

The third research question of the study concerned the replicability of L2 eye-tracking studies, whereby we investigated the reporting practices of eye-tracking experiments in the sample. The dataset consisted of 121 separate L2 eye-tracking experiments documented across 111 publications, which were coded for 33 items in six categories associated with the study design, methodology, data collection, and eye-tracking data pre-processing procedures. Frequencies and percentages of different study contexts, participant demographics, apparatus, and data pre-processing procedures were calculated (see Appendix H and Appendix I). To provide a general overview of reporting completeness,

Table 6. The Number of Categories of Information Provided in the Reviewed Studies.

Number of Categories Reported	Number of Studies	%
Three	24	19.8%
Four	57	47.1%
Five	30	24.8%
Six	10	8.3%
Total	121	100.0%

outlines the number of categories of information reported in the reviewed studies. It is apparent from this table that very few studies (n = 10; 8.3%) included all six categories of information (at least one item of each category) in the papers, indicating that over 90% of studies (n = 111; 91.7%) failed to specify at least one category of information. In addition, all reviewed studies reported more than two of the six categories, and nearly half of the studies (n = 57; 47.1%) indicated four categories of information.

We subsequently examined the number of studies that reported full information within each category, the results of which are presented in Figure 7. What stands out in the table is that none of the studies in the sample reported all necessary items. Across the categories, the software used in the L2 eye-tracking studies was specified in 54 studies (44.6%), followed by visual stimuli (n = 19; 15.7%). Surprisingly, no study reported all 10 items in the category of “study context and participant demographics”, and only one study presented information concerning the five items under “data pre-processing procedures” (see Appendix I for a full presentation of the reporting practices).

Finally,

Table 7. Replicability Index.

The Percentage of Variables Reported in Each Study	Number of Studies	%
Below 50.0%	43	35.5%
50.0% to 70.0%	73	60.3%
Above 70.0%	5	4.1%

illustrates the replicability index which is a function of the number of variables reported in each study over the total number of variables necessary for replicability purposes. The number of variables expected to be reported by each study varies slightly based on the modality of the visual stimuli used in the study. Specifically, text-based studies consist of 30 variables and image-based studies include 28 variables, while the number of variables for studies using both text and image stimuli is 31 variables.

It can be seen that only five studies (4.1%) reported more than 70.0% of the variables, while most studies (n = 73; 60.3%) provided between 50.0% and 70.0% of the variables. Notably, 43 studies (35.5%) specified less than 50.0% of the items in their articles, underscoring the prevalent issue of incomplete reporting in previous eye-tracking studies in L2 research. The six groups of variables are unpacked in the following sections.

8. Discussion

The present study synthesized the publications involving eye-tracking in L2 research (N = 111). Below, we will discuss our research findings and their implications for L2 eye-tracking research.

8.1. Research Question 1: Areas of Application of Eye-Tracking

The first research question aimed to investigate the types of eye-tracking measures that had to date been used in different research areas. Firstly, eight major research areas emerged together with one mixed area, showing that eye-tracking has been widely adopted as a tool to explore various language skills and linguistic components by L2 researchers. However, the areas appear not to have been investigated with the same depth and thus, certain areas remain under-researched.

Grammar and vocabulary are the most widely investigated areas of research, resonating with the findings of Godfroid (2019). The results indicate that there has been a high level of interest in applying eye-tracking to study L2 processing at the local level (e.g., word level and sentence level). This pattern may be attributed to two main factors. Firstly, experimental designs in word and sentence processing research are vast and varied: there are a myriad of experimental paradigms, input modes and different language learning stages that could be incorporated with eye-tracking to inform lexico-grammatical processing and learning (Godfroid 2019; Keating and Jegerski 2015). In terms of experimental paradigms for examining grammatical processing, there are four types of design: anomaly detection, ambiguity resolution, syntactic dependency formation, and referential processing (Godfroid 2019; Keating and Jegerski 2015), all of which have been adopted across the L2 studies reviewed here. Furthermore, processing and/or learning grammar and vocabulary have been examined through multiple input modes, including reading, listening, reading-while-listening, and viewing, thus allowing for a considerable flexibility in research design. Eye-tracking has also been utilized to investigate both the learning process (e.g., Montero Perez et al. 2015; Winke 2013) and the learning outcome (e.g., Pellicer-Sánchez et al. 2021a; Wong and Ito 2018). This has significantly extended the applicability of eye-tracking across a variety of research designs.

The second reason for the widespread use of eye-tracking in grammar and vocabulary research is the popularity of studies on the effect of attention on language learning and input processing in L2 research (Indrarathne and Kormos 2017; Issa and Morgan-Short 2019). Eye-tracking can capture the attention paid by learners to various linguistic features with an unprecedented level of precision in a natural and non-invasive way. A growing body of eye-tracking research has emerged in the domains of grammar and vocabulary acquisition and instruction, which employs eye-tracking to examine not only the locus but also the amount of attention paid to various target features. This has enabled the eye-tracking method to function as a viable substitute for traditional and less natural methods such as underlining, think-aloud protocols, and mouse-clicking (Duchowski 2017; Godfroid et al. 2013).

It has also been shown that the number of studies on L2 reading and listening has demonstrated a steady increase in recent years. This may be attributed to eye-tracking’s potential to shed light on how L2 learners process written and auditory input in real-time within naturalistic settings (Dussias 2010; Roberts and Siyanova-Chanturia 2013). Comparatively, eye-tracking has not achieved widespread adoption in the study of L2 writing. This may be attributed to the ongoing exploration by writing researchers regarding the applicability of eye-tracking in studying the writing process, and only recently have some studies aimed to illustrate the affordance of eye-tracking in this area (e.g., Ranalli et al. 2019). The analysis of participants’ eye movements during the composition processes commonly combines a digital screen recording of the process with visualizations of eye movements (Révész et al. 2019). Under these circumstances, the resultant video streams need to be manually annotated by creating moment-by-moment segments. This is a time-consuming procedure, and this methodological complexity can limit the development of L2 eye-tracking writing research. Fixing the screen areas where participants can type in their texts might be offered as a solution, but this can also diminish the authenticity of experiments. It is particularly the case that, in source-based academic writing—wherein authors frequently alternate between texts and/or scroll up and down in texts—an array of cognitive processes such as reading, viewing, skimming, scanning, confusion, confirmation, rebutting and so on are involved which would be extremely difficult, if not impossible, to disentangle in the gaze behavior data. In such cases, writing researchers need to utilize several methods to tap into these processes; for example, Révész et al. (2019) combined keystroke logs, eye-tracking and stimulated recall comment, demonstrating the methodological complexity required in this line of inquiry.

It was also found that the amount of eye-tracking research on validity, speaking and phonology is very limited. In validity research, eye-tracking has emerged as a useful means for validating language tasks or evaluating the construct validity of readability formulas (Nahatame 2021; Révész et al. 2014), illustrating new directions regarding the use of this method. While research on eye-tracking from psychology has provided basic knowledge and assumptions about the application of this method, L2 studies have fundamentally explored phenomena different from other fields, which entails unique experimental designs and features. Thus, it is suggested that more research should be conducted to establish the validity of eye-tracking in L2 research and advance field-specific methodological knowledge. With respect to speaking and phonology, only two studies were identified in each area (speaking, Flecken et al. 2015; Lee and Winke 2018; phonology, Esteve-Gibert and Muñoz 2021; Stone et al. 2018), demonstrating a relative lack of attention to these domains in L2 eye-tracking research and that more scholarly attention should be paid to them. In all, a wealth of research opportunities remains for future studies where the analysis of gaze behaviors can be informative.

8.2. Eye-Tracking Measures Adopted

As discussed earlier, we adapted Lai et al.’s (2013) and Godfroid’s (2019) classifications of eye-tracking measures to identify the measures used in our sample. It is evident from the results that there is an uneven distribution of the types of eye-tracking measures used in L2 research. While a substantial portion of studies in the sample adopted fixation measures, many other measure types were used far less frequently, such as regressions, saccades, blinks, and pupil measures.

The wide application of fixations in a range of subareas of L2 studies may be attributed to the fact that they can reveal much about various cognitive operations and online language processing, such as attention (e.g., Godfroid et al. 2013; Issa and Morgan-Short 2019), reading processes (e.g., Traxler et al. 2021), and listening processes (e.g., Kim and Grüter 2021; Mitsugi 2017). Similar to previous studies in general educational research (e.g., Lai et al. 2013), the results of this study demonstrate that fixation temporal measures have been predominantly applied by L2 researchers, followed by fixation count measures. Specifically, the two research designs that have been widely adopted—reading and the visual world paradigm—mainly focused on the analysis of fixation temporal and count measures, respectively (Godfroid 2019; Holmqvist et al. 2011). We found that the distribution of measure types in the field varies based on the topics under investigation. Again, in topics investigated more often through reading (e.g., grammatical processing and reading behaviors), fixation temporal measures are dominant. By contrast, in areas wherein the visual world paradigm is frequently applied (e.g., predictive language processing), fixation count measures such as the proportion of fixations are commonly used. Therefore, the choice of eye-tracking measures should be informed by research questions, research designs, and theories. In other words, there is no one-size-fits-all paradigm in designing eye-tracking research.

A few types of eye-tracking measures were used in a limited number of L2 studies. In particular, saccades, one of the fundamental measures of eye movements, were one of the least frequently used measure indices alongside eye blinks and pupil measures in L2 studies. However, there is one positive trend showing an increase in the application of saccadic eye-movement measures (e.g., Hung et al. 2020; Pellicer-Sánchez et al. 2021b). This is in contrast to the findings of Godfroid (2019) who found no saccadic measures used in her sample of publications (number of papers reviewed = 84). The increasing use of saccades may be attributed to researchers’ recognition that saccades can reveal participants’ processing of multimodal input (Alemdag and Cagiltay 2018). The results also demonstrate that L2 researchers have expanded the affordances and span of eye-tracking to issues that are commonly addressed using traditional research designs such as interviews and think-aloud protocols.

Combining different gaze behavior measures can better inform different aspects of L2 processing. Nahatame (2021) noted that the information on the global characteristics of gaze patterns during reading is typically indicated by a combination of fixation duration, saccade length, skipping rate and regression rate. However, this pattern is primarily based on findings from L1 reading studies, while L2 research on readers’ global reading behavior remains limited. The application of different measure types has the potential to extend the scope of current research beyond its current fronts in investigating L2 processing and comprehension at the local level (e.g., word level and clause level).

So far, in L2 studies, many eye-tracking measures have been underutilized; this is particularly the case for pupil and blink measures which none of the reviewed studies have employed. As shown in the study by Schmidtke (2018), pupil size can be applied in auditory and orthographic language processing and speech production research. Although less discussed in L2 research, blink rate can reveal processes underlying learning and goal-oriented behavior (Eckstein et al. 2017) or the mental workload (Holmqvist et al. 2011; Perkhofer and Lehner 2019). Overall, more research is needed to shine a light on whether and how the underutilized gaze measures can provide L2 researchers with new evidence.

8.3. Research Question Two

We found that L2 researchers investigated three types of cognitive mechanisms through eye-tracking: attention, higher cognitive processes, and cognitive load.

8.3.1. Attention

As the results show, a large number of the L2 studies (n = 51) utilized eye-tracking to gauge participants’ attention to various visual stimuli, indicating that attention is a central area of application of eye-tracking in L2 research in terms of cognitive mechanisms. This is attributed to the fact that gaze behavior is a more direct, continuous, and objective measure of overt visual attention compared with other available research techniques (Duchowski 2017; Issa and Morgan-Short 2019). Notably, eye-tracking provides researchers with not only the concurrent distribution or spatial information of attention but also the amount of attention paid or temporal distribution, thus offering a unique quantitative and continuous measure for attention, which cannot be obtained through other research methods such as note-taking or mouse-clicking (Duchowski 2017; Godfroid et al. 2013; Son et al. 2021).

With respect to eye-tracking measures of attention, studies included in this review demonstrated a strong preference for adopting temporal and count measures. It indicates that eye-tracking was used to measure the amount of attention allocated to certain areas (e.g., specific linguistic features in the written input, image and word areas in the multimodal input), rather than the sequence and direction of participants’ allocation of visual attention as can be reflected in spatial scale measures. The advantage of measuring attention with fixation and dwell measures using the temporal and count scales is that the interpretations can be straightforward: a higher number of fixations/dwells and longer fixation/dwell durations typically indicate a larger amount of attention (e.g., Alhazmi et al. 2019; Bax 2013; Indrarathne and Kormos 2017). Most studies indeed treat longer fixation time as an indication of higher levels of attention (e.g., Son et al. 2021; Warren et al. 2018). Similarly, higher numbers of fixation counts (e.g., Lee and Jung 2021; Pellicer-Sánchez et al. 2020), longer dwell durations (e.g., Batty 2021; Bax and Chan 2019), and higher numbers of dwell counts (e.g., Bax and Chan 2019; Lee and Révész 2020) are viewed as indicators of higher levels of attention.

There are some other types of gaze behaviors that have been adopted to measure attention, although less frequently, including gaze patterns (e.g., Hung et al. 2020), skipping (e.g., Lee and Révész 2020), regression counts (e.g., Montero Perez 2019), and saccade spatial measures (e.g., Hung et al. 2020). Gaze patterns integrate both spatial and temporal aspects of gaze behaviors, thus providing useful information to draw inferences on participants’ patterns of attention (Rahal and Fiedler 2019). For example, Hung et al. (2020) visualized eye movements to represent attention, reporting that fixation-time-based heat maps provided a holistic view regarding L2 readers’ visual attention to science text with visuals. Nevertheless, this approach is relatively uncommon, possibly because of the time-consuming nature of manually inspecting the gaze patterns of every participant, likely resulting in bias.

Critically, it is important to realize that eye-tracking is restricted to monitoring foveal vision, while humans can process visual stimuli with their parafoveal vision but cannot be recorded by the eye tracker (Godfroid 2019; Henderson 2003). It is on this basis that words and AOIs that are processed parafoveally may be skipped (Godfroid and Hui 2020). Accordingly, it would be beneficial to measure the duration of fixations on words and AOIs that are fixated by participants, as fixations provide positive evidence of attention (Godfroid and Hui 2020).

In summary, it may be said that there is no fixed measure in eye-tracking to investigate attention across different research designs. The data suggests that researchers can employ various types of eye-tracking measures to gain more insights into attention allocation (see Orquin and Holmqvist 2018, for a recent example). Thus, researchers investigating attention are recommended to consider adopting multiple measures in representing attention, such as fixation temporal and count, dwell temporal and count, and gaze pattern. This will help researchers to better delineate the time course of attention and its oscillation in time and space.

8.3.2. Higher Cognitive Processes

Higher cognitive processes were less frequently examined than attention in L2 eye-tracking studies, likely because it is challenging to properly and confidently link gaze behaviors to hypothesized undergirding cognitive processes (Strohmaier et al. 2020), while there is a more direct and perhaps solid relation between overt visual attention and eye movements (Godfroid 2019; Holmqvist et al. 2011).

A broader range of measure types was applied to explore higher-level cognitive processing. The most frequently used measure type is fixation count followed by fixation temporal measures. This is attributable to the frequent application of the visual world paradigm, in which the proportion of fixation is assessed to infer language comprehension processes (Dussias 2010; Huettig et al. 2011). On the other hand, fixation temporal measures are commonly used in reading-based studies to examine the moment-by-moment comprehension processes underlying reading (e.g., Keating 2009; Traxler et al. 2021). Using fixation temporal measures to gauge cognitive processing in L2 offers several advantages. First, research shows that fixation duration can be reliably used as a proxy for representing the depth of processing (Son et al. 2021). During L2 reading, longer fixations can be indicators of deeper levels of processing (Son et al. 2021), although they could also indicate difficulty in comprehension. Second, eye movements during reading can be carved up into early (e.g., first fixation duration) and late (e.g., total fixation duration) measures. It enables researchers to use a range of early and late measures to uncover the temporal dynamics of processing (Godfroid and Hui 2020).

Regression is also a relatively common measure to infer higher cognitive processes during L2 reading (e.g., Elgort et al. 2018; Fujita and Cunnings 2021; Keating 2009). A notable advantage of using eye-tracking over self-paced reading to examine the reading process is that, reading with eye-tracking allows for capturing and measuring regressions (Godfroid 2019). Regression is considered to represent reanalysis and reflect processing difficulties (Rayner 2009). For example, regression path duration is often interpreted as the time required to resolve a processing challenge (Godfroid 2019). In terms of the eye-tracking results, higher regression rates or longer regressions can be representative of difficulties in lexical access or text comprehension (Elgort et al. 2018; Keating 2009). Based on such understanding, measures of regressive eye movements have been utilized to investigate the process of, for example, anomaly detection during online sentence comprehension (Keating 2009) and semantic integration in contextual word learning from reading (Elgort et al. 2018). However, it seems that our knowledge of regressions in L2 processing is insufficient, which may impede the operationalization and interpretation of this measure in L2 research. For instance, it is possible that people regress to and refixate on a previous part because that word or AOI is of high relevance to the task (Orquin and Holmqvist 2018). Nonetheless, this interpretation is rarely mentioned in the L2 literature compared to comprehension difficulties or semantic integration as possible causes of regressions (e.g., Roberts and Siyanova-Chanturia 2013). A possible reason for the limited understanding of regressions in L2 processing is that “regressions are not particularly well understood because it is difficult to control them experimentally” (Rayner 2009, p. 1460). Overall, the absence of a reliable understanding of regressions may jeopardize the validity of inferences drawn from regressive eye movements.

In sum, L2 researchers have leveraged numerous eye-tracking measures to tap different cognitive processes. Nevertheless, using eye-tracking to draw inferences concerning higher cognitive processes can be challenging because of the simultaneous effects of multiple unobservable cognitive processes on eye movements. The same measure has been found to be interpreted in different ways (see, e.g., Bax 2015; Meghanathan et al. 2015). Moreover, two people could fixate on an AOI for the same amount of time but for different reasons. Take the fixation duration as an example; this metric can be associated with both the number of distractors that enter the mind and the amount of attention deployed to process the target stimuli (Meghanathan et al. 2015), which can pose a problem to the interpretation of fixation in multimodal reading if fixation duration is taken as a proxy for integration and comprehension. Even in unimodal environments, the interpretation of fixation duration is not straightforward and tightly controlled experimental designs are needed to draw accurate inferences from fixation duration and other metrics. As L2 researchers are exploiting the new possibility of eye-tracking in the studies of L2 processing, greater efforts need to be devoted to disentangling different processes underlying eye movements.

8.3.3. Cognitive Load

Cognitive load refers to “the load that performing a particular task imposed on the participant’s cognitive system” (Paas et al. 2003, p. 64). A smaller number of L2 studies (n = 3) have probed cognitive load using eye-tracking, even though cognitive load can be more easily derived from gaze behaviors (Meghanathan et al. 2015). This construct is considerably under-investigated in the L2 literature, too, and there is much room for development and innovation in this line of research. Although only three studies examining cognitive load have been found, a variety of measure types have been used. Fixation temporal and count measures have been utilized more frequently, and a higher fixation rate or a longer fixation duration indicates a greater cognitive load (Aryadoust et al. 2022; Révész et al. 2014). The other measure types have also proven to be useful in measuring cognitive load, although their application has been far and few between, such as dwell temporal, dwell count, skip, regression count and saccade spatial measures. The choice of eye-tracking measure and the interpretation of the collected metrics will depend on the type of the task used. In a study about measuring cognitive load imposed by test methods, Aryadoust et al. (2022) used dwell temporal and dwell count measures, wherein lower visit rates and higher normalized visit duration were treated as indications of lower cognitive load in the while-listening-performance tests.

Another useful measure of cognitive load is pupil diameter, whose relationship with cognitive load was discovered several decades ago by Hess and Polt (1964). Following this discovery, pupil size has been widely used in a variety of research designs (see van der Wel and van Steenbergen 2018), although being underappreciated in L2 research. Task-evoked pupil dilation has been found to be sensitive to cognitive load (Beatty and Lucero-Wagoner 2000), such that the size of the pupil enlarges proportionally in states of high mental load (Perkhofer and Lehner 2019).

Nevertheless, we note that using pupil diameter as a measure of cognitive load is not without limitations. The cognitive effects on pupil diameters are small but changes in pupil dilation can be triggered by a variety of extraneous factors, such as luminance and off axis-distortion (Holmqvist et al. 2011; Krejtz et al. 2018). Small changes in pupil diameter can easily be drowned in the large changes due to variations in light intensity, and thus the use of this measure will require the implementation of a tight experimental setup and design (Holmqvist et al. 2011).

8.4. Research Question 3: Replicability of L2 Eye-Tracking Studies

The third research question sought to provide a survey of the transparent and replicable reporting practices of L2 eye-tracking studies. We developed a list of 33 items critical to replicating an L2 eye-tracking study, which was informed by previous eye-tracking reporting guidelines (Carter and Luke 2020; Fiedler et al. 2020; King et al. 2019), methodological handbooks (Godfroid 2019; Holmqvist et al. 2011) and several empirical L2 eye-tracking studies (e.g., Plonsky and Gonulal 2015; Traxler et al. 2021).

The results revealed that the majority of the reviewed studies did typically not specify a number of categories and features of their studies. Taken together, the results from different analyses largely converged on the fact that L2 eye-tracking studies are not sufficiently transparent and complete in their reporting practices. Similar findings have been documented in previous reviews evaluating the reporting practices of eye-tracking in other fields, such as mathematics research (Strohmaier et al. 2020), communication science (King et al. 2019), and behavioral decision-making (Fiedler et al. 2020). Relatedly, the results also mirror those of the previous reviews examining the reporting of statistical methods or research tools in L2 research (Crowther et al. 2021; Plonsky and Gonulal 2015), where much of the critical information of each method was overlooked in the articles.

This worrying finding may be attributed to two main reasons. First, L2 eye-tracking studies have not established an agreed-upon reporting guidelines for L2 researchers to follow in the past years. Due to different considerations with regard to the nature of the respective research fields, reporting guidelines are also often biased towards specific research fields, and most of the prior reporting guidelines of eye-tracking are incomplete and inconsistent regarding what information to report (Holmqvist et al. 2023). Second, journal articles are often limited in length, and thus some of the information cannot be sufficiently specified in publications.

Eye-tracking studies in this review have shown a high level of variability and flexibility in methodological aspects regarding the apparatus, the analysis software, procedures and the parameters used. With such high degrees of freedom, eye-tracking researchers should make an effort to ensure the replicability of their studies by transparently reporting their research. This effort would contribute to the reliability and generalizability of the results obtained from the eye-tracking method and the scientific progress of the field. It should be noted that this study has explored replicability from the perspective of the extent to which L2 eye-tracking studies could be performed again by other researchers following the information and procedures documented in the original publications. The reasons for why a study cannot be replicated are complex and many, but transparent reporting with a high level of details from study design, sampling, choices of hardware and software, techniques and parameters used to process data will no doubt enhance replicability.

9. Conclusions

This systematic review sought to scope the application of eye-tracking in L2 research and examine the extent to which L2 eye-tracking studies are replicable from the perspective of reporting practices. The findings indicate a growing adoption of eye-tracking in L2 research. Eye-tracking was frequently used in grammar and vocabulary research; fixation temporal and count measures were the most frequently used measures; and three cognitive mechanisms were investigated, as previously discussed. However, our review also highlights that eye-tracking is just beginning to emerge in many areas, such as L2 writing and speaking. Furthermore, our classification of eye-tracking measures enabled us to demonstrate the limited utilization of available eye-tracking measures across various domains of L2 research. We suggest that future researchers explore these under-researched areas as well as cognitive mechanisms to attain a comprehensive understanding of L2 learners’ gaze behaviors during language processing and learning. Finally, the evidence of insufficient transparency emphasizes the need for more detailed reporting practices in future eye-tracking studies.

This review is subject to several limitations. One issue with the current review is that the studies reviewed here may not represent the entire L2 eye-tracking literature, since the data were chosen from tier-1 (quartile-1) English journals in the field. We suggest that future researchers should investigate articles published in other journals or sources (e.g., tier-2, tier-3, and even tier-4 journals), so that a more comprehensive review of the application and uses of eye-tracking in the field will be generated. In addition, future researchers should consider reviewing the articles that are published in bilingualism and multilingualism journals. It should also be noted that this study did not extract information from every possible source relevant to each reviewed study, which may impact the replicability index of L2 eye-tracking studies. Due to word limit requirements imposed by journals, researchers may need to selectively report information while potentially providing supplementary details through other channels. For instance, with the promotion of open research practices, researchers are making their materials and data accessible through repositories like IRIS (Instruments and Data for Research in Language Studies) (Marsden and Morgan-Short 2023). Nevertheless, our list of variables for replicability purposes remains critical, and the absence of such information in the paper could signify reduced replicability (Derrick 2016). We hope that the results of this study will encourage better design and reporting practices in future eye-tracking research.