Next Article in Journal
Effect of Enriched Environment on Cerebellum and Social Behavior of Valproic Zebrafish
Previous Article in Journal
“I Wouldn’t Even Know What to Ask for”: Patients’ and Caregivers’ Experiences of Psychological Support for Huntington’s Disease in Italy
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
NeuroSci
Volume 5
Issue 2
10.3390/neurosci5020008
neurosci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Comparing Self-Report vs. Performance Measures of Attentional Control and Efficiency

by
Mohammad Ahsan Khodami
1,*,
Luca Battaglini
1,
Maryam Jansarvatan
1,†,
Sofia Kireeva
2,† and
Seiran Bagheri
3,†
1
Department of General Psychology, University of Padua, 35131 Padua, Italy
2
Graduate School of Economics and Management, Ural Federal University, 62002 Ekaterinburg, Russia
3
Departmen of Psychology, Payame Noor University, Tehran 19395-4697, Iran
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
NeuroSci 2024, 5(2), 114-127; https://doi.org/10.3390/neurosci5020008
Submission received: 27 February 2024 / Revised: 25 March 2024 / Accepted: 27 March 2024 / Published: 4 April 2024
Browse Figures
Versions Notes

Abstract

:
Background: The Attention Control Scale (ATTC) is a widely used self-report measure of attentional control capacities. However, research questions whether it accurately substitutes for objective attention control tasks. This study investigated ATTC’s correlation with the Attention Network Test (ANT) across alerting, orienting, and executive control networks. We also used the Inverse Efficiency Score (IES) as an additional factor to check ATTC using ANT. Methods: We administered 143 participants who completed the ATTC questionnaire and ANT behavioral test assessing network efficiencies. Results: The results showed non-significant ATTC-ANT correlations across all networks. In an additional analysis, while the ATTC demonstrated factorial validity, subjective control was disconnected from actual attention regulation efficiency. A small male advantage emerged for executive control. Conclusions: Dissociations likely stem from attention complexity and method variances rather than overlap. The findings do not support the ATTC as a stand-alone proxy for performance-based measurement. Multifaceted assessments are essential for comprehensively capturing attentional control.

1. Introduction

Human behavior and cognition emerge from complex interactions between multiple mental processes. Attention represents a core capacity that selects, maintains, and shifts focus among all the stimuli bombarding our senses [1]. Attention control is the cognitive process of regulating attention through monitoring, selectivity, and flexibility. This multidimensional construct consists of specific attentional networks like alerting, orienting, and executive control [2]. Attention also allows priming relevant information while suppressing irrelevant noise. This filtering mechanism is essential for higher-order functions like learning, memory, reasoning, and decision-making to operate on a manageable subset of inputs [3]. While attention can be reflexively captured by salient stimuli, voluntary attentional control provides additional flexibility to direct focus based on current goals, expectations, and knowledge [4].
Attention control is the principal cognitive construct regulating various sub-cortical and cortical processes that effectively select and filter sensory information [5]. These interrelated sub-processes notably include the top-down modulation of attentional orientation reflecting current goals, the active inhibition of external and internal distractions, rapid flexibility in allocating attention between competing stimuli and changing situational demands, and continuous monitoring and timed updating of working memory representations [6]. Interpreting the precise neurocognitive mechanisms mediating these sub-processes and their dynamic integration remains an active area of scientific inquiry [7].
Attentional control abilities vary across individuals and relate to real-world outcomes. For example, deficits in controlling attention have been associated with clinical conditions like ADHD, anxiety, and depression [8,9]. Enhancing attentional control through training can improve cognitive performance and reduce symptomatology [10]. Given the importance of attentional control, accurately measuring it has implications for basic science and applied settings. However, assessment is complicated by the multifaceted nature of attention [11]. Focusing, shifting, and executive control represent different components that may not behave uniformly.
Questionnaire methods offer one approach to assess attentional control. Among them, the Attention Control Scale (ATTC or ACS) is a widely used self-report measure requiring individuals to place their capacity to focus and shift attention flexibly [12]. While convenient and easy to administer, this questionnaire is popular and widely used in studies (for review, see [13]). Some work has questioned the reliability of ATTC. Some studies show congruent relationships, while others do not [14,15], suggesting that establishing the convergence between subjective and objective attentional control measures has important theoretical and practical implications. A meta-analysis by [13] showed that the ATTC is not significantly associated with objective measures of attention control, suggesting that it may not accurately reflect attention control capacities.
Emerging computer-based assessments provide an alternative approach to objectively evaluating attentional control. Integrating subjective questionnaires and objective computer-based tasks could provide complementary insight into attentional control and its neural and cognitive mechanisms. Assessing questionnaires alongside behavioral and computerized tasks is vital for thoroughly understanding human behaviors and cognitive processes [16].
Questionnaires provide subjective self-reports, capturing individuals’ perceptions, attitudes, and self-assessments of behaviors, which are invaluable for understanding psychological states and traits. However, they can be limited by biases such as social desirability and self-awareness [17,18]. Behavioral and computerized tasks complement this by objectively measuring performance and responses on controlled grounds. These tasks can demonstrate cognitive and behavioral patterns that must be evident or accurately reported in questionnaires. By combining both methods, researchers can acquire a more potent and holistic perspective, enhancing the validity and reliability of their findings in psychological and cognitive research.
The current study investigates whether the ATTC can measure the same aspects observable through behavioral tasks. The primary objective was to determine whether ATTC is a dependable method for assessing attentional control or, as suggested by [13], if it lacks validity. Given the impracticality of employing every available task, the Attentional Network Task (ANT) was selected to examine the ATTC Scale.
Although some research in this area has been carried out, most of these studies focused on anxiety and mood as factors. Also, studies in this area were mostly carried out via presentation in the laboratory or via presence; the author of [19] further emphasizes this point, noting that the artificial environment of the lab can warp effect sizes, making them less applicable. Also, the author of [20] adds a phenomenological perspective, arguing that reducing problem-solving behavior to a cognitive mechanism in the lab overlooks important experiential aspects. Considering most of these findings, we have decided to administer this task in an online environment, which is less stressful and more based on the participants’ favorite time to do the task.
The Attentional Network Task (ANT) is used in cognitive psychology, particularly in studying attention. While there is limited direct information on using ANT to validate ATTC specifically, we can infer some reasons based on their functions and characteristics. First, it is a self-report measure that assesses individual differences in attentional control, focusing on the ability to maintain and shift attention between tasks [21]. On the other hand, ANT is a performance-based task that measures the efficiency of different attention networks (alerting, orienting, and executive control) in the brain. Using ANT to validate ATTC can be advantageous as it complements self-reported data with objective performance-based data, providing a more holistic view of an individual’s attentional control capacities [22,23].
In some studies involving functional magnetic resonance imaging (fMRI), ANT validated ATTC [24]. This suggests that ANT can be effectively used in different settings to validate the ATTC, which is important for generalizing the findings to various contexts. Also, the authors of [22,23,25] mention that the ANT is a suitable choice for observing ATTC due to its ability to provide objective, performance-based data that complement the self-reported nature of ATTC, its coverage of a broad range of attentional processes, and its potential to demonstrate the neural correlates and broader implications of attentional control.

2. Materials and Methods

2.1. Procedure

2.1.1. Participants

A power analysis was conducted using R Software [26] and the pwr package [27]. Before data collection to determine the required sample size for detecting medium correlations between the Attention Control Scale total and sub-scale scores (ATTC_TOT, ATTC_FOC, ATTC_SHI) and the Attention Network Test outcomes for the alerting, orienting, and executive control networks (alerting, orientation, and executive). The significance level was adjusted to 0.0056 (0.05/9) using a Bonferroni correction for the nine planned analyses to account for the multiple correlations. This adjustment ensures 80% power to detect an effect size of 0.30 for each test. The prior power analysis indicated that a sample size of 138 was required.

2.1.2. Procedure

For this research, we emailed a list of students who registered on the voluntary system to participate in the LinguaMed learning group. Also, we provided QR codes, direct links, and short links in the group chats to recruit participants. Each participant was directed to complete a Machform (http://www.matchform.com (accessed on 1 January 2023)) survey hosted on a personal server, accessible via the link. To ensure a diverse representation, we limited participation to up to two users per platform, device, or IP address. If a participant attempted to open the survey link on a smaller screen, such as a mobile phone, they were automatically presented a message asking them to complete the survey on a computer screen instead.
Before proceeding, participants were required to read and either accept or decline a consent form. In our consent form, we asked our participants to confirm if they had any psychological diagnoses. Upon confirmation and receiving, they were directed to provide demographic information, including age, gender, and handiness. Subsequently, they completed the ATTC questionnaire. Following this, participants were redirected to the Pavlovia (https://pavlovia.org/ (accessed on 1 January 2023)) platform, where they engaged in the ANT task online. Participants first completed 15 ANT practice trials to familiarize themselves with the task requirements. This was followed by 120 ANT test trials assessing attentional network efficiencies through reaction time (RT) and accuracy indices. Participants could withdraw their participation through consent and study procedures; incomplete datasets were excluded from analysis.
Participants were informed beforehand that no prize, credit, or results would be provided at the task’s conclusion. Furthermore, they were advised of their right to withdraw from participation at any stage during both the questionnaire and the ANT task.

2.1.3. Tasks

Attention Control Scale ATTC
The Attention Control Scale (ATTC) is a 20-item self-report psychometric instrument developed to assess perceived capacity for voluntary attention control across two key domains: attentional focusing and attentional shifting. The ATTC requires respondents to rate the typical frequency of experiences relating to concentrating, ignoring distractions, and flexibly directing focus using a 4-point Likert scale format (1 = almost never; 4 = always). Initial validation analyses indicated adequate internal reliability for the composite ATTC ( α = 0.88) and respective Attentional Focusing ( α = 0.80) and Attentional Shifting ( α = 0.65) sub-scales [12]. Principal components analysis verified the two-factor structure, which cumulatively accounted for 29% of the variance. Test-retest reliability over a 2-week interval was also acceptable (r = 0.76). Higher ATTC scores theoretically index greater attention control.
Attentional Network Task ANT
The Attention Network Test (ANT) is a behavioral procedure developed by Fan and colleagues [25] to evaluate the efficiency of three attentional networks—alerting, orienting, and executive control—within a single integrated task. The task requires participants to indicate the direction of a central arrow target flanked by congruent, incongruent, or neutral arrows. Warning cues presented before target onset manipulate alertness and spatial attention. By comparing performance (reaction times, accuracy) across cue and flanker conditions via subtractions, the ANT provides indexes of alerting, orienting, and conflict resolution capacities, demonstrating moderate test-retest reliability. The ANT cue-flanker manipulations also interact, with alerting cues enhancing distractor interference, suggesting functional interactions between networks [25,28]. We reproduced the exact experiment as Fan et al. did (for review [25]).

3. Data Analysis

We used Python code to calculate the total score, attentional shift, and attentional focus based on the questionnaire detail provided [12]. Since the main package lmsupport had technical problems calculating varScore, we adopted the varScore code from the main package and reran it to calculate the Attentional focus, shift, and total attention score see Appendix code (https://osf.io/8ptx9 (accessed on 20 December 2023)). All statistics and analyses were carried out using Python and statistical packages.
Before parametric statistical analyses, the normality assumption was assessed using Shapiro–Wilk tests to assess the three attitude measures—ATTC_TOT, ATTC_FOC, and ATTC_SHI. For the total attitude score ATTC_TOT, the Shapiro–Wilk test produced a W statistic of 0.9901 and a non-significant p-value of 0.4103, indicating that the distribution of ATTC_TOT scores does not significantly deviate from a normal distribution. Similarly, for ATTC_FOC and ATTC_SHI, the Shapiro–Wilk tests yielded non-significant p-values of 0.07126 and 0.09507, respectively. As all p-values are greater than the 0.0056 significance level, the null hypothesis that the data are normally distributed cannot be rejected. Therefore, the assumption of normality was deemed to have been met for using these variables in parametric statistical tests.

4. Results

4.1. Demographic Detail

In this study, we recruited a cohort of 143 participants, comprising 71 females and 72 males, with an average age of 26.15 years (SD = 5.04). Eighty-four of the participants were right-handed. Notably, due to the lack of a finishing questionnaire or odd pattern of answering (e.g., all answers were a specific number or the task still needed to be finished), we eliminated 17 participants from our results to conduct analysis.

4.2. Attentional Questionnaire Scale Result

In our study, the Attentional Control Scale (ATTC) results for Total Attention (ATTC_TOT) had a mean score of 50.38 with a standard deviation of 4.63; the Focus sub-scale (ATTC_FOC) had a mean score of 22.74 with a standard deviation of 3.48; and, lastly, the Shift sub-scale (ATTC_SHI) had a mean score of 27.64 with a standard deviation of 3.48.
The results revealed no significant gender differences in focus scores ( t ( d f ) = 1.18 , p = 0.24 ), total scores ( t ( d f ) = 0.49 , p = 0.62 ), or shift scores ( t ( d f ) = − 0.46 , p = 0.64 ). Furthermore, the small Cohen’s d effect sizes (range −0.08 to 0.20) and confidence intervals for mean differences containing zero (−1.45 to 1.77) indicated negligible differences between genders across all facets of attentional control. Table 1 shows the questionnaire results, and Figure 1 below illustrates the data distribution among participants for this task.

4.3. ANT Task Result

A descriptive analysis examined response time performance across the ANT conditions. Mean response times were faster for the double cue condition (M = 467.11, SD = 19.47) than the no cue condition (M = 582.03, SD = 10.31), indicating enhanced alerting with the availability of an alerting cue. Additionally, response times were slower for incongruent flanker trials (M = 570.57, SD = 18.23) than congruent trials (M = 525.81, SD = 12.95), suggesting greater executive control demands and conflict resolution for incongruent stimuli. Differences between other trial types were negligible. In sum, the expected effects of alerting and executive attention were evidenced in the descriptive results, whereby alerting cues sped responses through enhanced readiness, and incongruent conditions elicited responses to conflict and slower reactions. Figure 2 and Table 2 show a description of this analysis.
The current results demonstrate significant effects of cue type on reaction times, with the double “both” cue producing the fastest reaction times overall compared to the spatially neutral “center” and no cue "blank" conditions. This incremental speeding of reaction time suggests enhancement of alertness and spatial orienting when temporal and spatial cues are provided before the target.
Additionally, as Table 3 and Figure 3 show, reaction times were slowest in response to incongruent stimuli relative to congruent and neutral target conditions. This pattern indicates that executive control processes are recruited more when conflicting stimulus-response mappings induce competition, reflecting enhanced demands on selective attention. Participants took longer to respond to incongruent stimuli for all cue types than congruent and neutral conditions. This demonstrates increased cognitive effort and reduced processing efficiency when the attentional focus was directed to overcome incongruence.

4.3.1. Inverse Efficiency Score IES

The Inverse Efficiency Score (IES) is a widely used composite measure, originally proposed by [29]. It is generally calculated as the ratio of mean correct response times (RTs) to the proportion of correct responses (CRs), as defined by [29] and later by [30]:
I E S i , j = R T i , j ¯ C R i , j
where R T i , j ¯ represents the mean RT of participant i for correct responses in condition j, and C R i , j represents the proportion of correct responses of participant i in condition j. While the IES has mostly been calculated using only correct RTs in empirical studies, the authors of [31]’s original concept suggested considering all RTs, including those from erroneous trials. However, this analysis focuses on the more commonly reported version, which excludes incorrect RTs [32]. The IES is a valuable index in psychological research because it offers a holistic assessment of task performance efficiency, sensitive to individual differences, reflective of cognitive processing strategies, comparable across conditions, and supported by empirical evidence [33].

4.3.2. Calculate ANT Sub-Scales

Data from the ANT test drive attention sub-scales. Based on [25]:
  • Alerting is calculated by subtracting the mean RT of Both cue conditions from the mean of RT for the None Cue condition. This is because neither of these conditions provided information about the target or whether the stimulus would appear above or below the fixation point. In contrast, in the None condition, attention also remains diffused between it.
  • Orientation is calculated by subtracting the RT mean for Both minus center; this is because, in both conditions, we have either the top or down in which, in this condition, cue serves as predictive spatial information that allows the subject to begin Orientation-appropriate locations before the target arrives.
  • Executive control is calculated by subtracting all conditions where the cue is congruent from the total RT of incongruent data.
An independent samples t-test analyzed gender differences. As seen in Table 4, there were no significant differences between males and females on either alerting, t ( d f ) = 0.18 , p = 0.86 , Cohens d = 0.03, 95% CI = [−6.53, 7.82], or orienting, t ( d f ) = 0.47 ,   p = 0.64 , Cohens d = 0.08, 95% CI = [−6.69, 10.96]. However, a small-moderate difference emerged for executive control network efficiency, t ( d f ) = − 2.34 ,   p = 0.02 , Cohens d = −0.39, 95% CI = [−14.99, −1.33], with males demonstrating significantly enhanced executive functioning compared to females. While alerting and orienting performance was equivalent across genders, males showed an advantage in executive attentional control. While small in magnitude, this effect may have implications for males’ superior regulation of cognition and behaviors relative to females [34].

4.3.3. Correlational Analysis

The correlation analysis revealed varying degrees of association among the study variables, with ATTC_FOC showing a strong positive correlation with ATTC_TOT (r = 0.65, p < 0.001) and other notable correlations, as detailed in Table 5.
Figure 4 shows the correlation mconsidered variables study, which does not suggest any strong correlation between our considered variables.
Also, for IES, we could not find any correlational significance suggesting that data obtained from the ATTC questionnaire cannot predict the outcomes of the ANT task.

5. Additional Analysis

After finishing our analysis, we decided to conduct another additional analysis. This analysis aimed to check some other aspects of our data, so a confirmatory factor analysis (CFA) was conducted on the Attention Control Scale (ATTC) items to validate the hypothesized two-factor structure measuring attentional focus and shifting ability.
The CFA supported the expected two-factor model ( χ 2 ( 0 ) = 0.00 ,   p > 0.05 ) with fit based on standard criteria (CFI = 1.00, TLI =1.00, RMSEA = 0.00, and SRMR = 0.00). All ATTC items loaded significantly onto their specified factors ( p < 0.001 ). The “focus” factor consisted of three items related to one’s capacity to intentionally maintain attentional focus (std. loadings 0.33–0.41). The “shifting” factor contained two items measuring ease of switching focus (std. loadings 0.35–0.36). The factors showed a small, non-significant negative correlation ( r = − 0.09 ,   p = 0.30 ).
Even with the Attentional Control Scale (ATTC) showing solid factorial validity in the confirmatory factor analysis, we still do not see significant correlations between the ATTC scores and Attention Network Test (ANT) performance.
Research on the correlation between psychological questionnaires and behavior reveals mixed findings. Ref. [35] suggests that certain psychological features, such as a sense of being undervalued, can be linked to specific behaviors. However, ref. [36] warns that people tend to overestimate the stability of behavior across circumstances, leading to inaccurate correlations. Ref. [37] further problematizes the issue by finding that the Questionnaire for the Measurement of Psychological Reactance, a widely used tool, provides unreliable estimates. Ref. [38] adds another layer of complexity, highlighting the potential for measurement operations to influence correlations among measures. These studies collectively suggest that while psychological questionnaires may offer some insight into behavior, their accuracy and reliability are not guaranteed.
We also ran a multiple regression study to check if we could find any correlation between factors or not; based on the multiple regression results, we could not see any significant relationships between attentional control as measured by the ATTC factor scores and performance on the objective ANT as assessed by reaction time metrics of the alerting, orienting, or executive control networks. Specifically:
In the model predicting alerting network efficiency (alerting reaction time), neither the ATTC focus nor shifting coefficients were significant predictors (ATTC_FOC: b = − 0.699 , p = 0.20 ; ATTC_SHI: b = − 0.143 ,   p = 0.782 ).
Similarly, for the orienting network model, the ATTC factors did not significantly predict orienting efficiency scores (ATTC_FOC: b = − 0.319 ,   p = 0.636 ; ATTC_SHI: b = − 0.115 , p = 0.858 ). In the executive control model, while the ATTC focus factor was close to the significance level ( b = − 0.956 ,   p = 0.068 ), overall, the subjective attention measures did not reach significance for explaining executive network reaction times either. So, in all three attentional domains, an individual’s attentional control showed little connection to how efficiently they completed the ANT behavioral tasks engaging those specific control processes.
This aligns with the lack of previous Pearson correlations between the ATTC and ANT metrics. The reasons above, like differences between explicit perceptions versus implicit behaviors, likely explain the continued lack of significant relationships in the regression models.

6. Discussion

The current study tried to connect self-reported attentional control on the Attentional Control Scale (ATTC) and performance-based attentional control efficiency assessed by the Attention Network Test (ANT). However, the results did not support any connection, revealing non-significant relationships between ATTC scores and ANT outcomes across alerting, orienting, and executive control networks. This finding is consistent with [13], who reported that ATTC scores do not reliably predict objective attention control, suggesting the ATTC measures subjective perceptions rather than actual control abilities. Our study further questions the validity of the ATTC as a proxy for attentional control efficiency.
This dissociation between subjective self-reports and objective behavioral measures aligns with theoretical mo dels proposing distinct pathways for explicit, conscious self-evaluation versus implicit cognitive processing and performance [39,40]. While self-report scales like the ATTC assess a person’s explicit beliefs about their attentional capacities, these metacognitive judgments may not accurately track the implicit allocation of attentional resources during demanding cognitive tasks like the ANT. This divergence could stem from limitations in self-awareness, whereby individuals have incomplete introspective access to their own attentional control abilities.
Despite these results, the ATTC exhibited good factorial validity in a confirmatory analysis, indicating it reliably measures respondents’ perceived focusing and shifting capacities. Nevertheless, this insight seems disconnected from actual attention regulation in behavioral tasks. This dissociation could be due to the multidimensional nature of attention, as discussed by [41], making it challenging for a single task like the ANT to capture attention’s complexity fully. Moreover, inherent variances between self-report methods, computerized testing, and real-world functioning [42] may contribute to this discrepancy, compounded by limitations in self-awareness.
The present findings raise broader questions about how self-report and behavioral measures of cognitive processes will converge or diverge. Existing theories propose that subjective self-reports index stable trait-level characteristics better, while objective task performance is more sensitive to state-level fluctuations [43,44]. From this perspective, the ATTC may capture trait-level beliefs about attentional control abilities that remain stable across situations, whereas performance on the ANT is more variable and context-dependent. This could explain the observed disconnect, with implications for when each type of measure is more appropriate for assessment.
Furthermore, methodological differences between self-report and objective tests warrant consideration. Personal biases often influence subjective measures, while performance-based tests offer a more controlled evaluation environment. Investigating the neurocognitive support of attention control could clarify the discrepancies between these measures as the brain regions and networks involved may respond differently in subjective versus objective assessments.
Considering behavioral data and questionnaires, behavioral and psychological questionnaires often do not align due to various factors. Ref. [45] highlights the limitations of behavioral data in capturing the psychological processes that underpin group behavior. Ref. [46] further emphasizes this point, showing a modest overlap between psychometric and self-reported definitions of executive function deficits in individuals with ADHD.
Individual differences also play a critical role [9,47,48]. Variability in personality traits; cognitive abilities, especially in attention in different tasks [49,50,51]; and mental health status might influence the relationship between self-reported and objectively measured attention control. This aspect underscores the importance of considering a broad range of individual characteristics in future research.
Given these complexities, our findings do not support using the ATTC as a sole measure for attention control efficiency. Instead, a more nuanced approach is required, integrating various methods and perspectives. This includes longitudinal studies to track changes in attention control over time and alternative measurement instances encompassing broader psychological constructs. Such multifaceted approaches are essential for comprehensively understanding attention control and its assessment.

7. Conclusions

The current study found no significant correlation between the Attentional Control Scale (ATTC) and the Attention Network Test (ANT) across alerting, orienting, and executive control networks. This result aligns with previous findings questioning the validity of the ATTC as an accurate proxy for attentional control abilities. While the ATTC may reliably measure subjective perceptions of control, these metacognitive insights appear parted from actual attention regulation efficiency during behavioral testing.
Dissociations between subjective self-reports and objective performance arise due to attention’s intrinsically complex, multidimensional nature. Variability in what specific components’ different assessments emphasize can contribute to poor correlations: individual differences, limitations in self-awareness, and contrasts between testing methods and further obscure relationships.
By examining the correspondence between the ATTC and established attention control tasks, this study enhances our understanding of the conditions under which conflicting reports will arise based on assessment selections. The findings of [52,53,54,55] indicate that ATTC is not a valid measurement to assess attentional control, and using it requires more careful consideration.
Also, this work raises the question of whether a questionnaire alone is a good measure for psychological measurement. This can reconcile past correlational ATTC research contradictions through methodological and measurement dissociation. In addition, as [13,56,57] mention, we are calling for further emphasis on the need for careful consideration in using the ATTC.
The current results do not support using ATTC as a stand-alone substitute measure for attention control. A more nuanced, multimodal approach is necessary to fully capture attention’s complexities encompassing first-person experiences and third-person behaviors. Longitudinal studies tracking within-person changes and alternative measurement models may clarify boundaries between subjective self-reports and objective levels of control. Integrating multiple methods remains essential for elucidating the intricacies of attention control and advancing its assessment.
Furthermore, the reliability of questionnaires often hinges upon comparisons with preceding instruments, a point of significant importance given that the authors of [58] underline the imperative for regular estimation and disclosure of reliability in behavioral assessments within the field of psychological science. Such practices are crucial to solidify the validity of conclusions derived from statistical analyses. Similarly, the authors of [59] mention that endorsing standardized practices is essential for ensuring methodological integrity.

8. Limits of This Study

While providing evidence regarding the lack of correspondence between the ATTC and ANT, this study had limitations that could be addressed in future research. Our sample consisted solely of adults in a restricted age range, gathered predominantly online. The use of online surveys for data collection may face challenges such as unreliable email lists (in our case) and participant willingness. To minimize these variations, consider improving the design of search and delivery tools and engaging with variability during data preparation [60,61,62].
Further studies with larger, more diverse cohorts from broader populations and age groups are needed to generalize conclusions. We used a single self-report scale paired with one behavioral test. Employing multiple complementary subjective measures and objective tasks related to attention control would allow for a more robust evaluation of relationships. Data collection occurred wholly online, contributing to variability. Added experimental control could strengthen the reliability of the results. Although these findings question the interchangeability of the ATTC and performance indices like the ANT, continuing investigation with enhanced sampling, multimodal measurement, and tighter methodological control is vital to elucidate boundaries between perceived and actual attentional control capacities.

Author Contributions

Conceptualization, M.A.K.; methodology, M.A.K. and L.B.; software, M.J., M.A.K. and S.K.; validation, M.A.K., S.K. and S.B.; formal analysis, M.A.K. and M.J.; writing, review and editing, S.K., S.B. and M.A.K.; visualization, M.A.K. and L.B.; supervision, M.A.K. and L.B.; and funding acquisition, M.A.K. All authors have read and agreed to the published version of the manuscript.

Funding

M.A.K. has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 101034319 and from the European Union—NextGenerationEU. The other authors declare no funding.

Institutional Review Board Statement

The Declaration of Helsinki conducted the study.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Participants, before starting the whole experiment, had to read an information form and consent to continue the procedure.

Data Availability Statement

Data files, including the Analysis in R, detailed participants data, IES score, calculation of the ANT task, and code for calculating the ATTC questionnaire, are available at https://osf.io/rk4qa/ (accessed on 25 February 2024).

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Knudsen, E.I. Fundamental components of attention. Annu. Rev. Neurosci. 2007, 30, 57–78. [Google Scholar] [CrossRef] [PubMed]
  2. Callejas, A.; Lupiáñez, J.; Funes, M.; Tudela, P. Modulations among the alerting, orienting and executive control networks. Exp. Brain Res. 2005, 167, 27–37. [Google Scholar] [CrossRef] [PubMed]
  3. Chun, M.M.; Golomb, J.D.; Turk-Browne, N.B. A taxonomy of external and internal attention. Annu. Rev. Psychol. 2011, 62, 73–101. [Google Scholar] [CrossRef] [PubMed]
  4. Awh, E.; Aeschbach, D. Optimizing Training and Transfer of Attentional Control. Curr. Dir. Psychol. Sci. 2018, 27, 472–478. [Google Scholar]
  5. Posner, M.I.; Petersen, S.E. The attention system of the human brain. Ann. Rev. Neurosci. 1990, 13, 25–42. [Google Scholar] [CrossRef] [PubMed]
  6. Miyake, A.; Friedman, N.P.; Emerson, M.J.; Witzki, A.H.; Howerter, A.; Wager, T.D. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cogn. Psychol. 2000, 41, 49–100. [Google Scholar] [CrossRef] [PubMed]
  7. Suchy, Y. Executive functioning: Overview, assessment, and research issues for non-neuropsychologists. Ann. Behav. Med. 2009, 37, 106–116. [Google Scholar] [CrossRef] [PubMed]
  8. Derakshan, N.; Eysenck, M.W. Anxiety, processing efficiency, and cognitive performance: New developments from attentional control theory. Eur. Psychol. 2009, 14, 168–176. [Google Scholar] [CrossRef]
  9. Khodami, M.A.; Hosseini, S.M.; Bagheri, S.N.; Kireeva, S. Associations Between Recurrent COVID-19, Attention and Mental Health: A Longitudinal Study. 2023. Available online: https://www.researchsquare.com/article/rs-3670569/v3 (accessed on 15 March 2024).
  10. Tamm, L.; Epstein, J.N.; Peugh, J.L.; Nakonezny, P.A.; Hughes, C.W. Preliminary data suggesting the efficacy of attention training for school-aged children with ADHD. Dev. Cogn. Neurosci. 2016, 4, 16–28. [Google Scholar] [CrossRef]
  11. Steel, A.; Wharton, Z.; Allan, K. Examining the relationship between attention control and emotion recognition skills. Motiv. Emotion 2018, 42, 860–866. [Google Scholar]
  12. Derryberry, D.; Reed, M.A. Anxiety-related attentional biases and their regulation by attentional control. J. Abnorm. Psychol. 2002, 111, 225. [Google Scholar] [CrossRef] [PubMed]
  13. Clarke, P.J.; Todd, J. Lessons Unlearned: A Conceptual Review and Meta-Analysis of the Relationship between the Attention Control Scale and Objective Attention Control. Cogn. Emot. 2021, 35, 1447–1459. [Google Scholar] [CrossRef] [PubMed]
  14. Reinholdt-Dunne, M.L.; Mogg, K.; Bradley, B.P. Effects of anxiety and attention control on processing pictorial and linguistic emotional information. Behav. Res. Ther. 2009, 4, 410–417. [Google Scholar] [CrossRef] [PubMed]
  15. Forster, S.; Lavie, N. Pupil response indices of attentional selection in anxious individuals. Behav. Brain Funct. 2013, 9, 1–9. [Google Scholar]
  16. Posner, M.; Rothbart, M. Research on attention networks as a model for the integration of psychological science. Annu. Rev. Psychol. 2007, 58, 1–23. [Google Scholar] [CrossRef] [PubMed]
  17. Spector, P.E. Using self-report questionnaires in OB research: A comment on the use of a controversial method. J. Organ. Behav. 1994, 15, 385–392. [Google Scholar] [CrossRef]
  18. Fernandes, M.F.; Randall, D. The Nature of Social Desirability Response Effects in Ethics Research. Bus. Ethics Q. 1992, 2, 183–205. [Google Scholar] [CrossRef]
  19. Baumeister, R. Do Effect Sizes in Psychology Laboratory Experiments Mean Anything in Reality? PsyArXiv Preprint. 1 February 2020. [CrossRef]
  20. Wendt, A.N. Is There a Problem in the Laboratory? Front Psychol. 2018, 9, 2443. [Google Scholar] [CrossRef]
  21. Williams, P.G.; Rau, H.K.; Suchy, Y.; Thorgusen, S.R.; Smith, T.W. On the validity of self-report assessment of cognitive abilities: Attentional control scale associations with cognitive performance, emotional adjustment, and personality. Psychol. Assess 2017, 29, 519–530. [Google Scholar] [CrossRef]
  22. Martin, J.; Mashburn, C.A.; Engle, R.W. Improving the validity of the Armed Service Vocational Aptitude Battery with measures of attention control. J. Appl. Res. Mem. Cogn. 2020, 9, 323–335. [Google Scholar] [CrossRef]
  23. Morillas-Romero, A.; Tortella-Feliu, M.; Balle, M.; Bornas, X. Spontaneous emotion regulation and attentional control. Emotion 2015, 15, 162–175. [Google Scholar] [CrossRef]
  24. Markett, S.; Reuter, M.; Montag, C.; Voigt, G.; Lachmann, B.; Rudorf, S.; Elger, C.E.; Weber, B. Assessing the function of the fronto-parietal attention network: Insights from resting-state fMRI and the attentional network test. Hum. Brain Mapp. 2014, 35, 1700–1709. [Google Scholar] [CrossRef]
  25. Fan, J.; McCandliss, B.D.; Sommer, T.; Raz, A.; Posner, M.I. Testing the Efficiency and Independence of Attentional Networks. J. Cogn. Neurosci. 2002, 14, 340–347. [Google Scholar] [CrossRef] [PubMed]
  26. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2023; Available online: https://www.R-project.org/ (accessed on 1 February 2024).
  27. Champely, S.; Ekstrom, C.; Dalgaard, P.; Gill, J.; Weibelzahl, S.; Anandkumar, A.; Ford, C.; Volcic, R.; De Rosario, H. pwr: Basic Functions for Power Analysis. R Package Version 1.3-0. 2020. Available online: https://CRAN.R-project.org/package=pwr (accessed on 1 February 2024).
  28. Salo, R.; Gabay, S.; Fassbender, C.; Henik, A. Distributed Attentional Deficits in Chronic Methamphetamine Abusers: Evidence from the Attentional Network Task (ANT). Brain Cogn. 2011, 77, 446–452. [Google Scholar] [CrossRef]
  29. Akhtar, N.; Enns, J.T. Relations between covert orienting and filtering in the development of visual attention. J. Exp. Child Psychol. 1989, 48, 315–334. [Google Scholar] [CrossRef]
  30. Bruyer, R.; Brysbaert, M. Combining speed and accuracy in cognitive psychology: Is the inverse efficiency score (IES) a 491 better dependent variable than the mean reaction time (RT) and the percentage of errors (PE)? Psychol. Belg. 2011, 51, 5–13. [Google Scholar] [CrossRef]
  31. Townsend, J.T.; Ashby, F.G. Stochastic Modelling of Elementary Psychological Processes; Cambridge University Press: New York, NY, USA, 1983. [Google Scholar]
  32. Liesefeld, H.R.; Janczyk, M. Combining speed and accuracy to control for speed-accuracy trade-offs(?). Behav. Res. 2019, 51, 40–60. [Google Scholar] [CrossRef] [PubMed]
  33. Zilberg, N.; Weiss, D.; Horowitz, M. Impact of Event Scale: A cross-validation study and some empirical evidence supporting a conceptual model of stress response syndromes. J. Consult. Clin. Psychol. 1982, 50, 407–414. [Google Scholar] [CrossRef]
  34. Halpern, D.; Wright, T.M. A process-oriented model of cognitive sex differences. Learn. Individ. Differ. 1996, 8, 3–24. [Google Scholar] [CrossRef]
  35. Dragoş, D.; Ojog, D.G.; Tănăsescu, M.D. Questionnaire about psychology/disease correlation – II. J. Med. Life 2011, 4, 356–363. [Google Scholar] [PubMed]
  36. Kunda, Z.; Nisbett, R.E. The Psychometrics of Everyday Life. Cogn. Psychol. 1986, 18, 195–224. [Google Scholar] [CrossRef] [PubMed]
  37. Donnell, A.J.; Thomas, A.; Buboltz, W.C. Psychological Reactance: Factor Structure and Internal Consistency of the Questionnaire for the Measurement of Psychological Reactance. J. Soc. Psychol. 2001, 141, 679–687. [Google Scholar] [CrossRef] [PubMed]
  38. Feldman, J.M.; Lynch, J.G. Self-Generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior. J. Appl. Psychol. 1988, 73, 421–435. [Google Scholar] [CrossRef]
  39. Hofmann, W.; Gawronski, B.; Gschwendner, T.; Le, H.; Schmitt, M. A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Personal. Soc. Psychol. Bull. 2005, 31, 1369–1385. [Google Scholar] [CrossRef] [PubMed]
  40. Wilson, T.D.; Dunn, E.W. Self-knowledge: Its limits, value, and potential for improvement. Annu. Rev. Psychol. 2004, 55, 493–518. [Google Scholar] [CrossRef] [PubMed]
  41. Habekost, T.; Petersen, A.; Vangkilde, S. Testing Attention: Comparing the ANT with TVA-Based Assessment. Behav. Res. Methods 2013, 46, 81–94. [Google Scholar] [CrossRef] [PubMed]
  42. Chaytor, N.; Schmitter-Edgecombe, M. The Ecological Validity of Neuropsychological Tests: A Review of the Literature on Everyday Cognitive Skills. Neuropsychol. Rev. 2003, 13, 181–197. [Google Scholar] [CrossRef]
  43. Debusscher, J.; Hofmans, J.; De Fruyt, F. The effect of state core self-evaluations on task performance, organizational citizenship behaviour, and counterproductive work behaviour. Eur. J. Work. Organ. Psychol. 2016, 25, 301–315. [Google Scholar] [CrossRef]
  44. Sjåstad, H.; and Baumeister, R.F. The future and the will: Planning requires self-control, and ego depletion leads to planning aversion. J. Exp. Soc. Psychol. 2018, 76, 127–141. [Google Scholar] [CrossRef]
  45. Reynolds, K.J. Advancing Group Research: The (Non) Necessity of Behavioral Data? Small Group Res. 2011, 42, 359–373. [Google Scholar] [CrossRef]
  46. Biederman, J.; Petty, C.R.; Fried, R.; Black, S.; Faneuil, A.; Doyle, A.E.; Seidman, L.J.; Faraone, S.V. Discordance between Psychometric Testing and Questionnaire-Based Definitions of Executive Function Deficits in Individuals with ADHD. J. Atten. Disord. 2008, 12, 92–102. [Google Scholar] [CrossRef]
  47. Khodami, M.A.; Sheiban, L. An Investigation on Negative Activity, Alexithymia, Emotion Regulation, and Internet Addiction in a Sample of High School Students: A Randomized Controlled Trial. Ann.-MéDico-Psychol. Rev. Psychiatr. 2020, 178, 624–631. [Google Scholar] [CrossRef]
  48. Unsworth, N. Consistency of attentional control as an important cognitive trait: A latent variable analysis. Intelligence 2015, 49, 110–128. [Google Scholar] [CrossRef]
  49. Paneri, S.; Gregoriou, G. Top-Down Control of Visual Attention by the Prefrontal Cortex. Functional Specialization and Long-Range Interactions. Front. Neurosci. 2017, 11, 545. [Google Scholar] [CrossRef]
  50. Crova, C.; Marchetti, R.; Struzzolino, I.; Forte, R.; Pesce, C. Training Attention in Physical Education: Effects on Typically Developing and DCD Children. Procedia Soc. Behav. Sci. 2014, 116, 15091512. [Google Scholar] [CrossRef]
  51. Konishi, M.; Brown, K.; Battaglini, L.; Smallwood, J. When Attention Wanders: Pupillometric Signatures of Fluctuations in External Attention. Cognition 2017, 168, 16–26. [Google Scholar] [CrossRef]
  52. Quigley, L.; Wright, C.A.; Dobson, K.S.; Sears, C.R. Measuring attentional control ability or beliefs? Evaluation of the factor structure and convergent validity of the attentional control scale. J. Psychopathol. Behav. Assess. 2017, 39, 742–754. [Google Scholar] [CrossRef]
  53. Soltani, S.; van Ryckeghem, D.M.L.; Vervoort, T.; Heathcote, L.C.; Yeates, K.; Sears, C.; Noel, M. Attentional biases in pediatric chronic pain: An eye-tracking study assessing the nature of the bias and its relation to attentional control. Pain 2020, 161, 2263–2273. [Google Scholar] [CrossRef]
  54. Todd, J.; Notebaert, L.; Clarke, P.J.F. The association between self-report and behavioural measure of attentional control: Evidence of no relationship between ACS scores and antisaccade performance. Personal. Individ. Differ. 2022, 184, 111168. [Google Scholar] [CrossRef]
  55. Reinholdt-Dunne, M.L.; Mogg, K.; Benson, V.; Bradley, B.P.; Hardin, M.G.; Liversedge, S.P.; Pine, D.S.; Ernst, M. Anxiety and selective attention to angry faces: An antisaccade study. J. Cogn. Psychol. 2012, 24, 54–65. [Google Scholar] [CrossRef]
  56. Rawlings, D.; Goldberg, M. Correlating a Measure of Sustained Attention with a Multi-Dimensional Measure of Schizotypal Traits. Personal. Individ. Differ. 2001, 31, 421–431. [Google Scholar] [CrossRef]
  57. Judah, M.R.; Grant, D.M.; Mills, A.C.; Lechner, W.V. Factor Structure and Validation of the Attentional Control Scale. Cogn. Emot. 2013, 28, 433–451. [Google Scholar] [CrossRef] [PubMed]
  58. Parsons, S.; Kruijt, A.-W.; Fox, E. Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements. Adv. Methods Pract. Psychol. Sci. 2019, 2, 378–395. [Google Scholar] [CrossRef]
  59. Taylor, J.; Aldridge, J. Exploring the rewards and challenges of paediatric palliative care work—A qualitative study of a multi-disciplinary children’s hospice care team. BMC Palliat. Care 2017, 16, 73. [Google Scholar] [CrossRef]
  60. Wu, M.; Turpin, A.; Zobel, J. An investigation on a community’s web search variability. In Proceedings of the Australasian Computer Science Conference (2008), Wollongong, Australia, 22–25 January 2008. [Google Scholar]
  61. Wilkerson, M.H.; Lanouette, K.; Shareff, R.L. Exploring Variability during Data Preparation: A Way to Connect Data, Chance, and Context When Working with Complex Public Datasets. Math. Think. Learn. 2021, 24, 312–330. [Google Scholar] [CrossRef]
  62. Lefever, S.; Dal, M.; Matthíasdóttir, Á. Online Data Collection in Academic Research: Advantages and Limitations. Br. J. Educ. Technol. 2007, 38, 574–582. [Google Scholar] [CrossRef]
Neurosci 05 00008 g001
Figure 1. Box plot of measures for ATTC: ATTC_TOT represents the Attentional Control Scale Total; ATTC_FOC denotes the Attentional Control Scale Focus sub-scale; and ATTC_SHI signifies the Attentional Control Scale Shift sub-scale.
Figure 1. Box plot of measures for ATTC: ATTC_TOT represents the Attentional Control Scale Total; ATTC_FOC denotes the Attentional Control Scale Focus sub-scale; and ATTC_SHI signifies the Attentional Control Scale Shift sub-scale.
Neurosci 05 00008 g001
Neurosci 05 00008 g002
Figure 2. Box plot of response time (RT) for different conditions in the ANT task.
Figure 2. Box plot of response time (RT) for different conditions in the ANT task.
Neurosci 05 00008 g002
Neurosci 05 00008 g003
Figure 3. Reaction time “resp.rt” in ms for the ANT task based on the category of warnings provided to participants for ALL trials.
Figure 3. Reaction time “resp.rt” in ms for the ANT task based on the category of warnings provided to participants for ALL trials.
Neurosci 05 00008 g003
Neurosci 05 00008 g004
Figure 4. The correlation map between the ATTC scores (ATTC_TOT, ATTC_FOC, ATTC_SHI) and the ANT outcomes (alerting, orientation, executive, and IES).
Figure 4. The correlation map between the ATTC scores (ATTC_TOT, ATTC_FOC, ATTC_SHI) and the ANT outcomes (alerting, orientation, executive, and IES).
Neurosci 05 00008 g004
Table 1. Table with gender changes and mean (SD) format.
Table 1. Table with gender changes and mean (SD) format.
VariableMale M (SD)Female M (SD)tp-ValueCohens d95% CI
ATTC_TOT50.38 (4.63)49.99 (4.79)0.490.620.08[−1.16, 1.93]
ATTC_FOC22.74 (3.48)22.07 (3.27)1.180.240.20[−0.44, 1.77]
ATTC_SHI27.64 (3.48)27.92 (3.67)−0.460.64−0.08[−1.45, 0.90]
Table 2. Mean and Standard Deviation (SD) of response times.
Table 2. Mean and Standard Deviation (SD) of response times.
ConditionM (SD)
Mean RT531.353 (8.967)
Mean RT (Both)467.11 (19.47)
Mean RT (Blank)582.03 (10.31)
Mean RT (Center)494.23 (19.77)
Mean RT (Congruent)525.81 (12.95)
Mean RT (Incongruent)570.57 (18.23)
Table 3. Results based on cue and warning type for all trials and correct trials.
Table 3. Results based on cue and warning type for all trials and correct trials.
CueALLCorrect
BlankDoubleCenterBlankDoubleCenter
Congruent582 (39)493 (114)471 (99)582 (39)491 (114)470 (98)
Incongruent637 (80)491 (118)542 (86)637 (81)494 (114)542 (85)
Neutral560 (92)409 (78)496 (115)560 (93)409 (75)497 (115)
Table 4. Table with gender changes and mean (SD) format.
Table 4. Table with gender changes and mean (SD) format.
VariableMale M (SD)Female M (SD)tp-ValueCohens d95% CI
Alerting−114.60 (21.84)−115.24 (21.92)0.180.860.03[−6.53, 7.82]
Orientation−26.06 (25.83)−28.20 (27.93)0.470.640.08[−6.69, 10.96]
Executive−48.81 (20.22)−40.65 (21.42)−2.340.02−0.39[−14.99, −1.33]
Table 5. Correlation matrix of study variables.
Table 5. Correlation matrix of study variables.
ATTC_TOTATTC_FOCATTC_SHIIESAlertingOrientationExecutive
ATTC_TOT1
ATTC_FOC0.65 ***1
ATTC_SHI0.70 ***−0.091
IES0.052.91 × 10−30.061
alerting−0.09−0.11−0.010.021
orientation−0.04−0.04−0.01−0.130.60 ***1
Executive−0.02−0.160.130.090.19 *−1.55 × 10−31
* p < 0.05, and *** p < 0.001.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Khodami, M.A.; Battaglini, L.; Jansarvatan, M.; Kireeva, S.; Bagheri, S. Comparing Self-Report vs. Performance Measures of Attentional Control and Efficiency. NeuroSci 2024, 5, 114-127. https://doi.org/10.3390/neurosci5020008

AMA Style

Khodami MA, Battaglini L, Jansarvatan M, Kireeva S, Bagheri S. Comparing Self-Report vs. Performance Measures of Attentional Control and Efficiency. NeuroSci. 2024; 5(2):114-127. https://doi.org/10.3390/neurosci5020008

Chicago/Turabian Style

Khodami, Mohammad Ahsan, Luca Battaglini, Maryam Jansarvatan, Sofia Kireeva, and Seiran Bagheri. 2024. "Comparing Self-Report vs. Performance Measures of Attentional Control and Efficiency" NeuroSci 5, no. 2: 114-127. https://doi.org/10.3390/neurosci5020008

APA Style

Khodami, M. A., Battaglini, L., Jansarvatan, M., Kireeva, S., & Bagheri, S. (2024). Comparing Self-Report vs. Performance Measures of Attentional Control and Efficiency. NeuroSci, 5(2), 114-127. https://doi.org/10.3390/neurosci5020008

Article Metrics

Back to TopTop