Analysis of Reliability and Generalizability of One Instrument for Assessing Visual Attention Span: MenPas Mondrian Color

: Attention is one skill related to processes such as memory or learning, so, its evaluation is very interesting in areas such as clinical, educational or sports. The aim of this paper is to analyze the reliability and generalizability of one online computerized tool, named MenPas Mondrian Color, that has been developed for the visual attention span assessing and training. In addition, it has been intended to determine any existing relationships among the di ﬀ erent parameters of the tasks performed in order to check the coherence of the results obtained in the executions. In 11,540 analyzed executions of 1064 users from di ﬀ erent American, African and European countries, 6543 of them were performed by women (56.70%) and 4997 by men (43.30%). The age distribution showed that all of the participants were aged 18–55 years, with an average of 25.50 ± 8.91 years. The analyzed tool is called MenPas Mondrian Color which is included in the MENPAS 1.0 platform. Reliability (Cronbach’s Alpha), variance components and generalizability analyses were carried out in order to analyze the quality of the data gathered by this tool. The obtained results indicated optimal scores in the analyses performed, suggesting that the data gathered are reliable, precise and statistically generalizable to a larger population. Likewise, correlation analyses indicated that the di ﬃ culty of the task is related to the e ﬀ ectiveness in its executions, indicating that this is a highly sensitive tool.


Introduction
Attention is a brain function related to numerous processes such as perception, memory, or learning [1]. It contributes to the selection and adequate treatment of sensory information stored in the memory, facilitating the output of appropriate responses [2,3]. Therefore, it is considered as an ability linked to the processes that allow for human adaptation to the environment, and also what makes its study be especially interesting in areas such as clinical, educational, public health, labor or sports [4][5][6][7]. An adequate level of cognitive functioning and, specifically, a good attentional capacity, is an essential factor in human functioning, which affects their well-being [1,3,5].
Currently, most researchers argue that it is not a unitary mechanism, but rather a highly complex neuroanatomical and neurofunctional system, acting on various processes and taking shape anatomically in a wide range of brain areas [8,9]. In fact, it is based on an extensive network of cortical and subcortical connections, where structures such as the thalamus, limbic system, basal ganglia, anterior reticular system, parietal cortex, prefrontal cortex, anterior cingulate cortex, the upper midbrain collicles or the cerebellum are involved [10][11][12][13].
to obtain a more exact response time, knowing exactly the time where one stimulus is displayed or the time interval between one stimulus and the following one [56]. In addition, computers allow the use of flexible and creative resources that increase the motivation during exercise (such as sounds, colors, images, etc.), which could be an important point in some cases [57,58].
This increment in the available resources consolidate the possibility of offering alternative and complementary routes to the pharmacological ones in the intervention on certain situations and population groups, which is one aspect where there is a high demand [59]. Additionally, it facilitates as the data management as the flow of information between the professionals using them, by optimizing the processes of intervention on these cognitive abilities [47]. However, as a result of the proliferation of these types of tools and their increasingly widespread use, one of the questions raised is about the reliability of these types of tools and the quality of the data obtained by them.
Therefore, and based on the related background, the purpose of this work is to present preliminary data about reliability, generalizability and some correlation results of several configuration variables of the MenPas Mondrian Color (MenPas Cell, MenPas) task. Therefore, this study will try to determine if: (a) the analyzed instrument generates reliable, precise and generalizable data, and (b) the exercise results are altered by the instrument configuration handling.

Participants
A total of 11,540 executions were carried out by 1064 users from different countries in America (n = 651; 5.64%), Africa (n = 36; 0.31%) and Europe (n = 10,853; 94.04%)-most of them from Spain (n = 10,609; 91.93%). Of the total executions, 6543 were taken by female (56.70%) and 4997 by male (43.30%). Participants' ages were between 18 and 55 years, where the average age was 25.50 ± 8.91 years. The inclusion criteria were: (a) carry out the test between 2009 and 2013; (b) be between 18 and 55 years old; (c) make less than 40 errors during the execution of the exercise; (d) perform the exercise from 30 to 300 s; (e) perform the exercise with a displayed time from 5 to 30 s; (f) use a matrix size from 2 × 2 to 5 × 5.

Instruments and Measures
MenPas Mondrian Color. This computerized tool is used to evaluate and train the visual attention span, although it also involves other cognitive aspects such as short-term memory and working memory. This tool allows the creation of a color matrix, making up what is known as a Mondrian. The task consisted in visualizing a set of colors, located in a specific position of the matrix, and after a time of exposure, pointing to where they were located. It was hosted in MenPas Cell, downloadable from the MenPas online Psychosocial Evaluation Platform (www.menpas.com) [47,48,51]. This software is a Windows desktop application developed under the NET platform in the C# programming language and with the Visual Studio programming Integrated Development Environment (IDE). When starting the application, the program asks for the MenPas platform username and password. The test configurator allows one to set the size of the matrix (rows and columns), the number of colors, the displayed time and the time of the test. When the test starts, the test matrix appears with colors located in a position (see Figure 1). The colors disappear when the displayed time ends. Starting from here, the participant must point out where each color was located (see Figure 2). In order to do this, the participant must click by using the mouse on a color (the set of possible colors appears below the matrix) and then on the rectangle that he/she thinks was the color. Hits and errors of each run are stored in a database, which can be retrieved later.

Procedure
Data have been gathered by the research team collecting information from many MenPas Mondrian Color runs from January 2009 to December 2013. The request to participate in the study and the use of this tool was disseminated through the MenPas platform where the program is hosted. Thus, the participants could perform their exercises from home. Further, these data were stored in the database of the online platform. In order to get an informed consent, end-users were informed about what data are being gathered, who will have access to them, and how they will be used when users are registered on the platform, where their approval is mandatory for storing and processing those scores. Data processing is, obviously, anonymous. The exercises recorded took from 30 to 300 seconds, had a matrix size from 2 × 2 to 5 × 5, used 10 colors maximum and a displayed time from 5 to 30 s. The ethical principles of the Declaration of Helsinki [60] were respected throughout the research process. The study was approved by an ethics committee (CEUMA, no. 243, 19-2015-H) of the University of Málaga (Spain).

Procedure
Data have been gathered by the research team collecting information from many MenPas Mondrian Color runs from January 2009 to December 2013. The request to participate in the study and the use of this tool was disseminated through the MenPas platform where the program is hosted. Thus, the participants could perform their exercises from home. Further, these data were stored in the database of the online platform. In order to get an informed consent, end-users were informed about what data are being gathered, who will have access to them, and how they will be used when users are registered on the platform, where their approval is mandatory for storing and processing those scores. Data processing is, obviously, anonymous. The exercises recorded took from 30 to 300 seconds, had a matrix size from 2 × 2 to 5 × 5, used 10 colors maximum and a displayed time from 5 to 30 s. The ethical principles of the Declaration of Helsinki [60] were respected throughout the research process. The study was approved by an ethics committee (CEUMA, no. 243, 19-2015-H) of the University of Málaga (Spain).

Procedure
Data have been gathered by the research team collecting information from many MenPas Mondrian Color runs from January 2009 to December 2013. The request to participate in the study and the use of this tool was disseminated through the MenPas platform where the program is hosted. Thus, the participants could perform their exercises from home. Further, these data were stored in the database of the online platform. In order to get an informed consent, end-users were informed about what data are being gathered, who will have access to them, and how they will be used when users are registered on the platform, where their approval is mandatory for storing and processing those scores. Data processing is, obviously, anonymous. The exercises recorded took from 30 to 300 s, had a matrix size from 2 × 2 to 5 × 5, used 10 colors maximum and a displayed time from 5 to 30 s. The ethical principles of the Declaration of Helsinki [60] were respected throughout the research process. The study was approved by an ethics committee (CEUMA, no. 243, 19-2015-H) of the University of Málaga (Spain).

Results
In order to analyze the quality, reliability and generalizability of the data obtained (hits and errors for a specific execution (structure of the task matrix = rows × columns)), they were estimated according to the variables: gender (model 1), origin (model 2), age (model 3) and level of study (model 4).
Model 1: Hits, errors and structure according to the gender. Table 1 shows the analyses carried out according to gender, as well as the reliability estimated using Cronbach's Alpha. As can be seen, both male and female reliability indices are greater than 0.70. , the error variance in both procedures is the same, so it is assumed that the sample is normal, linear and homoscedastic [71,72]. Furthermore, the model and all its facets and interactions are significant (<0.0001) and explains 71.38% of the variance; and a Coefficient of Variation of 62.08, indicates that the data set is homogeneous. When a generalizability analysis was carried out, the following results have been obtained ( Table 2), indicating that data are reliable and generalizable. Model 2: Hits, errors and structure according to the origin of its participants. Table 3 shows the analyses carried out according to the origin of the participants, as well as the reliability estimated using Cronbach's Alpha. As can be seen, reliability indices are greater than 0.70, except for the executions carried out in America where the index is 0.68. , the error variance in both procedures is the same, all its facets and interactions are significant (<0.0001), with the exception of the interaction c×e×s, and explains 71.07% of the variance and a Coefficient of Variation of 62.02. Therefore, in both cases, the homogeneity of the data set can be assumed, and also that the sample is normal, linear and homoscedastic [71,72]. In addition, generalizability analyses have been carried out ( Table 4), indicating that data are reliable and generalizable. Model 3: Hits, errors and structure according to age Table 5 shows the analyses carried out according to the age of the participants, as well as the reliability estimated using Cronbach's Alpha. As can be seen, reliability indices are greater than 0.71 for all age intervals. , the error variance in both procedures is the same, all its facets and interactions are significant (<0.0001), explains 76.76% of the variance, and shows a Coefficient of Variation of 56.54. Therefore, in both cases the homogeneity of the data set can be assumed, as well as the sample being normal, linear and homoscedastic [71,72]. Additionally, generalizability analyses have been carried out ( Table 6), indicating that data are reliable and generalizable.  Table 7 shows the analyses carried out according to the educational level of the participants, as well as the reliability estimated using Cronbach's Alpha. As can be seen, reliability indices are greater than 0.71 for all age intervals. , the error variance in both procedures is the same, so it can be assumed that the sample is normal, linear and homoscedastic [71,72]. Furthermore, in the model, all its facets and interactions are significant (<0.0001), except for t×h×e×s, and explains 70.30% of the variance, and a Coefficient of Variation of 63.46, indicating that the data set analyzed is homogeneous. When a generalizability analysis has been carried out, the following results are obtained ( Table 8), suggesting that data are reliable and generalizable. Analysis of correlations between the waiting time variables and the number of colors with hits and errors depending on the size of the matrix. Table 9 shows the descriptive statistics and normality of the variables' successes, errors, displayed time and number of colors used during the task according to the size of the matrix used. Structures used for these analyses have been executed in more than 100 cases. Specifically, 2 × 2 (n = 968), 2 × 3 (n = 686), 3 × 3 (n = 6209), 3 × 4 (n = 689), 4 × 4 (n = 1884), 4 × 5 (n = 225) and 5 × 5 (n = 470) matrices have been analyzed. These structures, all together, are the 96.46% of the total. The correlation values between the hit and error variables with displayed time and number of colors are also shown. As can be seen, the values of asymmetry, kurtosis and Kolmogorov-Smirnov indicate that the data distributions were not normal. Thus, Spearman's test has been used for correlation analysis. As can be seen, the number of colors is positively related to errors and negatively related to hits. Taking into account the displayed time, in smaller structures (2 × 2, 2 × 3 and 3 × 3), there is a positive relationship with the hits and a negative relationship with the errors. However, it is not significantly related to the number of errors in the rest of the structures, because it is positively related to the hits in the 3 × 4 and 4 × 5 matrix, but negatively in the 4 × 4 and 5 × 5 structures.

Discussion
The aim of the present work is to analyze the reliability and generalizability of the data obtained by the MenPas Mondrian Color software (MenPas Cell, MenPas). In order to do that, Cronbach's Alpha analysis, variance and generalizability components have been carried out. The results show optimal values, which suggest an adequate quality of the data obtained. We also have tried to determine any relationships between the difficulty of the task (the number of colors in each matrix and the displayed time of the exercise prior to its execution have been analyzed) and the hits/errors made, observing that there is a congruent association between these parameters.
In order to evaluate the psychometric characteristics of the tool, different models have been tested, with the aim of checking the sensitivity of the tool in a wide spectrum of executions and diverse characteristic participants. All of them have shown that data are accurate and reliable, as well as generalizable, suggesting that it is an appropriate tool to be used for that evaluation. In addition, these results show an appropriate response to different variables such as age, gender, origin or education level, which allows us to infer that its capacity to provide quality data when assessing diverse populations is not affected. Among other reasons, this tool lacks cultural elements [73], because it uses simple stimuli common to all cultures, such as colors. This feature is important, as has been highlighted in previous studies with instruments of the same nature [74]. It is not influenced by prior learning, nor is it necessary to interpret a particular knowledge in a context. Therefore, it is considered a useful tool to be widely used.
Data found also indicated that a greater size structure of the used matrix generates a higher level of hits and a higher level of errors, maintaining a proportional and coherent relationship with the structure of the matrix used. The aim of using this tool is that when the size of the task matrix increases, then the participant is forced to retain a greater number of items. Therefore, the probability of making a greater number of errors is increased. That difficulty, and the ability of the tool to force more errors when the structure increases, allows for differentiation of expertise at each level of work. If this effect will not be generated, the discriminatory capacity of the tool would decrease and would make it less suitable for measuring this capacity. In addition, the phenomenon described above allows the training of the attentional amplitude. When the size of the task increases, a greater number of stimuli must be recognized and there is a greater probability of error, causing greater stimulation and effort to execute the task correctly. The possibility of voluntarily modifying these elements facilitates the control of the evaluation and training, encouraging the possibilities of using this tool [55].
The number of colors and the displayed time of the group of colors to be remembered later can also be manipulated. In addition to what we previously commented on about the versatility of the task, it allows the participant to tune and condition the user's motivation [55,58]. It has been observed that the increment in the number of colors makes the hits more difficult and increases the number of errors, which suggests that the possibility of manipulating these parameters becomes a positive aspect of the instrument by being able to condition the learning and evaluation of the participants. In previous studies, in which computerized attention assessment instruments have been used, it also has been observed that increasing the stimuli that must be managed during the test determines errors and fewer hits [50]. Likewise, it has been observed how the previously displayed time of the task is related to the successes and errors. It has been observed that up to nine items have a positive association with the successes and negative association with the errors, indicating that this exposure time facilitates the subsequent search for the stimulus. However, when up to 12 items, the relationships are not so clear, and this pattern disappears. This could suggest that the initial visualization during a longer time facilitates the memorization and later visualization of the objective stimuli, although it disappears when the amount of items to be attended is increased. This is consistent with theories that have described the limited capacity of people to retain and use a certain number of units of information [75]. The ability to retain visual information is limited, although it is determined by various parameters such as prior training, the type of information to be retained, or the precision with which the stimulus is recalled. Hence, it is difficult to make exact parallels between performances between tests. Still, it is perceived that based on a quantity of information, there is a pattern in the test presented in this study that shows a decline in the ability to use previously memorized information accurately, which coincides with what has been postulated by other authors [75,76].
The results found show that this instrument is reliable and could be used in various situations as a useful tool for training and cognitive evaluation. Furthermore, it is sensitive to different configurations, which is why it is suggested that this is a test that can be adapted to the needs of users. This test could be applied in schoolchildren, the elderly, cognitive training in athletes or workers who need to improve their cognitive performance. Even in pathologies with cognitive impairment, it could be used as a complementary instrument to assess the cognitive level, even though it is not a diagnostic test.
This work has limitations inherent to the fact that it is a preliminary study of the psychometric characteristics of the tool evaluated. This is a tool with a lot of characteristics that can be modulated, which increases the difficulty of validating all of them. Therefore, future research will try to solve these questions, determining the quality of the data offered by MenPas Mondrian Color. It will also be necessary to analyze its convergent validity to delve into its ability to assess the amplitude of attention. However, these preliminary results are promising, and the results obtained show that it could be a useful, versatile and appropriate tool to explore this capacity. Future works will try to analyze the validity of this instrument in order to confirm that this instrument can evaluate the attentional span.

Conclusions
Reliability (Cronbach's Alpha), variance components and generalizability analyses showed that the data were reliable, precise and generalizable. Additionally, correlation analyses indicated that some task characteristics were related to execution effectiveness. Therefore, a greater number of colors decrease the hits and the errors, and the initial visualization of items facilitates the memorization of the objective stimuli when the matrix is small or moderate in size.
The results obtained in this investigation indicated that the analyzed instrument offers quality data and could be appropriate to be used in evaluations and cognitive training, although it is necessary to develop a validity analysis to confirm the properties of this tool. For example, it would be appropriate to perform convergent and discriminant validity analyses. If these analyses were appropriate, we would be facing a test that could be used for evaluation or intervention in areas such as education, sports or the clinic, among others. Since it is a computerized test, it facilitates its application and data collection. Thus, this research offers interesting data on a new resource to be applied in the field of cognitive functioning.

Conflicts of Interest:
The authors declare no conflict of interest.