Complex Thinking and Sustainable Social Development: Validity and Reliability of the COMPLEX-21 Scale

: Thinking skills are essential to achieve sustainable social development. Nonetheless, there is no speciﬁc instrument that assesses all of these skills as a whole. The present study aimed to design and validate a scale to assess complex thinking skills in adult people. A scale of 22 items assessing the following aspects: analysis and problem solving, critical analysis, metacognition, systemic analysis, and creativity, in ﬁve levels, was created. This scale was validated in 626 university students from Peru. In total, 16 experts in the ﬁeld helped to determine the content validity of the scale (Aiken’s V value higher than 0.8). The conﬁrmatory factor analysis allowed the evaluation of the structure of the ﬁve factors theoretically proposed and the goodness of ﬁt indexes was satisfactory. An item was eliminated during the process and the scale resulted in 21 items. The composite reliability for the different factors was ranged between 0.794 and 0.867. The invariance between genders was also checked and the concurrent validity was proved. The study concludes that the content validity, construct validity, concurrent validity, and composite reliability levels of the COMPLEX-21 scale are appropriate.


Introduction
The development of complex thinking in citizens and communities is essential to achieve sustainable social development. An Australian study found that higher levels of complex thinking are positively associated with the prevention of fires in the community through better communication processes and more stable and less extreme attitudes [1]. The association between complex thinking and the development of math and natural sciences has also been established [2]. Several studies have assessed, from a complex point of view, the improvement in the medical treatment of patients with autoimmune diseases such as HIV [3], and it has been proposed as a reference point to better understand diabetes [4] Complex thinking also allows a better comprehension of the relationship between education and knowledge exchange [5]. Research carried out in the field of education based on complex thinking have improved student learning by helping them to develop skills for using computational programs [6] In the last decades, progress in the field of intellectual or thinking skills [7] has allowed the design and validation of scales to assess these skills, with a special focus on critical thinking, creativity, metacognition, and problem solving, among others. Some of these instruments are general, meant to assess several skills at the same time (for example, the scale of Hanlon et al. [8] or the instrument of Peeters et al. [9] Others are highly specific and aimed at assessing certain thinking skills, such as the scale of Tran et al. [10], which is focused on the assessment of creativity in lessons for students, or the A-E scale of Martisen and Furnham [11] to assess aspects of motivation and problem solving. However, the number of instruments focused on the construction of complex thought itself within the

Complex Thinking as High Order Thinking
Another branch in the study of complex thinking is established by the philosophy of cognitive science and the contributions of Lipman [16], who defines it as high order thinking that connects critical and creative thinking with problem solving or addressing situations. Critical thinking, as Lipman understands it, refers to critical reasoning based on arguments, whereas creative thinking refers to the generation of ideas or actions in the nondiscursive field. In daily life, both kinds of thinking have elements in common and must connect and complement each other. Studies in this field have found a relationship between the critical and creative processes [17][18][19][20]. This is supported by several neuroscience studies that show an interaction between certain cerebral areas that allow both kinds of thinking in human beings. [21][22][23][24]. Simple thinking, unlike complex thinking, and according to Lipman, is mechanical and a routine; it responds to algorithms and fails to connect different skills [15].

Complex Thinking as a Macro-Competence
Another branch in the study of complex thinking understands it as a multidimensional macro-competence consisting of other competencies or skills [25]. These competencies can be defined as "effective behaviors and skills to achieve or carry out successful projects in the future and which allow self-sustainable growth and a more equal development" [26] (p. 233). These competencies have been suggested as "a dynamic combination of knowledge, comprehension, skills, and capabilities" [27] (p. 3).
Recently, the consideration of complex thinking has changed the definition of competence [28]. For this reason, complex thinking has been included in the definition of competence, as can be observed in the approach of Cuadra-Martínez et al. [29], who defines complex thinking as a "competence to develop, in which the student and future worker makes a non-naïve "use" (conscious, explicit and reflexive) of his theories by differentiating or connecting them when needed and according to the professional context" [29] (p. 25).
Nonetheless, universities continue to teach a set of independent and non-related skills. This can be easily observed in the implementation of the Tuning Education Structures in Europe and Alfa Tuning Latin America Projects, which aim to equalize higher education in Europe and Latin America, respectively. In them, the competence of complex thinking is dispersed in a wide set of generic competencies such as the ability to block out the surroundings; analysis and synthesis; critical and self-critical ability; the ability to act properly in new situations; creative ability; the ability to detect, deal with and solve problems; and decision-making ability [30,31].

Complex Thinking as an or Comprehensive Performance
The last line of research is that of socioformation, a curricular, didactic, and assessable approach that aims at educating citizens to face the future challenges of sustainable social development (DSS) [32,33]. This consists of a process in which the members of society enjoy greater and better living conditions, allowing them, in turn, to prosper by means of economic welfare, collaborative work, inclusion, equity, health, and knowledge [34][35][36]. This in turn allows them to consider the construction of a new kind of society, the knowledge society [37]. The following features compose the DSS: (1) collaborative work is essential to creating communities that self-manage their development and consider the protection of the environment and biodiversity [38]; (2) problem solving is intended to promote science and technology to find renewable sources of energy and new means of production, construction, and transport [39]; (3) by empowering the ethical project of life, it is intended to empower, in turn, the citizens to implement urgent actions for an improved quality of life and coexistence with each other and the environment [35], and (4) to face the growing complexity of the new tasks that human beings face due to the emergence of artificial intelligence. To accomplish this, the development of certain skills such as creativity, innovation, and critical analysis needs to be promoted in people and communities [40].
Socioformation is a new educative, curricular, pedagogic, and assessable approach [41] to training citizens, organizations, and communities in sustainable social development. It entails implementing transversal and inclusive projects to overcome environmental challenges by means of ethical choices, collaborative work, entrepreneurship, knowledge co-creation, digital technology, and complex thinking. It is an alternative to other recent educative approaches such as social constructivism and meaningful learning, or even to connectivism. It is distinctive for its aim at the implementation of sustainability in the society, culture, economy, technology, environmental and educative processes.
In socioformation, complex thinking is an action aimed at solving contextual problems by connecting different kinds of knowledge, with creativity, critical thinking, systemic analysis, and metacognition, and by perceiving reality with flexibility, open-mindedness, and confrontation of uncertainty [42]. Socioformation features in the educative models of several Latin American universities, such as the Indo American University of Ecuador [43], the University Centre CIFE in Mexico [44], the National University Hermilio Valdizan in Peru [45], and the National University Federico Villar [46].
According to the socioformative approach, connecting critical analysis [47] with creativity is not enough to develop complex thinking. Three additional skills are necessary to direct the evaluation and development of complex thinking in the strict sense; namely, analysis and problem solving [48]; metacognition; and systemic thinking [49], this last focused on the comprehension of the contextual challenges and processes as dynamic systems demanding a multi-, inter-, and transdisciplinary approach.

Scales to Assess Complex Thinking
Several scales have been designed to assess some skills or dimensions of complex thinking, as shown in Table 1, which shows some of the most recent scales. Some of the scales shown in Table 1 are general, meant to assess a wide range of skills, whereas others are more specific and only focused on one or two abilities. By assessing the following examples, the following conclusions are expected: (1) there are no scales aimed at assessing exclusively the complex thinking construct in itself; (2) there are no scales allowing the assessment of complex thinking together with the contributions of the socioformative approach; (3) most of the scales have more than 30 items, making them of limited practicality due to the excessive number of questions; and (4) systemic thinking is barely considered in the instruments designed up to now. Regarding this last conclusion, we focus now on the case of the Study Engagement Questionnaire (SEQ), designed by Kember and Leung [50] and adapted to the Spanish population by Gargalloet al. [51] (2018). This scale assesses some of the dimensions of complex thinking such as critical thinking, creative thinking, self-managed learning, adaptability, and problem solving. Its purpose is to establish the elements involved in teaching and learning, but it fails in not accounting for systemic thinking or metacognition.

Participants
In total, 626 university degree or pre-degree students from four public universities in Peru participated in the study. Of the participants, 64.1% and 35.9% were women and men, respectively. Their ages ranged between 16 and 33 years old, with an average age of 20.78 DS + 3.3). Additionally, 32.2% of the participants claimed to work complementarily to their studies at the university. The selection of the sample was non-probabilistic, by open call and email.

Procedure
An instrumental study was carried out around the design of a new instrument to evaluate complex thinking, based on the Likert scale. For this, four stages were carried out: (1) design and peer review of the instrument; (2) content validity analysis by expert judges; (3) study of construct validity by confirmatory factor analysis; and (4) analysis of the evidence of convergent and discriminant validity. The stages and participants are described below.
Stage 1. Design and peer review process. Based on the theoretical references described in the literature review (Morin's epistemological approach, Lipman's higher-order thinking, complex thinking as macrocompetence, and the contributions of socioformation), the essential complex thinking skills that he had to tackle the new instrument. Some recent instruments on the subject were also analyzed in Table 1. Based on this, a draft of the instrument was made by the authors, which was later improved with the support of three experts in the area. The three experts presented the following characteristics: (1) doctorate in psychology, education, or social sciences; (2) experience of more than 15 years in the design and validation of instruments; and (3) have at least five publications on cognitive processes.
Stage 2. Assessment of the content validity. After the scale was designed with the support of the experts, this was assessed by 16 judges with experience in the field of complex thinking. At this stage, every subscale was assessed with two indicators; appropriateness of the questions and clarity in the writing. A Likert-kind scale with four levels from 1 to 4 (in which 1 is the lowest level and 4 the highest one) was used. The judges were also asked to make suggestions to improve the quality of each scale, such as adding or removing questions, improving the readability, and adding new scales.
Afterward, the 16 judges were asked to evaluate the instrument as a whole, with the same indicators of appropriateness and writing. At this stage, a third indicator was added, satisfaction, which was measured on a scale of 1 to 5 (from a very low satisfaction (1) to a high one (5)). To assess the judges' level of agreement, Aiken's V was used and the values accepted were the ones higher than 0.8 [66]. All of the judges were previously contacted via email, in which they received all the necessary information regarding the purpose of the instrument, and were asked to make suggestions to remove, add or improve the questions. In selecting the judges, all were required to have provable experience in the field of complex thinking skills, to be investigators and university professors, to have at least one Master's degree, to have published regarding the topic, and to have experience in the review or design of instruments of this nature. The features of the judges are described in Table 2. Stage 3. Construct validity. The whole sample of university students (n = 626) was used to carry out a confirmatory factor analysis with the approach of the maximum likelihood estimation. This technique was used since there are theoretically five factors and it was intended to establish if these factors were verified in the study. For this reason, two models have been tested: the first one with four factors and the second one with five. Finally, Cronbach s alpha [67] was used to determine the reliability and the composite reliability coefficient [68]. In the same way, and by using a multigroup analysis, a factor invariance analysis by gender was carried out. The confirmatory factor analysis was carried out in the AMOS 27 program.
Stage 4. Convergent and discriminant validity. To assess the convergent validity of the model, that is to say, if the constructs evaluated were effectively assessed by the instrument, three criteria were used. Firstly, the factor weight of the questions in their respective factors had to be higher than 0.50 [69]; secondly, the composite reliability index to be higher than 0.70 [69], and thirdly, the average variance extracted (AVE) to be higher than 0.50 [70]. A sample of 626 students was used to determine the convergent validity. Additionally, the assessment of the discriminant validity was carried out with the same program AMOS 27.

Ethical Aspects
The Mexican Law of Personal Data Protection was followed over the course of the research since the participants emails were requested in order to send them the instrument link. Additionally, all participants were informed of the aim of the study and each one signed an online consent letter. All participants were free to leave the process at any time and without consequences. After answering the questions, they were allowed to know the result of the research to benefit from the process and to implement improvements if needed. The study was approved by the Institutional Ethics Committee. Table 3 describes the complex thinking skills chosen for the current study based on the review of scientific literature. A total of five complex thinking skills were chosen: problem solving, systemic thinking, creativity, metacognition, and flexibility. Each of these skills was evaluated on a frequency scale from 1 to 5, in which 1 is "Never" and 5 "Always". The Likert-like scale of 22 items is shown in Appendix A. Systemic analysis "Is a methodology for analyzing and solving problems using systemic research and comparison of alternatives that are performed on the basis of the cost-to-cost ratio for their implementation and the expected results."

Stage 1. Design and Peer Review Process
Velkovski [74] (p. 322) 3 Creativity "Creatively is a divergent thinking process, which is the competences to provide alternative answers based on the information provided" Amrina et al. [75] (p. 129) 4

Stage 2. Content Validity
All 16 judges agreed with the suitable level of appropriateness and comprehension of the Complex Thinking Skills Scale regarding its four subscales and the instrument as a whole, as shown in Table 4. The Aiken s v values were higher than 0.8 in the two aspects assessed on the five scales. There was also agreement regarding the level of satisfaction of the global scale (V ≥ 0.8), thus, showing the validity of the instrument [66] (Table 4). Some judges made suggestions to improve several writing aspects, and these were implemented prior to the application to the general sample.

Stage 3. Construct Validity
Confirmatory factor analysis was used to validate the scale. The distribution of all items is similar to the normal one, which was assessed by using the kurtosis approach and asymmetry (Table 5), and by using the values proposed by Curran et al. [76]: The asymmetry and kurtosis coefficients were within the range of +1 and −1, and the sample had a normal distribution, according to the Shapiro-Wilk test. With these data, it is possible to proceed with the factor analysis itself. The convenience of using the maximum likelihood estimation as an extraction method is confirmed [77]. The first step in the confirmatory factor analysis was to determine the factor weights, which are described in Table 6. All of the factor weights of the items were higher than 0.5, and hence are considered significant [69], except for item 6 ("Do you question the facts to find opportunity areas and to implement improvements?") with a factor weight of only 0.425. For this reason, this item was eliminated and the confirmatory factor analysis was carried out again. Next, the 5-factor model postulated in the instrument was tested. Table 7 shows the goodness of fit indexes; they were positive and confirmed the theoretical model proposed. Since there is a wide range of such indexes, techniques suggested by Hair et al. [69] were used: a combination of chi-square reduced (χ 2 /gl), Tucker-Lewis index (TLI), comparative fit index (CFI), and the root mean square error of approximation (RMSEA). Even using the strictest criteria [78], requiring the TLI and CFI to be higher than 0.95, and the RMSEA lower than 0.6, the model still fits the data. Itis important to point out that, even though the chi-square test was significant, this index is especially sensitive to sample size, unlike the TLI for example, which turns it into a less-reliable index in this case [79] (Figure 1).  Finally, a multigroup analysis assessed the factor invariance of the instrument by gender. By following the most usual recommendations, configure, metrical, and scalar invariance was estimated, while the residual variance was discarded due to its low practical value [80]. Using the method of Cheung y Rensvold [81], differences lower than 0.01 in the CFI index not considered to be enough to discard the invariance hypothesis, whereas the differences between the RMSEA of the configurational model and the metrical and scalar models were not higher than 0.15 [82]. The results show no major differences between men and women (Table 8). Finally, a multigroup analysis assessed the factor invariance of the instrument by gender. By following the most usual recommendations, configure, metrical, and scalar invariance was estimated, while the residual variance was discarded due to its low practical value [80]. Using the method of Cheung y Rensvold [81], differences lower than 0.01 in the CFI index not considered to be enough to discard the invariance hypothesis, whereas the differences between the RMSEA of the configurational model and the metrical and scalar models were not higher than 0.15 [82]. The results show no major differences between men and women (Table 8).  Table 9 shows the results of the convergent validity. The average extracted variance was higher than 0.5 [83]. Additionally, the composite reliability was higher than 0.7, which is a positive indicator [84]. Thus, we conclude that the questions assess effectively the constructs established in each factor and demonstrate convergent validity. Additionally, the results of the discriminant validity, which assesses whether the factors are clearly different from each other, were more complex. Discriminant validity was assessed using the method proposed by Fornell and Larcker [70], in which the square root of the average extracted variance must be higher than the correlations matrix of that factor with the other factors. In this study, the correlation of the factors was high (see Figure 1) and we expect the correlations matrix to have higher scores than the square of the average extracted variance (Table 10). These results are varied. However, this is not necessarily problematic, since the scale assesses related factors. Thus, its purpose is not to reach a specific conclusion about the differences, but a conclusion of complex thinking practice itself. Note. Items in bold are the square root of the average extracted variance (AVE). These values must exceed the correlations between factors (off-diagonal Items) for adequate discriminant validity.

Discussion
Complex thinking, in the socioformative approach, is an or comprehensive performance that citizens and communities must carry out to achieve sustainable social development. This is the main challenge that humankind faces nowadays, and encompasses other related challenges such as disease prevention and health promotion; quality of life and socioeconomic development; inclusive economy; housing; transport and education; and pollution prevention, mitigation of global warming, and biodiversity preservation. Therefore, developing this comprehensive performance would allow facing the context and his problems with a systemic vision. For example, the case of the current COVID-19 pandemic [85]. This context has highlighted the lack of training in complex thinking in many leaders and citizens. This issue needs to be understood as a consequence, prevention efforts in many countries have not had the expected impact (see, for example, Roozenbeek et al. [86]). In this specific problem, the need to generate transdisciplinary actions has been suggested [87], which also applies to the climate crisis and other problems of humanity.
These are the reasons why the current research offers a new instrument to efficiently assess the complex thinking skills in adult people, named the Complex Thinking Essential Skills Scale (COMPLEX-21), and to direct better the educative actions of human talent. To this end, the analysis of a group of 14 experts in the field proved the content validity of the COMPLEX-21, with Aiken's V values higher than 0.80 [66]. This means that this scale is a useful, appropriate, and understandable instrument for potential users.
Additionally, the validity of the instrument was demonstrated, since the five factors postulated at a theoretical level for complex thinking (problem solving, critical analysis, metacognition, systemic analysis, and creativity) were assessed, and the results showed that the goodness of fit indexes met the criteria established in this field [88]. In addition, most of the items had a suitable factor weight and higher than 0.5. There was only one question with a non-suitable factor weight and it was eliminated from the instrument for this reason. Besides, the scale showed no difference by gender and. Therefore, it can be recommended for practical use.
The major problem is the one referred to the discriminant validity, in other words, the clear differentiation between the constructs assessed for each factor. We explain this in several ways. A possible explanation might be that the five factors are closely related to each other and thus, cannot be sufficiently separated. This would not be a single case (see Disabato et al. [89]; Gau [90]). Future studies of this scale will address this aspect to come to more decisive conclusions.
There is plenty of information and lots of proposals regarding complex thinking and its skills. Nonetheless, the instruments are quite general and do not address "Complex thinking" itself as a construct; many of these skills are assessed separately and systemic analysis does not seem to be that important. Nowadays, it is essential to assess these skills as a whole since complex thinking is increasingly a skill that every citizen must develop, including students, teachers, and university principals [43]. For this reason, the current study offers an appropriate scale with good initial psychometric properties to help meet this need in universities and encourage the practice of complex thinking in graduates. The end goal is the creation of a better and more environmentally friendly society.
In addition, the present study helps to understand the structure of complex thinking as a practice, just as socioformation proposes [41]: a new pedagogic approach, created with the support of leaders in education, community, and organizations to face the challenges of sustainable social development. Thus, the socioformative approach comprehends the complex thinking construct as an or comprehensive performance to solve the problems in the way of sustainable social development. This different from the assumption of the epistemological approach [12], which tends not to go beyond the philosophical plane; or from high order thinking [15] which does not consider the improvement of life conditions and biodiversity; or from macro-competence [25], which does consider the skills, attitudes, and knowledge but, does not go beyond a fragmented view of the concept, spends excessive time in the planning of processes and has no connection to sustainable development. Nonetheless, the contributions of these three perspectives have been considered in the socioformative approach but in the framework of a new proposal, more focused on the social aspect and nature.
Increasingly, Ibero-American Universities and educative institutions are implementing the socioformation in their systems [43][44][45][46]. Thus, the instrument validated in the present study will contribute to its consolidation by allowing university students, teachers, and principals a better and more efficient validation of complex thinking as a practice (or comprehensive performance). The current study provides information about the practical structure of complex thinking, which is composed of five essential skills interconnected to each other. Within the socioformation approach, and despite the changes taking place in higher education, it is essential to implement actions that promote complex thinking for all students. This is because many organizations keep insisting on teaching strategies that do not enhance complex thinking but the skills and abilities of simplistic thinking [48].
The COMPLEX-21 scale differs from other instruments assessing specific skills of complex thinking, such as the Study Engagement Questionnaire (SEQ) [50,51], the Watson-Glaser, the Critical Thinking Appraisal [52,53] and the California Critical Thinking Disposition Inventory [54,55] because these assess several dimensions of complex thinking but not the construct as a whole; these scales have a large number of items (around 70 items) and systemic analysis is relegated to a second plane.
The current scale has similar features to other instruments such as the study carried out by Gargallo et al. [51], which assesses, as a whole, the learning-teaching process in the university to provide feedback to the teachers and institutions so they can improve these processes. In that study, 805 people from three Valencian universities were evaluated with a survey assessing the abilities of several students and the ability of the teacher to create a suitable learning environment. Another related instrument is the one designed by Amrina et al. [75] which assesses the competence of the students in achieving the development of logic, critical and creative thinking by employing a Like scale (1-5 points). One difference between the study carried out by Amrina et al. [75] and ours is that our COMPLEX-21 is designed for several fields, not only for math. Additionally, their research did not consider systematic thinking. Amrina et al. [75] are similar to the COMPLEX-21 since both assume that both critical and creative thinking are necessary for the problem-solving process, just like the scales used by Heilat and Seifert [91], and Martinsen and Furnha [11]. Nonetheless, the COMPLEX-21 has its features, such as its focus on the complex thinking itself and not on the learning-teaching process.
The present study was exploratory; thus, it has limitations. Firstly, we considered only a subset of the possible skills of complex thinking due to the limited information available about this topic and the non-agreement among experts, who considered the five skills described above as essential and who decided not to add new ones to the list. The same happened with the judges during the content validity stage, since three of these judges proposed more skills, did not come into agreement. the two of the proposed skills were conceptual analysis and knowledge management, which have been associated with complex thinking [92]. For future studies, we recommend including an analysis of these two skills and integrating them with specific items. Secondly, the validity process would have been improved by joining it with analysis of other processes, such as concurrent validity with other similar instruments [93], predictive validity, and test-rest stability [94]. These studies were not carried out since this research was part of another research in university students. As a result, it was not possible to add further tests or tests on a larger subsample of students.
Although this is an exploratory study and new research is needed around the characteristics of the COMPLEX-21, some practical implications are described below: (1) the new scale will help to assess complex thinking as an integral performance from the framework of socioformation, because although complex thinking is considered a relevant axis for education [95]; There are no instruments to evaluate this process from the references discussed throughout this study; (2) COMPLEX-21 may contribute to reducing the high number of competencies studied in the training of citizens, since many skills integrated in complex thinking are addressed as separate competencies [30,31]; (3) the new instrument could help to determine more precisely the factors associated with better management to achieve sustainable social development, considering complex thinking as an integrative performance, as proposed from the socioformative approach; (4) based on COMPLEX-21, the educational models of educational institutions and universities could have more clarity regarding this process and its associated factors and (5)

Conflicts of Interest:
The authors declare no conflict of interest.  17. Do you intend to join forces with others to understand and solve problems of context more efficiently?
18. Do you intent to identify uncertain situations while addressing problems and do you face them with flexible strategies? 20. Are you the one proposing solutions to the problems and are these different from the ones already established in the context and the reported ones in bibliography resources?
21. Do you change the way in which you explain and solve a problem, through a different synthesis, a question that changes the analysis or even a new solution?
22. Do you intend to make a great impact in the problem-solving process regarding what has been done up to now and by following new strategies?
Note: the questions marked as * were eliminated from the scale as a result of the confirmatory factor analysis.