Complex Thinking and Sustainable Social Development: Validity and Reliability of the COMPLEX-21 Scale

Abstract

Thinking skills are essential to achieve sustainable social development. Nonetheless, there is no specific instrument that assesses all of these skills as a whole. The present study aimed to design and validate a scale to assess complex thinking skills in adult people. A scale of 22 items assessing the following aspects: analysis and problem solving, critical analysis, metacognition, systemic analysis, and creativity, in five levels, was created. This scale was validated in 626 university students from Peru. In total, 16 experts in the field helped to determine the content validity of the scale (Aiken’s V value higher than 0.8). The confirmatory factor analysis allowed the evaluation of the structure of the five factors theoretically proposed and the goodness of fit indexes was satisfactory. An item was eliminated during the process and the scale resulted in 21 items. The composite reliability for the different factors was ranged between 0.794 and 0.867. The invariance between genders was also checked and the concurrent validity was proved. The study concludes that the content validity, construct validity, concurrent validity, and composite reliability levels of the COMPLEX-21 scale are appropriate.

1. Introduction

The development of complex thinking in citizens and communities is essential to achieve sustainable social development. An Australian study found that higher levels of complex thinking are positively associated with the prevention of fires in the community through better communication processes and more stable and less extreme attitudes [1]. The association between complex thinking and the development of math and natural sciences has also been established [2]. Several studies have assessed, from a complex point of view, the improvement in the medical treatment of patients with autoimmune diseases such as HIV [3], and it has been proposed as a reference point to better understand diabetes [4] Complex thinking also allows a better comprehension of the relationship between education and knowledge exchange [5]. Research carried out in the field of education based on complex thinking have improved student learning by helping them to develop skills for using computational programs [6]

In the last decades, progress in the field of intellectual or thinking skills [7] has allowed the design and validation of scales to assess these skills, with a special focus on critical thinking, creativity, metacognition, and problem solving, among others. Some of these instruments are general, meant to assess several skills at the same time (for example, the scale of Hanlon et al. [8] or the instrument of Peeters et al. [9] Others are highly specific and aimed at assessing certain thinking skills, such as the scale of Tran et al. [10], which is focused on the assessment of creativity in lessons for students, or the A-E scale of Martisen and Furnham [11] to assess aspects of motivation and problem solving. However, the number of instruments focused on the construction of complex thought itself within the socioformative framework is remarkably low or non-existent. There are currently no scales aimed at assessing complex thinking construction found on SCOPUS or Web of Science using the search equation: TITLE–ABS-KEY ((scale OR questionnaire OR rubric) AND (“complex thinking”) AND (validity OR reliability OR relevance)).

Thus, the aim of this study was: (1)to design a brief scale to assess the essential skills of complex thinking in one instrument by considering a socioformative approach and the suggestions of a group of experts in the field; (2) to test the validity of the scale with the support of a group of experts from several institutions who assess the appropriateness, writing, and satisfaction of the instrument; and (3) to determine the construct validity and concurrence, as well as the reliability of the new scale.

2. Literature Review

2.1. The Complex Thinking as an Epistemological Approach

Complex thinking has several definitions, which need to be explained in any study of this kind. For Morin [12], who represents the most epistemological line, complex thinking consists of the study of social and environmental events in a continuous organization and reorganization process resulting from the connection between different elements and economic, political, sociological, and mythological dimensions [13]. To this author, the network of elements and processes is the complex part [13] and not if it is either complicated or difficult [14]. Nonetheless, to understand the world in its totality, a complex mind is required because, otherwise, we could easily make assumptions about reality in a rigid, one-dimensional, lineal, and separated way.

A well-organized or complex mind [15] is made up of certain skills such as contextualization, systemic analysis, flexibility, connection, and confrontation of uncertainty. These allow the human being to withdraw from one dimensionality, linearity, reductionism, or non-willingness to change, which are typical characteristics of simplistic thinking.

2.2. Complex Thinking as High Order Thinking

Another branch in the study of complex thinking is established by the philosophy of cognitive science and the contributions of Lipman [16], who defines it as high order thinking that connects critical and creative thinking with problem solving or addressing situations. Critical thinking, as Lipman understands it, refers to critical reasoning based on arguments, whereas creative thinking refers to the generation of ideas or actions in the non-discursive field. In daily life, both kinds of thinking have elements in common and must connect and complement each other. Studies in this field have found a relationship between the critical and creative processes [17,18,19,20]. This is supported by several neuroscience studies that show an interaction between certain cerebral areas that allow both kinds of thinking in human beings. [21,22,23,24]. Simple thinking, unlike complex thinking, and according to Lipman, is mechanical and a routine; it responds to algorithms and fails to connect different skills [15].

2.3. Complex Thinking as a Macro-Competence

Another branch in the study of complex thinking understands it as a multidimensional macro-competence consisting of other competencies or skills [25]. These competencies can be defined as “effective behaviors and skills to achieve or carry out successful projects in the future and which allow self-sustainable growth and a more equal development” [26] (p. 233). These competencies have been suggested as “a dynamic combination of knowledge, comprehension, skills, and capabilities” [27] (p. 3).

Recently, the consideration of complex thinking has changed the definition of competence [28]. For this reason, complex thinking has been included in the definition of competence, as can be observed in the approach of Cuadra-Martínez et al. [29], who defines complex thinking as a “competence to develop, in which the student and future worker makes a non-naïve “use” (conscious, explicit and reflexive) of his theories by differentiating or connecting them when needed and according to the professional context” [29] (p. 25). Nonetheless, universities continue to teach a set of independent and non-related skills. This can be easily observed in the implementation of the Tuning Education Structures in Europe and Alfa Tuning Latin America Projects, which aim to equalize higher education in Europe and Latin America, respectively. In them, the competence of complex thinking is dispersed in a wide set of generic competencies such as the ability to block out the surroundings; analysis and synthesis; critical and self-critical ability; the ability to act properly in new situations; creative ability; the ability to detect, deal with and solve problems; and decision-making ability [30,31].

2.4. Complex Thinking as an or Comprehensive Performance

The last line of research is that of socioformation, a curricular, didactic, and assessable approach that aims at educating citizens to face the future challenges of sustainable social development (DSS) [32,33]. This consists of a process in which the members of society enjoy greater and better living conditions, allowing them, in turn, to prosper by means of economic welfare, collaborative work, inclusion, equity, health, and knowledge [34,35,36]. This in turn allows them to consider the construction of a new kind of society, the knowledge society [37]. The following features compose the DSS: (1) collaborative work is essential to creating communities that self-manage their development and consider the protection of the environment and biodiversity [38]; (2) problem solving is intended to promote science and technology to find renewable sources of energy and new means of production, construction, and transport [39]; (3) by empowering the ethical project of life, it is intended to empower, in turn, the citizens to implement urgent actions for an improved quality of life and coexistence with each other and the environment [35], and (4) to face the growing complexity of the new tasks that human beings face due to the emergence of artificial intelligence. To accomplish this, the development of certain skills such as creativity, innovation, and critical analysis needs to be promoted in people and communities [40].

Socioformation is a new educative, curricular, pedagogic, and assessable approach [41] to training citizens, organizations, and communities in sustainable social development. It entails implementing transversal and inclusive projects to overcome environmental challenges by means of ethical choices, collaborative work, entrepreneurship, knowledge co-creation, digital technology, and complex thinking. It is an alternative to other recent educative approaches such as social constructivism and meaningful learning, or even to connectivism. It is distinctive for its aim at the implementation of sustainability in the society, culture, economy, technology, environmental and educative processes.

In socioformation, complex thinking is an action aimed at solving contextual problems by connecting different kinds of knowledge, with creativity, critical thinking, systemic analysis, and metacognition, and by perceiving reality with flexibility, open-mindedness, and confrontation of uncertainty [42]. Socioformation features in the educative models of several Latin American universities, such as the Indo American University of Ecuador [43], the University Centre CIFE in Mexico [44], the National University Hermilio Valdizan in Peru [45], and the National University Federico Villar [46].

According to the socioformative approach, connecting critical analysis [47] with creativity is not enough to develop complex thinking. Three additional skills are necessary to direct the evaluation and development of complex thinking in the strict sense; namely, analysis and problem solving [48]; metacognition; and systemic thinking [49], this last focused on the comprehension of the contextual challenges and processes as dynamic systems demanding a multi-, inter-, and transdisciplinary approach.

2.5. Scales to Assess Complex Thinking

Several scales have been designed to assess some skills or dimensions of complex thinking, as shown in

Table 1. Recent scales to assess components of complex thinking.

Name of the Scale or Questionnaire	Type	Dimensions or Aspects That it Evaluates	Number of Items	Reference
Study Engagement Questionnaire (SEQ)	General	Critical thinking; creative thinking; self-managed learning; adaptability; problem solving; communication skills; interpersonal skills; active learning; teaching for understanding; assessment; coherence of curriculum; teacher–students relationship; feedback to assist learning; relationship with other students; cooperative learning	35	Kember and Leung [50] Gargallo et al. [51]
Watson–Glaser Critical Thinking Appraisal	General	Inference; Recognition of assumptions; Deduction; Interpretation; Evaluation of arguments	80	Watson-Glaser [52] y D’Alessio et al. [53]
California Critical Thinking Disposition Inventory	General	pen-mindedness; Self-confidence; Maturity; Analyticity; Systematicity; Inquiry; Truth seeking	75	Facione and Facione [54] y Bayram et al. [55]
California Measure of Mental Motivation (CM3)	General	learning orientation; creative problem solving; cognitive integrity; scholarly rigor; and technological orientation	74	Insight Assessment [56] y Heilat and Seifert [57]
A-E scale	General	Motivation; understanding; problem solving; cognitive process	30	Kaufmann and Martinsen [58], y Martinsen and Furnham [11]
Knowledge Integrator Project (PIS-1)	General	Critical analysis; problem solving; knowledge integration; promotion of different learning activities; generation of learning; ICT promotion; academic performance.	28	Toapanta-Pinta et al. [59]
Cornell Critical Thinking Tests	Specific	Critical thinking	71	Leach et al. [60]
R & D research	Specific	Critical thinking	9	Amrina et al. [61]
Critical Thinking Disposition Scale	Specific	Critical thinking	20	Poondej and Lerdpornkulrat [62]
Torrance Tests of Creative Thinking (TTCT)	Specific	Critical thinking	18	Said-Metwaly et al. [63]
Metacognitive inventory (MCI)	Specific	Metacognition	18	Burin et al. [64]
Problem Solving Strategy Steps scale (PSSS)	Specific	Problem Solving	25	Gok [65]

, which shows some of the most recent scales. Some of the scales shown in Table 1 are general, meant to assess a wide range of skills, whereas others are more specific and only focused on one or two abilities. By assessing the following examples, the following conclusions are expected: (1) there are no scales aimed at assessing exclusively the complex thinking construct in itself; (2) there are no scales allowing the assessment of complex thinking together with the contributions of the socioformative approach; (3) most of the scales have more than 30 items, making them of limited practicality due to the excessive number of questions; and (4) systemic thinking is barely considered in the instruments designed up to now. Regarding this last conclusion, we focus now on the case of the Study Engagement Questionnaire (SEQ), designed by Kember and Leung [50] and adapted to the Spanish population by Gargalloet al. [51] (2018). This scale assesses some of the dimensions of complex thinking such as critical thinking, creative thinking, self-managed learning, adaptability, and problem solving. Its purpose is to establish the elements involved in teaching and learning, but it fails in not accounting for systemic thinking or metacognition.

3. Materials and Methods

3.1. Participants

In total, 626 university degree or pre-degree students from four public universities in Peru participated in the study. Of the participants, 64.1% and 35.9% were women and men, respectively. Their ages ranged between 16 and 33 years old, with an average age of 20.78 DS + 3.3). Additionally, 32.2% of the participants claimed to work complementarily to their studies at the university. The selection of the sample was non-probabilistic, by open call and email.

3.2. Procedure

An instrumental study was carried out around the design of a new instrument to evaluate complex thinking, based on the Likert scale. For this, four stages were carried out: (1) design and peer review of the instrument; (2) content validity analysis by expert judges; (3) study of construct validity by confirmatory factor analysis; and (4) analysis of the evidence of convergent and discriminant validity. The stages and participants are described below.

Stage 1. Design and peer review process. Based on the theoretical references described in the literature review (Morin’s epistemological approach, Lipman’s higher-order thinking, complex thinking as macrocompetence, and the contributions of socioformation), the essential complex thinking skills that he had to tackle the new instrument. Some recent instruments on the subject were also analyzed in Table 1. Based on this, a draft of the instrument was made by the authors, which was later improved with the support of three experts in the area. The three experts presented the following characteristics: (1) doctorate in psychology, education, or social sciences; (2) experience of more than 15 years in the design and validation of instruments; and (3) have at least five publications on cognitive processes.

Stage 2. Assessment of the content validity. After the scale was designed with the support of the experts, this was assessed by 16 judges with experience in the field of complex thinking. At this stage, every subscale was assessed with two indicators; appropriateness of the questions and clarity in the writing. A Likert-kind scale with four levels from 1 to 4 (in which 1 is the lowest level and 4 the highest one) was used. The judges were also asked to make suggestions to improve the quality of each scale, such as adding or removing questions, improving the readability, and adding new scales.

Afterward, the 16 judges were asked to evaluate the instrument as a whole, with the same indicators of appropriateness and writing. At this stage, a third indicator was added, satisfaction, which was measured on a scale of 1 to 5 (from a very low satisfaction (1) to a high one (5)). To assess the judges’ level of agreement, Aiken’s V was used and the values accepted were the ones higher than 0.8 [66]. All of the judges were previously contacted via email, in which they received all the necessary information regarding the purpose of the instrument, and were asked to make suggestions to remove, add or improve the questions. In selecting the judges, all were required to have provable experience in the field of complex thinking skills, to be investigators and university professors, to have at least one Master’s degree, to have published regarding the topic, and to have experience in the review or design of instruments of this nature. The features of the judges are described in

Table 2. Data of the judges who participated in the content validity.

N	16 Judges
Gender (%)	Women: 50% Men: 50%
Age (mean ± standard deviation)	50.6 (±16.88)
Last degree (%)	Master’s degree: 50% PhD: 50%
Years of experience as a university professor	17.8 (±9.08)
Years of research experience (media ± standard deviation)	11.8 (±6.93)
Average of articles published in the area (media ± standard deviation)	16.7 (±18.25)
Average of books published in the area (media ± standard deviation)	3.8 (±7.21)
Average of book chapters published in the area (media ± standard deviation)	7.8 (±11.28)
Average of presentations (media ± standard deviation)	25.5 (±37.79)
Average of conferences in the area (media ± standard deviation)	46.4 (±36.07)
Reviewer experience, measured in a range of 1 to 4 (media ± standard deviation)	3.8 (±0.58)
Continuous training courses that have been taken in the area (media ± standard deviation)	7.9 (±7.12)

.

Stage 3. Construct validity. The whole sample of university students (n = 626) was used to carry out a confirmatory factor analysis with the approach of the maximum likelihood estimation. This technique was used since there are theoretically five factors and it was intended to establish if these factors were verified in the study. For this reason, two models have been tested: the first one with four factors and the second one with five. Finally, Cronbach´s alpha [67] was used to determine the reliability and the composite reliability coefficient [68]. In the same way, and by using a multigroup analysis, a factor invariance analysis by gender was carried out. The confirmatory factor analysis was carried out in the AMOS 27 program.

Stage 4. Convergent and discriminant validity. To assess the convergent validity of the model, that is to say, if the constructs evaluated were effectively assessed by the instrument, three criteria were used. Firstly, the factor weight of the questions in their respective factors had to be higher than 0.50 [69]; secondly, the composite reliability index to be higher than 0.70 [69], and thirdly, the average variance extracted (AVE) to be higher than 0.50 [70]. A sample of 626 students was used to determine the convergent validity. Additionally, the assessment of the discriminant validity was carried out with the same program AMOS 27.

3.3. Ethical Aspects

The Mexican Law of Personal Data Protection was followed over the course of the research since the participants´ emails were requested in order to send them the instrument link. Additionally, all participants were informed of the aim of the study and each one signed an online consent letter. All participants were free to leave the process at any time and without consequences. After answering the questions, they were allowed to know the result of the research to benefit from the process and to implement improvements if needed. The study was approved by the Institutional Ethics Committee.

4. Results

4.1. Stage 1. Design and Peer Review Process

Table 3. Skills assessed by the Complex Thinking Essential Skills Scale.

Complex Thinking Ability	Definition	References	Number of Items
Problem solving	“Capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one’s potential”	Borgonovi and Greif [71] (p. 3)	5
Critical analysis	“Is an intellectual process that is actively and skillfully conceptualizing, applying, analyzing, synthesizing, and or evaluating information collected, or produced by observing, reflecting, considering, or communicating, as a guide to trust and do”	Fitriani et al. [72] (p. 380)	6
Metacognition	“One’s knowledge concerning one’s own cognitive processes and products or anything related to them […] refers, among other things, to the active monitoring and consequent regulation and orchestration of these processes […] usually in the service of some concrete goal or objective”	Flavell [73] (p. 232)	4
Systemic analysis	“Is a methodology for analyzing and solving problems using systemic research and comparison of alternatives that are performed on the basis of the cost-to-cost ratio for their implementation and the expected results.”	Velkovski [74] (p. 322)	3
Creativity	“Creatively is a divergent thinking process, which is the competences to provide alternative answers based on the information provided”	Amrina et al. [75] (p. 129)	4

describes the complex thinking skills chosen for the current study based on the review of scientific literature. A total of five complex thinking skills were chosen: problem solving, systemic thinking, creativity, metacognition, and flexibility. Each of these skills was evaluated on a frequency scale from 1 to 5, in which 1 is “Never” and 5 “Always”. The Likert-like scale of 22 items is shown in Appendix A.

4.2. Stage 2. Content Validity

All 16 judges agreed with the suitable level of appropriateness and comprehension of the Complex Thinking Skills Scale regarding its four subscales and the instrument as a whole, as shown in

Table 4. Expert judgment results.

	Variable	Mean	SD	Aiken’s V	95% Confidence Interval
					Lower Limit	Upper Limit
Problem solving	Suitability	3.81	0.390	0.938	0.832	0.979
	Understandability	3.63	0.484	0.875	0.753	0.941
Critical analysis	Suitability	3.75	0.559	0.917	0.804	0.967
	Understandability	3.88	0.331	0.958	0.860	0.988
Metacognition	Suitability	3.65	0.310	0.85	0.76	0.91
	Understandability	3.54	0.40	0.85	0.72	0.92
Systemic analysis	Suitability	3.81	0.390	0.938	0.832	0.979
	Understandability	3.75	0.433	0.917	0.804	0.967
Creativity	Suitability	3.81	0.390	0.938	0.832	0.979
	Understandability	3.69	0.583	0.896	0.778	0.955
General scale Total	Suitability	3.69	0.583	0.896	0.778	0.955
	Understandability	3.75	0.559	0.917	0.804	0.967
	Satisfaction	4.50	0.791	0.875	0.772	0.935

Note. SD = Standard deviation. The lower and upper limits correspond to the confidence interval (95%) calculated with the formula proposed by Penfield, y Giacobbi [66].

. The Aiken´s v values were higher than 0.8 in the two aspects assessed on the five scales. There was also agreement regarding the level of satisfaction of the global scale (V ≥ 0.8), thus, showing the validity of the instrument [66] (Table 4). Some judges made suggestions to improve several writing aspects, and these were implemented prior to the application to the general sample.

4.3. Stage 3. Construct Validity

Confirmatory factor analysis was used to validate the scale. The distribution of all items is similar to the normal one, which was assessed by using the kurtosis approach and asymmetry (

Table 5. Descriptive data of the sample.

	Mean	Standard Deviation	Skewness	Kurtosis
Question 01	2.83	0.815	−0.039	−0.122
Question 02	3.04	0.786	−0.195	0.294
Question 03	3.04	0.872	−0.028	−0.157
Question 04	3.12	0.817	−0.081	−0.139
Question 05	3.12	0.892	0.069	−0.191
Question 06	3.34	0.819	−0.602	0.654
Question 07	2.70	0.937	0.135	0.129
Question 08	2.94	0.902	−0.085	0.136
Question 09	2.84	0.940	−0.014	0.472
Question 10	2.81	0.992	−0.082	−0.231
Question 11	2.68	0.944	0.030	−0.403
Question 12	3.03	0.864	−0.232	−0.189
Question 13	2.80	0.900	−0.153	−0.127
Question 14	2.98	0.880	0.063	0.197
Question 15	2.75	1.013	0.003	−0.083
Question 16	2.78	0.923	0.036	−0.190
Question 17	2.95	0.899	−0.009	−0.092
Question 18	2.82	0.945	0.235	−0.084
Question 19	2.97	0.890	0.039	−0.114
Question 20	2.82	0.953	0.200	−0.095
Question 21	2.81	0.896	0.226	−0.072
Question 22	2.93	0.945	0.179	−0.330

), and by using the values proposed by Curran et al. [76]: The asymmetry and kurtosis coefficients were within the range of +1 and −1, and the sample had a normal distribution, according to the Shapiro–Wilk test. With these data, it is possible to proceed with the factor analysis itself. The convenience of using the maximum likelihood estimation as an extraction method is confirmed [77].

The first step in the confirmatory factor analysis was to determine the factor weights, which are described in

Table 6. Factor loadings.

							95% Confidence Interval
Factor	Indicator	Symbol	Estimate	Std. Error	z-Value	p	Lower	Upper
Problem solving	ITEM 1	λ11	0.596	0.029	20.297	<0.001	0.538	0.653
	ITEM 2	λ12	0.560	0.029	19.581	<0.001	0.504	0.616
	ITEM 3	λ13	0.657	0.031	21.277	<0.001	0.596	0.717
	ITEM 4	λ14	0.605	0.029	20.798	<0.001	0.548	0.662
	ITEM 5	λ15	0.669	0.032	21.134	<0.001	0.607	0.731
Critical analysis	ITEM 6	λ21	0.425	0.032	13.410	<0.001	0.363	0.487
	ITEM 7	λ22	0.724	0.032	22.277	<0.001	0.660	0.787
	ITEM 8	λ23	0.646	0.032	20.029	<0.001	0.583	0.709
	ITEM 9	λ24	0.679	0.034	20.282	<0.001	0.614	0.745
	ITEM 10	λ25	0.774	0.034	22.567	<0.001	0.707	0.841
	ITEM 11	λ26	0.732	0.033	22.375	<0.001	0.668	0.796
Metacognition	ITEM 12	λ31	0.618	0.031	19.873	<0.001	0.557	0.679
	ITEM 13	λ32	0.676	0.032	21.209	<0.001	0.613	0.738
	ITEM 14	λ33	0.663	0.031	21.352	<0.001	0.602	0.723
	ITEM 15	λ34	0.798	0.035	22.731	<0.001	0.729	0.867
Systemic analysis	ITEM 16	λ41	0.740	0.032	23.173	<0.001	0.677	0.802
	ITEM 17	λ42	0.678	0.032	21.310	<0.001	0.616	0.740
	ITEM 18	λ43	0.778	0.032	24.139	<0.001	0.715	0.842
Creativity	ITEM 19	λ51	0.650	0.032	20.400	<0.001	0.587	0.712
	ITEM 20	λ52	0.758	0.033	22.979	<0.001	0.694	0.823
	ITEM 21	λ53	0.709	0.031	22.850	<0.001	0.648	0.770
	ITEM 22	λ54	0.762	0.032	23.527	<0.001	0.698	0.825

. All of the factor weights of the items were higher than 0.5, and hence are considered significant [69], except for item 6 (“Do you question the facts to find opportunity areas and to implement improvements?”) with a factor weight of only 0.425. For this reason, this item was eliminated and the confirmatory factor analysis was carried out again.

Next, the 5-factor model postulated in the instrument was tested.

Table 7. Fit measures.

Index	Expected Value (Hair et al., 2014)	Value Obtained
Chi-square (χ2)	Non-significative	409.513, p < 0.001
Degrees of freedom (df)	-	179
χ2/df ratio	<3.0	2.28
Tucker–Lewis index (TLI)	>0.90	0.963
Comparative fit index (CFI)	>0.90	0.971
Root-mean-square error of approximation (RMSEA)	<0.08	0.045

shows the goodness of fit indexes; they were positive and confirmed the theoretical model proposed. Since there is a wide range of such indexes, techniques suggested by Hair et al. [69] were used: a combination of chi-square reduced (χ²/gl), Tucker-Lewis index (TLI), comparative fit index (CFI), and the root mean square error of approximation (RMSEA). Even using the strictest criteria [78], requiring the TLI and CFI to be higher than 0.95, and the RMSEA lower than 0.6, the model still fits the data. Itis important to point out that, even though the chi-square test was significant, this index is especially sensitive to sample size, unlike the TLI for example, which turns it into a less-reliable index in this case [79] (Figure 1).

Finally, a multigroup analysis assessed the factor invariance of the instrument by gender. By following the most usual recommendations, configure, metrical, and scalar invariance was estimated, while the residual variance was discarded due to its low practical value [80]. Using the method of Cheung y Rensvold [81], differences lower than 0.01 in the CFI index not considered to be enough to discard the invariance hypothesis, whereas the differences between the RMSEA of the configurational model and the metrical and scalar models were not higher than 0.15 [82]. The results show no major differences between men and women (

Table 8. Factor invariance test.

Model	χ2	df	χ2/df	RMSEA	TLI	CFI	ΔRMSEA	ΔTLI	ΔCFI
Configural	635.341	358	1.775	0.036	0.954	0.964	-	-	-
Metric	663.238	374	1.773	0.035	0.954	0.963	0.001	0.000	0.001
Scalar	688.940	395	1.744	0.035	0.956	0.962	0.001	−0.002	0.002

).

4.4. Stage 4. Convergent Validity

Table 9. Convergent validity of the model.

Factor	# of Items	Composite Reliability	AVE	Standardized Factor Loads
Problem solving	5	0.862	0.56	Item01 (0.72); Item02 (0.71); Item03 (0.75); Item04 (0.77); Item05 (0.77)
Critical analysis	5	0.794	0.57	Item07 (0.78); Item08 (0.72); Item09 (0.75); Item10 (0.78); Item11 (0.76)
Metacognition	4	0.838	0.57	Item12 (0.71); Item13 (0.74); Item14 (0.75); Item15 (0.80)
Systemic analysis	3	0.833	0.63	Item16 (0.80); Item17 (0.75); Item18 (0.82)
Creativity	4	0.867	0.62	Item19 (0.74); Item20 (0.80); Item21 (0.80); Item22 (0.81)

Note. AVE = average variance extracted.

shows the results of the convergent validity. The average extracted variance was higher than 0.5 [83]. Additionally, the composite reliability was higher than 0.7, which is a positive indicator [84]. Thus, we conclude that the questions assess effectively the constructs established in each factor and demonstrate convergent validity.

Additionally, the results of the discriminant validity, which assesses whether the factors are clearly different from each other, were more complex. Discriminant validity was assessed using the method proposed by Fornell and Larcker [70], in which the square root of the average extracted variance must be higher than the correlations matrix of that factor with the other factors. In this study, the correlation of the factors was high (see Figure 1) and we expect the correlations matrix to have higher scores than the square of the average extracted variance (

Table 10. Discriminant validity of the model.

		1	2	3	4	5
Factor 1	1	0.746
Factor 2	2	0.818	0.757
Factor 3	3	0.844	0.858	0.752
Factor 4	4	0.800	0.839	0.870	0.791
Factor 5	5	0.780	0.940	0.865	0.821	0.787

Note. Items in bold are the square root of the average extracted variance (AVE). These values must exceed the correlations between factors (off-diagonal Items) for adequate discriminant validity.

). These results are varied. However, this is not necessarily problematic, since the scale assesses related factors. Thus, its purpose is not to reach a specific conclusion about the differences, but a conclusion of complex thinking practice itself.

5. Discussion

Complex thinking, in the socioformative approach, is an or comprehensive performance that citizens and communities must carry out to achieve sustainable social development. This is the main challenge that humankind faces nowadays, and encompasses other related challenges such as disease prevention and health promotion; quality of life and socioeconomic development; inclusive economy; housing; transport and education; and pollution prevention, mitigation of global warming, and biodiversity preservation. Therefore, developing this comprehensive performance would allow facing the context and his problems with a systemic vision. For example, the case of the current COVID-19 pandemic [85]. This context has highlighted the lack of training in complex thinking in many leaders and citizens. This issue needs to be understood as a consequence, prevention efforts in many countries have not had the expected impact (see, for example, Roozenbeek et al. [86]). In this specific problem, the need to generate transdisciplinary actions has been suggested [87], which also applies to the climate crisis and other problems of humanity.

These are the reasons why the current research offers a new instrument to efficiently assess the complex thinking skills in adult people, named the Complex Thinking Essential Skills Scale (COMPLEX-21), and to direct better the educative actions of human talent. To this end, the analysis of a group of 14 experts in the field proved the content validity of the COMPLEX-21, with Aiken’s V values higher than 0.80 [66]. This means that this scale is a useful, appropriate, and understandable instrument for potential users.

Additionally, the validity of the instrument was demonstrated, since the five factors postulated at a theoretical level for complex thinking (problem solving, critical analysis, metacognition, systemic analysis, and creativity) were assessed, and the results showed that the goodness of fit indexes met the criteria established in this field [88]. In addition, most of the items had a suitable factor weight and higher than 0.5. There was only one question with a non-suitable factor weight and it was eliminated from the instrument for this reason. Besides, the scale showed no difference by gender and. Therefore, it can be recommended for practical use.

The major problem is the one referred to the discriminant validity, in other words, the clear differentiation between the constructs assessed for each factor. We explain this in several ways. A possible explanation might be that the five factors are closely related to each other and thus, cannot be sufficiently separated. This would not be a single case (see Disabato et al. [89]; Gau [90]). Future studies of this scale will address this aspect to come to more decisive conclusions.

There is plenty of information and lots of proposals regarding complex thinking and its skills. Nonetheless, the instruments are quite general and do not address “Complex thinking” itself as a construct; many of these skills are assessed separately and systemic analysis does not seem to be that important. Nowadays, it is essential to assess these skills as a whole since complex thinking is increasingly a skill that every citizen must develop, including students, teachers, and university principals [43]. For this reason, the current study offers an appropriate scale with good initial psychometric properties to help meet this need in universities and encourage the practice of complex thinking in graduates. The end goal is the creation of a better and more environmentally friendly society.

In addition, the present study helps to understand the structure of complex thinking as a practice, just as socioformation proposes [41]: a new pedagogic approach, created with the support of leaders in education, community, and organizations to face the challenges of sustainable social development. Thus, the socioformative approach comprehends the complex thinking construct as an or comprehensive performance to solve the problems in the way of sustainable social development. This different from the assumption of the epistemological approach [12], which tends not to go beyond the philosophical plane; or from high order thinking [15] which does not consider the improvement of life conditions and biodiversity; or from macro-competence [25], which does consider the skills, attitudes, and knowledge but, does not go beyond a fragmented view of the concept, spends excessive time in the planning of processes and has no connection to sustainable development. Nonetheless, the contributions of these three perspectives have been considered in the socioformative approach but in the framework of a new proposal, more focused on the social aspect and nature.

Increasingly, Ibero-American Universities and educative institutions are implementing the socioformation in their systems [43,44,45,46]. Thus, the instrument validated in the present study will contribute to its consolidation by allowing university students, teachers, and principals a better and more efficient validation of complex thinking as a practice (or comprehensive performance). The current study provides information about the practical structure of complex thinking, which is composed of five essential skills interconnected to each other. Within the socioformation approach, and despite the changes taking place in higher education, it is essential to implement actions that promote complex thinking for all students. This is because many organizations keep insisting on teaching strategies that do not enhance complex thinking but the skills and abilities of simplistic thinking [48].

The COMPLEX-21 scale differs from other instruments assessing specific skills of complex thinking, such as the Study Engagement Questionnaire (SEQ) [50,51], the Watson–Glaser, the Critical Thinking Appraisal [52,53] and the California Critical Thinking Disposition Inventory [54,55] because these assess several dimensions of complex thinking but not the construct as a whole; these scales have a large number of items (around 70 items) and systemic analysis is relegated to a second plane.

The current scale has similar features to other instruments such as the study carried out by Gargallo et al. [51], which assesses, as a whole, the learning–teaching process in the university to provide feedback to the teachers and institutions so they can improve these processes. In that study, 805 people from three Valencian universities were evaluated with a survey assessing the abilities of several students and the ability of the teacher to create a suitable learning environment. Another related instrument is the one designed by Amrina et al. [75] which assesses the competence of the students in achieving the development of logic, critical and creative thinking by employing a Like scale (1–5 points). One difference between the study carried out by Amrina et al. [75] and ours is that our COMPLEX-21 is designed for several fields, not only for math. Additionally, their research did not consider systematic thinking. Amrina et al. [75] are similar to the COMPLEX-21 since both assume that both critical and creative thinking are necessary for the problem-solving process, just like the scales used by Heilat and Seifert [91], and Martinsen and Furnha [11]. Nonetheless, the COMPLEX-21 has its features, such as its focus on the complex thinking itself and not on the learning–teaching process.

The present study was exploratory; thus, it has limitations. Firstly, we considered only a subset of the possible skills of complex thinking due to the limited information available about this topic and the non-agreement among experts, who considered the five skills described above as essential and who decided not to add new ones to the list. The same happened with the judges during the content validity stage, since three of these judges proposed more skills, did not come into agreement. the two of the proposed skills were conceptual analysis and knowledge management, which have been associated with complex thinking [92]. For future studies, we recommend including an analysis of these two skills and integrating them with specific items. Secondly, the validity process would have been improved by joining it with analysis of other processes, such as concurrent validity with other similar instruments [93], predictive validity, and test-rest stability [94]. These studies were not carried out since this research was part of another research in university students. As a result, it was not possible to add further tests or tests on a larger subsample of students.

Although this is an exploratory study and new research is needed around the characteristics of the COMPLEX-21, some practical implications are described below: (1) the new scale will help to assess complex thinking as an integral performance from the framework of socioformation, because although complex thinking is considered a relevant axis for education [95]; There are no instruments to evaluate this process from the references discussed throughout this study; (2) COMPLEX-21 may contribute to reducing the high number of competencies studied in the training of citizens, since many skills integrated in complex thinking are addressed as separate competencies [30,31]; (3) the new instrument could help to determine more precisely the factors associated with better management to achieve sustainable social development, considering complex thinking as an integrative performance, as proposed from the socioformative approach; (4) based on COMPLEX-21, the educational models of educational institutions and universities could have more clarity regarding this process and its associated factors and (5) educational institutions could assess the level of development of complex thinking in students with the new scale and thus implement actions that help with that, which is a purpose in various educational centers [43,44,45].