Systematic Review of Instruments to Assess Computational Thinking in Early Years of Schooling

Abstract

Computational thinking (CT) is considered a key competence in today’s digital era. It is an emerging construct that relates to critical thinking and creativity. Research on its assessment is in the process of consolidation. This systematic review aims to analyze studies that have used CT assessment instruments for children and adolescents aged 4 to 16 years in order to identify which variables, they assess and their psychometric properties. The search and analysis were carried out following the PRISMA statement protocol, analyzing 50 articles published between 2006 and March 2023. An increase in the publication of CT measurement instruments is observed, with 54% of them supported by evidence of validity and 88% by reliability, highlighting construct validity, followed by content and criteria validity. China leads in the number of publications, while Asia and Europe concentrate most of the research. There is a noticeable contribution from South America, evidencing the lack of participation from Central and South American countries in this field of study.

1. Introduction

Computational thinking (CT) has been established as a key skill in today’s digital era [1,2], relating to critical thinking and creativity [3]. It is considered an emerging construct, consolidating from theoretical, operational, and measurement perspectives. The lack of consensus on its operationalization implies its implementation in various ways in the student learning process and hinders the development of assessment methodologies and tools [4].

The literature shows a diversity of approaches aimed at fostering the development of CT through integration strategies in the educational environment. At the same time, it highlights the importance of deepening the understanding of teaching and assessment processes from the early years of schooling [5].

In the article “Computational Thinking” [6], the conceptual foundations and fundamental characteristics of CT are established; it is conceived as a process that involves problem-solving, system design, and understanding human behavior through the basic principles of computer science. Román’s work [7] outlines CT definitions from different perspectives: generic, operational, psychological-cognitive, and educational-curricular. It also presents an operational definition, expressing that it involves the ability to approach problems using the fundamental principles of computing and the logic of programming languages. Likewise, Tang and collaborators [8] propose a classification of definitions based on two aspects: the first focuses on characteristics linked to programming and computational concepts, while the second refers to the skills and competencies students must acquire in specific areas of knowledge and the ability to effectively tackle problems.

In the educational field, there are various frameworks for introducing CT into school curricula, including the three-dimensional framework by Karen Brennan and Mitchel Resnick [9] and the CSTA Standards Task Force [10]. These provide a structure that guides the teaching of this construct. This integration can be transversal, incorporating it into various curricular areas, or independent, assigning it as an autonomous subject or study area. This has shown the need to implement assessment methodologies and tools with validity and reliability [4] that can adapt to various purposes and uses [11]. Additionally, some research concludes that there are few agreements on the strategies to evaluate it, and few instruments have psychometric properties for use with young children [8,9,12,13,14,15].

As CT is a recent field of study, there are various assessment proposals in different formats, involving students in different types of activities and educational contexts [14]. These include questionnaires [13,16,17,18], task-based tests [19], coding tests [20], observation [21], and one-on-one interviews [22]. On the other hand, Poulakis and Politis [23], based on their systematic review findings, classify CT assessment approaches into three categories: (1) assessment through specific programming environments, (2) evaluation criteria and psychometric instruments, and (3) multiple assessment methods, consisting of project portfolios, participant observation, and artifact-based student interviews.

Various studies express the need for multiple or complementary assessment systems [24,25] that address both cognitive and non-cognitive aspects of CT to obtain a comprehensive view of student learning. Grover and Pea [24] analyzed the use of an assessment system encompassing multiple strategies to measure the growth of algorithmic thinking skills and their transfer, as well as non-cognitive aspects, such as perceptions of computer science and communication of CT ideas.

Román and collaborators [25] also emphasize the importance of employing more than one tool to obtain a comprehensive assessment of CT. They classify assessments into several types: diagnostic assessments to identify areas for improvement, summative assessments to measure final performance, formative-iterative assessments to adjust teaching during the process, data mining assessments to analyze learning patterns, skill transfer assessments to evaluate practical application, perception assessments to understand attitudes toward CT, and vocabulary assessments to evaluate the mastery of technical terms related to the construct.

This systematic review aims to identify research that has used CT assessment instruments for children and adolescents aged 4 to 16 years in order to identify which variables they assess and their psychometric properties.

2. Materials and Methods

The search and analysis of the literature were carried out considering the protocol of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA in Supplementary Materials) statement [26,27]. The review protocol includes eligibility criteria, sources of information and search strategy, study selection, data extraction, and information elements.

The following research questions were posed, associated with both bibliometric and interest variables:

Q1. What bibliometric characteristics are identified in the analyzed articles?

Q2. Which instruments to measure CT have been used in their original version or have adjustments?

Q3. What are the ages targeted by the CT assessment instruments?

Q4. What is the number of items and most commonly used response options?

Q5. What are the elements related to CT evaluated in the analyzed instruments?

Q6. What psychometric properties are evidenced in the instruments, and what methods were used for the analysis of psychometric properties?

Through the analysis of the articles, these questions are answered by identifying for the bibliometric variables: the source of information, the year of publication, common words in the article title and keywords, the co-citation network of authors, and the country where the study was conducted; and regarding the interest variables: the instruments used, the population under study, the number of items and response options, the theoretical basis used, the elements related to CT evaluated, the evidence of reliability and validity, the methods used to determine validity, and the reliability coefficients implemented. Finally, techniques for data analysis and visualization were applied.

a.

Eligibility Criteria

Research articles published in scientific journals from January 2006 to March 2024 were included. Articles that assess the construct using instruments in children and adolescents aged 4 to 16 years were considered. Articles addressing other topics or evaluating CT using a tool other than an instrument (surveys, observation rubrics, interviews) were excluded. Duplicate articles, reviews, editorials, review articles, books, summaries, or conference proceedings were also excluded.

b.

Databases, Information Sources, and Search Strategy

Research articles were searched in Scopus and Web of Science (WoS). Selection and extraction were carried out in parallel by two researchers during March 2024. Based on the review objective and research questions, CT was established as the axial category, and evaluation and the population under study as complementary categories. Synonyms for the terms were searched, and the following search string was obtained: “Computational Thinking” (Topic) AND assessment OR evaluat* OR instrument OR measur* OR apprais* OR scale OR test (Topic) AND “elementary school” OR “primary school” OR “primary education” OR “elementary education” OR kindergarten OR “early childhood education” (Topic) AND Article (Document Types). The search strategy was documented transparently and reproducibly to allow others to replicate the process.

c.

Study Selection

When searching WoS and Scopus, 398 articles were found for the systematic literature review. After removing duplicates (148 articles), 250 articles were obtained for analysis. By applying the exclusion and inclusion criteria, 171 articles were discarded. This led to a sample consisting of 42 studies. Additionally, 8 articles were directly included, considered relevant for meeting the inclusion criteria of this review, and mostly referenced in the 42 articles obtained in the search, leading to a total of 50 articles comprising this review. The flow diagram for literature selection is presented in Figure 1.

d.

Data Extraction and Information Elements.

Once the selection process was completed, all 50 articles were thoroughly analyzed, extracting data of interest for the present study [28], including the following elements: title and research objective, year of publication, abstract, keywords, country where the research was conducted, publishing journal, name of the instrument for assessing CT, elements related to CT evaluated, theoretical basis for instrument development, ages or school level of evaluated students, number of items, type of item responses, and evidence of reliability and validity.

e.

Bibliometric Analysis and Data Visualization

Based on the 50 articles comprising the sample, an analysis was conducted for each study variable. The analysis approached the information based on: (1) Bibliometric variables: information source, year of publication, article title, keywords, authors, and country where the research was conducted; and (2) Variables of interest: population under study, number of items, response options, authors providing theoretical basis for the instrument, elements related to CT evaluated, evidence of reliability and validity, and methods used to determine them. VOSviewer [29] was used for bibliometric analysis.

3. Results

3.1. Analysis of Bibliometric Variables

Information Source. The search was conducted in two databases, WoS and Scopus. Fifty articles were identified for this systematic literature review, where it was found that both sources contributed approximately the same number of articles.

Year of publication. Article selection was conducted from 2006 to March 2024 (18 years). Although 2006 was the starting year, the search revealed that the first publication related to assessment instruments appeared in 2017 with two articles [7,30], and in 2023, the highest number of publications was reached, with fourteen articles [18,31,32,33,34,35,36,37,38,39,40,41,42,43], as evidenced in Figure 2.

Title of the article and keywords. The analysis of article titles and keywords revealed that the words “computational”, “thinking”, “test”, “education”, “primary”, and “school” are most frequently repeated, as shown in Figure 3. These results confirm that the articles are related to the objective set for this review.

In Figure 4, a map based on the words from the titles and keywords of the articles is shown. For this, terms that appeared at least 5 times out of a total of 70 words were considered. The graph reveals 5 groups, composed of 32, 15, 11, 9, and 3 elements, respectively. Group 1 includes words such as “assessment”, “validity”, “tool”, and “effect”, among others (with link 62 and a total link strength of 655).

Co-citation of authors. Figure 5 refers to the co-citation network of authors. Authors who appeared most frequently were Román-González, M. [7,17,44,45,46], Zapata-Cáceres, M. [17,46,47,48,49], and Martín-Barroso, E. [17,46,47,48,49], each in 5 articles. El-Hamamsy, L. [46,47,49] appeared in 3 articles, and Butz, M. [44] appeared in one article. Four groups were identified: the first group with 6 elements is led by El-Hamamsy, L., and Zapata-Cáceres, M. (links 9, total link strength 14). The second group with 6 elements is led by Butz, M. (links 6, total link strength 6). The third group with 5 elements is led by Román-González, M. (links 16, total link strength 17). The fourth group with 3 elements includes Martín-Barroso, E. (links 62, total link strength 72). Additionally, a publication authored by Román-González, M., Zapata-Cáceres, M., El-Hamamsy, L., and Martín-Barroso, E. was identified [46].

Country where the study was conducted. The country with the most publications is China with nineteen articles, followed by Spain and the United States with eight each, Turkey with four, and Switzerland with two publications. Finally, Germany, Austria, France, Indonesia, Italy, the Netherlands, Portugal, the Czech Republic, and Uruguay each have one publication. Regarding continents, Asia and Europe have the highest representation, as shown in Figure 6.

3.2. Analysis of Variables of Interest

In the review of the 50 articles, a total of 22 different instruments for assessing CT were identified, which are presented in

Table 1. Identified instruments.

#	Article	Instrument Used	Citation
1	Computational thinking test for beginners: Design and content validation	BCTt	[17]
2	The competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary School	cCTt	[49]
3	Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students	CTA-CES	[50]
4	The effect of an unplugged coding course on primary school students’ improvement in their computational thinking skills	CTST	[34]
5	Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test	CTt	[7]
6	Development and Validation of the Computational Thinking Test for Elementary School Students (CTT-ES): Correlate CT Competency With CT Disposition	CTT-ES	[51]
7	Computational Thinking Test for Lower Primary students: Design principles, content validation, and pilot testing	CTtLP	[52]
8	Development and Predictive Validity of the Computational Thinking Disposition Questionnaire	Questionnaire of CT disposition	[53]
9	Formation of computational identity through computational thinking perspectives development in programming learning: A mediation analysis among primary school students	Scale of CT perspectives	[54]
10	Measuring coding ability in young children relations to computational thinking, creative thinking, and working memory	Assessment through a card-based game	[19]
11	A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades	Assessment of computational thinking concepts	[55]
12	Design and validation of learning trajectory-based assessments for computational thinking in upper elementary grades	Early assessment	[56]
13	Assessing elementary students’ computational thinking in everyday reasoning and robotics programming	Instrument with emphasis on robotics programming	[30]
14	Validating a computational thinking concepts test for primary education using item response theory: An analysis of students’ responses	Instrument for testing CT concepts	[57]
15	Can computational thinking be improved by using a methodology based on metaphors and scratch to teach computer programming to children?	PCNT	[58]
16	Computational Thinking Evaluation Tool Development for Early Childhood Software Education	Computational Thinking Test	[59]
17	Developing scientific literacy-based teaching materials to improve students’ computational thinking skills	Integrated CT competence test	[60]
18	Unplugged or plugged-in programming learning: A comparative experimental study	Test of computational concepts mastery	[61]
19	Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts	CT practices test	[62]
20	Possibilities of diagnosing the level of development of students’ computational thinking and the influence of alternative methods of teaching mathematics on their results	Didactic CT test	[33]
21	TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education	TechCheck	[13]
22	TechCheck-K- A Measure of Computational Thinking for Kindergarten Children	TechCheck-K	[63]

.

Furthermore, in 11 articles, adjustments were made to some of these instruments, such as translation, expert judgment, pilot testing, and/or verification of psychometric properties. This results in a total of 33 different instruments for measuring CT skills.

Table 2. Referenced instruments used without adaptation and with adaptation.

Base Instrument	Article Where the Instrument Is Used	Adjustment Evidence				No Adjustment	Comments
		Translate	Expert Judgment	Pilot Testing	Psychometric Properties
BCTt [17]	[42]					X	The instrument is applied in its original version and translated into Portuguese.
	[46]			X	X		The psychometric properties of two instruments are compared: the Beginner Computational Thinking Test (BCTt), developed for grades 1–6, and the Competent Computational Thinking Test (cCTt), validated for grades 3–4.
	[48]					X	The instrument is applied in its original version.
	[64]					X	The instrument is applied in its original version.
CTt [7]	[45]					X	The instrument is applied in its original version.
	[65]	X		X	X		It is translated into Chinese, and the reliability is analyzed.
	[44]					X	It is translated into German, and the number of items in the instrument is reduced.
	[66]				X		The instrument is applied in its original version, and its internal consistency is determined.
cCTt [49]	[46]			X	X		The psychometric properties of two instruments are compared: the Beginner Computational Thinking Test (BCTt), developed for grades 1–6, and the Competent Computational Thinking Test (cCTt), validated for grades 3–4.
	[47]					X	The instrument is applied in its original version.
CTS (The original article where the instrument appears is not part of the 50 articles in the review.) [67]	[38]	X			X		It is translated into Chinese.
	[65]	X		X	X		It is translated into Chinese.
	[40]	X		X	X		It is translated into Chinese, adjusted, and simplified for comprehension and application.
	[41]					X	It is adjusted to the age range of the study.
	[68]	X			X		It is translated into Chinese and expressions are adjusted.
	[69]	X	X		X		It is translated into Chinese and adjusted to reading levels by a research and teaching team.
CTtLP [52]	[18]				X		The instrument is applied in its original version. CFA is conducted.
	[39]		X	X	X		The psychometric properties of the instrument are validated.
	[43]					X	The instrument is applied in its original version.
CT questionnaire (The original article where the instrument appears is not part of the 50 articles in the review.) [70]	[71]					X	The instrument is applied in its original version.
	[72]					X	It is adjusted to the age range of the study.
CT practices test [62]	[32]				X		A partial matrix sampling approach was used to distribute this comprehensive test into multiple shorter forms with fewer items, aiming to reduce the testing burden on students.
TechCheck [13]	[31]					X	The instrument is applied in its original version.
	[35]					X	The instrument is applied in its original version.
	[37]					X	The translation performed by one of the authors is used [73].
	[73]	X	X	X	X		The test items were translated into Turkish.
TechCheck-K [63]	[31]					X	The instrument is applied in its original version.

presents the instruments used as a basis in the research and the adjustments made.

It was found that the instruments BCTt [17], CTt [7], CTS [67], and TechCheck [13] are the most referenced in other research for assessing CT, either directly or with adjustments.

Furthermore, several studies used questions from different years and countries of the Bebras Tasks to construct instruments that allowed them to measure CT skills of interest for the study [19,34,36,59,74,75]. Additionally, one article uses questions from a repository at Peking University [76] to assess this construct.

It should be noted that several studies used or adjusted more than one instrument. Relkin, Johnson, and Bers [31] used five instruments in their research (TechCheck, TechCheck-k, TechCheck-1, TechCheck-2, TechCheck-PreK); El-Hamamsy et al. [46] adjusted two instruments (BCTt and cCTt); Ma et al. [65] adjusted two instruments (CTt and CTS).

Target population. In the various reviewed studies, age ranges varied between 4 and 16 years.

Table 3. Age distribution for each instrument.

Year	Article	Instrument Used and/or Adjusted	Age
			5	6	7	8	9	10	11	12	13	14	15	16
2017	Assessing elementary students’ computational thinking in everyday reasoning and robotics programming [30]	Instrument with emphasis on robotics programming						X	X
2017	Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test [7]	CTt						X	X	X	X	X	X	X
2019	Developing scientific literacy-based teaching materials to improve students’ computational thinking skills [60]	Integrated CT competence test					X	X
2020	Can computational thinking be improved by using a methodology based on metaphors and scratch to teach computer programming to children? [58]	PCNT					X	X	X	X
2020	Formation of computational identity through computational thinking perspectives development in programming learning: A mediation analysis among primary school students [54]	Scale of CT perspectives					X	X	X	X
2020	TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education [13]	TechCheck	X	X	X	X	X
2020	Computational thinking test for beginners: Design and content validation [17]	BCTt	X	X	X	X	X	X	X	X
2020	Development and Predictive Validity of the Computational Thinking Disposition Questionnaire [53]	Questionnaire of CT disposition						X	X
2021	Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students [50]	CTA-CES				X	X	X	X	X
2021	Effect of Scratch on computational thinking skills of Chinese primary school students [68]	CTS						X	X
2021	Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts [62]	CT practice test					X	X	X
2021	Promoting pupils’ computational thinking skills and self-efficacy: a problem-solving instructional approach [65]	CTt						X	X
		Self-Efficacy Scale for CT						X	X
2021	Computational Thinking Test for Lower Primary students: Design principles, content validation, and pilot testing [52]	CTtLP			X	X
2021	Design and validation of learning trajectory-based assessments for computational thinking in upper elementary grades [56]	Early assessment				X	X	X
2021	TechCheck-K- A Measure of Computational Thinking for Kindergarten Children [63]	TechCheck-K	X	X
2021	Measuring coding ability in young children relations to computational thinking, creative thinking, and working memory [19]	Assessment through a card-based game	X	X
2021	A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades [55]	Assessment of computational thinking concepts					X	X	X
2021	Computational Thinking Evaluation Tool Development for Early Childhood Software Education [59]	CT Test Using Bebras Tasks	X
2022	Adaptation into turkish of the computational thinking test for primary school students [73]	TechCheck			X	X	X
2022	Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners’ CT test (BCTt) and the competent CT test (cCTt) [46]	BCTt				X	X	X
		cCTt			X	X	X
2022	Development and Validation of the Computational Thinking Test for Elementary School Students (CTT-ES): Correlate CT Competency With CT Disposition [51]	CTT-ES							X	X
2022	Exploring the characteristics of an optimal design of non-programming plugged learning for developing primary school students’ computational thinking in mathematics [69]	Perception Questionnaire Adapted from CTS					X	X
2022	The competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary School [49]	cCTt			X	X	X
2022	Unplugged or plugged-in programming learning: A comparative experimental study [61]	Test of computational concepts mastery						X	X
2022	Validating a computational thinking concepts test for primary education using item response theory: An analysis of students’ responses [57]	CT Concepts Test Instrument for Primary Education Based on an ECD Approach				X	X	X	X
2023	Developing and Testing a Design-Based Learning Approach to Enhance Elementary Students’ Self-Perceived Computational Thinking [38]	CTS					X	X
2023	Development and validation of a computational thinking test for lower primary school students [39]	CTtLP		X	X	X	X	X
2023	Effect of Reverse Engineering Pedagogy on Primary School Students’ Computational Thinking Skills in STEM Learning Activities [40]	CTS						X	X
2023	Monitoring cognitive development through the assessment of computational thinking practices: A longitudinal intervention on primary school students [32]	CT practices test				X	X	X	X	X
2023	Possibilities of diagnosing the level of development of students’ computational thinking and the influence of alternative methods of teaching mathematics on their results [33]	Didactic CT test					X	X	X
2023	The effect of an unplugged coding course on primary school students’ improvement in their computational thinking skills [34]	CTST				X	X	X	X
2023	Unravelling the underlying mechanism of computational thinking: The mediating role of attitudinal beliefs between personality and learning performance [18]	CTtLP				X	X	X	X
2024	The effect on computational thinking and identified learning aspects: Comparing unplugged smartGames with SRA-Programming with tangible or On-screen output [66]	CTt				X	X	X	X	X	X
Total			5	5	7	15	22	26	21	8	2	1	1	1

shows the ages at which the 33 identified instruments were applied. Appendix A presents the age table for the 50 analyzed articles.

In Figure 7, the age distribution of the 50 studies included in the review and the 33 studies where distinct instruments to assess CT were identified is shown. It can be observed that there is a greater availability of instruments in the age range of 8 to 11 years.

Number of Items. From the reviewed studies, it was identified that the “CT Practices Test”, used in two studies [32,62], has the highest number of items, with a total of forty-seven. On the other hand, both the “Assessment through a card-based game” [19] and the “Early Assessment” [56] have the fewest items, each with nine in total.

Response Options. Regarding the distribution of response formats used in the instruments, multiple-choice is the most implemented format, used in 68% of the reviewed articles, followed by the 5-point Likert scale in 20% of the articles. Open-ended questions combined with multiple-choice and fill-in-the-blank short responses are each used in 3% of the articles. Additionally, the response format was not specified in 6% of the cases. It is important to note that two instruments have two different response formats: the CT questionnaire [72], which includes fill-in-the-blank and short responses, and the Computational Thinking Test using Bebras Problems [36], which includes both open-ended and multiple-choice questions.

Theoretical Foundation. Figure 8 presents the authors and/or associations that have contributed to the construction of the instruments identified in the review, providing theoretical frameworks, definitions, and taxonomies. The most widely used model is the “3D” model proposed by Brennan and Resnick [9].

Skills, attitudes, and/or computational concepts evaluated. The most commonly evaluated skills identified are abstraction, debugging and evaluation, problem-solving, algorithms, and algorithmic thinking; likewise, the most commonly evaluated computational concepts are sequences, loops, and conditionals. Additionally, attitudes such as cooperation and creativity are also considered. The frequency with which these categories are evaluated is shown in Figure 9.

In most of the research studies, skills are evaluated, followed by concepts, attitudes, and perspectives. Some studies also evaluate skills and concepts, or skills and attitudes, while only one study evaluates skills, concepts, and perspectives, as shown in Figure 10.

Table 4. Assessed skills, concepts, perspectives, and attitudes.

Year	Citation	Skills														Concepts									Perspectives			Attitudes
		Abstraction	Algorithmic Thinking	Critical Thinking	Logical Thinking	Algorithms	Coding	Problem- Solving	Decomposition	Modularization	Debugging and Evaluation	Generalization	Remix and Reuse	Representation	Pattern Recognition	Sequences	Directions	Loops	Conditionals	Control Structures	Events	Parallelism	Operators	Data	Expressing	Connecting	Questioning	Cooperativity	Creativity	Sensitivity	Self-efficacy	Inclination
2017	[30]						x	x
2017	[7]						x									x		x	x
2019	[60]					x		x						x	x
2020	[58]										x		x			x		x	x		x	x	x	x	x	x	x
2020	[17]															x		x	x
2020	[53]						x	x																						x	x	x
2020	[54]																								x	x	x
2020	[13]					x				x	x			x						x
2021	[55]						x									x		x	x		x	x
2021	[59]	x				x		x						x
2021	[52]															x	x	x	x
2021	[56]						x		x		x					x		x	x
2021	[50]	x				x		x	x							x		x				x
2021	[68]		x	x				x																				x	x
2021	[62]	x	x							x	x		x
2021	[19]					x	x			x								x	x
2021	[65]		x	x				x								x	x	x	x									x	x
2021	[63]					x				x	x			x						x
2022	[73]					x				x	x			x						x
2022	[46]															x		x	x
2022	[51]	x				x			x		x	x
2022	[69]	x	x	x				x	x																			x
2022	[49]															X		x	x
2022	[61]					x	x											x	x
2022	[57]															x		x	x
2023	[38]		x	x				x																				x	x
2023	[39]															x	x		x
2023	[40]		x	x				x																				x	x
2023	[32]	x	x							x	x		x
2023	[33]	x	x				x		x		x	x
2023	[34]	x				x			x		x	x
2023	[18]						x									x		x	x
2024	[66]																	x	x
	Total	8	8	5	0	10	9	10	6	6	10	3	3	5	1	13	3	15	15	3	2	3	1	1	2	2	2	5	4	1	1	1

presents the skills, concepts, perspectives, and attitudes assessed in the 33 different instruments identified for evaluating the CT.

Evidence of reliability and validity, and methods used to determine them.

Table 5. Articles associated with the instruments and the evidence of validity and reliability.

Year	Article	Instrument Used and/or Adjusted	Validity					Reliability
			Content	Construct	Criterion	Other	No evidence	Coefficient
2017	Assessing elementary students’ computational thinking in everyday reasoning and robotics programming [30]	Instrument with emphasis on robotics programming					X	Cronbach’s alpha
2017	Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test [7]	CTt			X			Cronbach’s alpha
2018	Extending the nomological network of computational thinking with non-cognitive factors [45]	CTt	X		X			Cronbach’s alpha
2019	Developing scientific literacy-based teaching materials to improve students’ computational thinking skills [60]	Integrated CT competence test	X				X	Cronbach’s alpha
2019	Educational Robotics in Primary School: Measuring the Development of Computational Thinking Skills with the Bebras Tasks [74]	Bebras Tasks					X	No evidence
2020	Can computational thinking be improved by using a methodology based on metaphors and scratch to teach computer programming to children? [58]	PCNT					X	No evidence
2020	Computational thinking through unplugged activities in early years of Primary Education [75]	Bebras Category Instrument					X	No evidence
2020	Formation of computational identity through computational thinking perspectives development in programming learning: A mediation analysis among primary school students [54]	Scale of CT perspectives	X	X				Cronbach’s alpha
2020	TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education [13]	TechCheck	X		X			Cronbach’s alpha
2020	Computational thinking test for beginners: Design and content validation [17]	BCTt	X		X			Cronbach’s alpha
2020	Development and Predictive Validity of the Computational Thinking Disposition Questionnaire [53]	Questionnaire of CT disposition		X				Cronbach’s alpha
2021	Cognitive abilities and computational thinking at age 5: Evidence for associations to sequencing and symbolic number comparison [72]	CT questionnaire					X	Cronbach’s alpha
2021	Collaborative Game-Based Environment and Assessment Tool for Learning Computational Thinking in Primary School: A Case Study [48]	BCTt					X	Cronbach’s alpha
2021	Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students [50]	CTA-CES	X	X	X			Cronbach’s alpha
2021	Does learning to code influence cognitive skills of elementary school children? Findings from a randomized experiment [71]	CT questionnaire					X	No evidence
2021	Effect of Scratch on computational thinking skills of Chinese primary school students [68]	CTS		X				Cronbach’s alpha
2021	Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts [62]	CT practices test	X	X		X		Empirical
2021	Promoting pupils’ computational thinking skills and self-efficacy: a problem-solving instructional approach [65]	CTt					X	Cronbach’s alpha
		Self-Efficacy Scale for CT					X	Cronbach’s alpha
2021	Computational Thinking Test for Lower Primary students: Design principles, content validation, and pilot testing [52]	CTtLP	X	X	X			Cronbach’s alpha
2021	Design and validation of learning trajectory-based assessments for computational thinking in upper elementary grades [56]	Early assessment				X		Cohen’s kappa
2021	TechCheck-K- A Measure of Computational Thinking for Kindergarten Children [63]	TechCheck-K			X			No evidence
2021	Measuring coding ability in young children relations to computational thinking, creative thinking, and working memory [19]	Assessment through a card-based game	X	X	X			Cronbach’s alpha
2021	A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades [55]	Assessment of computational thinking concepts		X				Cronbach’s alpha
2021	Computational Thinking Evaluation Tool Development for Early Childhood Software Education [59]	CT Test Using Bebras Tasks	X					No evidence
2022	A cognitive definition of computational thinking in primary education [44]	CTt					X	Cronbach’s alpha
2022	Adaptation into turkish of the computational thinking test for primary school students [73]	TechCheck					X	KR-20 Split half Spearman-Brown
2022	Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners’ CT test (BCTt) and the competent CT test (cCTt) [46]	BCTt		X				Cronbach’s alpha
		cCTt		X				Cronbach’s alpha
2022	Development and Validation of the Computational Thinking Test for Elementary School Students (CTT-ES): Correlate CT Competency With CT Disposition [51]	CTT-ES		X	X			Cronbach’s alpha
2022	Effect of Unplugged Programming Teaching Aids on Children’s Computational Thinking and Classroom Interaction: with Respect to Piaget’s Four Stages Theory [76]	CT capacity test					X	Cronbach’s alpha
2022	Exploring the characteristics of an optimal design of non-programming plugged learning for developing primary school students’ computational thinking in mathematics [69]	Perception Questionnaire Adapted from CTS	X				X	Cronbach’s alpha
2022	The competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary School [49]	cCTt	X	X	X			Cronbach’s alpha
2022	The Effect of Robotics-Based Storytelling Activities on Primary School Students’ Computational Thinking [64]	BCTt					X	No evidence
2022	Unplugged or plugged-in programming learning: A comparative experimental study [61]	Test of computational concepts mastery					X	McDonald’s ω
2022	Validating a computational thinking concepts test for primary education using item response theory: An analysis of students’ responses [57]	CT Concepts Test Instrument for Primary Education Based on an ECD Approach	X	X				Cronbach’s alpha
2023	A Normative Analysis of the TechCheck Computational Thinking Assessment [31]	TechCheck					X	Cronbach’s alpha
2023	Computational Literacy: Unplugged musical activities around Bebras International Challenge [36]	Computational Thinking Test using Bebras Problems					X	Cronbach’s alpha
2023	Computational thinking in primary school: effects of student and school characteristics [37]	TechCheck					X	Cronbach’s alpha
2023	Developing and Testing a Design-Based Learning Approach to Enhance Elementary Students’ Self-Perceived Computational Thinking [38]	CTS		X				Cronbach’s alpha
2023	Development and validation of a computational thinking test for lower primary school students [39]	CTtLP	X	X	X			Cronbach’s alpha
2023	Effect of Reverse Engineering Pedagogy on Primary School Students’ Computational Thinking Skills in STEM Learning Activities [40]	CTS		X				Cronbach’s alpha
2023	Effects of robotics STEM camps on rural elementary students’ self-efficacy and computational thinking [41]	Survey adapted from CTS					X	Cronbach’s alpha
2023	Effects of Scratch-Based Activities on 4th-Grade Students’ Computational Thinking Skills [42]	BCTt	X		X			Cronbach’s alpha
2023	Exploring the underlying cognitive process of computational thinking in primary education [43]	CTtLP					X	Cronbach’s alpha
2023	Monitoring cognitive development through the assessment of computational thinking practices: A longitudinal intervention on primary school students [32]	CT practices test	X	X				Cronbach’s alpha
2023	Possibilities of diagnosing the level of development of students’ computational thinking and the influence of alternative methods of teaching mathematics on their results [33]	Didactic CT test		X	X			No evidence
2023	The effect of an unplugged coding course on primary school students’ improvement in their computational thinking skills [34]	CTST	X					KR-20
2023	Think together, design together, code together: the effect of augmented reality activity designed by children on the computational thinking skills [35]	TechCheck					X	No evidence
2023	Unravelling the underlying mechanism of computational thinking: The mediating role of attitudinal beliefs between personality and learning performance [18]	CTtLP		X			X	Cronbach’s alpha
2024	A Bebras Computational Thinking (ABC-Thinking) program for primary school: Evaluation using the competent computational thinking test [47]	cCTt		X				Cronbach’s alpha
2024	The effect on computational thinking and identified learning aspects: Comparing unplugged smartGames with SRA-Programming with tangible or On-screen output [66]	CTt					X	Cronbach’s alpha

details the 50 articles, along with the instrument used, and also provides information on the validity and reliability evidence mentioned in the articles.

It was found that 54% of the reviewed articles have evidence of validity, and 88% present evidence of reliability. Among the validity types, it was found that 17 articles used content validity, followed by construct validity in 21 articles and criterion validity in 13 articles. Two articles referenced other types of validity: apparent validity [62] and argument-based validity [56]. Additionally, 23 articles were identified without validity testing. Regarding reliability, Cronbach’s alpha was used in 39 articles, followed by KR-20 in 2 articles, Spearman-Brown, McDonald’s omega, split-half, Cohen’s kappa, and empirical reliability in 1 article each. However, 12 articles do not present evidence of reliability.

4. Discussion

The aim of this systematic review was to identify research that has utilized CT assessment instruments in children and adolescents aged between 4 and 16 years to identify the variables they assess and their psychometric properties.

Through the search chain, 42 articles were identified, and an additional 8 articles were directly included, totaling 50 articles for analysis. Although the first definition of CT was presented in 2006, the first instruments to measure this construct were found starting in 2017 [7,30]. The year 2023 recorded the highest number of these instruments, with fourteen articles [18,31,32,33,34,35,36,37,38,39,40,41,42,43], indicating a growing interest in the scientific community in measuring CT skills.

Most instruments used multiple-choice as the response format, followed by a 5-point Likert scale. In terms of item count, two instruments had nine items [19,56] and one had forty-seven [32]. China had the highest number of publications with nineteen articles, followed by Spain and the United States with eight articles each, Turkey with four, and Switzerland with two publications. Asia and Europe are the continents where most research is conducted. Only one article from South America was found in this systematic literature review [72].

The most referenced authors were Marcos Román-González [7,17,44,45,46], María Zapata-Cáceres [17,46,47,48,49], and Estefanía Martín-Barroso [17,46,47,48,49], appearing in five articles each. Laila El-Hamamsy [46,47,49], was mentioned in three different articles. Furthermore, for the design of most instruments analyzed in this review, the three-dimensional (3D) model proposed by Brennan and Resnick [9] was the most used.

To determine validity and reliability, it was necessary to specify the methods used, which can be addressed through classical test theory (CTT) and item response theory (IRT). CTT relies on methods that evaluate test quality by measuring internal consistency and validity (content, criterion, and construct). In contrast, IRT offers a more advanced approach by considering individual item and participant characteristics, allowing for a more precise estimation of assessed skills and a more sensitive performance assessment. Integrating CTT and IRT provides a more comprehensive and reliable test assessment, facilitating informed decision-making in various educational and professional contexts.

The review protocol posed questions to deepen the discussion:

Which CT assessment instruments have been used in their original version or with adjustments? Among the 50 articles analyzed, 22 instruments designed to evaluate CT were identified, some of which have been used in other studies in their original version and in another 11 articles with adjustments such as translation, expert judgment, pilot testing, and verification of psychometric properties. Notably, the CTS [67] served as a reference in six investigations, while the BCTt [17], TechCheck [13], and CTt [7] were used in four investigations each. This underscores the importance of these instruments in the field of CT assessment. Additionally, the use of Bebras Tasks was noted in six articles as a basis for generating instruments and evaluating PC skills of interest in the research.

It was also observed that several studies used or adjusted more than one instrument. For example, Relkin, Johnson, and Bers [31] used five instruments (TechCheck, TechCheck-k, TechCheck-1, TechCheck-2, and TechCheck-PreK); El-Hamamsy, Zapata-Cáceres, Marcelino, and colleagues [46] adjusted two instruments (BCTt and cCTt), and Ma H. et al. [65] adjusted two instruments (CTt and CTS).

It is important to note that using an unadapted instrument may run counter to the need expressed by several authors that instruments should be contextually relevant [77] and that methodologies and tools must possess psychometric properties [4].

What are the age ranges targeted by the CT assessment instruments? In the analysis of the 50 reviewed articles, a total of 33 different instruments designed to assess CT in children within the age range of interest, spanning from 4 to 16 years, were identified.

The instrument with the broadest age range is the “Computational Thinking Test for Beginners” (BCTt), which covers ages from 5 to 12 years [17]. It is followed by the CTt, which covers ages from 5 to 12 years [7], and in [66], the CTt was used in a population aged 7 to 12 years.

It was observed that the age range with the highest availability of instruments ranges from 8 to 11 years. In contrast, for 4-year-olds, only two instruments were found: TechCheck in its TechCheck-PreK version [31] and the CT questionnaire [72]. For the 14, 15, and 16-year-old population, only one instrument, CTt [7], was identified.

What elements related to CT are assessed in the analyzed instruments? Most of the analyzed instruments assess CT from the cognitive component. The findings align with previous reviews [15,78,79,80,81,82], where the most commonly assessed skills include abstraction (13 instruments), problem-solving (11 instruments), algorithmic thinking (13 instruments), and specific computational concepts such as algorithms (14 instruments), sequences (22 instruments), loops, and conditionals (24 instruments each). This is consistent with the findings of [78,83,84,85,86].

The majority of investigations assess skills (37 instruments), followed by concepts (32 instruments), attitudes (7 instruments), and computational perspectives (2 instruments). Additionally, 17 articles were identified where both skills and concepts were assessed, 7 where skills and attitudes were assessed, and only one investigation assessed skills, concepts, and perspectives. This finding highlights the need to propose comprehensive approaches to assess CT in children and adolescents aged 4 to 16 years [24,25,86] and the fact that disconnected tests do not allow for the measurement of computational perspectives [25].

What psychometric properties are evidenced in the instruments? Analyzing the psychometric properties evidenced in the 50 articles regarding validity and reliability, it was found that 54% of the articles have evidence of validity, and 88% presented evidence of reliability, with construct validity predominating, followed by content and criterion validity. Additionally, two articles referenced other types of validity: apparent validity [62] and argument-based validity [56]. However, 12% of the articles provided no evidence of either validity or reliability. These results contrast with the findings in [8], which express that many CT assessments lack evidence of reliability and validity.

What methods were used for analyzing psychometric properties? Statistical methods such as correlation, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA) were employed to determine validity in most cases. Regarding reliability, 78% of the articles used Cronbach’s Alpha coefficient for evaluation. However, 24% of the articles did not provide evidence of having tested this psychometric property.

The evaluation of CT has experienced significant growth in recent years. However, there are still studies that use assessment instruments without reported psychometric validity. Additionally, current instruments tend to focus on specific skills such as abstraction, debugging, and problem-solving, as well as on CT-related concepts such as sequences, loops, and conditionals, while neglecting key components such as non-cognitive skills, which are also essential for holistic development.

There are few assessment instruments aimed at children between the ages of 4 and 6, a critical stage for cognitive and non-cognitive development, early detection of skills, and identification of areas for improvement. The lack of specific tools for this age group limits the possibility of establishing a solid foundation for future learning. Therefore, the design of appropriate instruments for this age range would help promote educational processes that prepare children to face the educational challenges of the future.

Given these limitations, several lines of future research could be drawn to improve the understanding and evaluation of CT. Among them, the development of assessment instruments adapted to different educational and demographic contexts to ensure equity in evaluation is essential. Likewise, conducting longitudinal studies to understand how CT skills develop and transform over time would provide a deeper insight into their evolution.

Furthermore, the integration of non-cognitive skills into evaluations will offer a more comprehensive view of student development and comparing pedagogical approaches could help identify educational practices that optimize CT learning in various contexts. Additionally, analyzing teacher training is key, as teachers are the main facilitators of CT in the classroom, making it essential to investigate how they are prepared and updated in this field.

Finally, research on the perception of CT among students, teachers, and society will allow for adjustments in pedagogical strategies to increase acceptance and motivation towards its learning, thus contributing to a more effective implementation of computational thinking in educational systems.

5. Conclusions

The systematic review analyzed 50 research articles focusing on CT assessment instruments published in WoS and Scopus databases from 2006 to March 2024. It was identified that, although the number of instruments to assess this construct has increased in recent years, in some cases, the contextual relevance and psychometric properties are not adequately considered. The review delineated the assessed skills, concepts, perspectives, and attitudes associated with CT.

The search strategy was documented transparently and reproducibly to allow others to replicate the process. Bibliometric and research interest variables were defined based on the research questions posed. Through bibliometric variables, it was determined that there was an increase in research aimed at evaluating CT through an instrument in 2023. China has conducted the most studies in this area, followed by the United States and Spain. In the Latin American context, only one study was found. Authors such as Brennan and Resnick, Marina Bers, Özgen Korkmaz, and Marcos Román Gonzales served as foundational figures in instrument construction. The most frequently used or adapted instruments in research include CTS, BCTt, CTt, and TechCheck. Additionally, a significant number of instruments incorporate questions from various years and countries’ Bebras Tasks. Instrument adjustments included item translation, expert judgment, pilot testing, and determining psychometric properties.

The age groups with the highest number of instruments for evaluating CT are between 8 and 11 years, contrasting with only two instruments identified for 4-year-olds and a single instrument for 14 to 16-year-olds. Regarding assessed skills, abstraction, problem-solving, evaluation and debugging, algorithms, and algorithmic thinking were identified as the most common, followed by sequences, loops, and conditional concepts. Some studies also evaluated computational perspectives and certain attitudes. This finding underscores the need for a comprehensive assessment of CT.

Most instruments provide evidence of content, construct, or criterion validity, with reliability predominantly assessed through Cronbach’s Alpha coefficient.