Leveraging Systems Thinking, Engagement, and Digital Competencies to Enhance First-Year Architecture Students’ Achievement in Design-Based Learning

Abstract

In recent years, teaching and learning practices have transformed greatly due to emerging technologies. Despite various pedagogical and technological innovations, the learning effectiveness of the new learning environments is still being debated. Systems thinking concepts and methods are needed regarding how to accommodate digital technology to optimize the efficacy of students’ learning, especially when student cohort specificities are addressed. For the purpose of this study, we used an empirical research design supported by a bibliometric analysis. Multiple regression using dummy coding of the predictor variables was conducted to compare the prediction models across different groups of first-year students, while a sequential mediation model was used to examine the students’ perceptions of systems thinking, engagement in the design course, and information communication technology (ICT) self-concept in relation to academic achievements. The results indicate that systems thinking centered around the understanding of feedback behaviors and causal sequences in the system has a direct effect on the design outcome and ICT self-concept related to problem solving and cognitive engagement, while, indirectly, systems thinking also mediates achievement in design courses. The ICT self-concept related to problem solving and cognitive engagement mediates the relationship between systems thinking and design course achievement. This study highlights the importance of leveraging learning system dynamics factors in diverse student cohort design courses and provides implications for developing a high-performance digital education sustainable ecosystem.

1. Introduction

Unarguably, teaching and learning practices are experiencing a deep transformation driven by ICT, which has experienced enormous expansion, not only in the educational sector but also in the industrial and service sectors. Education and training have become more accessible, manageable, and visible, by any standard. The exploitation of the potential of ICT has caused the number of learners and trainees to increase, resulting in larger classes (physical and virtual) and a number of new faculties or, even, universities [1]. Students are becoming increasingly sophisticated, as the potential of ICT and other educational tools offers a host of new or redefined opportunities for learning and completing new tasks that were previously unthinkable [2]. In this manner, new learning dynamics (flexibility, efficiency, engagement, adaptivity, reflectiveness) should be established [2]. The effectiveness and potential impact of these new learning dynamics are the subject of ongoing discussion in different sectors of the literature (economics and management, education and educational technology, social studies, etc.) [1,3].

During the time of COVID-19 restrictions, both students and educators were immersed in ICT, and the experiences gained in that situation may generate new beliefs and values which could present barriers for the effective use of ICT in the future [3,4]. For the effective introduction and integration of different ICT and tools in classrooms, or in a wide range of learning environments, new methods and approaches should be developed and used to cope with challenges in education and surrounding environments and for assisting in the externalization of personal beliefs [4]. In 2020, the European Commission (EC) adopted the Digital Education Action Plan (DEAP) (2021–2027) envisaging two priorities: (1) fostering the development of a high-performing digital education ecosystem and (2) enhancing digital skills and competences for the digital transformation in which 13 actions, in total, are foreseen [5]. The DEAP should be a key enabler in achieving the skills and competences needed for the 21st century and should facilitate recovery and resilience for a greener, more digital, and resilient European Union [5]. One of these actions is determining how to foster digital literacy and cope with the anomalies, pseudoscience, and disinformation generated by different digital systems [5].

The DEAP also affects higher education, where almost 90% of institutions are already working on realizing the plan [6]. These institutions are much better acquainted with fully online, blended, and hybrid forms of learning than before [6]. Nowadays, emerging technologies pedagogically facilitate affordance and opportunities for learning in different environments and forms [7], and knowledge and skills have become more complex, which increases target engagement in information processing and knowledge attainment in general [8]. As stated by Kautz et al. [9], a special focus should be placed on fostering both cognitive and noncognitive skills (communication, empathy, emotions, collaboration, teamwork, perseverance, grit, etc.) which have the same, or even greater, power in the prediction of learning outcomes.

Despite the acceleration of digitization in education in the last few years, digital inequalities have emerged in distance, face-to-face, and hybrid learning environments to an even greater extent than in the past, when the diversity and accessibility of educational technology and the effect of the accompanying educational methods were limited, as Gottschalk and Weise reported [10]. Although it was expected that modern ICT would effectively accommodate diverse groups of students, no matter the group, cohort size, or cultural and other background, the evidence has shown a lack of strategic and target integration and that the use of digital technologies and systems may result in less competitive outcomes [10]. Digital divides should be treated and mitigated seriously, since they may take place across different levels or stages of knowledge and skill attainment [10]. Thus, systemic understanding of the subject is needed, where distinctions, systems, relationships, and perspectives with their underlying skills are fundamental rules in systems thinking [11].

In contemporary ICT-based learning environments, fostering skills and knowledge transfer might be facilitated with a wide range of data attributes per student. This is evident in larger cohorts of students engaged in learning [12]. In smaller cohorts, there is a limited set of student data (attendance, interactions, use and diversity of learning material, sources and concepts generated, number and types of feedback), but, nevertheless, these limited attributes successfully predict students’ interim and final marks, as argued by Wakelam et al. [12]. During the study, students faced different stressors which might affect poor academic performance and lower completion rates, which is more evident in smaller cohorts and among first-year students where the number of peer interactions is smaller and value sharing is reduced [13]. Moreover, a larger diversity across students’ age, gender, and even culture may provide an environment in which students from underprivileged communities (physical or virtual) may experience robust social engagement in learning [13]. Mauldin et al. [13] also found that students in small cohorts are more frequently physically isolated from their peers than their counterparts in larger cohorts (in terms of lab work and time spent out of campus). In addition, they found that individualistic departmental cultures may present a barrier in peer relationships. On the other hand, Kara et al. [1] reported that larger classrooms in higher education are associated with lower grades of students in science, technology, engineering, and mathematics (STEM) subjects. Moreover, small classes are beneficial to students with lower socioeconomic status, in general, but in STEM fields, high-capacity male students also benefit [1]. These cohort-specific factors may stymie engagement in learning, development of systems thinking, and ICT self-concept and can result in lower achievement marks [11,12,13,14].

Because of the educational benefits of systems thinking, digital competences, and improved engagement in design-based learning, architecture study programs should seek to facilitate the relationships toward competitive design projects as a learning outcome. Understanding the factors predicting design project achievements in different-sized groups of students may help faculties develop interventions and programs to nurture their students’ systems thinking, ICT self-concept, and engagement in learning.

1.1. Theoretical Framework

All scientific disciplines, including architecture education, face exponential growth in the volume of the literature published in last 20 years. Thus, in the context of scientific information overload, combining bibliometrics and grounded theory will facilitate reviews that have a descriptive, comprehensive, or explanatory aim, as proposed by Walsh and Rowe [15]. Therefore, for the purpose of this study, bibliometrics were complemented with a grounded theory approach in the literature review.

1.1.1. Bibliometric Analysis

As argued by Zupic and Cater [16], bibliometric methods aim at revealing the performance and structure of the input data. We used the Web of Science (WoS) database to identify architecture education research articles. The following search strategies were adopted in this study: keywords, “(TS = (Architecture) OR TS = (architecture education)) AND ((TS = (systems thinking) OR TS = (design) OR TS = (system))) AND (TS = (engagement))”; publication time, 1 January 1993 to 5 August 2023. In the 30-year period from 1993 to 2023, there were 1279 English publications indexed in the WoS Core Collection online database. Then, we removed the publications (374 documents) other than articles, and only original articles (905 in total) were included in the bibliometric analysis. After checking for anomalies in eligible records retrieved from the WoS, a bibliometric analysis was performed on the final records using bibliometrix 4.0 (Naples, Italy) (https://www.bibliometrix.org/, accessed on 5 August 2023) [3]. The Biblioshiny app was used to provide a graphical web interface in the RStudio environment, version 4.3.1 (https://rstudio.com, accessed on 5 August 2023).

The bibliometric analysis was performed in a two-step procedure: (1) performance analysis and (2) science mapping. The performance analysis consists of publication- and citation-related metrics, while the science mapping focuses on citation analysis, bibliometric coupling and co-word analysis as suggested by Donthu et al. [17].

As suggested by Walsh and Rowe [15], we first defined boundaries in the field of the study, then managed bibliographic data and conducted structural analysis. Next, after an analysis of the relationship between categories and concepts, we prepared a conceptual model, identified research gaps, and prepared propositions. During the workflow, we also made adjustments (e.g., type of keyword selection, search engine keys, etc.) between some consecutive steps.

The results of the bibliometric analysis can be enriched by three enrichment paths based on a network analysis in the form of network metrics, clustering, and visualization as argued by Donthu et al. [17]. Network metrics highlight the importance of research constituents by enhancing the discussion of research domains through measures such as degree centrality (number of relational connections), betweenness centrality (information transfer between unconnected groups of nodes), and closeness centrality (effective information transfer through proximity to other nodes) [17]. Clustering allows us to create thematic clusters that provide insight into the most important topics with underlying intellectual structure and evolution over time in a research area [17]. For the purpose of this study, the science mapping includes topic and keywords trends. A thematic analysis was conducted to detect the main research topics of architecture education using a co-occurrence network, and thematic evolution was conducted using a thematic map. A co-occurrence network of keywords is shown in Figure 1. It shows the relatedness of items based on the number of documents in which keywords occur together. The size of a node and label indicates the frequency of a keyword in the dataset, whereas the thickness of an edge indicates the co-occurrence frequency between keywords. The closely related keywords are labeled with greater thickness of the line and vice versa, whereas the color of the node shows the cluster with which the keyword is associated.

As shown in Figure 1, two main clusters represented the keywords and corresponding links of research on the topic. The largest cluster is represented by the red color, and it is centered around the node of architecture, the main topic in the research. The research on architecture seems to be most strongly related with “engagement” and “design”, followed by “systems” and “technology”, as can be seen in Figure 1, where these relations are labeled with thicker connectors.

The keyword “engagement” is related also to “technology”, “performance”, and “learning impact” in design-based learning. The keywords “performance” and “quality” might be bridging connectors with the second cluster, represented by the blue color, which has two strong nodes, “management” and “system”, followed by the keywords “network” and “challenges”. The keyword “engagement” from the first cluster directly relates to “management”, which strengthens the impact of the “engagement” keyword. A keyword system which also emphasizes systems thinking can be a crucial mediator and catalyst between knowledge creation and knowledge management, allowing us to cope with future challenges implicit in creating a sustainable built and natural environment.

The keyword “engagement” is also related to “technology”, “performance”, and “learning impact” in design-based learning. The keywords “performance” and “quality” might be bridging connectors with the second cluster, represented by the blue color, which has two strong nodes, “management” and “system”, followed by the keywords “network” and “challenges”. The keyword “engagement”, from the first cluster, directly relates to “management”, strengthening the impact of the “engagement” keyword. A keyword system which also emphasizes systems thinking can be a crucial mediator and catalyst between knowledge creation and knowledge management, allowing us to cope with future challenges implicit in creating a sustainable built and natural environment.

Another important representation of keywords is their evolution over the last 30 years. In the themes’ evolution, the 30-year period was divided into five stages: 1993–2015, 2016–2018, 2019–2020, 2021–2022, and the last milestone, 2023 (Figure 2). The first stage was the initial stage of basic domain research, with few keywords and loose research topics. Following the second stage, targeted studies were conducted on mind tools and architecture itself, but a system view on the topic was representative. For the first time, engagement rose; this dominated throughout the next stages. In the third and fourth stages, in-depth studies were conducted on social media, quality, behavior, education, prediction, and engagement. This year, the keywords of “challenges”, “engagement”, “validation behavior”, and “strategies” dominate the discussion.

With the development of the quality of architecture education and the number of papers published in last years, no keyword design, as a basic design studio activity, has been identified. This suggests that the number of architecture education articles has increased over time, and the topic has evolved through further research within architecture education into systems, behavior, predictions, engagement with ICT-enhanced learning, and performance outcomes.

1.1.2. Learning Dynamics in Technology-Enhanced Education Ecosystems

The current structure and dynamics of learning as a process of acquisition of knowledge, skills, attitudes, behaviors, etc., by different means (e.g., study, experience, practice) require much more meaningful learning, since we are faced with conflicting information via the Internet, social media, or other digital sources which are replacing traditional sources of learning (e.g., teachers, role models, peers) [18]. Much more than before, we face pseudoscience, misconceptions, misinterpretations, and misapplications which might hinder meaningful learning [19]. Thus, multimodal thinking must be constantly encouraged, as suggested by Bryce and Blown [18]. Thereby, a perceived change in concept, which can be viewed as ability and skill, should rather be seen as a process of conceptual prioritizing, revision, and selection, and not only as a replacement [18]. To enhance learning and retention of new and/or complex concepts, Ausubel has proposed advance organizers which focus student attention during learning [19], leverage existing knowledge through activity-based instruction, and create higher-order structures inside learning institutions [18]. The advance organizers should operate at a higher level of abstraction, generality, and inclusiveness, and can be seen as comparative and expository organizers, according to the concept of Subsumption theory [20]. As argued by Elfeky et al. [21], comparative organizers can be used as reminders to bring into working memory information which may not appear to be relevant, for example, by activating existing schemas, while expository organizers may relate what the learner already knows with the new and unfamiliar material to make it more plausible to the learner. Thus, using advanced organizers, students’ active involvement and participation in learning activities increases. In addition, students’ reactions to and interactions with the learning material will increase, as the material is embedded in a surrounding environment of knowledge creation and cocreation [22]. As an advance organizer for learning systems thinking and for sustainable knowledge transfer, Green et al. [23] suggest the introduction of different ICT and digital tools (e.g., simulation, prediction models, functional analysis diagrams).

Student engagement is a key learning focus of student-centered active learning, where it can also predict students’ performance outcomes in the course [24] and the effectiveness of teaching and learning platforms [7]. Due to the heterogenous nature of engagement, a meta-construct of engagement is rather complex, consisting of several dimensions utilizing cognitive, behavioral, emotional, and social processes and, as such, a meta-construct complements self-regulation and vice versa [22].

In the case of the architecture or art disciplines, there is another important dimension of engagement—aesthetic engagement—which also involves creativity and somatic engagement. Both of these dimensions employ both higher-level cognition of aesthetic objects (e.g., rethinking interpretations, discovering affective resonance) and low-level forms (perceptual engagement) [25]. When forming aesthetic judgments, we deploy perceptual, cognitive, and emotional processes [25]. The aesthetic response can be multifaceted: (1) cognitive, (2) somatic, (3) emotional, and (4) spiritual [26]. In addition, this response is related to personal motivational attitudes [27].

The context of cognitive dimension is, rather, seen as “the psychological investment students make towards learning” [28] across cognitive taxonomic levels [29], and, as argued by Barlow et al. [28], meaningful learning is predicated on quality cognitive engagement. Moreover, as argued by Green [30], deep cognitive engagement has predictive value in students’ higher-level cognitive outcomes.

Behavioral engagement is seen as the effort or involvement students make in the classroom, school, or other learning environments when learning [24]. Students’ multidimensional behavior can be reflected in asking questions, collaboration, intensive communication, hands-on activities, active experimentation, attending classes and school, following classroom rules, and interacting positively and appropriately with teachers and peers [31]. Behavioral engagement is a predictor of students’ drop-out rate from study and of their long-term achievement [31].

Affective engagement reflects students’ reactions to their learning environment, and it may emphasize different emotions or emotional states (e.g., boredom, enthusiasm, value, personal connection). As argued by Kotluk and Tormey [32], emotions, through their five components, are reflected in the appraisal of a situation (cognitive), bodily changes (neuropsychological), action tendencies (motivational), facial expressions (psychomotor), and intuition (subjective feeling). Thus, emotional engagement enhances students’ motivation, resilience, and will to invest effort in their learning [24]. Thus, both positive (curiosity, surprise, etc.) and negative achievement-related emotions (confusion, frustration, etc.) may be triggered by a task or assignment during learning [22]. This calls for finding a balance between students’ capacity and the difficulty of an assigned task to produce an optimal experience [33].

Social engagement, as a dimension of engagement, can be seen as a part of interpersonal skills related to collaboration and interaction with different agents in the learning process (peers, teachers, shared content, and digital systems) [7,34]. Moreover, social engagement, together with emotional and cognitive engagement, plays a crucial role when tasks deal with the problems and contexts of complex social relationships, when students are directed to work with others, and when value systems conflict [24,35], especially during study abroad [36]. As argued by Shany et al. [37], socioemotional encounters involve a resonance of others’ affective states and attribution of mental states to others where empathy can integrate both states. Moreover, emotional aspects of the experience gained from tasks or other work assignments can be used for visualizing and, consequently, trigger somatic engagement with learning [37].

Considering a complex structure of engagement, Sinatra et al. [38] do not recommend artificially splitting the dimensions, since dimensions of engagement co-occur and contribute to each other. Thus, when measuring engagement, some overlap in the scales can be expected, and the interpretation of results should be carried out with caution, using the systems view. Moreover, since all parts of teaching and learning are part of the educational system, we should also view an ability to engage in systems thinking as a key component in the success of quality improvement initiatives and critical to systems-based practice, as argued by Dolansky et al. [39].

The potential of systems thinking for coping with current needs and wants in surrounding environments and the digital education ecosystem is great [11,39,40,41]. Systems thinking can be seen as an educational approach which enhances one’s “ability to recognize, understand, and synthesize the interactions, and interdependencies in a set of components designed for a specific purpose” [42] (p. 5). The dimensions of systems thinking can be reflected in the ability to manage a sequence of events, causal sequence, multiple possible causations, variation in different types (random/special), feedback and interrelations of factors, and patterns of relationships [42]. Systems thinkers also demonstrate an increased capacity for the community-minded components of cognitive and affective empathy [43]; thus, systems thinking might be seen as a key influencer in dynamic knowledge transfer [44,45] when employing the four essential systems thinking skills: making distinctions, organizing systems, recognizing relationships, and taking multiple perspectives [11]. In a meaningful educational setting, learning instructors, when using a systems thinking approach, must leverage wide prior knowledge to fit the phenomena best [46]. Moreover, to obtain much of the benefits of systems thinking, attention should be paid to unintended areas, parts, boundaries, and dynamic relationships in the system [46]. According to sociocultural theory, the social environment also facilitates knowledge construction and transfer, and scaffolding learning activities should be directed toward higher levels of systems thinking taxonomy (creating simulation models, testing policies, etc.) [23,36,47,48] where students may develop self-regulated behaviors [39,46]. Social interactions can help learners move from the unknown to known in the context of systems thinking [36,46]. For the effective implementation of systems thinking in the classroom, several models can be proposed, including models for the improvement of fluency in complex reasoning, a framework consisting of the problem, perspective, and time [49]; models for developing metacognition, a strategy of distinction, systems, relationship, and perspectives (DSRP) [11]; a heuristic model with dimensions of declarative and conceptual knowledge, modeling systems, solving problems using system models, and the evaluation of system models [50]; a four-domain model (mindset, content, structure, behavior) [51]; a systems literacy model (systems language, methods, and practices) [41]; and others.

Systems thinking could be seen as an effective complement for design thinking, and vice versa, since systems thinking encompasses both facets of insights (gaining and using) [51], especially in disciplines where creativity, art, and design dominate [52]. Architecture education and design studio work can deploy both approaches to improve the system dynamic and user design experience [52]. Moreover, in the complex and dynamic environment in which we create and use design today, systems thinking may improve our understanding of future designer thinking by changing perceptions and mental models, inversing aspects in design and, thus, viewing them in a new light, interconnecting patterns in all natural things [53], and creating a more inclusive student-centered experience for enhancing engagement with learning [54], decision making [55,56,57] and leadership [58,59,60]. Systems thinking can play a key role in the shift to sustainability by focusing on interconnected changes in technologies, social practices, business models, regulations, and social norms [55]. Therefore, education for sustainable development can be considered as a main tool to achieve the Sustainable Development Goals defined by the United Nations [61], and the competence of systems thinking can be considered as a key competence [62].

Despite the advantages of complementary integrated design and systems thinking, more research is needed regarding how to address complex sustainability challenges in the world we live in and make through the principles of the good, just, useful, and satisfying in human experience [63]. As argued by Buchanan [63], failing to consider these working principles in systems and design thinkers may lead to the failures of technological or social platforms.

One of the priorities of the DEAP is enhancing digital skills and competences for the digital transformation [5], which should be realized through different actions. Action number 7 deals with fostering digital literacy and tackling disinformation through education. In recent years, especially during the COVID-19 pandemic, countries around the world invested large amounts of assets in ICT and digital systems, in general, but findings from the scientific literature indicated mixed effects of ICT on academic achievements. Thus, curriculum designers, policy makers, and other educators who employ ICT in educational settings must rethink ICT and digital systems from different perspectives.

For the purpose of this study, we examined ICT-enhanced education through the lenses of activity theory [64], as suggested by Kwong and Churchill [65] and the perspective of ICT self-concept [66]. Activity systems deal with subject, object, and community (learning environment) where the subject is placed to accomplish tasks with tools to fulfil outcomes (object) [65]. ICT might function as a mediator in learning, and activities can be distributed among different groups or teams [65,67]. Regarding how students perceive ICT competences and their motivation to use and interact with it, a competence self-concept can be an important predictor in performance and behavior in ICT-enhanced learning [14,66]. Moreover, ICT-enhanced learning and knowledge sharing, according to the connectivism theory [68], can be seen as a network phenomenon where the diversity of opinion drives learning through different information sources and may also be found in nonhuman appliances, as argued by Siemens [69] and Downes [70].

Some recent positive effects of digital systems were reported for the personal level (endorsing thinking and self-management) and the social level (encouraging socialization and communication) [65], while association between ICT use and students’ boredom was moderated by educator enthusiasms and whether or not educators use ICT [71]. The literature also suggests that cognitive–emotional engagement in ICT mediates the effect of ICT use on academic achievement [67]. The ICT self-concept, in terms of basic computer skills, was an important predictor in performance, mediated by the interest in ICT, while social engagement mediates the relationship between computer self-concept and content knowledge [14].

Systems thinking has a long tradition in computing education and greatly relates to computational thinking [72]. Thus, it is expected that systems thinking may affect engagement in learning using ICT [73] and ICT self-concept might moderate the relationship between systems thinking and design course achievement [74]. Systems thinking skills can be enhanced using different activities supported by ICT and other digital tools, such as modeling, simulations, 3D printing, and scanning [23,75]. When systems thinking is combined with design thinking, ICT use on higher taxonomic levels may predict performance in design-based learning [75], especially when the teaching/learning material and environment are created in a way in which they can be modified (notably, ICT permits tasks to be redesigned) and redefined (ICT permits the creation of new, previously inconceivable tasks) according to the SAMR model [76]. The SAMR model (substitute, augmentation, modification, and redefinition) is widely used for the assessment of ICT integration in education and educational practices [76]. When using the SAMR model, for effective ICT integration in teaching/learning, we should be aware of the barriers. These barriers are primarily the teachers’ beliefs and attitudes that influence their experience of ICT use and classroom integration [3]. The next factor which might affect ICT integration in a classroom using SAMR is computer-based assessment. The literature suggests that immediate performance feedback on responses can benefit both cognition and metacognition in learning, but when negative responses are provided, it may decrease student motivation and engagement with tasks [77]. Moreover, feedback behavior after a correct response does not differ from the student’s behavior in the case of no feedback [77]. Thus, when modeling complex behavior using systems thinking, it is necessary to carefully generate causal loops within respective systems by leveraging processes for reinforcing, balancing, and delaying feedback, depending on the gap between the desired and actual state [78]. When investigating a cohort as a system, learning occurs through the construction and crossing of networks established by students [68]; thus, feedback loops must be purposefully mapped, monitored, and modified to prevent system failure due to loops that interfere with learning objectives [77].

1.2. Study Context, Aim, and Research Questions

This study was carried out in the contexts of architecture education. In Poland, the Master of Architecture degree can be obtained after completing uniform master’s studies or two-cycle studies (first degree: Bachelor of Architecture, engineering studies; second degree: Master of Architecture, master’s studies) in the field of architecture. The field of study of architecture may have a general academic profile or a practical profile. The study of architecture may be conducted at public or private universities, where the size of cohorts may differ greatly. The observations which were made up to this time show that students of large faculties perceive studies differently compared to those from smaller faculties [79]. As a result, student behavior and academic performance may also differ, despite the same curriculum, the same admission requirements, a comparable level reached in the examination at the beginning of the study, and the time allocated to each subject. In addition, in some cases, part of the teaching staff may also be shared between universities [79].

Driven by the research gap concerning the cohort specificity in knowledge creation and transfer in and among first-year students, this study’s aim was twofold: (1) to determine valid measures which capture abilities which enhance a group’s knowledge creation and transfer and (2) to examine which student cohort dynamics are conditioned by a study program carried out with different-sized groups. As for the literature review, we complemented bibliometrics with grounded theory to improve the transparency and depth of analysis. A bibliometric analysis in architecture education provides us with a co-occurrence network, Keyword Plus, which may support the selection and determination of influencers in the dynamics of knowledge creation and transfer in the field.

Based on the findings of the literature review, the present study addresses the following research questions (RQs), which guided the entire research:

RQ1: What is the reliability and validity of measurement for capturing the dynamics of knowledge creation and transfer?

RQ2: What are the first-year students’ level of systems thinking, engagement in learning, ICT self-concept, and design thinking, and what are the differences among the students of different-sized universities?

RQ3: Do systems thinking, the students’ engagement in learning, ICT self-concept, and design thinking influence the effect of different-sized groups on the achievement in design projects?

RQ4: What is the relationship between systems thinking, ICT self-concept, engagement in a design studio, and the design project final marks when controlling for prior achievement?

Whereas the present study is a part of two multi-year projects, namely, “Architecture education for the 21st century” (Faculty of Architecture, Cracow University of Technology, Cracow, Poland) and “Developing the 21st century skills needed for sustainable development and quality education in the era of rapid technology-enhanced changes in the economic, social and natural environment” (Faculty of Education, University of Ljubljana, Ljubljana, Slovenia), the main contribution of the study is providing researchers, curriculum designers, and educators in the field with a valid methodology for studying learning dynamics from a systemic perspective in technology-enhanced educational settings for digital equity and effective inclusion in education.

2. Materials and Methods

The researchers used an empirical research design using survey methodology with a quantitative approach.

2.1. Architecture Education Basic Settings

Firstly, we carefully investigated architecture study programs and found that the most similar study programs are for the first year of the first cycle (Bachelor Studies). Therefore, we used a research group of first-year students at three universities in Poland as our research subjects. This fulfilled all the basic requirements for our study of learning dynamics, that subjects must (1) attend public universities, (2) conduct two-cycle studies (Bachelor and Master, separately), and (3) carry out studies with a general academic profile but differing in the number of students. Therefore, for this study of architecture education learning dynamics, we selected the Cracow University of Technology (CUT), which has about 250 students enrolled in their first year, the Poznań University of Technology (PUT), which has approximately 160 students enrolled in their first year, and the Kielce University of Technology (KUT), which has approximately 30 students enrolled in the first year [79].

At CUT, the largest university in this study, students are divided into larger or smaller groups according to subject. In the design studio and ICT classes, there are around 15 students; in math, geometry, and language courses there may be around 30 students in each group. All 250 students meet for lectures or seminars; thus, they can share their projects, ideas, and inspirations.

We found similar circumstances at PUT [80]. At KUT, where 30 students are enrolled in the study year, learning activities are structured to place students in small interactive groups. This enables them to start together and finish together while exchanging ideas and thoughts in the process [81].

In the first year of study, students learn all the basics necessary in the sustainable architectural world. They take their first steps in urban planning, architectural design and all related disciplines, starting with the more creative side such as freehand drawing courses, but also history, mathematics, physics, construction, building structures, engineering, sustainability, and even necessary software skills. The program of study is based on a transdisciplinary connection between most subjects, society, and the sustainable development policy. Students learn to work both individually and in teams, solving sustainable design problems at all levels in a variety of contexts, most of them multidisciplinary. The work in the design studio is largely based on issues of sustainable development, respect for the natural environment, and the integration of people and the environment—elements that are in line with the current socio-economic policies of the European Union. By promoting the principles of sustainable design, pro-ecological architecture, and universal design, students create design projects that use at least three elements of sustainable design. These selected principles are closely related to the SDGs, which are a set of interconnected goals and global targets for transforming our world [61]. The students first gain knowledge of sustainable design in lectures and then apply this knowledge in their pavilion projects and later in interior sustainable design projects. An example of a set of sustainable design principles might be as follows: design a building with renewable energy sources, solar panels, photovoltaic glass facades, or green roofs, captures rainwater, involves the 3Rs—reduce, reuse, recycle—cost-effective solutions, durable design solutions, product life cycle, enhances indoor environmental quality, optimizes operational and maintenance practices, and optimizes energy use where sometimes wind energy and geothermal are considered. The principles chosen by the students usually affect the shape of the pavilion and the materials they use for the design, and improve this design in the next step when they choose an open space for the pavilion and design its interior. The students used systems thinking in their work by exploring multiple perspectives, looking at the whole and the parts, making good decisions that are sustainable even in confusing situations, considering problems appropriately, recognizing and respecting architectural systems and adhering to boundaries, distinguishing and evaluating each part of a system, noting and understanding relationships, understanding feedback behaviors, understanding past system behaviors and predicting future behaviors, and responding to changes over time. Students will be able to apply a variety of skills to change a system to achieve goals or learning objectives.

2.2. Procedure and Sample

The survey, which consists of four instruments, was designed and developed by the authors with the feedback of four experts in the field of architecture to assess the face validity of the instruments early on. Next, the content validity was assessed by architecture faculty teachers (6) who were familiar with the concept under investigation. They evaluated the items with respect to problems, ambiguity, proper use of terms, and comprehensibility.

When the evidence of face and content validity was provided, a survey study using the instruments was conducted. The data were collected in June 2022 through an online survey hosted on the Google platform. All involved participants were informed about the study on the survey’s front page—before proceeding with the questions—where clear instructions regarding how to fill all four questionnaires were also provided. All concerns and questions raised by the participants were addressed to the authors’ email addresses and resolved promptly before the students took part in the study. Informed consent was provided by the participants for the collection of personal information, e.g., sex, age, institution, year of study, and grade point average (GPA). All consent forms were collected by the online portal Google Forms and archived at the Cracow University of Technology.

Since this study involved human participants from different universities, all three universities had it approved by their review boards or ethics committee, who also certified that the study was performed in accordance with the ethical standards presented in the Declaration of Helsinki. The ethical statement with approval was collected in May 2022 from all three universities involved in the study.

After verifying the consent and cleaning the data, the final analytical sample consisted of 138 first-year students who successfully completed all four questionnaires, while the total base of first-year architecture students from all three universities was approximately 440 students. All four instruments were prepared in Polish. The final analytical sample consisted of 36 males (26.09%) and 102 females (73.91%); the students’ average age was 20.1 years (SD = 1.4). The distribution of students across the universities was as follows: 57 from CUT (41.3%), 51 from PUT (37%), and 30 students from KUT (21.7%).

2.3. Measures

This study examined the variables of students’ systems thinking, their engagement with learning, ICT self-concept, and design thinking ability; these were assessed using a Likert scale, while the variable of GPA was reported by students based on the average grades they received after completion of the semester courses together with a design subject grade. The GPA and design subject grade ranged from 1 to 5, while a grade threshold for successful design work was 2. All four measures, with corresponding final instruments, are attached as Supplementary Materials titled Student Survey final questionnaires.

2.3.1. Systems Thinking

For the construction and measurement of systems thinking skills, the 20-item Systems Thinking Scale (STS) developed by Moore et al. [42] was used as the basis. This instrument has been validated in international studies that confirmed the unidimensionality and reliability of the scale [39,40,42,43,82]. Before we translated the questionnaire into Polish, we adapted the items towards settings in architecture education and modified the response scale from five points (original) to six points, where the responses ranged from 0—never to 5—most of time. Thus, we applied an exploratory factor analysis (EFA) to discover the new factor structure. To retain the number of factors, we performed Velicer’s minimum average partial (MAP) test [83,84], which revealed three factors that were also confirmed with a revised MAP r² test [85]. A parallel analysis for the current study was run in IBM SPSS (v. 25), utilizing the map.sps script developed by O’Connor [85] (https://oconnor-psych.ok.ubc.ca/nfactors/nfactors.html, accessed on 31 July 2023). After applying the Zwick and Velicer rules [84] to retain the components, the EFA was rerun, and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was 0.80, indicating the reliability of the principal component analysis and the compactness of the correlations, confirming the sample’s ability to produce distinct components (Bartlett’s test of sphericity was significant, p < 0.001). All construct items had high communalities (>0.5) and loadings higher than 0.5, as suggested by Tabachnik and Fidel [86]. The components were also supported by a scree plot that yielded three clear components. The first factor had five items related to understanding the feedback behaviors, analyzing the causal sequence, and the variation in different types (random/special) in the system. The second factor, with four items, was closely related to the student’s ability to discover the interrelations of the factors and recognize the multiple causations possible in the systems. The third factor contains six items related to recognizing a sequence of events, causal sequence, and interrelations of factors. The reliability of each scale was estimated using McDonald’s ω, which revealed moderate reliability for all three factors (0.77, 0.73, and 0.71, respectively).

The final instruments for measuring the systems thinking consisted of 15 items, and the item scores were summed to provide a total STS score. Scores could range from 0 to 75. For the subscales, a mean (M) and standard deviation (SD) was calculated. There were no reverse-coded items.

2.3.2. Student Engagement

The student engagement was conceptualized as the students’ perceptions of their engagement resulting from their interactions in design subject learning, the learning environment, and its agents (e.g., peers and educators) and measured along the dimensions of behavioral, cognitive, emotional, social, aesthetic, and somatic student engagement, with scales adapted from previous research. These original engagement dimensions have been used in different educational settings, e.g., chemistry education (behavioral, cognitive, emotional, and social), and psychology (aesthetic and somatic). Therefore, where necessary, these were modified to capture the features of architecture design-based learning.

The behavioral (four items), cognitive (four items), emotional (five items), and social engagement (four items) were measured using items adapted from Naibert and Barbera [24]. We kept most of the original items from Naibert and Barbera [24], but items were adapted and modified in consideration of the students’ engagement in the design studio activity.

The aesthetic engagement may play a crucial role in the architecture design and urban planning activities, since it involves active participation in the appreciative process, sometimes by overt physical action but always by creative perceptual involvement [87], and alongside cognitive and affective processes that students actively deploy on their way to generating aesthetic judgement [25]. As argued by Schummer et al. [88], the aesthetic quality of places is of central importance to the quality of people’s lives. The aesthetic engagement was measured using three items adopted from Diessner et al. [26].

A six-point Likert scale was used to assess the students’ engagement, from 1—strongly disagree to 6—strongly agree. The negatively worded items were reverse coded before analysis.

For identifying the underlying factors that drive common variance, an EFA was conducted and a five-factor solution was discovered with a 64.1% explained variance. The KMO statistics were 0.81, indicating an adequate sample. Bartlett’s sphericity test statistics were 1104 (p < 0.001), indicating that the correlations in the data were strong enough to use dimension reduction.

Velicer’s MAP test for determining the number of components was used, and it indicated five components. The reliability of each scale was estimated using McDonald’s ω, which revealed moderate to high reliability for all scales (0.82, 0.79,0.81, 0.80, 0.75, and 0.76, respectively).

2.3.3. ICT Self-Concept

For assessment of ICT self-concept, we used the ICT-SC scale developed by Schauffel et al. [66]. An original 25-item ICT-SC scale with six dimensions was validated in the English and German languages. For the purpose of this study, we made some modifications; namely, for assessment, we chose a six-point Likert scale (the original scale has five points) ranging from 1—strongly disagree to 6—strongly agree. Next, we positioned the questionnaire in the context of the higher education settings of the architecture subject matter and design studio, while the original ICT-SC questionnaire, instead, aims to capture, at a general level, the heterogenous groups included.

An EFA was conducted using the principal component analysis extraction method in combination with an Oblimin rotation. A three-factor solution explained 71.2% of the total variance. The results confirmed that each item had a clear primary loading on one factor (factor loadings > |0.50|). The components were also supported by a scree plot that yielded three clear components. Bartletts’ test of sphericity for the 25-item instrument was 3360 (p < 0.001) and the Kaiser–Meyer–Olkin measure of sampling adequacy was 0.93, indicating the reliability of the principal component analysis and the compactness of the correlations, confirming the sample’s ability to produce distinct components. Both Velicer’s MAP [83] and the MAP r² [85] test indicated a three-factor solution.

To establish the reliability and internal consistency, we also calculated McDonald’s ω coefficients, which yielded 0.95, 0.90, and 0.94 for the three components, respectively, and 0.97 for the 25-item instrument in general. The first factor (11 items) represented self-concept toward ICT-enhanced communication, processing, and store and content generation. The second factor (four items) was related to students’ self-concept about safe applications, while the third factor (nine items) represented the use of ICT in general and for problem solving.

In total, 24 items were used for analysis, while one item from the original ICT-SC scale was excluded. The item deleted from the original instrument read, “It is easy for me to prepare digital data, information, and content for others”.

2.3.4. Design Thinking

The questionnaire contained items related to the students’ design thinking abilities in the design studio activities. The items were adapted from the Dosi et al. [89] mindset and assessed on a six-point Likert scale, from strongly disagree (1) to strongly agree (6). All 71 items were administered to the students, followed by an examination of the new factor structure. The EFA revealed an eight-factor structure, wherein the items accounted for 61.5% of the total variance above the threshold of 0.5 suggested by Hair et al. [90] and Pituch and Stevens [91]. Bartletts’ test of sphericity for the 71-item instrument was 7015 (p < 0.001), and the KMO measure of sampling adequacy was 0.84, indicating the reliability of the principal component analysis and the compactness of the correlations, confirming the sample’s ability to produce distinct components. Both Velicer’s MAP [83] and the MAP r² [85] test indicated an eight-factor solution.

A new DT mindset consists of eight factors, with 37 items in total, which are labeled as follows (

Table 1. New DT mindset factor structure with McDonald’s ω reliability coefficients.

Factor	Number of Items	McDonald’s ω
DT1—Abductive thinking, creativity, and envisioning new things/future knowledge	8	0.91
DT2—Embracing risk and being comfortable with uncertainty	5	0.81
DT3—Empathy	5	0.82
DT4—Teamwork and collaboration	4	0.72
DT5—Experiential intelligence	3	0.75
DT6—Learning-oriented and optimistic that they will have an impact	6	0.85
DT7—Problem reframing	3	0.80
DT8—Open to different perspectives/diversity	3	0.78

).

2.4. Data Analysis

2.4.1. Descriptive Statistics and Normality Tests

Means and standard deviations were calculated for the scores of all constructs. A measure of the skewness and kurtosis has been reported in analytics. To test the normality, the Shapiro–Wilk test was conducted. IBM SPSS Statistics software (version 25) was used.

2.4.2. Validity Tests and Inferential Statistics

The convergent and discriminant validity of the constructs was conducted using the ADAN-CO 2.3 software (https://www.composite-modeling.com/ (accessed on 13 August 2023)) while, for the reliability estimation, McDonald’s omega was calculated using Hayes’ Omega macro for SPSS downloaded from www.afhayes.com (accessed on 12 August 2023). In empirical research such as ours, McDonald’s omega is favored since several assumptions can be violated (equal factor loadings, uncorrelated errors, unidimensionality, etc.), and omega provides a better alternative than Cronbach’s alpha [92,93].

To detect the differences between the groups of students, multivariate analysis of covariance (MANCOVA) was used, while multiple regression with interaction terms were used as dummy variables in the model to further distinguish the different-sized groups of students in achievements at design-based learning. A sequential mediation analysis was performed to examine the key measures which might also have complementary effects on the design project grade controlled with prior knowledge.

For the effect size, different measures were used. As a measure of the effect size where a between-group effect was detected, eta squared (η²) was used with the following interpretation: size from 0.01 to 0.05 = a small effect; size from 0.06 to 0.14 = a medium effect; and size of 0.14 and over = large effect [94].

In the mediation analysis, a measure of the effect size, Cohen’s f², was used for direct effects. The Cohen’s f² categorized the effect size as small (≥0.02), medium (≥0.15) or large (≥0.35) [95]. For estimating and interpreting the indirect effect size, we used υ², where the squared standardized υ effect should be greater than 0.175 for a large effect, 0.075 for a medium effect, and 0.01 for a small effect, making them more appropriate for indirect effects, as proposed by Lachowicz et al. [96], Ogbeibu et al. [97], and Gaskin et al. [98].

3. Results

3.1. Convergent and Discriminant Validity

The validity of measures in statistics is essential for producing reliable, accurate, and meaningful data that support informed decision making, build credible knowledge, and contribute to the advancement of various fields of study [99]. Although we used questionnaires that had been previously validated, the convergent and discriminant validity must be reconfirmed to verify whether adapted items and changed assessment scales may distort the measures.

3.1.1. Systems Thinking Skills

As evidence for systems thinking measures, convergent validity was provided, as shown in

Table 2. Composite reliability (CR), the square root of the average variance extracted (AVE) (in bold), and correlations among systems thinking (ST) constructs (off-diagonal).

Latent Constructs	CR	AVE	ST 1	ST 2	ST 3
ST 1	0.83	0.53	0.72
ST 2	0.82	0.62	0.36	0.79
ST 3	0.81	0.51	0.52	0.46	0.71

. All three measures significantly loaded onto the contemplation trait and were reliable enough, whereas McDonald’s ω and the composite reliability (CR) of the constructs were >0.70 [90,91].

As is shown in Table 2, all AVE values were above the threshold of 0.5, while the square root of the AVE was larger than 0.7 (bold-diagonal), the threshold suggested by Hair et al. [90]. Moreover, the inter-construct correlation values (off-diagonal) ranged from 0.36 to 0.52, indicating that the measures converged to a medium to large degree, according to the strength of the correlation represented as large, ≥0.5; medium, 0.3–0.5; and small, 0.1 to 0.29, as suggested by Carlson and Herdman [100]. Thus, our results suggest the convergent validity of the adapted constructs, and the high convergent validity suggests that all dimensions of systems thinking should be retained.

Next, we also examined the discriminant validity using the heterotrait–monotrait (HTMT) approach proposed by Henseler et al. [101] and Shaffer et al. [102].

Table 3. Heterotrait–monotrait ratio of correlations (HTMT) and Fornell–Larcker criterion results (in parentheses) for systems thinking scale. AVE is shown in the diagonal.

Latent Constructs	ST 1	ST 2	ST 3
ST 1	0.53
ST 2	0.40 (0.11)	0.62
ST 3	0.70 (0.25)	0.53 (0.14)	0.51

shows that the HTMT ratio of correlations was less than the threshold of 0.85 [102].

Discriminant validity was also evaluated against the Fornell and Larcker criterion [103] as a control. This criterion is commonly employed [99,104]. This criterion suggests that AVE (see Table 3) is greater than the shared variance (in parentheses) [99]. Examining the results of the validation generated by the ADAN-CO software [105], it has been found that all shared correlation values are markedly lower than the AVE of each factor.

Our results suggest that all variables of systems thinking included in the analytics demonstrate evidence of discriminant validity.

3.1.2. Student Engagement

The same procedure was applied in the engagement measures. The convergent validity of the engagement scales was examined analytically for the AVE, factor loadings, and interfactor correlations (

Table 4. Composite reliability (CR), the square root of the average variance extracted (AVE) (in bold), and correlations among engagement constructs (off-diagonal).

Latent Constructs	CR	AVE	Behavioral	Cognitive	Emotional	Social	Aesthetic
Behavioral	0.89	0.67	0.82
Cognitive	0.86	0.62	0.61	0.79
Emotional	0.86	0.58	0.55	0.44	0.76
Social	0.85	0.62	0.11	0.12	0.15	0.79
Aesthetic	0.85	0.68	0.42	0.35	0.31	0.24	0.83

).

As is shown in Table 4, all the AVE values are above the threshold of 0.5, while the square root of the AVE is larger than 0.7 (bold-diagonal), the threshold suggested by Hair et al. [90]. Moreover, the inter-construct correlation values (off-diagonal), ranging from 0.10 to 0.52, indicate that the measures converge in a small to large manner, according to the strength of the correlation represented as large, ≥0.5; medium, 0.3–0.5; and small, 0.1 to 0.29, as suggested by Carlson and Herdman [100]. An especially weak correlation was found between the social engagement and cognitive, emotional, and behavioral values. Thus, our results suggest the convergent validity of the adapted constructs, and good convergent validity suggests that we retain all five dimensions of the students’ engagement with learning in design studio.

The discriminant validity of the construct was tested using the HTMT approach and controlled by the Fornell and Larcker criterion (

Table 5. Heterotrait–monotrait ratio of correlations (HTMT) and Fornell–Larcker criterion results (in parentheses) for engagement measures. AVE is shown in the diagonal.

Latent Constructs	Behavioral	Cognitive	Emotional	Social	Aesthetic
Behavioral	0.67
Cognitive	0.69 (0.31)	0.62
Emotional	0.65 (0.28)	0.54 (0.14)	0.58
Social	0.11 (0.04)	0.10 (0.03)	0.15 (0.02)	0.62
Aesthetic	0.51(0.16)	0.44 (0.12)	0.37 (0.09)	0.26 (0.05)	0.68

).

Our results suggest that all the variables of the student engagement with learning included in the analytics demonstrate evidence of discriminant validity.

3.1.3. ICT Self-Concept

Table 6. Composite reliability (CR), the square root of the average variance extracted (AVE) (in bold), and correlations among ICT-SC constructs (off-diagonal).

Latent Constructs	CR	AVE	ICT-SC 1	ICT-SC 2	ICT-SC 3
ICT-SC 1	0.94	0.68	0.82
ICT-SC 2	0.93	0.78	0.65	0.88
ICT-SC 3	0.96	0.69	0.81	0.69	0.83

shows the results from a test of convergent validity of ICT-SC constructs. The same procedure as that of the two previous measures was applied.

As is shown in Table 6, all the AVE values are above the threshold of 0.5, while the square root of the AVE is larger than 0.7 (bold-diagonal), the threshold suggested by Hair et al. [90]. Moreover, the inter-construct correlation values (off-diagonal), ranging from 0.65 to 0.81, indicate that the measures converge greatly according to the strength of the correlation [100].

The discriminant validity of the construct was tested using the HTMT approach and controlled by the Fornell and Larcker criterion (

Table 7. Heterotrait–monotrait ratio of correlations (HTMT) and Fornell–Larcker criterion results (in parentheses) for ICT-SC measures. AVE is shown in the diagonal.

Latent Constructs	ICT-SC 1	ICT-SC 2	ICT-SC 3
ICT-SC 1	0.68
ICT-SC 2	0.70 (0.42)	0.78
ICT-SC 3	0.86 (0.68)	0.75 (0.48)	0.69

).

As shown in Table 7, the HTMT ratio of the correlation between Factor 1 and 3 is just above the threshold of 0.85. Thus, we applied the HTMT ratio of the correlations more liberally, using a threshold value of 0.90 [106], and all values met this criterion. Moreover, Henseler et al. [101] stated that even when the inter-construct correlations were as high as 0.95 HTMT, it failed to detect discriminant validity violations.

Our results suggest that all the variables of the student ICT self-concept included in analytics demonstrate evidence of discriminant validity.

Despite the validity of the ICT-SC measures, we also checked whether the dataset was contaminated by common-method bias. First, we performed a widely used Harman’s single factor test through EFA. The result indicated that a single-factor solution accounted for more than 50% (54.3%) of the variance. This showed that the dataset could be contaminated by common-method bias [107,108]. In addition to Harman’s test, we conducted a full collinearity test where the occurrence of all variance inflation factors (VIFs) was lower than 3.3, indicating that the model could be considered free of common-method bias [107]. In this study, a latent factor ICT-SC 3 had a VIF value greater than 3.3 (3.7) but lower than the more liberal threshold value of 5 proposed by Hair et al. [109]. The respondents were also assured of anonymity and confidentiality [110] to reduce common-method variance. Since the threat of common-method bias exists, the results and interpretations of the findings should be treated with caution.

3.1.4. Design Thinking

The convergent validity of the DT constructs was assessed against the same criterion as all previous measures in this study. The results from the analysis showed (

Table 8. Composite reliability (CR), the square root of the average variance extracted (AVE) (in bold), and correlations among DT constructs (off-diagonal).

Latent Constructs	CR	AVE	DT1	DT2	DT3	DT4	DT5	DT6	DT7	DT8
DT1	0.92	0.59	0.77
DT2	0.87	0.58	0.24	0.76
DT3	0.90	0.70	0.53	0.12	0.84
DT4	0.82	0.54	0.43	0.33	0.39	0.73
DT5	0.84	0.65	0.36	0.11	0.31	0.18	0.81
DT6	0.87	0.69	0.60	0.24	0.53	0.38	0.42	0.83
DT7	0.87	0.69	0.44	0.10	0.33	0.43	0.37	0.42	0.83
DT8	0.88	0.59	0.61	0.32	0.43	0.41	0.33	0.51	0.41	0.77

) that the AVE values ranged from 0.58 to 0.70. Moreover, in determining the composite reliability (CR) of the constructs where the suggested threshold value is =0.7 [90], the CR values ranged from 0.82 to 0.92. This means that the constructs used in this study met the thresholds of convergent validity.

The discriminant validity of the DT constructs was tested using the HTMT approach and controlled by the Fornell and Larcker criterion (

Table 9. Heterotrait–monotrait ratio of correlations (HTMT) and Fornell–Larcker criterion results (in parentheses) for DT measures. AVE is shown in the diagonal.

Latent Constructs	DT1	DT2	DT3	DT4	DT5	DT6	DT7	DT8
DT1	0.59
DT2	0.25 (0.04)	0.58
DT3	0.60 (0.26)	0.12 (0.02)	0.70
DT4	0.52 (0.17)	0.41 (0.11)	0.49 (0.15)	0.54
DT5	0.42(0.12)	0.07 (0.01)	0.38 (0.09)	0.25 (0.03)	0.65
DT6	0.71(0.35)	0.30 (0.05)	0.65 (0.28)	0.51 (0.15)	0.54 (0.17)	0.69
DT7	0.52(0.19)	0.06 (0.03)	0.39 (0.11)	0.57 (0.19)	0.46 (0.14)	0.54 (0.18)	0.69
DT8	0.72(0.36)	0.38 (0.10)	0.51 (0.18)	0.52 (0.16)	0.39 (0.11)	0.63 (0.25)	0.52 (0.17)	0.59

).

Our results suggest that all the variables of design thinking included in the analytics demonstrate evidence of discriminant validity.

3.2. Descriptive Statistics and Normality Tests

The descriptive statistics include variables of the systems thinking ability, engagement with learning, ICT self-concept, and design thinking reported for first-year students from all three universities in the study. To verify the normality, a measure of symmetry (skewness, S) and the tailedness of the distribution (kurtosis, K) are also shown in

Table 10. First-year architecture students’ self-reported average scores expressed with mean (M) and standard deviation (SD) across the subscales of systems thinking, engagement, ICT-SC, and DT along with a measure of skewness (S) and kurtosis (K) (n = 138). Data from assessment of students’ learning achievements are expressed with GPA and design project grade.

	Variables	CUT				PUT				KUT
		M	SD	S	K	M	SD	S	K	M	SD	S	K
Students’ grades	GPA value	4.30	0.39	−0.42	−0.15	4.32	0.31	−0.26	−0.97	4.20	0.26	−0.19	−0.31
	Design project grade	4.67	0.43	−0.96	0.75	4.48	0.51	−0.74	0.05	4.28	0.55	−0.51	0.19
Systems thinking	ST 1	3.91	0.79	−0.19	−0.75	3.75	0.81	−0.38	−0.77	3.26	0.91	−0.71	−0.18
	ST 2	3.85	0.75	−0.75	0.97	3.51	0.68	0.09	−0.45	3.08	0.82	0.09	−0.49
	ST 3	3.31	0.61	−0.15	−0.98	3.25	0.78	−0.61	0.26	3.29	0.80	−0.16	−0.17
	Total ST	54.75	8.26	−0.18	−0.86	52.39	8.69	−0.25	−0.20	48.41	9.91	−0.49	0.53
Engagement	Behavioral	5.13	0.76	−0.96	0.81	4.86	0.99	−0.65	−0.66	4.71	0.86	−0.19	−0.96
	Cognitive	5.09	0.62	−0.73	0.09	4.79	0.73	−0.34	−0.58	4.47	0.73	−0.03	−0.87
	Emotional	4.42	0.97	−0.37	−0.46	4.23	1.04	−0.71	0.26	4.75	0.67	−0.98	0.99
	Social	3.53	0.75	0.22	−0.35	3.66	0.89	−0.22	−0.03	3.81	1.06	−0.15	−0.99
	Aesthetic	3.91	0.85	0.61	−0.10	4.30	0.85	−0.61	0.97	4.20	0.77	−0.19	0.02
ICT self-concept	ICT-SC 1	4.28	1.02	−0.26	−0.40	4.21	0.96	−0.66	0.73	4.76	0.98	−0.98	0.97
	ICT-SC 2	3.60	1.02	0.16	0.11	3.68	1.23	−0.49	−0.66	4.50	0.94	−0.14	−0.54
	ICT-SC 3	4.25	1.01	−0.15	−0.70	3.77	1.07	−0.61	−0.18	3.54	1.02	−0.97	0.99
Design thinking	DT1	4.45	0.84	0.48	−0.76	4.60	0.71	0.08	−0.72	4.55	0.71	0.21	−0.98
	DT2	3.27	1.08	0.34	−0.01	3.52	0.97	−0.09	−0.03	3.63	0.91	−0.10	−0.64
	DT3	4.56	0.85	−0.21	0.51	4.94	0.66	−0.57	0.22	4.81	0.83	−0.63	−0.57
	DT4	3.89	0.77	0.28	0.77	4.03	0.89	−0.32	−0.03	4.36	0.86	−0.26	−0.28
	DT5	4.92	0.87	−0.57	−0.13	4.71	0.85	−0.86	0.15	4.34	0.80	−0.64	0.98
	DT6	4.61	0.86	0.01	−0.85	4.80	0.71	−0.38	−0.73	4.73	0.80	−0.41	−0.47
	DT7	4.74	0.81	−0.18	−0.98	4.87	0.76	−0.83	0.72	4.86	0.87	−0.70	−0.14
	DT8	4.97	0.91	−0.72	−0.09	4.77	0.84	−0.38	−0.48	4.18	0.95	−0.23	−0.99

.

A combination of a visual inspection, an assessment using skewness and kurtosis (Table 10), and a formal normality test (Kolmogorov–Smirnov) was used to assess whether the assumption of normality was acceptable or not. Since all values of skewness were between −2 and +2, and kurtosis was between −7 and +7, the data were considered to be normal, as argued by Hair et al. [90] and Byrne [111]. The Kolmogorov–Smirnov test produced nonsignificant test statistics (p > 0.05), suggesting that the dataset possessed normal distribution.

MANCOVA was used to test the statistical significance of the effect of the group of students as an independent variable on a set of dependent variables according to the measures of these dependent variables. As we suspect that the unobserved ability affects the GPA which students obtained after the study semester, the GPA was used as a covariate in the study.

Before we conducted the MANCOVA, we confirmed the homogeneity of the regression slopes to see if there was an interaction between our covariant (GPA) and independent variable (Group). This procedure was applied for all measures in this study (systems thinking, engagement, self-concept, and design thinking). Box’s test of equality of covariance matrices of dependent variables was not significant (p > 0.05) and Pillai’s trace was also nonsignificant for the interaction effect (p > 0.05). Next, Leven’s test of equality of the variances indicated that the error variance of the dependent variable was equal across groups (p > 0.05). Thus, we satisfied our assumption of the homogeneity of the regression slopes and ran a full factorial MANCOVA.

First, the MANCOVA was conducted to determine the effect of a group controlled by GPA on systems thinking across three factors. The results of the MANCOVA showed that there were significant differences among the three study groups on the dependent measures ST1 and ST2 (F (2,138) = 4.58, p = 0.012, partial η² = 0.07; F (2,138) = 7.91, p = 0.001, partial η² = 0.11, respectively). The examination of the pairwise comparisons and means revealed that in ST1, there was a significant difference between the mean scores of the CUT and PUT (p = 0.009) groups and, in ST2, between the CUT and KUT groups (p < 0.001). No significant differences were found in ST3. The differences in factors ST1 and ST2 were also reflected in the total scores on systems thinking between the CUT and KUT students (p = 0.016).

The same procedure was applied for the engagement measures, where differences were found in three dimensions: (1) cognitive engagement (F (2,138) = 6.27, p = 0.002, partial η² = 0.09), (2) emotional engagement (F (2,138) = 3.50, p = 0.033, partial η² = 0.05), and (3) aesthetic engagement (F (2,138) = 3.18, p = 0.048, partial η² = 0.04). The pairwise comparisons indicated differences in the cognitive engagement: the CUT students outperformed both the PUT (0.039) and KUT students (p = 0.01). In emotional engagement, the KUT students outperformed the PUT students, while, in aesthetic engagement, the PUT students perceived higher levels of beauty in design tasks in contrast with the CUT students (p = 0.018).

The examination of the ICT self-concept, which might be an important predictor of performance, behavior, and motivation in the design studio, revealed interesting results. For ICT-SC1 (communication, material preparation, and storage) and ICT-SC2 (safe application), the first-year KUT students perceived this scale to be higher than their counterparts from CUT and PUT (F (2,138) = 4.12, p = 0.020, partial η² = 0.06; F (2,138) = 7.51, p = 0.001, partial η² = 0.10, respectively). Using ICT for enhancing higher-order thinking skills (problem solving, creative work, etc.), as represented by ICT-SC3, was rather more developed in the CUT students (F (2,138) = 4.80, p = 0.010, partial η² = 0.06), who perceived their ability in these skills to be higher than the PUT and KUT students (p = 0.048, p = 0.022, respectively).

The results of the MANCOVA for design thinking showed that there were significant differences among the three study groups on the four dependent measures, namely, DT3, DT4, DT5, and DT8 ((F (2,138) = 3.25, p = 0.042, partial η² = 0.05; F (2,138) = 3.20, p = 0.045, partial η² = 0.04; F (2,138) = 3.80, p = 0.026, partial η² = 0.05; F (2,138) = 5.51, p = 0.005, partial η² = 0.08, respectively). The examination of the pairwise comparisons and means revealed that there was a significant difference between the mean scores of the PUT and CUT groups (p = 0.041) in the achievement of DT3 (empathy); in DT4 (team work and collaboration), the CUT group outperformed the KUT group (p = 0.044). The CUT students also outperformed the KUT students in DT5 (experiential intelligence) (p = 0.02), and the same results were demonstrated for DT8 (p = 0.004). DT5 points toward action-oriented behavior in the design studio over discussion and conceptual or analytical behavior. DT8 is related to the diversity of or openness to different perspectives, which can be understood as encompassing collaboration in diverse teams—beyond the usual disciplines—to tap into knowledge and experiences, and the integration of diverse external perspectives throughout design thinking [89].

3.3. Do Systems Thinking, Students’ Engagement in Learning, ICT-SC, and Design Thinking Influence the Effect of Different-Sized Cohort Groups on Achievement in a Design Project?

The bibliometric analysis results in Table 10 show the importance of key educational factors, which might be crucial for enhancing the students’ achievements in the design studio. Thus, we decided to investigate whether systems thinking, engagement in learning, ICT-SC, university condition membership, and the interaction between them are significant predictors in the model. For a more nuanced investigation of the group differences, we used multiple regression with interaction terms and, to facilitate regression analysis, we created dummy variables for group membership, where the KUT membership condition was the reference group. Next, we conducted a multiple linear regression, regressing achievement in the design project onto educational factors, dummy condition variables, and interaction terms. A total score for systems thinking, ICT self-concept, each engagement type, and design thinking was used in the regression analysis.

3.3.1. Systems Thinking

Systems thinking is a critical interdisciplinary skill that is described as the cognitive flexibility needed to collaboratively work on problems facing society [49]. The ability to understand feedback behaviors, causal sequence, and variation in different types (random/special) and the ability to discover the interrelations of factors and recognize multiple causations possible in the systems did not seem to be evenly developed in students. In systems thinking, it is essential that students employ skills, make distinctions, organize systems, recognize relationships, and take multiple perspectives [11].

As detailed in

Table 11. The interactive effect of systems thinking and university membership group on design project grade (reference group is KUT membership).

Model	Unstandardized Coefficients		t	Sig. p-Value
	β	Std. Error β
Constant	2.76	0.40	6.30	0.000
PUT	1.13	0.55	2.13	0.042
CUT	1.05	0.57	1.75	0.072
Systems thinking	0.04	0.01	4.31	0.000
PUT × systems thinking	−0.03	0.01	−1.98	0.048
CUT × systems thinking	−0.02	0.01	−1.41	0.146

Note: adjusted R² = 0.26.

, the model explained a significant 26% of the variance in the design project grade, F (5, 132) = 9.03, p < 0.001.

As shown in Table 11, systems thinking and PUT membership condition, and the interaction between the two, were significant predictors in the model. Thus, our hypothesis was partially supported: systems thinking moderated some but not all the effects of university membership on design project grade. The PUT students scored, on average, 1.13 points higher than the KUT students, and this score was moderated by systems thinking. The CUT students also scored higher in the project grade than the KUT students, but this was not moderated significantly by systems thinking (p > 0.05).

3.3.2. Student Engagement in Learning

The same procedure was applied for engagement measures, and our hypothesized model was supported only by the behavioral and cognitive engagement while the emotional, social, and aesthetic engagement did not moderate the effects of group membership on the design project grade (p > 0.05).

For the behavioral engagement, the results indicated that the model explained a significant 25% of the variance in the design project grade, F (5, 132) = 8.92, p < 0.001. As detailed in

Table 12. The interactive effect of behavioral engagement and university membership group on design project grade (reference group is KUT membership).

Model	Unstandardized Coefficients		t	Sig. p-Value
	β	Std. Error β
Constant	2.09	0.44	4.76	0.000
PUT	1.86	0.55	3.40	0.001
CUT	2.06	0.60	3.42	0.001
Behavioral engagement	0.47	0.09	5.01	0.000
PUT × behavioral engagement	−0.36	0.11	−3.14	0.002
CUT × behavioral engagement	−0.37	0.12	−3.02	0.003

Note: adjusted R² = 0.25.

, the behavioral engagement, the condition of CUT or PUT membership, and the interaction between the two were significant predictors in the model. Thus, our hypothesis was supported: behavioral engagement moderated all the effects of group membership on the design project grade.

As shown in

Table 13. The interactive effect of cognitive engagement and university membership group on design project grade (reference group is KUT membership).

Model	Unstandardized Coefficients		t	Sig. p-Value
	β	Std. Error β
Constant	1.99	0.45	4.35	0.000
PUT	1.97	0.58	3.38	0.001
CUT	2.05	0.67	3.04	0.003
Cognitive engagement	0.52	0.10	5.12	0.000
PUT × cognitive engagement	−0.40	0.12	−3.22	0.002
CUT × cognitive engagement	−0.39	0.14	−2.79	0.006

Note: adjusted R² = 0.25.

, the model of predicting factors in the design project grade where cognitive engagement was included together explained a significant 25% of the variance in the design project grade, F (5, 132) = 8.94, p < 0.001.

The results in Table 13 indicate that the cognitive engagement moderated all effects of PUT and CUT group membership on the design project grade when the reference group was the KUT students.

In the scope of this research question, we also investigated whether the ICT self-concept and design thinking ability moderated the effects of PUT and CUT group membership contrasted with the effects of KUT group membership, but the hypothesis was not supported (p > 0.05). Thus, we were interested to determine if a sequence—and if so, what sequence—of affecting factors we examined may further explain the differences in the design project grade.

3.4. Testing Sequential Mediation of Systems Thinking Skills, ICT-SC, and Engagement in Learning on Students’ Achievements in Design Projects

The overall goal of this study was to provide evidence and deeper insights into the dynamics of knowledge creation and transfer in order to improve students’ achievements in the design studio. Thus, we further investigated variables where higher-graded students dominated using different types (and combinations) of thinking (systems thinking and design thinking), engagement, and developed ICT self-concept. Our proposed model presents systems thinking skills which model the ICT-enhanced engagement of students’ learning and practice in the design activity. It was observed that engagement influenced the project’s final mark.

A sequential mediation analysis was performed, using Model 6 in Hayes’ Process macro in SPSS (v. 25), to examine the effects of several mediators on the relationship between the first-semester GPA and the student design project grade [112]. We used a mediation analysis to bootstrap the indirect effect and test the mediation effect, as suggested by Preacher and Hayes [113] and Hair et al. [109]. Preacher and Hayes [113] also stated that no overlapping zero in-between should appear in the values for mediation, which means that the confidence interval (CI) did not include zero.

A series of regression analyses were completed to investigate any influences on the design project grade. Our final model, which explained 40.5% of the variance, included the variables of systems thinking, ICT-SC, and cognitive engagement with direct and sequential mediation effects on the design project grade.

The exploration of the relationship between systems thinking, ICT self-concept, cognitive engagement, and achievement began with a review of the relevant correlations (Figure 3). For the ICT self-concept, a construct of ICT-SC3 was used, since this concept reveals students’ behavior towards the use of ICT to enhance higher-order thinking skills (e.g., problem solving, critical thinking, decision making, creativity).

The results indicated both direct and mediated effects. The first-semester GPA significantly (p < 0.05) affects systems thinking and the design project grade (β = 0.63, Cohen’s f² = 0.32; β = 0.34, Cohen’s f² = 0.17, respectively), while nonsignificant effects (p > 0.05) emerged between the GPA and ICT-SC3 and cognitive engagement. As expected, systems thinking has significant (p < 0.05) and positive effects on ICT-SC3, cognitive engagement, and project grade (β = 0.41, Cohen’s f² = 0.18; β = 0.26, Cohen’s f² = 0.25; β = 0.08, respectively). The use of ICT to enhance higher-order thinking skills, as assumed, has a significant (p < 0.05) and positive effect on the cognitive engagement and achievement in the design project (β = 0.24, Cohen’s f² = 0.06; β = 0.10, Cohen’s f² = 0.04, respectively). Given the effect size, systems thinking skills would be classified as having a medium effect on this relationship, while ICT-SC3 and cognitive engagement would be classified as having a small effect.

The results indicate that systems thinking mediates the relationship between the GPA and project grade (indirect effects = 0.14, CI [0.006, 0.250], υ² = 0.020), while three sequential mediation paths were revealed, namely, (1) GPA -> systems thinking -> cognitive engagement -> design project grade (indirect effects = 0.05, CI [0.004, 0.108], υ² = 0.003), (2) GPA -> systems thinking -> ICT-SC 3 -> design project grade (indirect effects = 0.04, CI [0.001, 0.087], υ² = 0.002), (3) GPA -> systems thinking -> cognitive engagements -> ICT-SC 3 -> design project grade (indirect effects = 0.02, CI [0.003, 0.033], υ² = 0.0004). The effect size of systems thinking as a mediator for the project grade would be classified as small, while the effect size of complimentary mediation would be classified as less than small in this relationship. As suggested by Gaskin et al. [98], evidence supports significant indirect effects (p < 0.05) despite some values of υ² being less than 0.01, for example, in a case where the sample size is smaller than 400 and the research context and phenomenon of interest are still underdeveloped.

4. Discussion

The primary contribution of this study was to create a learning dynamics model where the direct and indirect effects of three dominant paradigms in architecture design studio learning outcomes of diverse students’ cohorts, namely, systems thinking, ICT self-concept, and engagement in learning, were examined. For several decades, when different-sized groups’ or cohorts’ achievement were investigated, diverse and inconsistent findings were revealed [1,12,13]. In the present study, when we used a systemic approach for leveraging key enablers, results revealed a significant heterogeneity of measures which contribute to achievement in design courses.

4.1. Reliability and Validity of Measurement for Capturing Dynamics of Knowledge Creation and Transfer

For the purpose of the present study, we carefully selected respective measures. The extensive validation resulted in the confirmation of all measures we used.

The original STS [42], after adaptation and necessary modifications, produced a 15-item questionnaire that converged in three valid constructs. All constructs had a reliability above the threshold of 0.7 [86]. Evidence of the discriminant and convergent validity is also provided, and the strength was estimated as medium to large [100]. All constructs together successfully encompass all six dimensions of systems thinking as they were described in the literature [11,39,40,48,51].

The student engagement scales were developed based on [24,26], resulting in a 20-item questionnaire. All scales demonstrated medium to large reliability [91], while similar conclusions have been drawn for the convergent and discriminant validity [100,101].

The ICT self-concept was measured using a 24-item questionnaire adapted from [66], but the EFA revealed only three valid constructs, which were further used for the interpretation of results. In a study by Schauffel at el. [66], six dimensions were used, and the convergent and discriminate validity were not confirmed as correlating with other related scales in the ICT self-concept. Thus, in the present study, we highlighted the critical value of the scales to discriminate the validity due to multicollinearity. Using a three-factor solution, in this study, we mitigated critical issues to discriminate the validity, while the findings about the convergent validity and evidence of reliability are consistent with the findings of Schauffel et al. [66].

The design thinking was measured by 37 items which explained eight constructs. All constructs were moderately reliable [91] and valid [100,105], which is consistent with Avsec [75].

For the determination of the factors to retain in the EFA, a revised Velicer’s MAP [85] combined with a scree test was found to be appropriate for the purpose of this study.

4.2. First-Year Students’ Level of Systems Thinking, Engagement in Learning, ICT Self-Concept, and Design Thinking

The mean scores of systems thinking do not differ significantly (p > 0.05) across the groups of first-year architecture students. These scores are comparable to those of medical students [42], while they are lower than those of healthcare professionals with work experience greater than 5 years [42] and those of in-service teachers [82]. It seems that the ability to understand different feedback behaviors and causal sequence (ST1) can be decisive in systems thinking learning when the different group size is addressed. It could be that this systemic ability was underdeveloped in the KUT students, which may result in lower course achievements, consistent with the findings of Kuklick et al. [77] and Senge [78]. Moreover, it seems that the feedback behavior can be more easily shaped in larger cohorts, when the student attributes are more diverse and present in a larger number, which evokes intensive critical thinking, since in smaller cohorts the students’ system is smaller and understanding of concepts can easily converge with one of the prevailing concepts, whether it is correct or not, which further supports the findings of Kuklick et al. [77]. The systems thinking ability—understanding the interrelations of factors and recognizing multiple causations possible in the systems (ST2)—was most developed in the largest cohort, the CUT students, where a wider range of data attributes for each student was possible than for those in the small cohorts, which may facilitate learning and knowledge transfer, consistent with the findings of Wakelam et al. [12] and Mauldin et al. [13]. In smaller groups, there are limited social interactions and less learning nodes and connections between them, which may result in lower scores in the factors ST1 and ST2, which is consistent with the findings of Goldie [68], Tarrant et al. [36], and Pazicni and Flynn [46].

Examining the differences among the different groups of students in the engagement in learning, we observed that the larger groups outperformed the smaller groups in cognitive engagement. It seems that more meaningful learning using different advanced organizers (simulations, modeling, graphs, diagrams, etc.) was encompassed in the learning process of the CUT students, since, as argued by Barlow et al. [28], a transition from shallow to meaningful cognitive processing more likely occurs in larger groups, especially when learning is enhanced using different types of ICT or digital systems [7]. Moreover, cognitive engagement might be induced by a higher level of systems thinking in larger groups, consistent with the findings of Dolansky et al. [39], and a higher level of self-regulation, as argued by Gunness et al. [7]. The groups with a higher level of cognitive engagement also had higher final project marks. It could be that engagement in learning is able to mediate ICT use in the design activity and course achievement, consistent with the findings of Li and Zhu [67]. Notably, regarding emotional engagement, the smaller cohorts outperformed the larger cohorts, pointing to the triggering of more positive emotions (curiosity and belonging) being more likely in the smaller groups by the assignment, consistent with the findings of Boekaerts [22] and Lonngren et al. [35]. The results also indicated nonsignificant differences (p > 0.05) in the behavioral and social engagement in learning, pointing to comparable student involvement in learning environments across groups and collaborative social interactions in design assignments which deal with contexts and problems with similar social relationships. Thus, we confirmed the findings of Naibert and Barbera [24] and Bergdahl et al. [34].

The ICT self-concept, through its constructs, might be an important predictor of cognitive, behavioral, and emotional outcomes. Thus, the results yielded in this study are of special importance and interest. The students from smaller cohorts outperformed their counterparts from the larger cohorts in self-perceived concept regarding the use of ICT for communication, generating learning material, and safe use of ICT. This might be attributed to working in smaller groups or even to physical isolation from peers, which may induce the use of ICT on a personal (self-management) and social level (communication), which is consistent with the findings of Kwong and Churchill [65]. The results from the students in larger cohorts indicated that the use of ICT for problem solving and creative work was a primary goal when engaged in learning. It may be that students in larger cohorts can create a so-called society of knowledge or student educational resources hub, confirming the findings of Wakelam et al. [12].

For design thinking, larger cohorts promote collaboration and diversity in teamwork as action-oriented behavior. It seems that a systemic approach, in larger groups, can manipulate more attributes than a systemic approach in smaller cohorts. This confirms the findings of Elsawah et al. [41], who found that systems language structured by systems methods was applied efficaciously by competency in systems practices (procedural knowledge). This systems language (concepts and principles) seems to affect the feedback loops in larger cohorts more, and it seems easier to reinforce, balance, and modify feedback, due to a wider set of attributes, in a larger cohort [41]. It could be that the value and limitations of systems thinking can be demonstrated more easily in larger cohorts which provide a valuable input for design principles and designing itself, as argued by Buchanan [63]. Larger cohorts can typically have extensive information sharing and searching when collaborating on problem-solving tasks. In the design studio, particularly, this flow might go beyond teams; consequently, the engagement in information processing is stronger, as argued by Duan et al. [8].

4.3. Systems Thinking, Students’ Engagement in Learning, ICT Self-Concept, and Design Thinking Influence Achievement in Design Projects

The systems thinking analysis results, when conditioned by membership in different cohorts, showed a slight difference when comparing the medium-sized cohort to the small-sized cohort in the prediction of the final project grade, while the final project grades of students in the larger cohorts did not differ significantly (p > 0.05) from the others. For the measure of systems thinking, a total score was used. It may be that the factors of systems thinking (with their respective dimensions) have significant different prediction values for design achievement; this is the subject of the final research questions we address in this study. It seems that unidimensional systems thinking ability is not a sufficient predictor of the learning dynamics in different-sized cohorts, which points to multidimensional treatment when measuring and interpreting systems thinking, as has been stated by Dolansky et al. [39], Grohs et al. [49], Cabrera and Cabrera [11], and Elsawah et al. [41].

The results from the analysis of the students’ engagement in learning indicate the behavioral and cognitive engagement conditioned by cohort membership is significant in the prediction of the final project grade when larger cohorts were compared with smaller cohorts. Engagement is inherent in all learning processes with the different levels of self-regulation involved [22]. A lack of self-regulation may decrease a student’s awareness of learning, use of learning strategies, number of activities, ability to control learning and deliver effective feedback, communication, and collaboration interactions in learning; this may inhibit engagement [7]. It seems that the students in the larger cohorts perceived more involvement when working on the design studio projects, which affected their final grade. This is consistent with the findings of Hospel et al. [31], whereby they argue that when more reflections, collaboration, intensive communication, hands-on activities, and active experimentation are involved, students behave more effectively and the commitment to task finalization is greater. It might be that the perception of the usefulness of ICT for solving problems is greater, which increases all types of engagement, as argued by Gunness et al. [7]. Moreover, these results can be connected with a greater externalization of beliefs toward the use of ICT in larger cohorts, which is consistent with the findings of Bicalho et al. [3].

Similar findings were revealed regarding cognitive engagements: the larger cohorts significantly leverage this engagement for meaningful learning, which explains differences in the final project grade, consistent with the findings of Barlow et al. [28]. The cognitive engagement might be induced by a greater level of ICT self-concept related to the problem-solving ability, which confirms the findings of Gunnes et al. [7] and points towards the systemic integration of ICT for providing effective feedback. Together, these factors may help to reduce the misinformation, misconceptions, and pseudoscience generated in learning. It might be that the larger cohorts successfully accommodate and encourage multimodal thinking using different advance organizers, which is consistent with the findings of Bryce and Blown [18], and, by doing so, improve meaningful learning, as argued by Ausubel [19]. When combining both the behavioral and cognitive engagement in learning, the risk of failure can be reduced, especially in larger cohorts, as argued by Mouldin et al. [13], since students are more likely to better self-regulate their learning toward higher-order thinking outcomes, as argued by Boekaerts [22]. Students who are keener on novelties in ICT will be more likely to have higher predispositions to explore and experiment with new technologies [7] and become more engaged in learning with behavioral and emotional engagement, while those who perceive the usefulness of ICT will score higher on all components of engagement (cognitive, behavioral, social, emotional, and reflective), as suggested by Gunnes et al. [7].

Moreover, ICT use for problem solving may affect students’ self-efficacy and self-regulatory processes and, thereby, their cognitive engagement in learning to improve learning achievements, as argued by Li and Zhu [67]. According to the connectivism theory, there is no real concept of knowledge transfer but, instead, knowledge emerges from connections that are formed during network activity, as argued by Downes [70]. Thus, students’ cognitive engagement evoked by systems thinking may further stimulate localization of information sources and the recurrence and development of learning patterns which might be adequately durable for the emergence of knowledge—in our case, high grades on a design project—which confirms the findings of Downes [70] and Goldie [68].

Notably, no moderating effects of the social, emotional, and aesthetic engagement were found which may predict a student’s final grade. Since several authors provide evidence that the use of ICT may affect all types of engagements for improving academic achievements [7,24,34,67], it could be that ICT and the digital systems used in the design studio evenly accommodate the students’ social, emotional, and aesthetic engagement in learning, no matter the size of the cohort. Next, we confirmed our previous statement, since the results of the analysis of the ICT self-concept and design thinking moderating abilities indicated no significant effects in the larger cohorts compared with a smaller cohort used as a reference. It could be that the approach of design thinking which is largely exploited in design tasks creates student engagement in tasks itself, as argued by Grau and Rockett [54].

4.4. Sequential Mediation of Systems Thinking Skills, ICT-SC, and Engagement in Learning on Students’ Achievements in Design Project

For a nuanced investigation of the direct and indirect effects on the final project achievement controlled with prior knowledge (GPA), a sequential mediation was used.

The positive and significant association between the effect of the first-semester GPA and design project grade is in line with the activity theory, as these findings indicate that students who have better cognitive abilities (GPA) to comprehend subject matter in the first semester are more likely to be motivated in project work and reflective learning as a part of systems thinking. It seems that a wide range of prior knowledge should be leveraged to fit systems thinking best, as argued by Pazicni and Flynn [46]. Surprisingly, the GPA level is not predictive of the use of ICT for problem solving, either in cognitive engagement nor in design-based learning, but increased problem-solving ability in students evokes students’ cognitive engagement and directly and indirectly affects their success in design tasks, confirming the findings of Li and Zhu [67] and Schauffel et al. [66]. It seems that the conative, instead of the cognitive, dimension of the attitude toward ICT might be decisive in the use of ICT for problem solving and experimentation, as argued by Gunness et al. [7].

Our study’s findings indicate that the most important mediator in the model is systems thinking’s ability to understand feedback behaviors and causal sequences. Induced by cognitive ability, this may directly affect the use of ICT for problem solving, evoke cognitive engagement in learning, and improve the design-based learning outcomes. Moreover, the cognitive engagement in learning mediates ICT use for problem solving and achievement in design courses, which is consistent with the findings of Li and Zhu [67].

Even though prior research has shown that the perceived usefulness of ICT positively affects all types of student engagement, we provide evidence that only the targeted use of ICT (problem solving and creative work) enhances higher-order thinking and students’ cognitive–motivational strength when engaged in design tasks, which confirms the findings of Avsec [75]. It seems that advance organizers, because they build information modeling, 3D modeling, and various simulations used in the course along with functional analysis diagrams and flowcharts, might enhance meaningful learning when students are cognitively and behaviorally engaged at higher levels, which is consistent with the findings of Eidin et al. [72] and Green at al. [23].

Our study results also provide evidence for sequential mediation relationships. The GPA increased systems thinking ST1 and further increased ICT-SC3, followed by increased cognitive student engagement which, in turn, influenced design-based learning achievements. Thus, ST1, ICT-SC3, and cognitive engagement, in that order, encouraged the positive relationship between the GPA and student achievement in the design courses. It might be that concept change is not just a replacement for a previous concept, but rather a process of conceptual prioritizing, revision, and selection whereby students are intently engaged in problem solving in a system-based intensive reflective practice, as argued by Bryce and Blown [18]. In addition, systems thinking, by articulating problems through different perspectives, expands time and space constraints to mitigate unintended consequences and improve our decision making [55,56]. A combination of both approaches improves student’s empathy (cognitive and emotional), leading to a greater engagement in learning [43,54]. When systems and design thinking are combined, it may provoke users to exert both ways of thinking at the same time, and by doing this, enables designers, by the quality feedback that helps one to better understand complex problems, to create awareness and stimulate creative thinking for better and sharpened problem definition [57]. By integrating the systems and design thinking approaches, the leadership capacity, as part of innovation skills, may increase [58], since it was perceived as low in students practicing remote technology-enhanced learning [59]. Moreover, a system-based development of transformational leadership capacity may benefit the ICT self-concept and engagement in learning [60].

In line with the activity theory, our research shows that in the design studio learning environment, students (subjects), when applying principles of systems thinking (rules) for using ICT (tools), are more likely to reach the intended outcomes (object) on both the individual level (endorsing thinking and self-management) and the social level (socialization and communication). Thereby, students who successfully engaged in learning were successful when they managed design tasks and, when engaged, multiagent feedback was provided. This further supports the findings of Kwong and Churchill [65] and Gunness et al. [7] when investigating technology-enhanced learning environments. The core abilities in ICT-enhanced learning which should be developed in the learner are establishing and maintaining connections in the network, where different fields, ideas, and concepts are perceived in the context of a system which generates up-to-date knowledge and decision making [69], especially important skills in design activities. When members of a cohort create an effective network or learning community, they must consider the diversity, autonomy, openness, and connectivity of the participants engaged in distributive knowledge creation [70].

In the context of the activity theory, this result indicates the possibility of approaching the object of the design as a system and the designer as a systems thinker, while the whole design community can be seen as an innovation system, which supports the findings of Mononen [53].

Next, two additional sequential effects were found: (1) the GPA increased systems thinking ST1 and, further, increased ICT-SC3, resulting in higher levels of design project achievements, and (2) GPA increased ST1 and, further, evokes cognitive engagement in learning, resulting in higher design project grades. The latter is consistent with the findings of Engström et al. [50]. The finding that ST1 affects the use of ICT for problem solving and, in turn, influences students’ higher-order thinking achievements is consistent with the findings of Green et al. [23], while the finding that systems thinking in combination with design thinking not only improves the student empathy but may enhance the engagement in learning, supports the findings of Davis et al. [43] and Grau and Rockett [54]. Moreover, this finding might be helpful for articulating designers’ cognition as a dynamic system and for understanding the interactions between designer, design, and consumer, as argued by Mononen [53]. Thus, conceptual change can be seen as a process of conceptual prioritizing, revision, and selection, and not only replacement, which is consistent with the findings of Bryce and Blown [18].

4.5. Limitations of the Study and Future Work

Despite providing new insights into the learning dynamics of first-year architecture students for improving achievements in design courses in different-sized cohorts, there are also some limitations of the present study which might be addressed in future studies.

Firstly, our conclusions are limited to the learning environment’s factors in one academic year. Because the three universities were not randomly selected, they may not be representative of the overall population of architecture students, and the study should be repeated to test the external validity.

Secondly, since the study focused on the factors of students, future research should also include educator-based factors, especially factors such as the leadership type and capacity of teachers/educator, which may affect the learning dynamics.

Thirdly, for a better understanding of the learning dynamics with deeper insights, aspects of other students from undergraduate programs should also be included, and respective factors should be mapped in a cross-sectional study where the present conceptual model should, therefore, be tested.

Fourthly, methodologically, the number of observations per factor should be larger to provide more power in the statistical test for stronger arguments and in order for structural equation modeling to be used for multigroup analysis. For future studies, we propose a mixed method for data gathering and self-reported questionnaires complemented with different tests or observations in the learning environment. The survey should be administered in two parts with a time delay, since four questionnaires administered in a one-shot study, as in our case, reduces the completion rates and representativeness of the data, especially when the survey is conducted through mobile devices.

5. Conclusions and Implications

The present study aims for (1) validation of the set of measures to map learning dynamics in different-sized cohorts, and (2) for the development of a conceptual model which best explains system-based learning dynamics in different-sized cohorts.

Our scale development efforts were informed by different studies, since no relevant research was found which investigated the system-based learning dynamics in different sizes of cohorts in architecture education. When the relevant literature was identified, with the support of bibliometrics, we used different frameworks for each of the observed variables to situate the purpose and scope of our instruments. All scales were validated for content and construct validity and, despite the different number and structure of new factors compared to the previous research, we provided moderate to strong evidence for the validity. All our instruments provide new perspectives on the knowledge creation and transfer framework and extend its application for scalability in different contexts or disciplines of study.

To achieve the second aim of the study, we highlighted the importance of systems thinking, ICT self-concept, and engagement for enhancing design-based learning outcomes, controlled for prior knowledge. Despite the small number of observations per group (<100), preventing the use of structural equation modeling for multigroup comparison, the two-stage method used in this study successfully identified and deployed factors for effective learning dynamics. Thus, we contributed new insights into the systems thinking theory, self-concept theory, and engagement theory, in the wider context of the activity theory and connectivism theory, as frameworks for technology-enhanced, design-based, and meaningful learning. Analyzing the data collected with the four instruments, we found changes in the different cohorts represented by the corresponding university, with some measures tied to the cohort size (ST1, ST2, ICT3, cognitive and behavioral engagement) and others not (design thinking, ST3, ICT1, ICT2, emotional, social, and aesthetic engagement). Our results have two important theoretical implications. First, they describe how different dimensions of systems thinking, ICT self-concept, engagement, and design thinking need to be fostered in first-year students to achieve the optimal adaptation to design-based learning. Second, they highlight the need for a broader range of measurement tools for design-based learning outcomes, as knowledge emerges from connections formed during network activity.

In addition to the theoretical implications already mentioned, this study also has practical implications. First, educators, curriculum designers, and educational management in higher education could benefit from understanding the relationships we tested using a sequential meditation analysis. As higher education stakeholders shape the digital ecosystem of education, they can select promising approaches to effectively digitize curricula, pedagogy, and assessment. In addition, education management can select promising approaches for creating an effective guidance and regulatory framework, especially when the education system is dealing with cohorts of different sizes. As a key factor which might improve the design-based learning in smaller cohorts, the understanding of feedback behaviors and causal sequences was found in the system, complemented with ICT use for problem solving and other high-order thinking skills’ development, while cognitive engagement induced by systems thinking significantly enhanced first-year students’ achievements in the design studio. In this way, higher education stakeholders can communicate the intended use of ICT to students before they use it.

Second, the finding of the positive indirect effects of the structure of systems thinking (identifying and characterizing feedback loops) on the performance in the design project suggests that providing students with more opportunities to use ICT to promote higher-level thinking in formal and informal contexts helps develop their self-determination, judgment, and cognitive engagement in ICT use, creating the conditions under which better learning outcomes are more likely.

Third, educational management should ensure that adequate resources are available to support faculty in their design studio responsibilities, especially when they need to explore different sustainable options in the course, in order to foster students’ systems and design thinking skills. In addition, the students should be provided with sufficient resources to support their initial attempts at sustainable design-based learning projects. This is important to develop, explore, and utilize their systems thinking, digital competences, design thinking, knowledge building, collaboration and communication, creativity, and leadership skills. We are hopeful that this study will inspire further innovative research and development in the areas of student systems thinking, engagement in learning, ICT competences development, and design thinking, particularly in architecture education.