Next Article in Journal
Influence of Illumination Parameters on Night Sky Observation in Rural Areas
Next Article in Special Issue
The Impact of Accessing Education via Smartphone Technology on Education Disparity—A Sustainable Education Perspective
Previous Article in Journal
Characteristics of Droplet Behaviors during Spray Breakup Process
Previous Article in Special Issue
E-Learning Readiness Assessment Using Machine Learning Methods
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Exploring Factors Influencing the Acceptance of E-Learning and Students’ Cooperation Skills in Higher Education

Educational Technology Department, College of Education, King Saud University, Riyadh 11652, Saudi Arabia
*
Author to whom correspondence should be addressed.
Sustainability 2023, 15(12), 9363; https://doi.org/10.3390/su15129363
Submission received: 31 March 2023 / Revised: 25 May 2023 / Accepted: 6 June 2023 / Published: 9 June 2023

Abstract

:
This study investigates the relationship between the behavioral intention to use e-learning and academic achievement, using self-determination theory (SDT) and critical thinking as one of the 21st century skills. This study also examines how the behavioral intention to use e-learning, which mediates the effects of 21st century skills such as logical thinking and perceived utility, promotes academic performance. The approach is based on structural equation modeling using partial least squares (PLS-SEM). A survey question on the idea of self-determination and critical thinking in the 21st century was given to 346 students at King Saud University as the main method of collecting data. The obtained outcomes of students’ perceived usefulness, critical thinking in the 21st-century skills, and behavioral intention to utilize e-learning indicate a positive effect on their academic achievement in higher education institutes, and all of the surveyed students were completely satisfied with the effect of critical thinking in the 21st-century skills on behavioral intention to use e-learning. This study indicates that self-determination theory and critical thinking in the 21st-century skills, as well as communication skills over e-learning systems, enhance the students learning activities and enable the sharing of knowledge, information, and discussions, and, hence, we recommend that students utilize e-learning systems at educational institutions throughout Saudi Arabia for the purpose of learning and that they should be encouraged to do so through lecturers at higher level education institutions.

1. Introduction

Information and communication technologies (ICTs) have had a significant impact on education, which has led to the development of e-learning systems [1]. E-learning is a critical tool to meet the demand for favorably qualified specialists in today’s technologically evolved society [2]. E-learning, which has been properly developed and validated, is the transfer of knowledge and skills via online platforms, such as intranets, extranets, and e-learning systems. Since the COVID-19 pandemic, and because there has been a compelling need to switch from in-person to online education, e-learning systems have been widely adopted by educational providers, including, notably, higher education institutions [3]. Nowadays, e-learning is the main format to deliver education in almost all educational institutions across the globe [4], and due to its importance at this time, the number of studies exploring different angles for e-learning systems to deliver effective teaching and learning under the COVID-19 situation has been rapidly growing [5]. For instance, Su and Guo [5] investigated students’ online learning experiences and found that the quality of information, course design, peer interaction, and self-discipline have beneficial effects on students’ learning outcomes and satisfaction. There has been some further study on the challenges of e-learning in the COVID-19 setting [6], the significance of motivation [7], students’ welfare [5], teaching approaches [8], students’ engagement and learning satisfaction [9], and learning performance [10]. One of the crucial parts of e-learning that has not been well researched is the factors that affect students’ adoption and use of e-learning technology [11]. Students’ use and acceptance of e-learning are crucial to its success. If students were to refuse to use e-learning tools or use those systems against their will, the effectiveness of e-learning would be questionable [12]. One of the keys to the success of e-learning is how creatively it is used. The phrase “innovative utilization of e-learning” refers to the manner in which students make use of particular e-learning components or use e-learning in fresh and different ways [13].
The creative application of e-learning has a considerable impact on its effectiveness [14]. Investigating the elements that affect students’ continuous use of digital learning is essential as well. Students are reported to use e-learning indefinitely after their initial experience [15]. In the research on the continuous use of data systems, the frequent and ongoing use of a typical example by students is referred to as “continuous use” [16]. E-learning must be used frequently in order to be productive and have a positive impact on students [12]. Many studies have demonstrated that a user’s continued use of an e-learning program is a measure of that project’s success, and this topic has received more research in recent years than it did in the past [17,18].
Academic achievement is insufficient to thrive in the 21st century [19]. Educators, employers, and policymakers [19] continually emphasize the extra talents needed, often known as 21st century skills. Although these abilities are thought to be necessary for modern existence, they have actually been important throughout human history [20]. However, in the modern world, these abilities must be upgraded and learned while taking into account the demands of a globalizing society. Humanity has always needed the ability to communicate, just as it has needed the ability to think critically and solve problems. Professional success depends on having strong communication skills [21]. They serve as the cornerstone for covalent binding, as well as other 21st century skills, because these skills are built on social interaction [22]. One of the essential traits for success in the modern economic world is cooperation. Additionally, cooperation is required in order to reveal other skills [23]. In today’s digital world, digital literacy is also important [24]. It consists of both technological and cognitive abilities [25]. It is a basic skill for resolving technological, cognitive, social, and communicational issues, especially in the digital environment [26].
In the literature, studies have been done to show the connections between these essential 21st century skills. The goal of this research was to determine the levels of 21st century skills required for college students and teachers [27], school administrators [28], and K–12 students [29], as well as the relationships between these skills. Additionally, the research on the subject has examined how to build a model that predicts academic success based on a student’s level of 21st century skills [30], and how to take into account aspects such as academic progress and attitudes towards a course to predict 21st century skills [29].
Education and political leaders worldwide recognize critical thinking as an essential skill for 21st century learning which is crucial for equipping youth for the modern world. While certain research findings suggest effective methods to enhance students’ critical thinking skills and explore the impact of critical thinking on their academic success in online learning, other studies indicate that mere online instruction does not guarantee significant improvement in students’ critical thinking. For instance, AlMahdawi et al. [31] advocate for the implementation of online collaborative teamwork as a highly effective approach to enhance learners’ critical thinking, interactivity, and creativity. In contrast, other studies reveal that students often display limited levels of critical thinking and participation during online discussions, resulting in uncollaborative and superficial conversations [31]. However, Abdul Razzak [32] reports that utilizing a strategic online instruction approach, such as incorporating real-life simulations, can stimulate profound thinking, foster engagement, and facilitate the generation of innovative ideas. Analyzing the aforementioned findings in the literature demonstrates that, while some studies identify the factors that influence students’ critical thinking skills, others indicate that critical thinking itself affects student achievement in the realm of online learning. Reference [33] discusses the promotion of critical thinking skills through e-learning, underscoring its importance in engaging with online content and problem-solving. Collaboration and communication skills are also essential in the digital realm. Minan et al. [34] emphasize the value of collaborative learning in the online classroom, emphasizing its contribution to meaningful engagement and knowledge construction. Information management skills are crucial for effectively navigating and evaluating the vast amount of online information. In light of the aforementioned research, this essay investigated students’ academic success and e-learning in Saudi higher education [35]. The results were examined using the self-determination theory, which takes into account external factors. However, no studies on model development for revealing the levels of the 21st-century core skills to predict each other were found in the literature. The primary goal of this research is to investigate the connections between the behavioral intention to use e-learning and academic achievement, using self-determination theory (SDT) and critical thinking as one of the 21st century skills. The development of models that demonstrate the predictive levels of using e-learning for 21st century skills is what lends credibility to this study. In addition, this study will make a significant contribution to the literature, as it highlights the importance of cooperation and critical thinking in developing 21st century skills. Thus, this study is expected to fill a gap in the literature.
In view of the above, this paper investigated students’ academic achievement and the use of e-learning for learning in Saudi higher education. The findings were evaluated using the self-determination theory with external variables. The “Structural Equation Model” (SEM) was developed to investigate the interactions between the factors that affect academic attainment. The research model focuses on the effects of both collaborative design requirements on students’ academic performance, specifically how perceived independence, self-efficacy, perception relatedness, technology literacy, effective communication, cooperation, usability, critical rationale in the 21st century, and behavioral intention to use are implemented in e-learning. We employed IBM SPSS and Smart-PLS 3.3.3 as the primary statistical tools for this study, covering measure construction, convergent measurement validity, discriminant measurement validity, and structural model exploration. This study is organized as follows: Section 2 deals with the creation of a theoretical model and a hypothesis. Section 3 deals with research techniques. Section 4 presents the data that have been studied. Section 5 discusses the results. Section 6 deals with the conclusions (conclusion and future study direction).

2. Theoretical Model and Hypotheses Development

SDT has recently become quite popular in the scientific literature. Numerous studies have employed the lens of SDT to address research issues in a variety of contexts, such as mobile-based evaluation [36], computer game enjoyment [37], trying to predict usage behavior and acquiescence in cloud-based virtual classrooms [38], marker room continuation [39], quality of work [40], using social media platforms to advertise electronic word-of-mouth [41], and education and online learning. Chen and Jang [42] and Sørebø et al. [43] employed the SDT framework to determine how well Saudi Arabian students adopted e-learning, and found that autonomous motivation was an important predictor of the three TPB categories and that perceived behavioral control (PBC) had a substantial impact on student behavior (attitude towards behavior, subjective norm, and PBC). According to SDT, relatedness, autonomy, and competence are three basic psychological demands that all people have and that help to explain “the sensation of choice”.

2.1. Perceived Autonomy

Students who need autonomy in their online learning desire to control their own decisions and learning strategies [44]. Numerous studies have demonstrated the link between autonomy and successful outcomes. In the context of an organizational setting, Deci et al. [45] discovered that autonomy support has a considerable impact on satisfaction and corporate trust. According to Gagné et al. [46], it has been proven that managerial assistance helps to promote organizational acceptance of change. In fact, a number of studies have suggested that autonomy encourages both intrinsic and extrinsic drive, which in turn leads to beneficial effects. According to Williams and Deci [47], when students have more autonomy and support, they demonstrate a greater assimilation of the course’s values. The observed support from supervisors has an impact on a system’s perceived utility and simplicity of use in the IS domain. Inside the context of e-learning, the skycap predicts the perceived usefulness and perceived playfulness [48]. The perceived utility, in relation to employing online learning, is classified as extrinsic motivation by a prior study [42]. The enjoyment felt as a pleasant emotion in connection with the fun of online courses is defined as intrinsic motivation [48]. According to these theories, the observed value (extrinsic motivation) and reported enjoyment (intrinsic motivation) of online SRL should be positively connected, in the current study, with perceptions of independence in online learning. Therefore, this study posited the following hypotheses:
Hypothesis 1 (H1).
PA significantly influences PU.
Hypothesis 2 (H2).
PA significantly influences CT.

2.2. Perceived Competence

Students’ willingness to use an online tool effectively to raise their academic achievement indicates their need for competency during online learning. Zhao et al. [49] made the assumption that motivation levels are influenced by how well a competency need is met. Students who feel capable of participating in online SRL will feel empowered to control their own participation. A connection between this perception of competence and the intrinsic and extrinsic drives is anticipated. The concept of self-efficacy [50], which is concerned with how well people perceive their capacity to organize and carry out tasks in order to produce desired results [51], is linked to the concept of competence. According to Luo et al. [48], students’ perceptions of the value of e-learning are correlated with their level of social competence. Computer proficiency has been demonstrated in the IS literature to have an impact both on being extrinsically motivated (i.e., perceived usefulness) and intrinsically motivated (i.e., perceived enjoyment) [43]. Sørebø et al. [43] confirmed that the perceived usefulness and enjoyment of using e-learning technologies are related to the amount of self-efficacy involved in doing so. Therefore, this study posited the following hypotheses:
Hypothesis 3 (H3).
PC significantly influences PU.
Hypothesis 4 (H4).
PC significantly influences CT.

2.3. Perceived Relatedness

Students’ online SRL exhibits a demand for relatedness that shows their desire to feel as though their actions are connected to and supported by those near them [47]. People are more inclined to support their organization’s aim when they sense a connection to the other members of the group, according to SDT. If they are valued by pertinent others (such as peers, friends, family, superiors, or organizations), people continue to engage in certain uninteresting or unpleasant activities [52]. As a result, meeting their demands exemplifies a particular social influence that is comparable to the idea of facilitating conditions in the IS domain. In a previous stud, subjective standards, such as perceived fun or enjoyment, were shown to have an impact on perceived usefulness and intrinsic motivation [53]. According to Luo et al. [48], there is a positive correlation between a learning activity’s relatedness, usefulness, and usability. We contend that students’ perceptions of their relationships with other significant individuals should affect their perceptions of usefulness and enjoyment. Therefore, this study posited the following hypotheses:
Hypothesis 5 (H5).
PR significantly influences PU.
Hypothesis 6 (H6).
PR significantly influences CT.

2.4. Digital Literacy

Online learning can also have a significant impact on the outcomes of online distance learning. Digital literacy refers to a person’s capacity to use ICT and the Web to accomplish goals [54]. Digital literacy is defined as “the awareness, attitude, and capacity of individuals to adequately use digital tools and services to recognize, access, manage, incorporate, evaluate, and synthesize online technologies, build new knowledge, create digital gestures, and interact with others in the context of specific life situations in order to enable productive social action; and also to reflect on this process”, as cited in [55]. Digital literacy is one of the most crucial 21st century abilities [55]. There is no connection between digital literacy and critical thinking or problem-solving skills. Nonetheless, when asked to handle a problem, students are observed to follow the phases of evaluating the necessity of information and finding, organizing, analyzing, synthesizing, and presenting sources [56]. In addition, they look for these sources online [57]. Students must be digitally literate in order to evaluate the worth and reliability of the material they obtain online [58]. As a result, digital literacy significantly aids pupils’ ability to think critically and solve problems. Therefore, this study posited the following hypotheses:
Hypothesis 7 (H7):
DL significantly influences PU.
Hypothesis 8 (H8):
DL significantly influences CT.

2.5. Effective Communication

The ability of pupils to arrange their ideas, information, and conclusions, and to communicate them effectively—whether orally, in writing, or in the form of other media—is defined as communication. Humans are social creatures who engage in ongoing social interaction. As a result, effective communication is among the most important factors in social success [54]. Establishing a productive workplace requires effective communication. All educators should support their students in developing critical communication skills in this era of globalization by giving them time and chances to practice interpersonal relationships [59]. This is because people from different cultures are continuously mixing and engaging. Aizenkot Ben David [60] looked into how the application of blended learning in courses affected students’ 21st century abilities and GPA. Effective communication is a talent that is necessary in the 21st century. It is especially crucial in fields where teamwork is highly valued. This is due to the fact that, in order to handle a complex problem, a person must have strong communication and cooperation skills [61]. Practical communication abilities have been demonstrated to be extremely important for collaborative work [62]. Effective functional communicators are better able to complete their tasks collaboratively. As a result, effective communication skills can affect teamwork [63]. Therefore, this study posited the following hypotheses:
Hypothesis 9 (H9).
EC significantly influences PU.
Hypothesis 10 (H10).
EC significantly influences CT.

2.6. Cooperativity Skills

According to earlier research [64], using online networks in conjunction with cooperative learning has a greater effect on outcomes. The immediate nature of the tools, in the findings of [64], enhances learning outcomes as people study or interact while using them. For instance, according to Alavi [65], students who have used group decision-support systems have shown more passion, improved skill development, and received higher overall evaluations [66]. The literature contains studies on the relationship between cooperative and critical thinking skills. Most of these studies concluded that cooperation or cooperative activities either improve or heighten the capacity for critical thought. Chen and Swan [67], who offer a theoretical foundation related to critical thinking, present a four-step structure, with the final stage being “critical appraisal of others’ opinions”. The best places to investigate people’s opinions are also cooperative work environments. When cooperative work is developed, students are better able to employ their critical faculties in circumstances which require debate [66]. Therefore, this study posited the following hypotheses:
Hypothesis 11 (H11).
CS significantly influences PU.
Hypothesis 12 (H12).
CS significantly influences CT.

2.7. Perceived Usefulness

Perceived usefulness (PU) is defined, in the context of e-learning, as the degree to which users believe that e-learning may assist them in achieving their educational and professional goals. According to earlier studies, PU has the biggest effect on attitude [68]. Moreover, behavioral intentions to adopt e-learning were considerably impacted by PU. Several empirical studies have revealed that PU is the primary consideration when determining whether to employ a specific technology or not [69]. Students will not use an e-learning system unless they think doing so will improve their academic achievement. Perceived usefulness (PU) and behavioral control to utilize the e-learning system have a substantial positive link, according to prior research on students’ behavioral intention to use an e-learning system (BI) [69]. According to one notion, a person would become more upbeat as they began to value the e-learning system more [70]. According to the literature, there is a solid foundation of evidence for the relationship between PU and 21st century critical thinking skills [71]. Therefore, this study posited the following hypotheses:
Hypothesis 13 (H13).
PU significantly influences CT.
Hypothesis 14 (H14).
PU significantly influences BI.

2.8. Critical Thinking as a 21st Century Skill

Inferring, interpretating, analyzing, and self-regulatory judgment are the outcomes of critical thinking. It also clarifies the factual, conceptual, scientific, or contextual factors supporting a conclusion. The ability to think critically is crucial, liberating in the classroom, and a useful asset in one’s personal and civic life [72]. The technique called critical thinking seeks to help people choose wisely, when it comes to their beliefs and actions. It also requires higher-order cognitive skills [72]. Technology encourages critical thinking and simplifies the understanding of instructional materials [73]. The relationship between critical thinking and education is obvious, since one cannot think well without being able to learn well. Both academic and professional success depend on critical thinking. It is a psychological process used to create or resolve issues, come to conclusions, comprehend details, and discover solutions to issues [74]. Peer conversation enhances comprehension even when no one in a discussion forum has the right answer and the group is simply developing critical thinking skills [74]. One study found that students generally believed that they had high critical thinking and problem-solving skills, and were satisfied with their ability to think critically [75]. Additionally, critical thinking abilities are necessary for students to pinpoint the root of a problem and the best fix to enhance their performance [76]. Thus, student performance achievement and behavioral intention are related to critical thinking learning. Therefore, this study posited the following hypothesis:
Hypothesis 15 (H15).
CT significantly influences BI.

2.9. Behavioral Intention to Use

Learners’ behavioral intention (BI) with regard to e-learning is their intention to use e-learning systems consistently from on the time of the decision [77]. Numerous studies have demonstrated that behavioral intention directly and dramatically affects how well an e-learning system performs academically [78]. The authors of [79] posit that, once students make the decision to use an online education system, they will follow through and use it. Also, Davis et al., [80] assert that one of the key determinants of how people really use information technology (IT) or new technologies is business intelligence (BI). The authors of Bae [81] make the case that BI can be employed to assess a person’s propensity for engaging in a particular behavior. It has been established by researchers [71] that behavioral intention significantly improves real IT usage. It has also been proven by numerous e-learning studies [82] that BI has a favorable correlation with practical use. This study suggests that BI is positively connected to AU, which is consistent with the earlier literature. Therefore, this study posited the following hypothesis:
Hypothesis 16 (H16).
BI significantly influences AA.

2.10. Academic Achievement

The knowledge, skills, and behaviors that students acquire in educational settings are what define student achievement, and their learning outcomes reflect this [82]. On the other hand, it may be said that there are a lot of elements that will affect student accomplishment in online learning settings created with technology support. For instance, Cho and Shen [83] claimed that students’ ability to self-regulate in online classrooms had an impact on their academic performance. Additionally, research indicates that effective e-learning environments improve student performance [84]. Furthermore, according to Chang and Tsai [85], while a lack of communication and technical difficulties are challenges for students, course design and time management are crucial elements of successful online courses for those who learn this way.

3. Research Methodology

3.1. The Study’s Design

The major goal of the study was to create a clear and understandable theoretical framework for evaluating the acceptability of e-learning and its contributing components. The suggested model and survey were built, validated, and tested using a multistage testing technique. The participants in this study first examined 47 previously employed questions to gauge Saudi institutions’ adoption of 21st century skills-based education through e-learning, identifying 10 elements (cooperativity skills, academic achievement, behavioral intention to use, critical thinking in the 21st century skills, digital literacy, effective communication, perceived autonomy, perceived competence, perceived relatedness, and perceived usefulness) to evaluate e-learning. Second, information was manually gathered from 370 students who were randomly selected from Saudi Arabia’s King Saud University. The structural equation partial least squares model and IBM SPSS (version 26, Originlab Corporation, Northampton, MA, USA) were used to assess the collected data (PLS-SEM 3.3.3, IBM, Armonk, NY, USA).

3.2. Data Collection

This study’s target audience consisted of undergraduate students at King Saud University in Saudi Arabia. A questionnaire survey was used to collect the data. There were 370 people who participated in the poll. The sample size employed in this study is, therefore, sufficient to reflect Saudi Arabian students’ impressions about the deployment of the e-learning system. In total, 24 polls were disqualified due to missing data. A total of 370 surveys were given out, and 346 of them, or 93.5%, were returned by the respondents. In this study, a quantitative technique and a questionnaire survey were used. In order to collect the data, self-administrated questionnaires were distributed to King Saud University students between December 2022 and January 2023. The primary analysis of this study included 346 questionnaires with a 93.5% response rate, and this sample size was found to be acceptable based on Hair et al.’s (2010) recommendation that the minimum sample size for quantitative research is 335. Table 1 provides further details on the participants. The data were thus examined using SPSS from a total of 346 surveys. In Table 1, the students’ genders, ages, and specialties are displayed. Based on the survey’s demographics, 129 of the respondents were female, while 217 were male. The age range of the 51 respondents who were surveyed was between 18 and 20. A total of 67 respondents were aged from 21 to 24. A total of 129 of those polled were between the ages of 25 and 29. A total of 65 of those polled were between the ages of 30 and 34. Furthermore, 34 respondents were 35 years old or older. Finally, regarding specialization, 162 were from humanities colleges, 108 were from scientific colleges, and 76 were from medical colleges.

3.3. Instrument Development

Data were gathered using a five-point Likert scale, and the preexisting concepts of this research were used with the following adjustments to meet the context of this study. Perceived competence, relatedness, independence, and autonomy were adapted from the sample questionnaire from [86]. Effective communication was taken into consideration [87], Cooperativity skills and Critical thinking in the 21st century were adapted from the sample survey in [87], User satisfaction and Behavioral intent to use were adapted from [88], and Digital literacy was adjusted from the sample questionnaire in [89]. Finally, the example questionnaire from [90] was modified to reflect the academic achievement of students (see Appendix A). All of the variables and their sources are presented in Table 2. To analyze the data, the researchers used the Partial Least Square Structural Equation Modeling (PLSSEM) procedures. In this study, they utilized the Smart-PLS 3.3.3 software to evaluate both the measurement and structural models. The measurement model was used to compute the validity and reliability of the data. To ensure data validity, they reported the AVE, with a required value of 0.5 for convergent validity, and applied the Fornell–Larcker criterion, cross-loading, and HTMT to assess the discriminant validity. To measure the data reliability, an internal consistency and reliability process was conducted using two approaches, where both values should be >0.7. For the assessment model, we determined the significance of the relationship through the path coefficient, t-value, and p-value.

4. Result and Data Analysis

4.1. Measurement Model

The evaluation processes required to validate the precision and reliability of the measurements are referred to as the measurement model. Three different approaches were taken into account. Validity comes in three types: discriminant, convergent, and Internal Consistency Reliability and Indicator Loadings. Three actions are suggested by Hair et al. [91].

4.2. Internal Consistency Reliability and Indicator Loadings

In this study, the indicator loadings were calculated using PLS-SEM data. Table 2 displays the specifics of the loadings. The vast majority of the products had loading levels above 0.700 [91]. In the PLS-SEM investigation’s second phase, 50 indicators were evaluated. The examination results for the statistical coherence among the variables are referred to as internal consistency reliability. When evaluating internal reliability, one should use Cronbach’s alpha (CA) and composite reliability (CR). The criteria detailed in [91] were used to determine the CA and CR values in this study: >0.700 for CA and >0.700 for CR. The features of each measured value are displayed in Table 2. All constructions exhibit high levels of internal consistency for both CA and CR values, with CA reliability ranging from 0.773 to 0.882 and CR confidence from 0.843 to 0.914.

4.3. Convergent Validity

A statistical technique related to constructing validity is referred to as validity testing. The idea of convergent validity states that assessments using the same or similar conceptions should be closely related. Convergent validity must be expressed in terms of the AVE scores. PLS-SEM was used to calculate the SmartPLS scores. At minimum, 50% of the variability should be explained by the AVE, which should have a score greater than or equal to 0.500; over half of the discrepancy is explained by the obtained values for all projects, which are greater than 500 [92], as shown in Table 2.

4.4. Discriminant Validity

The degree to which a construct differs from other constructs, according to [91], is referred to as discriminant validity. The AVE score of a notion must be less than the average variance of all of the model constructs. The findings of this study demonstrate that the AVE score for each notion is lower than its shared variance (Table 3). The establishment of construct validity resulted from examining these Fornell–Larcker criteria. Cross-loadings are another method that can be used to assess a discriminant’s validity. Table 3 shows that the outer loading value of each indicator for each build was higher than the cross-loading value of the additional constructs. The cross-loading of fair reflection results in discriminant validity
Validity will also be clear when the heterotrait-monotrait (HTMT) is higher than 0.900. A HTMT value over 0.900 indicates that the construct is not legitimate. Table 4 shows that none of the HTMT values exceeded 0.900. The findings reveal that the numbers were significantly off from 1.

4.5. Structural Model Assessment

The structural equation evaluation is divided into various phases [91]. The process of computerizing collinearity started with the reporting of variance inflation factor (VIF) statistics. The link was researched in the following phase. Step three involved calculating the R2 significance level. In step four, it was revealed what the effect size of F2 meant for the construct’s applicability. An inquiry into the rationale behind the selected endogenous constructs was the aim of this investigation. When presenting the Q2 values, the R2 and F2 impact sizes were pegged in the PLS-SEM model to generate the data.

4.6. Coefficient of Determination (R2)

The degree of relevance (R2), which is the result of the linear regression and is defined as the percentage of variance in latent variables that are endogenous, and that the independent factor would be able to predict, is the output of the analysis. It assesses how well a model can forecast the future. It is calculated using the correlation coefficients’ cube. On the R2 scale, which runs from 0 to 1, a greater number represents a higher degree of R2. Using this scale, 0.25 is regarded as a low value, 0.50 as a moderate value, and 0.75 as a big value [91]. Table 5 shows the R2 result based on the investigation’s findings. Academic achievement (0.169), behavioral intention to use (0.240), 21st century critical thinking skills (0.350), and perceived value (0.511) are taken into account for the students. Table 5 displays the outcomes.

4.7. Effect Size (F2)

The correlation value, often known as the F2 statistic, is a statistical indicator of the degree to which a prediction construct is related to a certain variable. As an alternative, F2 is used to assess how external constructs affect endogenous constructs. The goal of F2 is to examine what happens to R2 when an external variable is taken out of the model. A small impact is one with an F2 value of 0.02, a medium impact is one with an F2 value of 0.15, and a significant effect is one with an F2 value of 0.35, according to [91]. Ten confirmed effect sizes were observed in this study’s data. According to Table 6, the association between behavioral intention to use and academic achievement was the lowest, while the correlation between behavioral intention to use and perceived usefulness was the highest, with an F2 value of 0.523.

4.8. Collinearity Issue

It is important to look at the time series between the predictor sets. The VIF value can be used to determine the serial correlation. Time series will be a problem if the VIF value is reported to be >3000 [91]. Cooperation skills predict both rational thought in the 21st century skills and perceived usefulness (VIF = 2.004 and 1.867, respectively); academic achievement predicts behavioral intention to use (VIF = 1.000); 21st century critical thinking skills predict behavioral intention to use (VIF = 1.234); online learning predicts 21st century critical thinking skills (VIF = 1.581) and perceived usefulness (PU) (VIF = 1.559); perceived autonomy (VIF = 1.633) and PU (VIF = 1.592) predict critical thinking in the 21st century skills; perceived competence (VIF = 1.379), PU (VIF = 1.345), perceived relatedness (VIF = 2.081), behavioral intention to use (PU) (VIF = 1.234), and usefulness (VIF = 2.045) predict rational reflection in the 21st century skills. All VIF readings are below three (Table 7). Therefore, integration is not a concern in this investigation [91].

4.9. Hypothesis Testing

For the purpose of calculating the route factor between exogenous and endogenous structures, the model was used to construct the sample using 5000 subsamples. Figure 1 depicts the assumption; Figure 2 shows the results of the path coefficient; and Figure 3 shows the results of the path coefficient (t-values). In this work, SEM analysis was used to evaluate the research model. Sixteen hypotheses (H1–H16) were tested in order to look into the connection between the self-determination hypothesis and environmental factors on usefulness, 21st century critical thinking skills, and behavioral intentions to use e-learning, which can affect the implementation of e-learning systems for educational purposes. The results were verified. All 16 of the assumptions were incorporated into the study model, as shown in Table 8.
Perceived usefulness and critical reasoning in 21st century skills are positively impacted by perceived autonomy (p = 0.141, t = 2.819; p = 0.155, t = 2.978). Therefore, H1 and H2 have statistical significance. When using an e-learning system for learning, perceived competence significantly improved perceived usefulness (p = 0.129, t = 2.997) and critical reasoning in the 21st century (p = 0.195, t = 3.688).
Consequently, H3 and H4 have statistical significance. Furthermore, perceived relatedness (p = 0.143, t = 2.036) and perceived utility (p = 0.151, t = 2.815) significantly improved analytical reasoning in the 21st century. H5 and H6 are statistically significant as a result. Table 8 shows that the digital literacy component of the e-learning strategy significantly improved top reasons (p = 0.102, t = 2.068) and critical reasoning in the 21st century skills (p = 0.143, t = 2.650). H7 and H8 are statistically significant as a result. Communicating effectively on PU (p = 0.174, t = 3.552) and critical reasoning in 21st century skills (p = 0.17, t = 2.689) are significantly positively correlated, as shown in Table 6.
Therefore, the H9 and H10 theories are accepted. According to Table 6, there is a strong positive correlation between cooperative learning skills and perceived usefulness (p = 0.259, t = 4.825) and 21st century critical thinking skills (p = -0.174, t = 2.846). The results also indicate a favorable association between 21st century critical thinking skills (p = −0.174, t = 2.846) and cooperation skills and perceived value (p = 0.259, t = 4.825). Therefore, H11 and H12 were supported. Similar to this, the 13th and 14th hypotheses (H13 andH14) proposed that perceived utility affected both behavioral intentions to utilize online learning (p = 0.433, t = 8.564) and critical thinking in the 21st century (p = 0.135, t = 2.033). As a result, the hypotheses (H13 and H14) are confirmed.
Additionally, Table 6 demonstrates a strong correlation between critical thinking skills for the 21st century (p = 0.109, t = 2.019) and behaviour intention to use online learning. This indicates that the gender of the pupils had an impact on their critical thinking abilities in the 21st century skills. As a result, the hypothesis (H15) is accepted. The Hypothesis (H16), that there is a relationship between behavioral intention to use online learning and academic achievement, were also investigated (p = 0.412, t = 7.637). Therefore, the hypothesis (H16) was confirmed. Table 8 displays the outcomes.

5. Discussion and Implications

Understanding the mechanics underlying how these networks affect academic accomplishments becomes more and more important as educational institutions incorporate e-learning into their educational environments to improve the learning process. Despite the recent growth in studies on the effects of e-learning on student learning, there are still significant gaps that prevent us from fully comprehending this phenomenon and providing academics with new opportunities. The primary objective of the current study is to examine how the learners’ skills affect how well they use e-learning in their learning processes. The conceptual framework of this study was developed based on prior research, and it addressed the elements and characters of skills through the concepts of perceived utility and critical thinking in 21st century abilities. Perceived autonomy, perceived competence, reported relatedness, technology skills, communication skills, and cooperativity skills are the six components that make up this notion, which determines behavioral intent to use e-learning and academic achievement.
Figure 1 shows how this study used self-determination theory in conjunction with external variables to assess the factors that influence students’ perceptions of the value of e-learning in higher education in Saudi Arabia, as well as their behavioral intention to use it. King Saud University students provided the data that were used in this study’s research. The 16 research hypotheses presented in Table 8 are supported by the findings of the study’s examination of structural equation modeling.
In light of this, the techniques used in this study indicate self-determination theory with exogenous factors to be the most important influence on academic accomplishment in terms of behavioral intention to utilize e-learning settings as an educational approach and critical reasoning in 21st century skills. The most recent findings firmly support the hypotheses (H1 and H2) that perceived autonomy increases outcomes and critical reasoning in 21st century skills. This is in line with earlier studies [43,48] that discovered a connection between claimed abilities and critical reasoning, and their perceived usefulness in 21st century skills. The study’s findings support the two hypotheses (H3 and H4) and demonstrate that critical thinking and perceived usefulness in 21st century abilities are positively impacted by perceived competence. This is strong support for the perceived competence variable. Previous studies that support this [43,48] have confirmed these correlations. This study’s findings also show that perceived usefulness and critical reasoning in 21st century abilities in educational institutions have a significant impact on observed relatedness, corroborating the hypotheses (H5 and H6). As a result, the results of this study support earlier findings about the relationships among various factors [43,48].
This study’s findings also corroborate the hypotheses (H7 and H8) and offer convincing proof that digital literacy positively affects 21st century skills such as critical thinking and perceived usefulness. To put it another way, higher digital literacy leads to more use of the critical thinking and perceived value skills that are essential for the 21st century. This supports earlier studies that found a strong correlation between digital literacy and perceived usefulness and 21st century skills, such as critical thinking [93]. The study’s findings offer compelling support for the importance of good communication, confirming (H9 and H10) and indicating that it has a big impact on perceived usefulness and critical thinking in the 21st century skills in educational institutions. However, the fact that e-learning is widely used by students in Saudi colleges, where there are enough of them to have a sizable impact on their peers can be used to explain this conclusion. The findings of this study strongly confirm past conclusions about the interrelationships of different factors [94]. However, these results were at odds with those of other investigations [95].
Both of the two hypotheses (H11 and H12) for the second variable, collaboration abilities, have a significant beneficial influence on perceived cooperation and critical thinking in the 21st century skills. This is consistent with the findings of other researchers who discovered that involvement, whether direct or indirect, had a positive effect on perceived utility and critical thinking abilities in the 21st century skills [94]. Behavioral intention to use e-learning as a sustainable growth avenue for higher education was found to be positively correlated with perceived usefulness (H13 and H14), critical thinking in the 21st century, and behavioral intention to use E-learning, which improved educational success when measuring educational sustainability. Hence, this study’s findings are consistent with past findings about varying correlations [88]. This study’s findings largely support hypothesis (H15), which states that rational reflection in 21st century skills promotes behavioral intention to employ e-learning constructively. In other words, when a classroom is convenient and acceptable, behavioral intentions to use e-learning rise, which promotes the development of 21st century critical thinking skills. This supports earlier studies [87] that discovered a connection between behavioral intention to use e-learning and critical reasoning in 21st century skills.
Behavioral intention to use e-learning had a significant beneficial influence on academic success as well as intention to use e-learning systems in higher education, according to study theories (H16) associated with the following factor. This is in line with a prior study that discovered a favorable relationship between actual performance in an e-learning system and academic success, as well as a behavioral intention to use e-learning. Hence, the outcomes of this study confirm past research which correlates numerous factors [90].

5.1. Theoretical and Practical Implications

The six components of 21st century e-learning platforms include computer skills, communication skills, perceived connectedness, perceived competence, and cooperativity. Skills were seen as potential constructions that might influence how positively students view e-learning systems and how prepared they are to use analytical reasoning in the 21st century. In order to quantify and evaluate its findings, this study combined self-determination theory with external variables. The following are some of the important outcomes and implications of this study:
  • The self-determination theory has been proved to be an ideal model for understanding perceived autonomy, perceived competence, and perceived relatedness, in improving academic achievement. Academic achievement subsequently could increase the student’s adoption of e-learning for learning.
  • The critical thinking in the 21st-century skills has provided evidence that it is an appropriate factor that helps to understand students’ collaboration and communication skills in accepting and using e-learning systems as the medium of teaching and learning.
Next, a theoretical model for e-learning using self-determination theory with external factors, and other related technologies, is being developed. Self-determination theory is enhanced by the study’s inclusion of external variables. Future e-learning systems can be used to improve learning and teaching results by applying the technology adoption strategy. The study’s main practical implications and benefits are thus accomplished by responding to the research questions. First, the model of technology acceptance demonstrated its suitability for obtaining independent variables to improve students’ perceived usefulness, 21st century critical thinking skills, and behavioral intention to use e-learning, which improve their academic success in higher education. Second, the self-determination theory offers support for modeling independent variables, boosting perceived utility, 21st century critical thinking abilities, and behavioral intention to use online learning. As a result, students in higher education are using e-learning more frequently. Because it identifies individual effects of utilizing e-learning on perceived benefits, 21st century thinking abilities, and behavioral intention to use e-learning, this research contributes significant theoretical value to earlier studies in these disciplines.

5.2. Limitations and Future Work

This research has some limitations that could be addressed in future studies. The model tested in this study focuses on the self-determination theory and three specific skills related to e-learning systems in the 21st century, which may restrict the generalizability of the findings to other theories and skills which are not discussed in the paper. Additionally, measuring 21st century skills with the critical thinking method, as they were examined by the study, can be considered a limitation because there are no objective measurement tools for these skills. Due to the large number of skills included in this study, short data collection tools were preferred. However, these tools may not have fully captured the complex constructs being measured, such as creativity. Therefore, in future studies, researchers could test the model developed within this study using different data collection tools that measure the skills in greater detail and more objectively. Another limitation of this work is that the study may be constrained due to the small sample size which was employed.

6. Conclusions

The findings of our study confirm the value of e-learning systems for the instruction of and learning by college students. All cultures can benefit from the proposed study of self-determination theory (SDT) and environmental factors (skills). This study demonstrates how adaptable SDT is and how it can be used to investigate the usage of e-learning in the 21st century, as well as perceived value, cognitive abilities, and behavioral intent, particularly in the Saudi setting. As for the usefulness of e-learning portal services from the perspectives of male and female students, the results of the current study show that men and women use the Saudi Arabian e-learning portal differently. Additionally, e-learning’s perceived utility, the ability to think critically, and the behavioral intention to utilize it all play a mediating role amongst the several independent elements that affect academic accomplishment. Effective communication has a great impact on critical thinking. In other words, encouraging efficient communication and critical thinking can directly enhance academic performance. Additionally, it was discovered that perceived autonomy had a direct and substantial influence on perceived usefulness. For this reason, it is important to foster students’ feelings of autonomy in order to improve their academic performances. Therefore, future studies can be improved by using a larger sample size and engaging of respondents from other countries (can be divided into stages/groups, e.g., Southeast Asia, and so on). Other than that, we suggest that a “multidisciplinary approach” (quantity and quality) could be used to provide more detailed explanations of the results of the current research, especially regarding those unrelated relationships. The models used in this study, self-determination theory and critical thinking as one of the 21st-century skills, can also still be upgraded in order to obtain better results. For example, the incorporation of different expectations (for example, information quality, system quality, and interaction) might expand the utilization of self-determination theory and critical thinking as one of the 21st-century skills in an assortment of innovative settings.

Author Contributions

Conceptualization, U.A. and A.A.; methodology, U.A. and A.A.; software, U.A. and A.A.; validation, U.A. and A.A.; formal analysis, U.A. and A.A.; investigation, U.A. and A.A.; resources, U.A. and A.A.; data curation, U.A. and A.A.; writing—original draft preparation, U.A. and A.A.; writing review and editing, U.A. and A.A.; visualization, U.A. and A.A.; supervision, U.A. project administration, U.A. and A.A.; funding acquisition, U.A. and A.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by Researchers Supporting Project number (RSP2023R159), King Saud University, Riyadh, Saudi Arabia.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Perceived competence (PC)
1.During E-learning use, I get many chances to show my capability.
2.During E-learning use, I have confidence in my ability to do things well.
3.During E-learning use, I am capable at what I do.
4.During E-learning use, I can competently achieve my goals.
5.During E-learning use, I can successfully complete difficult tasks.
Perceived autonomy (PA)
6.I have more control while using E-learning.
7.During E-learning use, I have a sense of freedom to make my own choices.
8.During E-learning use, there are many opportunities for me to decide for myself what and how I learn in e-learning system.
9.During E-learning use, I do what really interests me.
10.During E-learning use, my choices express who I really am as a student.
Perceived relatedness (PR)
11.E-learning gives me more chances to interact with others.
12.I feel close to others while using E-learning.
13.During E-learning use, My friends at online learning support me.
14.During E-learning use, I consider the people I work with to be my friends.
15.I have more opportunity to be close to other though E-learning.
Digital literacy (DL)
16.During E-learning use, I know to solve my own technical problems.
17.During E-learning use, I can learn new technologies easily.
18.During E-learning use, I keep up with important new technologies for learning.
19.During E-learning use, I am confident with my search and evaluate skills in regard to obtaining information.
20.During E-learning use, I have good ICT skills.
Effective communication (EC)
21.During E-learning use, Learning communication skills has improved my ability to communicate with students.
22.During E-learning use, Learning communication skills is fun.
23.During E-learning use, Learning communication skills is too easy.
24.During E-learning use, Developing my communication skills is just as important as developing my knowledge of teaching.
Cooperativity skills(CS)
25.I felt that using e-learning for active collaborative learning with peers was effective.
26.During E-learning use, I was able to develop study skills through member’s collaboration.
27.Active collaborative learning experience in the E-learning environment is better than in a face-to-face learning.
28.I think that collaborative learning with using of e-learning increases my understanding of how to perform tasks.
Perceived usefulness (PU)
29.E-learning use, I can enhance my teaching quality.
30.I find E-learning Platforms useful in my study / research.
31.Using E-learning Platforms enables me to accomplish tasks more quickly.
32.Using E-learning Platforms increases my productivity.
33.Overall, using e-learning Platforms enhances my effectiveness in my studies.
Critical thinking in E-learning (CT)
34.During E-learning use, I make use of a systematic method while comparing the options at my hand and while reaching a decision.
35.During E-learning use, I think about other possible ways of understanding what I am learning.
36.During E-learning use, I evaluate different opinions to see which one makes more sense.
37.During E-learning use, I decide what kind of information can be trusted.
Behavioral intention to use e-learning (BI)
38.I intend to use E-learning in my studies when it becomes available.
39.I intend to use E-learning in my studies as often as needed.
40.I predict I would E-learning use in my studies in the future.
41.I agree that E-learning should be adopted in learning for knowledge sharing.
42.I will frequently return to the E-learning use that I participate in the future.
Academic achievement (AA)
43.E-learning Platforms have improved my comprehension of the concepts studied.
44.E-learning Platforms have led me to a better learning experience.
45.E-learning Platforms activities have allowed me to better understand my studies.
46.E-learning Platforms activities are helpful in my studies and makes it easy to learn.
47.E-learning Platforms activities improve my academic achievement.

References

  1. Silva, S.; Fernandes, J.; Peres, P.; Lima, V.; Silva, C. Teachers’ Perceptions of Remote Learning during the Pandemic: A Case Study. Educ. Sci. 2022, 12, 698. [Google Scholar] [CrossRef]
  2. Vershitskaya, E.R.; Mikhaylova, A.V.; Gilmanshina, S.I.; Dorozhkin, E.M.; Epaneshnikov, V.V. Present-day management of universities in Russia: Prospects and challenges of e-learning. Educ. Inf. Technol. 2020, 25, 611–621. [Google Scholar] [CrossRef]
  3. Alkhwaldi, A.F.; Abdulmuhsin, A.A. Crisis-centric distance learning model in Jordanian higher education sector: Factors influencing the continuous use of distance learning platforms during COVID-19 pandemic. J. Int. Educ. Bus. 2022, 15, 250–272. [Google Scholar] [CrossRef]
  4. Al-Rahmi, A.M.; Shamsuddin, A.; Wahab, E.; Al-Rahmi, W.M.; Alyoussef, I.Y.; Crawford, J. Social media use in higher education: Building a structural equation model for student satisfaction and performance. Front. Public Health 2022, 10, 1003007. [Google Scholar] [CrossRef] [PubMed]
  5. Su, C.Y.; Guo, Y. Factors impacting university students’ online learning experiences during the COVID-19 epidemic. J. Comput. Assist. Learn. 2021, 37, 1578–1590. [Google Scholar] [CrossRef]
  6. Rahman, M.H.A.; Uddin, M.S.; Dey, A. Investigating the mediating role of online learning motivation in the COVID-19 pandemic situation in Bangladesh. J. Comput. Assist. Learn. 2021, 37, 1513–1527. [Google Scholar] [CrossRef]
  7. Mishra, L.; Gupta, T.; Shree, A. Online teaching-learning in higher education during lockdown period of COVID-19 pandemic. Int. J. Educ. Res. Open 2020, 1, 100012. [Google Scholar] [CrossRef]
  8. Pinho, C.; Franco, M.; Mendes, L. Application of innovation diffusion theory to the E-learning process: Higher education context. Educ. Inf. Technol. 2021, 26, 421–440. [Google Scholar] [CrossRef]
  9. Ansari, J.A.N.; Khan, N.A. Exploring the role of social media in collaborative learning the new domain of learning. Smart Learn. Environ. 2020, 7, 9. [Google Scholar] [CrossRef] [Green Version]
  10. Li, C.; He, L.; Wong, I.K.A. Determinants predicting undergraduates’ intention to adopt e-learning for studying english in chinese higher education context: A structural equation modelling approach. Educ. Inf. Technol. 2021, 26, 4221–4239. [Google Scholar] [CrossRef]
  11. Santo, L.D.; Peña-Jimenez, M.; Canzan, F.; Saiani, L.; Battistelli, A. The emotional side of the e-learning among nursing students: The role of the affective correlates on e-learning satisfaction. Nurse Educ. Today 2022, 110, 105268. [Google Scholar] [CrossRef]
  12. Akbari, M.; Danesh, M.; Rezvani, A.; Javadi, N.; Banihashem, S.K.; Noroozi, O. The role of students’ relational identity and autotelic experience for their innovative and continuous use of e-learning. Educ. Inf. Technol. 2022, 28, 1911–1934. [Google Scholar] [CrossRef]
  13. Chopra, G.; Madan, P.; Jaisingh, P.; Bhaskar, P. Effectiveness of e-learning portal from students’ perspective. Interact. Technol. Smart Educ. 2019, 16, 94–116. [Google Scholar] [CrossRef]
  14. Yang, J.; Peng, M.Y.P.; Wong, S.H.; Chong, W.L. How E-Learning Environmental Stimuli Influence Determinates of Learning Engagement in the Context of COVID-19? SOR Model Perspective. Front. Psychol. 2021, 12, 584976. [Google Scholar] [CrossRef]
  15. Dubey, P.; Pradhan, R.L.; Sahu, K.K. Underlying factors of student engagement to E-learning. J. Res. Innov. Teach. Learn. 2023, 16, 17–36. [Google Scholar] [CrossRef]
  16. Kew, S.N.; Tasir, Z. Developing a Learning Analytics Intervention in E-learning to Enhance Students’ Learning Performance: A Case Study. Educ. Inf. Technol. 2022, 27, 7099–7134. [Google Scholar] [CrossRef]
  17. Boyaci, D.B.; Atalay, N. A scale development for 21st Century skills of primary school students: A validity and reliability study. Int. J. Instr. 2016, 9, 133–135. [Google Scholar] [CrossRef]
  18. Habets, O.; Stoffers, J.; Van der Heijden, B.; Peters, P. Am i fit for tomorrow’s labor market? The effect of graduates’ skills development during higher education for the 21st century’s labor market. Sustainability 2020, 12, 7746. [Google Scholar] [CrossRef]
  19. Gkemisi, S.; Paraskeva, F.; Alexiou, A.; Bouta, H. Strengthening collaboration and communication skills in an online TPD program for 21st-century educators. Int. J. Learn. Technol. 2016, 11, 340–363. [Google Scholar] [CrossRef]
  20. Benbow, R.J.; Lee, C.; Hora, M.T. Exploring college faculty development in 21st-century skill instruction: An analysis of teaching-focused personal networks. J. Furth. High. Educ. 2021, 45, 818–835. [Google Scholar] [CrossRef]
  21. Hasanah, H.; Nasir Malik, M. Blended learning in improving students’ critical thinking and communication skills at University. Cypriot J. Educ. Sci. 2020, 15, 1295–1306. [Google Scholar] [CrossRef]
  22. Carlgren, T. Communication, Critical Thinking, Problem Solving: A Suggested Course for All High School Students in the 21st Century. Interchange 2013, 44, 63–81. [Google Scholar] [CrossRef]
  23. Olanrewaju Adebisi, R. Developing Students’ Critical Thinking Abilities about Environmental Change using the Discovery Method of E-Learning. J. Learn. Educ. Policy 2022, 3, 1–12. [Google Scholar] [CrossRef]
  24. van Laar, E.; van Deursen, A.J.A.M.; van Dijk, J.A.G.M.; de Haan, J. The relation between 21st-century skills and digital skills: A systematic literature review. Comput. Human Behav. 2017, 72, 577–588. [Google Scholar] [CrossRef]
  25. Neumeyer, X.; Santos, S.C.; Morris, M.H. Overcoming barriers to technology adoption when fostering entrepreneurship among the poor: The role of technology and digital literacy. IEEE Trans. Eng. Manag. 2021, 68, 1605–1618. [Google Scholar] [CrossRef]
  26. Tang, T.; Vezzani, V.; Eriksson, V. Developing critical thinking, collective creativity skills and problem solving through playful design jams. Think. Ski. Creat. 2020, 37, 100696. [Google Scholar] [CrossRef]
  27. Motallebzadeh, K.; Ahmadi, F.; Hosseinnia, M. Relationship between 21st century skills, speaking and writing skills: A structural equation modelling approach. Int. J. Instr. 2018, 11, 265–276. [Google Scholar] [CrossRef]
  28. Özdemir, S.; Çoban, Ö.; Bozkurt, S. Examination of the relationship between school principals’ 21st century skills and their strategic leadership according to teachers’ opinions. Pegem J. Educ. Instr. 2020, 10, 399–426. [Google Scholar] [CrossRef]
  29. Ng, D.T.K.; Leung, J.K.L.; Su, J.; Ng, R.C.W.; Chu, S.K.W. Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educ. Technol. Res. Dev. 2023, 71, 137–161. [Google Scholar] [CrossRef]
  30. Sulaiman, J.; Ismail, S.N. Teacher competence and 21st century skills in transformation schools 2025 (TS25). Univers. J. Educ. Res. 2020, 8, 3536–3544. [Google Scholar] [CrossRef]
  31. Almahdawi, M.; Senghore, S.; Ambrin, H.; Belbase, S. High school students’ performance indicators in distance learning in chemistry during the COVID-19 pandemic. Educ. Sci. 2021, 11, 672. [Google Scholar] [CrossRef]
  32. Abdul Razzak, N. Strategies for effective faculty involvement in online activities aimed at promoting critical thinking and deep learning. Educ. Inf. Technol. 2016, 21, 881–896. [Google Scholar] [CrossRef]
  33. Chusni, M.M.; Saputro, S.; Rahardjo, S.B. Suranto Student’s critical thinking skills through discovery learning model using e-learning on environmental change subject matter. Eur. J. Educ. Res. 2020, 10, 1123–1135. [Google Scholar] [CrossRef]
  34. McDonald, E.W.; Boulton, J.L.; Davis, J.L. E-learning and nursing assessment skills and knowledge—An integrative review. Nurse Educ. Today 2018, 66, 166–174. [Google Scholar] [CrossRef]
  35. Durak, H.Y.; Saritepeci, M. Analysis of the relation between computational thinking skills and various variables with the structural equation model. Comput. Educ. 2018, 116, 191–202. [Google Scholar] [CrossRef]
  36. Nikou, S.A.; Economides, A.A. Mobile-Based Assessment: Integrating acceptance and motivational factors into a combined model of Self-Determination Theory and Technology Acceptance. Comput. Human Behav. 2017, 68, 83–95. [Google Scholar] [CrossRef]
  37. Rogers, R. The motivational pull of video game feedback, rules, and social interaction: Another self-determination theory approach. Comput. Human Behav. 2017, 73, 446–450. [Google Scholar] [CrossRef] [Green Version]
  38. Hew, T.S.; Kadir, S.L.S.A. Predicting the acceptance of cloud-based virtual learning environment: The roles of Self Determination and Channel Expansion Theory. Telemat. Inform. 2016, 33, 990–1013. [Google Scholar] [CrossRef]
  39. Han, S.Y.; Yoo, J.; Zo, H.; Ciganek, A.P. Understanding makerspace continuance: A self-determination perspective. Telemat. Inform. 2017, 34, 184–195. [Google Scholar] [CrossRef]
  40. Zhang, J.; Zhang, Y.; Song, Y.; Gong, Z. The different relations of extrinsic, introjected, identified regulation and intrinsic motivation on employees’ performance: Empirical studies following self-determination theory. Manag. Decis. 2016, 54, 2393–2412. [Google Scholar] [CrossRef]
  41. Wang, T.; Yeh, R.K.J.; Chen, C.; Tsydypov, Z. What drives electronic word-of-mouth on social networking sites? Perspectives of social capital and self-determination. Telemat. Inform. 2016, 33, 1034–1047. [Google Scholar] [CrossRef]
  42. Chen, K.C.; Jang, S.J. Motivation in online learning: Testing a model of self-determination theory. Comput. Human Behav. 2010, 26, 741–752. [Google Scholar] [CrossRef]
  43. Sørebø, Ø.; Halvari, H.; Gulli, V.F.; Kristiansen, R. The role of self-determination theory in explaining teachers’ motivation to continue to use e-learning technology. Comput. Educ. 2009, 53, 1177–1187. [Google Scholar] [CrossRef]
  44. Alsadoon, E.; Alkhawajah, A.; Suhaim, A. Bin Effects of a gamified learning environment on students’ achievement, motivations, and satisfaction. Heliyon 2022, 8, e10249. [Google Scholar] [CrossRef] [PubMed]
  45. Deci, E.L.; Connell, J.P.; Ryan, R.M. Self-Determination in a Work Organization. J. Appl. Psychol. 1989, 74, 580–590. [Google Scholar] [CrossRef]
  46. Gagné, M.; Zuckerman, M.; Koestner, R. Facilitating acceptance of organizational change: The importance of self-determination. J. Appl. Soc. Psychol. 2000, 30, 1843–1852. [Google Scholar] [CrossRef]
  47. Williams, G.C.; Deci, E.L. Internalization of Biopsychosocial Values by Medical Students: A Test of Self-Determination Theory. J. Pers. Soc. Psychol. 1996, 70, 767–779. [Google Scholar] [CrossRef]
  48. Luo, Y.; Lin, J.; Yang, Y. Students’ motivation and continued intention with online self-regulated learning: A self-determination theory perspective. Z. Erzieh. 2021, 24, 1379–1399. [Google Scholar] [CrossRef]
  49. Zhao, L.; Lu, Y.; Wang, B.; Huang, W. What makes them happy and curious online? An empirical study on high school students’ Internet use from a self-determination theory perspective. Comput. Educ. 2011, 56, 346–356. [Google Scholar] [CrossRef]
  50. Hew, T.S.; Sharifah, S.L. Applying Channel Expansion and Self-Determination Theory in predicting use behaviour of cloud-based VLE. Behav. Inf. Technol. 2017, 36, 875–896. [Google Scholar] [CrossRef]
  51. Aljawarneh, S.A. Reviewing and exploring innovative ubiquitous learning tools in higher education. J. Comput. High. Educ. 2020, 32, 57–73. [Google Scholar] [CrossRef]
  52. Bandura, A. Social Foundations of Thought and Action. In The Health Psychology Reader; SAGE Publications Inc.: New Delhi, India, 2012; pp. 94–106. [Google Scholar]
  53. So, W.W.M.; Chen, Y.; Wan, Z.H. Multimedia e-Learning and Self-Regulated Science Learning: A Study of Primary School Learners’ Experiences and Perceptions. J. Sci. Educ. Technol. 2019, 28, 508–522. [Google Scholar] [CrossRef]
  54. Çetin, E. Digital storytelling in teacher education and its effect on the digital literacy of pre-service teachers. Think. Ski. Creat. 2021, 39, 100760. [Google Scholar] [CrossRef]
  55. Martin, A. A european framework for digital literacy. Nord. J. Digit. Lit. 2006, 1, 151–161. [Google Scholar] [CrossRef]
  56. Ibieta, A.; Hinostroza, J.E.; Labbé, C. Improving Students’ Information Problem-Solving Skills on the Web Through Explicit Instruction and the Use of Customized Search Software. J. Res. Technol. Educ. 2019, 51, 217–238. [Google Scholar] [CrossRef]
  57. Frerejean, J.; Velthorst, G.J.; van Strien, J.L.H.; Kirschner, P.A.; Brand-Gruwel, S. Embedded instruction to learn information problem solving: Effects of a whole task approach. Comput. Human Behav. 2019, 90, 117–130. [Google Scholar] [CrossRef]
  58. McGrew, S.; Byrne, V.L. Who Is behind this? Preparing high school students to evaluate online content. J. Res. Technol. Educ. 2020, 53, 457–475. [Google Scholar] [CrossRef]
  59. Kocak, O.; Coban, M.; Aydin, A.; Cakmak, N. The mediating role of critical thinking and cooperativity in the 21st century skills of higher education students. Think. Ski. Creat. 2021, 42, 100967. [Google Scholar] [CrossRef]
  60. Aizenkot, D.; Ben David, Y. An exploratory study of 21 st century skills of undergraduate Education students: A comparison between freshman, second, and graduation years. In Innovations in Education and Teaching International; Taylor & Francis Online: London, UK, 2022; pp. 1–10. [Google Scholar] [CrossRef]
  61. Lucas, C.; Schindel, T.J.; Saini, B.; Paslawski, T. Game changer: Pharmacy students’ perceptions of an educational “Party Hat” game to enhance communication and collaboration skills. Curr. Pharm. Teach. Learn. 2020, 12, 442–449. [Google Scholar] [CrossRef]
  62. Ma, C.; Shang, J.; Bott, M.J. Linking unit collaboration and nursing leadership to nurse outcomes and quality of care. J. Nurs. Adm. 2015, 45, 435–442. [Google Scholar] [CrossRef]
  63. Duncan, K.J. Examining the Effects of Immersive Game-Based Learning on Student Engagement and the Development of Collaboration, Communication, Creativity and Critical Thinking. TechTrends 2020, 64, 514–524. [Google Scholar] [CrossRef]
  64. Reychav, I.; Wu, D. Mobile collaborative learning: The role of individual learning in groups through text and video content delivery in tablets. Comput. Human Behav. 2015, 50, 520–534. [Google Scholar] [CrossRef]
  65. Alavi, M. Collaborative Learning: An Empirical Evaluation. MIS Q. 1994, 18, 159–174. [Google Scholar] [CrossRef]
  66. Kumar, G.; Banerjee, R.N.; Meena, P.L.; Ganguly, K.K. Joint planning and problem solving roles in supply chain collaboration. IIMB Manag. Rev. 2017, 29, 45–57. [Google Scholar] [CrossRef]
  67. Chen, C.C.; Swan, K. Using innovative and scientifically-based debate to build e-learning community. Online Learn. J. 2020, 24, 67–80. [Google Scholar] [CrossRef]
  68. Salloum, S.A.; Qasim Mohammad Alhamad, A.; Al-Emran, M.; Abdel Monem, A.; Shaalan, K. Exploring students’ acceptance of e-learning through the development of a comprehensive technology acceptance model. IEEE Access 2019, 7, 128445–128462. [Google Scholar] [CrossRef]
  69. Abdullah, F.; Ward, R. Developing a General Extended Technology Acceptance Model for E-Learning (GETAMEL) by analysing commonly used external factors. Comput. Human Behav. 2016, 56, 238–256. [Google Scholar] [CrossRef]
  70. Sukendro, S.; Habibi, A.; Khaeruddin, K.; Indrayana, B.; Syahruddin, S.; Makadada, F.A.; Hakim, H. Using an extended Technology Acceptance Model to understand students’ use of e-learning during COVID-19: Indonesian sport science education context. Heliyon 2020, 6, e05410. [Google Scholar] [CrossRef]
  71. Scherer, R.; Siddiq, F.; Tondeur, J. The technology acceptance model (TAM): A meta-analytic structural equation modeling approach to explaining teachers’ adoption of digital technology in education. Comput. Educ. 2019, 128, 13–35. [Google Scholar] [CrossRef]
  72. Facione, P.A. Critical Thinking: What It Is and Why It Counts. Insight Assess. 2011, 1, 1–23. [Google Scholar]
  73. Kim, K.; Sharma, P.; Land, S.M.; Furlong, K.P. Effects of Active Learning on Enhancing Student Critical Thinking in an Undergraduate General Science Course. Innov. High. Educ. 2013, 38, 223–235. [Google Scholar] [CrossRef]
  74. Bellaera, L.; Weinstein-Jones, Y.; Ilie, S.; Baker, S.T. Critical thinking in practice: The priorities and practices of instructors teaching in higher education. Think. Ski. Creat. 2021, 41, 100856. [Google Scholar] [CrossRef]
  75. Castaño, J.P.; Arnal-Pastor, M.; Pagán-Castaño, E.; Guijarro-García, M. Bibliometric analysis of the literature on critical thinking: An increasingly important competence for higher education students. Econ. Res. Istraz. 2023, 36, 2125888. [Google Scholar] [CrossRef]
  76. Zhang, Q.; Tang, H.; Xu, X. Analyzing collegiate critical thinking course effectiveness: Evidence from a quasi-experimental study in China. Think. Ski. Creat. 2022, 45, 101105. [Google Scholar] [CrossRef]
  77. Venkatesh, V.; Thong, J.Y.L.; Xu, X. Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Q. Manag. Inf. Syst. 2012, 36, 157–178. [Google Scholar] [CrossRef] [Green Version]
  78. Venkatesh, V.; Morris, M.G.; Davis, G.B.; Davis, F.D. User acceptance of information technology: Toward a unified view. MIS Q. Manag. Inf. Syst. 2003, 27, 425–478. [Google Scholar] [CrossRef] [Green Version]
  79. Alqahtani, M.A.; Alamri, M.M.; Sayaf, A.M.; Al-Rahmi, W.M. Exploring student satisfaction and acceptance of e-learning technologies in Saudi higher education. Front. Psychol. 2022, 13, 939336. [Google Scholar] [CrossRef]
  80. Davis, F.D.; Bagozzi, R.P.; Warshaw, P.R. User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Manag. Sci. 1989, 35, 982–1003. [Google Scholar] [CrossRef] [Green Version]
  81. Bae, M. Understanding the effect of the discrepancy between sought and obtained gratification on social networking site users’ satisfaction and continuance intention. Comput. Human Behav. 2018, 79, 137–153. [Google Scholar] [CrossRef]
  82. Pop, M.D.; Pugna, A.P.; Crețu, V.I.; Potra, S.A. Two Years of Hybrid Education in Romania: A Comparative Overview of the Students’ Expectations for the Online Educational Platforms. Sustainability 2022, 14, 4241. [Google Scholar] [CrossRef]
  83. Cho, M.H.; Shen, D. Self-regulation in online learning. Distance Educ. 2013, 34, 290–301. [Google Scholar] [CrossRef]
  84. Han, I.; Shin, W.S. The use of a mobile learning management system and academic achievement of online students. Comput. Educ. 2016, 102, 79–89. [Google Scholar] [CrossRef]
  85. Chang, Y.C.; Tsai, Y.T. The Effect of University Students’ Emotional Intelligence, Learning Motivation and Self-Efficacy on Their Academic Achievement—Online English Courses. Front. Psychol. 2022, 13, 203. [Google Scholar] [CrossRef]
  86. Sun, Y.; Ni, L.; Zhao, Y.; Shen, X.L.; Wang, N. Understanding students’ engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. Br. J. Educ. Technol. 2019, 50, 3156–3174. [Google Scholar] [CrossRef]
  87. Limna, P.; Siripipatthanakul, S.; Phayaprom, B.; Siripipattanakul, S. The Relationship Between Twenty-First-Century Learning Model (4Cs), Student Satisfaction and Student Performance-Effectiveness. Int. J. Behav. Anal. 2022, 2, 1–18. [Google Scholar]
  88. Mailizar, M.; Burg, D.; Maulina, S. Examining university students’ behavioural intention to use e-learning during the COVID-19 pandemic: An extended TAM model. Educ. Inf. Technol. 2021, 26, 7057–7077. [Google Scholar] [CrossRef]
  89. Saxena, C.; Baber, H.; Kumar, P. Examining the Moderating Effect of Perceived Benefits of Maintaining Social Distance on E-learning Quality During COVID-19 Pandemic. J. Educ. Technol. Syst. 2021, 49, 532–554. [Google Scholar] [CrossRef]
  90. Omar, H.A.; Ali, E.M.; Belbase, S. Graduate students’ experience and academic achievements with online learning during COVID-19 pandemic. Sustainability 2021, 13, 13055. [Google Scholar] [CrossRef]
  91. Hair, J.F.; Risher, J.J.; Sarstedt, M.; Ringle, C.M. When to use and how to report the results of PLS-SEM. Eur. Bus. Rev. 2019, 31, 2–24. [Google Scholar] [CrossRef]
  92. Fornell, C.; Larcker, D.F. Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. J. Mark. Res. 1981, 18, 39. [Google Scholar] [CrossRef]
  93. Hizam, S.M.; Akter, H.; Sentosa, I.; Ahmed, W. Digital competency of educators in the virtual learning environment: A structural equation modeling analysis. IOP Conf. Ser. Earth Environ. Sci. 2021, 704, 12023. [Google Scholar] [CrossRef]
  94. Tang, Y.M.; Chen, P.C.; Law, K.M.Y.; Wu, C.H.; Lau, Y.Y.; Guan, J.; He, D.; Ho, G.T.S. Comparative analysis of Student’s live online learning readiness during the coronavirus (COVID-19) pandemic in the higher education sector. Comput. Educ. 2021, 168, 104211. [Google Scholar] [CrossRef] [PubMed]
  95. Susilowati, N.; Mahmud, A.; Sari, P.N. Communication Skill, Student Engagement and Self-Efficacy: Flow on Effect on Student Online Learning. J. Educ. Technol. 2022, 6, 67–76. [Google Scholar] [CrossRef]
Figure 1. Research model (source: author).
Figure 1. Research model (source: author).
Sustainability 15 09363 g001
Figure 2. Findings for path coefficient.
Figure 2. Findings for path coefficient.
Sustainability 15 09363 g002
Figure 3. Path (t-Value) findings.
Figure 3. Path (t-Value) findings.
Sustainability 15 09363 g003
Table 1. The survey’s demographics for gender, age, and specialization.
Table 1. The survey’s demographics for gender, age, and specialization.
DemographicDescriptionNumber of Respondents
GenderFemale129
Male217
Age18–2051
21–2467
25–29129
30–3465
35 and above34
SpecializationHumanities Colleges162
Scientific Colleges108
Medical Colleges76
Table 2. Reflective indictor loadings, CR, CA, and AVE.
Table 2. Reflective indictor loadings, CR, CA, and AVE.
FactorsItemsLoadingCACRAVEFactorsItemsLoadingCACRAVE
Perceived autonomy
(PA)
PA_10.780.850.890.62Cooperativity skills (CS)CS_10.880.870.910.72
PA_20.74CS_20.89
PA_30.82CS_30.89
PA_40.83CS_40.71
PA_50.78
Perceived competence
(PC)
PC_10.810.880.910.68Perceived usefulness (PU)PU_10.730.770.840.52
PC_20.81PU_20.72
PC_30.84PU_30.74
PC_40.86PU_40.71
PC_50.80PU_50.71
Perceived Relatedness
(PR)
PR_10.790.860.900.65Critical thinking in the 21st-century skills
(CT)
CT_10.850.870.910.72
PR_20.86CT_20.86
PR_30.84CT_30.88
PR_40.82CT_40.81
PR_50.72
Digital literacy (DL)DL_10.850.870.910.66Behavioral intention to use e-learningBI_10.760.870.900.65
DL_20.87(BI)BI_20.80
DL_30.77 BI_30.82
DL_40.74 BI_40.82
DL_50.83 BI_50.82
Effective communication (EC)EC_10.820.860.910.71Academic achievement
(AA)
AA_10.820.870.910.66
EC_20.88AA_20.84
EC_30.87AA_30.86
EC_40.80AA_40.79
AA_50.75
Table 3. Discriminant validity (Fornell–Larcker criterion).
Table 3. Discriminant validity (Fornell–Larcker criterion).
CSAABICTDLECPAPCPRPU
Cooperativity Skills0.85
Academic achievement 0.390.81
Behavioral intention to use0.400.410.81
Critical thinking in the 21st century skills0.290.470.300.85
Digital literacy0.440.760.480.430.81
Effective communication0.520.420.360.430.410.84
Perceived autonomy0.530.410.410.400.450.380.79
Perceived competence0.330.400.370.430.450.360.350.82
Perceived relatedness0.600.460.430.430.460.590.500.370.81
Perceived usefulness0.600.450.480.440.480.540.510.430.570.72
Table 4. Heterotrait–monotrait ratio for discriminant validity (HTMT).
Table 4. Heterotrait–monotrait ratio for discriminant validity (HTMT).
CSAABICTDLECPAPCPRPU
Cooperativity Skills
Academic achievement 0.45
Behavioral intention to use0.460.47
Critical thinking in the 21st century skills0.340.540.34
Digital literacy0.510.760.550.49
Effective communication0.600.480.420.490.47
Perceived autonomy0.620.460.470.450.520.45
Perceived competence0.380.450.420.480.510.420.40
Perceived relatedness0.700.520.500.490.530.680.580.42
Perceived usefulness0.690.530.590.520.580.630.600.500.67
Table 5. Coefficient of determination R2.
Table 5. Coefficient of determination R2.
R SquareR Square Adjusted
Academic achievement 0.160.16
Behavioral intention to use0.240.23
Critical thinking in the 21st century skills0.350.33
Perceived usefulness0.510.50
Table 6. Results for F2.
Table 6. Results for F2.
BICTPU
Cooperativity Skills 0.420.37
Academic achievement0.20
Behavioral intention to use
Critical thinking in the 21st century skills0.31
Digital literacy 0.320.31
Effective communication 0.420.33
Perceived autonomy 0.420.42
Perceived competence 0.540.42
Perceived relatedness 0.310.32
Perceived usefulness0.520.31
Table 7. Variance inflation factor (VIF).
Table 7. Variance inflation factor (VIF).
BICTPU
Cooperativity Skills 2.001.86
Academic achievement 1.00
Behavioral intention to use
Critical thinking in the 21st century skills1.23
Digital literacy 1.581.55
Effective communication 1.761.69
Perceived autonomy 1.631.59
Perceived competence 1.371.34
Perceived relatedness 2.082.03
Perceived usefulness1.232.04
Table 8. Hypotheses assessment.
Table 8. Hypotheses assessment.
Βt-Valuesp = ValueResults
Perceived autonomy -----> Perceived usefulness (H1)0.142.810.005Accepted
Perceived autonomy -----> Critical thinking in the 21st century skills (H2)0.152.970.003Accepted
Perceived competence -----> Perceived usefulness (H3)0.122.990.003Accepted
Perceived competence -----> Critical thinking in the 21st century skills (H4)0.193.680.000Accepted
Perceived relatedness -----> Perceived usefulness (H5)0.152.810.005Accepted
Perceived relatedness -----> Critical thinking in the 21st century skills (H6)0.142.030.042Accepted
Digital literacy -----> Perceived usefulness (H7)0.102.060.039Accepted
Digital literacy -----> Critical thinking in the 21st century skills (H8)0.142.650.008Accepted
Effective communication -----> Perceived usefulness (H9)0.173.550.000Accepted
Effective communication -----> Critical thinking in the 21st century skills (H10)0.172.680.007Accepted
Cooperativity Skills -----> Perceived usefulness (H11)0.254.820.000Accepted
Cooperativity Skills -----> Critical thinking in the 21st century skills (H12)−0.172.840.005Accepted
Perceived usefulness -----> Critical thinking in the 21st century skills (H13)0.132.030.043Accepted
Perceived usefulness -----> Behavioral intention to use E-learning (H14)0.438.560.000Accepted
Critical thinking in the 21st century skills -----> Behavioral intention to use E-learning (H15)0.102.010.044Accepted
Behavioral intention to use E-learning -----> Academic achievement (H16)0.417.630.000Accepted
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Aldraiweesh, A.; Alturki, U. Exploring Factors Influencing the Acceptance of E-Learning and Students’ Cooperation Skills in Higher Education. Sustainability 2023, 15, 9363. https://doi.org/10.3390/su15129363

AMA Style

Aldraiweesh A, Alturki U. Exploring Factors Influencing the Acceptance of E-Learning and Students’ Cooperation Skills in Higher Education. Sustainability. 2023; 15(12):9363. https://doi.org/10.3390/su15129363

Chicago/Turabian Style

Aldraiweesh, Ahmed, and Uthman Alturki. 2023. "Exploring Factors Influencing the Acceptance of E-Learning and Students’ Cooperation Skills in Higher Education" Sustainability 15, no. 12: 9363. https://doi.org/10.3390/su15129363

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop