Surveying and Modelling 21st Century Online Learning Patterns of Medical Students

Medical education in the 21st century is shifting more toward online learning because of extensive application of information and communication technology (ICT). We surveyed medical students’ 21st century online learning experiences and modeled the interrelations among relevant dimensions of 21st century online learning. Based on the general themes proposed by multiple 21st century learning frameworks and current medical education emphases, a seven-factor instrument was developed for surveying 364 medical students’ learning process, thinking process, and basic science-related clinical ability. The associations among the seven factors and the structural relationships of how online learning practices and thinking processes affected basic science-related clinical ability were explored. The developed instrument was validated and possessed good reliability. The seven dimensions were interrelated. Specifically, meaningful learning with ICT was positively associated with other learning practices. The learning practices were positively associated with the thinking processes and the thinking processes were positively associated with students’ basic science-related clinical ability. Our findings suggested that students engaged in active and collaborative learning with technology would employ higher-order thinking and perceived better basic science-related clinical ability. The findings support engaging medical students with 21st century learning practices to strengthen students’ self-perception of clinical ability.


Introduction
A renewed focus of health care in the 21st century is required because of the changing environment [1]. The demands for renewal of medical knowledge and technological advancement trigger ongoing medical educational reforms to transform teacher-centric pedagogy. Over the past two decades, medical educators have embraced student-centered teaching modes with the use of information and communication technology (ICT) tools [2,3]. The shift has recently been accentuated towards extensive use of online or blended mode due to the COVID-19 pandemic. To cultivate students' ability to adapt to the rapid changes, an educational model that foster 21st century learning practices emphasizing collaborative and intentional construction of knowledge supported by online environments has been advocated [4]. Thus, effective use of ICT (e.g., using online learning platform for advanced and latest knowledge acquisition and construction) has become an indispensable for the 21st century medical education. In the study of Stoehr et al., there is an apparent discrepancy between medical students' preference and what actual curricula offer them [5]. The study indicates the necessity to investigate medical students' online learning practices for curriculum reforms. Nevertheless, there remained an absence of instrument to survey medical students' 21st century online learning.

Intercorrelations of the 21 CLS Components
The associations among various dimensions of the 21 CLS have been discussed. Sang et al. demonstrated strong intercorrelations among the six types of learning practices (rs > 0.50) [10], and the structural relations were re-verified in the later study [15]. However, the six learning practices present distinctive focuses, with MLT, CoL, SDL being considered as learning process denoting active learning within the socio-technological milieu, and CreT, CriT and APS being categorized as thinking process denoting higher-order cognitions. A plethora of studies have provided positive evidence on the impacts of learning process on thinking process enhancement. First, under online learning, use of ICT becomes the base of learning [16,17]. The employment of ICT enables the design of active and collaborative learning environments [18,19], and ICT as a learning medium can promote self-directed learning readiness [20,21]. Further, students engage in CoL that involve the active exchange of ideas would stimulate higher levels of CriT and CreT, which contribute to better problemsolving skills [22,23]. SDL plays a strategic role in planning, monitoring, and adjusting thinking and learning processes. It undergirds students' cognitive functioning [24]. Going forward, problem solving involves complex thinking strategies [25], CriT and CreT may in turn predict APS. The six learning strategies are intimately associated, and previous literature suggest some predictive relations among them.
Promising pedagogical design of learning strategies in medical education (i.e., MLT, CoL, SDL, CriT, CreT, APS) that enrich students' learning experiences in integrating, analyzing and applying scientific knowledge and information will equip students with intellectual capacity to use basic science knowledge and principles to clinical practice [26][27][28]. Thus, learning process and thinking process are likely to predict medical students' ability in applying basic science into medical practice. The relations can be interpreted by the "presage-process-product" (3P) model proposed by Biggs [29], in which learning processes fuel the thinking processes and foster ability development. Nonetheless, the mechanism of how medical students' practical ability can be enhanced through learning and thinking process have rarely been empirically investigated.
Medical students have been surveyed about their learning styles, such as learning approaches and orientations and conditions for performance [30]; however, little data exist on evaluating medical students' 21st century online learning practices, especially in the period that the emergence of ICT and the outbreak of the COVID-19 pandemic accelerated the switch from face-to-face to online or blended learning. This sudden switch might have caused problems in both curriculum framework and students' learning. Therefore, surveying medical students' online learning experiences is necessary. Additionally, notwithstanding medical students' practical ability has received increasing focus, there remained an absence in standardized tools for assessing medical students' competence in applying basic science knowledge to clinic setting [31], thus it is of importance to take medical students' ability in applying scientific knowledge to medial practice into consideration when designing the instrument. The proposed measurement can provide evidence to understand the important learning practices students are engaged in during online learning and throw light on curriculum design.
To fill the research gap, this study aimed to (1) develop and validate a 21 CL instrument to survey and model medical students' online learning practices; (2) explore the role of MLT plays in the development of other learning processes during online learning, and the mechanism of how learning process and thinking process influence the medical students' basic science-related clinical ability. Particularly, we examined whether MLT serves as a base for the development of CoL and SDL, whether learning process (CoL and SDL) can predict thinking process (CreT and CriT), and whether CreT and CriT can affect APS and in turn lead to changes in ABS ( Figure 1). ability in applying basic science into medical practice. The relations can be interprete the "presage-process-product" (3P) model proposed by Biggs [29], in which learning cesses fuel the thinking processes and foster ability development. Nonetheless, the m anism of how medical students' practical ability can be enhanced through learning thinking process have rarely been empirically investigated.
Medical students have been surveyed about their learning styles, such as lear approaches and orientations and conditions for performance [30]; however, little dat ist on evaluating medical students' 21st century online learning practices, especially in period that the emergence of ICT and the outbreak of the COVID-19 pandemic acceler the switch from face-to-face to online or blended learning. This sudden switch might h caused problems in both curriculum framework and students' learning. Therefore, veying medical students' online learning experiences is necessary. Additionally, notw standing medical students' practical ability has received increasing focus, there rema an absence in standardized tools for assessing medical students' competence in appl basic science knowledge to clinic setting [31], thus it is of importance to take medical dents' ability in applying scientific knowledge to medial practice into consideration w designing the instrument. The proposed measurement can provide evidence to un stand the important learning practices students are engaged in during online learning throw light on curriculum design.
To fill the research gap, this study aimed to (1) develop and validate a 21 CL in ment to survey and model medical students' online learning practices; (2) explore the of MLT plays in the development of other learning processes during online learning, the mechanism of how learning process and thinking process influence the medical dents' basic science-related clinical ability. Particularly, we examined whether MLT se as a base for the development of CoL and SDL, whether learning process (CoL and S can predict thinking process (CreT and CriT), and whether CreT and CriT can affect and in turn lead to changes in ABS ( Figure 1).

Materials and Methods
Data were collected from a 2021 to 2022 cohort. In view of the outbreak of COV 19, all medical students were experiencing online learning during this period. Base our investigation, Zoom was the primary tool for online teaching and learning, follo by Microsoft team and Blackboard. Medical students were provided with online lectu watched patient videos and case scenarios, had group discussions, practiced med training, submitted homework, and took examinations through the online learning forms.
The Associate Director of the Faculty of Medicine acted as a gatekeeper to rec ment, forwarding emails of invitation, consent form as well as the survey link to all 1 to Year 6 medical students on behalf of the research team. All the data were colle

Materials and Methods
Data were collected from a 2021 to 2022 cohort. In view of the outbreak of COVID-19, all medical students were experiencing online learning during this period. Based on our investigation, Zoom was the primary tool for online teaching and learning, followed by Microsoft team and Blackboard. Medical students were provided with online lectures, watched patient videos and case scenarios, had group discussions, practiced medical training, submitted homework, and took examinations through the online learning platforms.
The Associate Director of the Faculty of Medicine acted as a gatekeeper to recruitment, forwarding emails of invitation, consent form as well as the survey link to all Year 1 to Year 6 medical students on behalf of the research team. All the data were collected through the university online platform. Participants were recruited through mass email to the students in the Faculty of Medicine, those with interest were asked to click into the link to the 21 CLS and complete the online survey. Informed consent was first obtained from the participants prior to their participation in the study.

Instrument
The 21st Century Learning Survey (21 CLS) developed by Sang et al. was adapted in the current study [10]. The original 21 CLS comprised of six subscales: CoL, CriT, SDL, CreT, MLT, and APS. Previous study has demonstrated good construct validity and internal consistency of the original 21 CLS. In our study, ABS was added as an important indicator to survey the influence of 21st century online learning on medical students' clinical ability [14]. Thus, the developed 21 CLS is a seven-factor instrument that consists of 29 items, with each subscale containing 4 to 5 items, respectively. All items in the survey were screened and reviewed by three medical experts and two educational experts to establish face validity before actual implementation. Responses are provided on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree).

Data Analysis
The 21 CLS for medical students was developed and validated with a mixed method that involved literature review, classical theory test, expert input, and confirmatory factor analysis (CFA). Though the original 21 CLS has been well validated, due to the change of the application context and the emergence of specified items for medical students, revalidation was needed. We subjected the newly developed 21 CLS for medical students to the CFA procedures with the use of Mplus 8.0 [32]. Standardized skewness and kurtosis of the twenty-nine items provided evidence that the 21 CLS meet the assumption of the normality (<|1.50|) [33]. Maximum likelihood estimation was adopted to compute the model fit statistics. Specifically, CFA on the ABS subscale was first conducted to test its construct validity. Then, additional CFA on the correlated seven-factor solution was performed to test the model. Multiple fit indices were accounted for assessing the model, in particular, item should have interpretable loading (>0.50) and with no problematic cross-loading (>0.20). Comparative fit index (CFI) and Tucker Lewis index (TLI) above 0.90 and the root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) below 0.08 represented acceptable fit [34][35][36][37]. Additionally, the composite validity (CR) and average variance extracted (AVE) recommended by Hair et al. were computed, with CR valued above 0.60 and AVE valued above 0.50 demonstrating convergent validity [38,39].
Further, reliability test, item-scale correlations, and intercorrelations were performed to test the internal consistencies of the 21 CLS subscales as well as the whole 21 CLS, with Cronbach's alpha above 0.65, and significant item-scale correlations and significant intercorrelation among subscales indicating good internal consistency and content validity.
Path analyses with structural equation model (SEM) was later adopted to explore the role of MLT plays in the development of other learning process (CoL, SDL), and the paths of how learning processes are related to the thinking processes (CriT, CreT), and how CriT and CreT predict APS, and further, how thinking processes predict ABS ( Figure 2).    Table 1. Overall, the fit indices indicated that the 21 CLS for medical students possess construct validity.     Table 1. Overall, the fit indices indicated that the 21 CLS for medical students possess construct validity.  Table 1 summarizes the scale description and alpha coefficients for the 21 CLS for medical students. Alpha coefficients were above 0.90 for all subscales, and valued 0.95 for the total scale, indicating good internal consistency. Correlation analyses demonstrated strong internal consistency as well; specifically, each item correlated significantly with its corresponding subscale and items within the subscale (ps < 0.01), with CoL ranging from 0.72 to 0.91, CriT ranging from 0.66 to 0.91, SDL ranging from 0.73 to 0.91, CreT ranging from 0.78 to 0.95, MLT ranging from 0.52 to 0.91, APS ranging from 0.70 to 0.93, ABS ranging from 0.52 to 0.93. Agreement with each item and its corresponding subscale was obtained. Additionally, significant correlations were found among the subscales, ranging from 0.24 (MLT and CreT) to 0.68 (MLT and SDL), ps < 0.01 (see Table 2).

Discussion
The objectives of the current study were to develop and validate the 21 CLS that can be adapted to medical students and to identify medical students' 21st century online learning practices. Hence, the 7-factor solution 21 CLS with 29 items was constructed. The score agreement was acceptable to strong, revealing a good internal structure within each subscale. Reliability test and CFA on the whole scale and subscales indicated that the 21 CLS was reliable and valid. This study thus provides support for the 21 CLS being used as an instrument on understanding medical students' 21st century online learning practices, thinking processes and ability-based medical outcomes. It could serve as foundation for other researchers who need to assess medical students' online learning practices and to identify changes in any dimensions for evaluating the effectiveness of proposed 21st century online learning programs. Our findings also contribute to the literature on understanding the online learning practices of medical students from investigating the associations among learning process, thinking process, and learning outcome.
In line with previous literature, the correlations, internal coefficients, and interpretable fit statistics supported the 21 CLS constructs. The 21 CLS emphasizes 21st century medical online learning from the meaningful learning perspective with the use of ICT, collaborative teamwork, self-directed learning, critical and creative thinking skills, and problem-solving skills [3,4,7,40]. To ensure the online learning practices contributes to medical students' clinical competences, assessment on basic knowledge of principles of different diseases and technologies in both classroom and community-based settings are considered in the instrument [8,11,41], thus ABS is added for measuring medical students' ability in applying basic science knowledge to clinical practice.
When investigating the associations among varied aspects of medical students' online learning experiences and the structural relationships for their medical educational outcomes, significant relations among learning practice, thinking process, and learning outcome were obtained. The finding expanded previous study indicating that MLT plays an essential role in the cultivation of CoL and SDL, which subsequently leads to changes in thinking process [15]. During online learning, student isolation and individual-centric modes of learning could impede students' incorporating teamwork [22]. The emergence of ICT adds active elements to academic environment, providing opportunities for students to develop interpersonal skills, motivate and encourage the acquisition of knowledge and basic skills. ICT is thus considered as a powerful tool for enriching students' educational experience [16,17]. The active learning environment incorporates elements of CoL and SDL that contributes to higher-order thinking processes could further support authentic problem solving [22][23][24]. The findings reveal that in this period when there were frequent switches between modes of instruction (online versus face to face), MLT is an important competence to facilitate medical students to engage in collaboration and self-direction, which in turns shape their thinking processes [23,42,43]. The structural model indicates that curriculum design for 21st century online medical learning needs to address all constructs measured as they are interrelated. As implied by Al-Kadri et al., the teacher plays a key role in initiating students' learning process and determining what kind of learning strategies students adopt in clinical setting [43]. Medical educators will need to choose the ICT tools for specific authentic problems, structure SDL and COL, and provide scaffolding for the thinking processes. Future study may employ hierarchical modelling to investigate how medical teachers' design of online learning shape students' learning experiences.
Moreover, the improvement of APS would rely on the cultivation of one's critical thinking and creative mind [44][45][46], which further influenced students' basic science-related clinical ability (ABS) [8,9]. This finding is congruent with literature that thinking process can predict the change of ability-based medical educational outcomes [28]. Since the changes of thinking process are directed by learning process, the thinking process may act as a mediator in the link between learning process and basic science-related clinical ability. Future study may explore the mediating or moderating role of the thinking process by investigating the indirect effect of learning practices on ability-based educational outcomes through thinking process. The 21st century online learning experiences and the mechanism for change of thinking process and learning outcome of medical students in the current study showed similar patterns with previous literature [15] suggested that changes of educational outcomes could be driven by pedagogical strategies and thinking processes, and meaningful learning with ICT is essential under online learning [16,17]. However, the proposed associations of the online learning experiences of medical students in 21st century need to be further tested. Investigation would especially be meaningful if the teaching and learning practices have been intentionally designed to promote 21st century learning by medical educators, as suggested by Chai et al. [47].
As stated in the literature review, few studies have been targeted on identifying the online learning patterns of medical students in the 21st century. Thus, surveying medical students' online learning experiences, thinking mode and skills has practical implications for assessing the learning environment students perceived and surfacing the discrepancies between the pedagogical design and actual implementation and outcomes. Furthermore, assessing students' 21st century online learning could provide an insight on curriculum design and future educational policy making for medical students. In view of the rapid switch of online mode and the particularity of medical education, a deliberate curriculum framework for online learning remained absent in most of the medical schools [5]. A tailormade learning design for students has been highlighted by Chacko in filling in the gap of the expectations from teacher and student [48]. When designing learning program and activities to improve 21st century online learning, various dimensions, including students' learning maturity, needs in different stages, and their learning progress, should be taken into consideration to facilitate medical students to develop their professionalism. The results of path analysis also suggest that the associations of various dimensions can be considered in curriculum design for facilitating a specific dimension.
This study has several limitations. First, participants of the current study were voluntary-driven, thus the composition of the sample did not show equality regarding students of pre-clinical and clinical year and students of male and female genders. To extend the psychometric properties of the 21 CLS, future study could perhaps test measurement invariance on medical and health care students of pre-clinical and clinical year as well as students of different genders to examine if the proposed construct of the 21 CLS behaves equally among students with various demographic characteristics. Second, the data was collected from Hong Kong and may not be representative of other regions of different socioeconomic status. Third, the 21 CLS is a self-report survey that may have bias because of subjectivity. Future study can further test the validity of the instrument with a larger sample and different educational contexts and include measures such as in-depth interview and academic performance for surveying medical students' 21st century learning patterns.

Conclusions
In conclusion, this study develops and validates an instrument for surveying and modeling medical students' 21st century online learning patterns. Associations among various factors and their directions were explored, and our finding supported the hypothesized model and extended the previous literature.