Examining Self-Regulated Learning Strategy Model: A Measurement Invariance Analysis of MSLQ-CAL among College Students in China

Abstract

This study examined psychometric properties and measurement invariance of the Motivated Strategies for Learning Questionnaire for Chinese adult learners, learning strategy scale (MSLQ-CAL-LS). Data were collected from 2499 college students from 15 Chinese universities. Results from factor analysis suggested satisfactory psychometric properties of MSLQ-CAL-LS. We further identified strong evidence to support the configural, metric, scalar and strict invariance across the gender groups, confirming the appropriate use of MSLQ-CAL-LS that can accurately capture the construct of self-regulated learning (SRL) strategies among both female and male Chinese adult learners. This study provides one step forward to measure SRL outside the Western context. Recommendations for future research are discussed.

1. Introduction

Self-regulated learning (SRL) is an essential concept to understand the metacognitive, motivational, and behavioral aspects in the learning process [1] which has attracted researchers for decades. Empirical evidence indicates a strong correlation between students’ use of SRL strategies and their academic performance [2,3,4]. The global pandemic has led to interrupted education and an abrupt transition from face-to-face to an online learning format or a hybrid in tertiary institutions [5]. Such unexpected change might result in a need for learners to adjust their SRL strategies [6,7]. Research also shows that SRL strategies can foster students’ digital literacy (the ability to identify, evaluate, and communicate information via typing or media on various online platforms) when learning independently online, which has been considered as sustainable development in lifelong learning [8] and aligns with the UNESCO Education for Sustainable Development (ESD) goal [9].

To better understand learners’ application and adjustment of SRL strategies, the first step is to identify a psychometrically sound instrument that accurately captures the construct of SRL among the student population that is of interest. This step is particularly important in light of differences in student characteristics, education level, and cultural and learning contexts [10] because SRL originated from Western philosophy, and it encompasses different theories, models, and instruments. Tong et al. [11] examined the cross-cultural transferability of the translated version of the widely used SRL instrument, Motivated Strategies for Learning Questionnaire [12], among Chinese college students, and adapted the instrument for Chinese adult learners (MSLQ-CAL). Although factor analysis yielded satisfactory loadings and model fit, the authors presented limitations of MSLQ-CAL, and recommended validation of each scale (i.e., motivation and learning strategies) using more advanced statistical approaches, given the limited availability of SRL instruments in Chinese context.

Therefore, the purpose of this study is to extend Tong et al.’s work to further validate the subscale of learning strategies (MSLQ-CAL-LS) among 2499 Chinese undergraduate students who have completed one-semester of online learning during COVID-19 pandemic in spring 2020 and resumed face-to-face instruction in fall 2020. The validation process included exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and measurement invariance (female vs. male).

2. Literature Review

2.1. Self-Regulated Learning Model and Learning Strategies

Self-regulated learning (SRL) is regarded as a proxy for lifelong learning under the context of formal education [13] and plays a critical role in promoting students to be sustainable learners beyond schooling [8,14]. SRL is defined as “the degree to which individuals are metacognitively, motivationally, and behaviorally active participants in their own learning process [1] (p. 5). Pintrich and Zusho [15] emphasized that five components need to be included in the SRL model: students’ characteristics (e.g., age and gender), classroom context (e.g., instructional approach and instructor behavior), learning motivation (e.g., self-efficacy and control of learning beliefs), self-regulatory process (e.g., regulating cognition), and academic outcomes (e.g., persistence and achievement). Therefore, researchers need to give full consideration to these components when examining student learning strategies [16], particularly contextual factors [17] and students’ characteristics [18].

Strategies of cognitive, metacognitive, and resource management are three common strategies under the SRL model [19]. Cognitive strategies help learners process information and knowledge from texts and lectures [19]. The typical cognitive strategies are rehearsal, elaboration, and organization [12]. Meta-cognitive strategies help learners control, monitor, and regulate their own cognition and learning [19]. The resource management strategy helps learners take advantage of resources besides their cognition, such as peer learning, help-seeking, effort regulation, and time and study environment management [19].

Further, SRL strategies are used in different phases (i.e., forethought, performance, and self-reflection) to help learners plan, execute and assess their academic performance) [20] and acquire and retain knowledge in methodological and structured ways [21]. During such a learning process, learners intentionally use learning strategies to monitor and adjust learning activities to acquire new knowledge and meet their learning goals [22]. Empirical studies confirmed that students’ academic performance is positively related to learning strategies [2,3].

2.2. Previous Validation Studies on Learning Strategy Scales

Multiple learning strategy scales were developed based on SRL theories, including the online self-regulated learning questionnaire (OSLQ [23]), the motivated strategies for learning questionnaire (MSLQ [12]), and the Learning and Study Strategies Inventory (LASSI) [24]. Among these instruments, MSLQ, a self-report, domain-specific instrument, is one of the most commonly validated instruments across cultures and contexts. MSLQ contains two subscales, motivation and learning strategies. Although two subscales are normally used together to measure students’ motivation and learning strategies (e.g., [25,26]), researchers tend to validate and report psychometric properties separately for these two subscales (e.g., Rami´rez-echeverry et al.2016; Tong et al., 2020). For the purpose of this study, we reviewed validation studies that focused on the learning strategy subscale.

First, we found that factor analysis is the most commonly adopted statistical method to examine the psychometric properties of learning strategy scales. Most studies conducted EFA (e.g., [27]) or CFA (e.g., [28,29]), or both (e.g., [11,30,31]) to validate the instrument’s structure. In addition to factor analysis, researchers also examine measurement invariance (MI) of learning strategy instruments to confirm whether the same construct holds across groups. MI is a statistical property to measure the psychometric equivalence of a model construct across groups [32]. For example, Maun et al. [33] tested MI of seven subscales of MSLQ, including learning strategies, among four caste groups (i.e., Schedules Castes, Scheduled Tribes, Other Backward Castes, and General) and genders (i.e., female and male). Stevens and Tallent-Runnels [34] tested MI across gender and ethnic groups of LASSI for high school students (LASSI-HS). Both studies reported solid evidence to support strong configural, metric, and scalar invariance between gender, suggesting structural similarities and applicability of the instrument in both gender groups. Although we can only locate very few studies examining MI of learning strategy instruments, gender appears to be a popular grouping factor in MI analysis. Such popularity might be due to empirical evidence that male and female students significantly differ in their choice of learning strategies (e.g., [35,36,37,38]). It is not clear, however, that the gender difference results from the application of SRL strategies or from the measurement error between the two gender groups [39]. Hence, an MI test of gender will contribute to the understanding of the source of such difference.

Tong and colleagues [11] translated, adapted, and validated the original motivation and SRL instrument, MSLQ among adult Chinese learners. Psychometric properties of the MSLQ-CAL were tested via EFA and CFA. Results suggested that cross-cultural adaptation and modification were necessary. An earlier version of MSLQ-CAL was also applied to investigate Chinese undergraduate students’ motivation and learning strategies in different contexts (e.g., [37,40]).

2.3. Purpose of This Study

Tong et al. [11] suggested that large sample size is needed to further validate MSLQ-CAL with more sophisticated statistical analysis such as measurement invariance. They also suggested additional items in the learning strategy subscale because some of the factors only retained two items, which is less than the threshold of three to five items per factor as suggested by MacCallum et al. [41]. Moreover, the previous validation and application studies were conducted to measure learners’ motivation and SRL strategies in an in-person learning environment, and MI test has not been conducted across gender groups on SRL strategies, whereas gender difference in employing SRL strategies has been reported [42]. Taking into consideration the recommendation of Tong et al. [11] and the abrupt transition of the learning environment due to pandemic, in this study we attempted to validate MSLQ-CAL learning strategy subscale, MSLQ-CAL-LS, with larger sample size and test MI among Chinese undergraduates who experienced the abrupt change to their learning contexts impacted by the COVID-19 pandemic. Three research questions guided our study:

Research Question 1: What is the factor structure of the MSLQ-CAL-LS?

Research Question 2: How well does the MSLQ-CAL-LS model fit Chinese adult learners?

Research Question 3: Does the MSLQ-CAL-LS construct differ between female and male Chinese adult learners?

3. Method

3.1. Participants

The participants recruited in this study were 2499 undergraduate students currently enrolled in 15 public universities in a coastal city of China. All participants were native Chinese majoring in various disciplines, including mass media, computer science, education, and engineering. At the time of data collection, the mean age of these participants was 19.68 years old (SD = 1.3), with a range between 16 and 29. The majority (97.1%) of the students are within the range of 18–22. The gender distribution in the current sample is 56.4% female and 42.1% male, with 36 participants not disclosing gender information. Other than 81 students without accurate information regarding their years of enrollment, the final sample included 591 freshmen (24.4%), 884 sophomores (36.6%), 585 juniors (24.2%), and 358 seniors (14.8%). The SRL survey was distributed online via Wenjuan Xing links through Wechat, a multipurpose messaging social media platform with over one billion users in China. On the coverage page of the survey, participants were notified that their participation was completely voluntary, and their accurate responses would contribute to the understanding of Chinese college students’ learning strategies.

3.2. Learning Strategy Scale and Procedure on Revising

In Tong et al.’s [11] previous validation study, two subscales in the MSLQ-CAL survey—motivation and learning strategies—were validated separately, with the former composed of 23 items to measure students’ confidence about their ability to complete an academic task and anxiety during examinations, and the latter composed 29 items to evaluate students’ use of self-regulated learning strategies and management of various resources in learning. All items were measured on a 7-point Likert scale (1 = strongly disagree; 7 = strongly agree). Tong et al. [11] indicated that the learning strategies subscale was composed of nine factors: metacognitive self-regulation, effort regulation, time management, organization, peer learning, elaboration, rehearsal, critical thinking, and study environment management with internal consistency ranging from 0.46 to 0.74, RMSEA of 0.044, SRMR of 0.045, and CFI of 0.914.

In the current study, we decided to remove the three factors (i.e., elaboration, rehearsal, and effort regulation) because Chinese students seldom use elaboration strategies in their learning [43], and the Cronbach alpha of the rehearsal factor in the validation study of Tong et al. [11] was 0.48, the lowest among nine factors. Further, there is limited empirical evidence on a positive relationship between rehearsal and elaboration strategies and academic outcomes [21]. Finally, two out of three items in effort regulation strategy are reverse coded, and it is found that Chinese students do not respond well to reverse coded items [2,11]).

We then retained the remaining six factors for further validation: meta-cognitive self-regulation, time management, organization, peer learning, critical thinking, and study environment management for the following four reasons. First, metacognition, critical thinking, and peer learning are highly related to learners’ academic performance, especially in an online learning environment [21]. Second, although the relationship between students’ use of learning strategies and their academic performance was not significantly impacted by their learning modes (face-to-face vs. online), online learners tended to use strategies of organization, seeking help from peers, and metacognition less frequently than face-to-face learners [44]. Third, for Chinese adult learners, the study environment is crucial to learning and positively related to their use of learning strategies [45]. Finally, Li & Zhang [46] highlighted the importance of time management to facilitate Chinese adult learners’ academic engagement. Whole-semester exclusive online learning might affect students’ use of time management strategies.

As mentioned earlier, time management and study environment management factors in the MSLQ-CAL [11] only contained two items, which does not meet the minimum item number per factor suggested in the previous study [41]. Therefore, we reviewed items in Time Management Behavior Scale [47] and Study Environment Management scale [45] and adapted these items to fit the context of Chinese higher education. All items were translated into Chinese with back-translation method [48] as was in the validation of the original MSLQ-CAL. Definitions and examples of six learning strategies domains are listed below:

Mate-cognitive self-regulation (5 items): it refers to students’ awareness and control (i.e., planning, monitoring and regulating) of cognition. A sample item reads, When I study for this class, I set goals for myself in order to direct my activities in each study period.

Critical thinking (4 items): it refers to the extent to which students apply previous knowledge or information to solve problems or make an evaluation in a new situation. A sample item reads, I treat the course material as a starting point and try to develop my own ideas about it.

Organization (3 items): it refers to students’ selection of appropriate information and making structured connections among the information to be acquired. A sample item reads, When I study the readings for this course, I outline the material to help me organize my thoughts.

Time management (7 items): it refers to how students’ schedule and manage their study time. A sample item reads, During a workday I evaluate how well I am following the schedule I have set down for myself.

Study environment management (6 items): it refers to how students’ set up and organize their learning environment. A sample item reads, I always find a quiet area when I need to learn.

Peer learning (4 items): it refers to students’ collaboration or help-seeking with peers during learning. A sample item reads, I try to work with other students from this class to complete the course assignment.

3.3. Data Analysis

Four steps of validation analysis were conducted to examine the psychometric properties of the MSLQ-CAL-LS: internal consistency, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and measurement invariance (MI). SPSS was applied to calculate internal consistency and EFA, while MPlus was applied for conducting CFA and MI. We randomly split our sample into three subsamples based on a ratio of 2:4:4, which resulted in 509 for EFA, 1010 for CFA, and 980 for MI. Internal consistency was reported with Cronbach’s alpha for each factor as well as the whole scale of SRL.

More specifically, EFA was conducted to examine the equivalence of factor structure of MSLQ-CAL-LS to address the first research question. Based on Kaiser’s [49] criterion, we retained factors with eigenvalues greater than 1. We also applied Tabachnick and Fidell’s [50] rule of item selection of minimum loading at 0.32. Furthermore, we used Principal Axis Factor (PAF) as extraction method and Oblimin Oblique as rotation method, following the same EFA procedure in the MSLQ-CAL validation of Tong et al. [11].

To address the second research question, CFA was conducted to examine and confirm the factor structure of the revised model. To evaluate model fit, we reported the following commonly-used fit indices: root-mean-square error of approximation (RMSEA), standardized root mean squared residuals (SRMR), and comparative fit index (CFI). If the model meets the following criterion with a CFI value larger than 0.9 [51], SRMR value less than 0.07 [52], and RMSEA value less than 0.08 [53], the model fit is considered as acceptable. CFA was conducted with maximum likelihood estimation with fixing variances, and factors were correlated.

To address the last research question, we conducted measurement invariance to compare the factor structure of MSLQ-CAL-LS across gender groups. The process of MI also involves testing the fit of a set of increasingly restrictive models against the baseline model. We then compared the difference in CFI (ΔCFI) between nested equivalence models, which is considered a reliable measure of model fit in MI [54,55]. Specifically, we used multigroup confirmatory factor analysis (MGCFA) and applied the procedure of testing measurement invariance outlined by Putnick and Bornstein [32]. CFA was conducted separately for female and male groups on the proposed measurement model. If the same factor structure is found across female and male students, testing of configural invariance would be the next step. Configural invariance is the first and least stringent step in measurement invariance. This step aims at testing whether the constructs gave the same pattern of loading across groups [32]. It is a baseline model against the subsequent more restrictive models (e.g., metric invariance, scalar invariance; [56]). If the configural invariance is established and supported, the next step would be the metric invariance to test whether each item contributes to the latent factor at a similar degree across gender. Once the metric invariance is established, testing scalar invariance follows in order to examine whether the item intercept is equivalent across gender. If the metric invariance is established, the last step of strict invariance would be conducted to explore whether the residual variance is equivalent across gender.

Configural invariance is evaluated by the overall model fit, including a series of model fit indices, i.e., CFI, RMSEA, SRMR, and Chi-square (χ²). Moreover, the fit of metric and scalar invariance is generally evaluated by comparing the two nested models that are identical except for specific path restrictions in one [32]. For example, compared to the configural invariance model, although the metric invariance model has the same structure, it imposes the same constraints on the factor loading across groups (Putnick & Bornstein, 2016). The common nested model comparison includes computing the difference in fit indices (i.e., ΔCFI, Δχ²). Cheung and Rensvold [55] suggested that a ΔCFI value less than 0.01 is acceptable. In the current paper, we relied more on ΔCFI instead of χ² due to the latter’s sensitivity to minor parameter changes in large samples, as suggested by Chen [57].

4. Results

Research Question 1: What is the factor structure of the MSLQ-CAL-LS?

Six factors were generated through EFA, each with an eigenvalue greater than 1. There were three items (i.e., items 7, 37, and 53) across two factors with a loading that does not meet the threshold of 0.32 [50]. Moreover, four items (items 9, 24, 56, and 62) were eliminated from the model due to a lack of meaningful structure, as suggested by Costello and Osborne [58]. For example, item 9 (I attend class regularly) and item 24 (I make sure I keep up with the weekly readings and assignments for this course) belonged to the time management factor in MSLQ-CAL; however, they were found to load onto a separate factor (factor #6) in our analysis and failed to generate a meaningful theoretic explanation. It is possible that the participants rated these items based on how well they behave in this course (i.e., attendance and assignment completion) instead of time management. As a result, we removed this factor, which yielded the MSLQ-CAL-LS with five factors (i.e., time management, organization, critical thinking, peer learning, and study environment management), which accounted for 51% of the total variance.

Compared to the original learning strategy subscale structure in MSLQ-CAL, EFA results suggested a different structure of the original metacognitive self-regulation factor. For example, Item 22 (I ask myself questions to make sure I understand the material I have been studying in the class) under the original factor of metacognitive self-regulation unexpectedly loaded onto critical thinking. Because critical thinking refers to the extent to which students apply previous knowledge to solve new problems and make an evaluation, it is meaningful why this item loaded onto critical thinking factor with medium factor loading. Item 52 (When I study for this class, I set goals for myself in order to direct activities in each study period), under the original factor of metacognitive self-regulation, loaded onto time management. The word period translated into Chinese generally refers to a length of time, which might explain students’ understanding of the relevance with their management of time.

Therefore, the final revised MSLQ-CAL-LS included five factors and 22 items. The standardized factor loading estimates of the final learning strategy scale were significant and fell within the range from 0.40 to 0.86 (see

Table 1. Standardized factor loading estimates in learning strategy scale.

Item	Critical Thinking	Organization	Time Management	Study Environment	Peer Learning
% of variance	4.37	3.70	30.56	6.10	6.23
Eigenvalue	1.27	1.07	8.86	1.77	1.81
Cronbach’s alpha	0.77	0.79	0.81	0.75	0.74
Item14	0.52
Item22	0.45
Item29	0.67
Item44	0.48
Item55	0.72
Item10		0.65
Item25		0.58
Item40		0.54
Item39			0.40
Item52			0.65
Item63			0.42
Item68			0.63
Item71			0.75
Item15				0.53
Item30				0.54
Item45				0.86
Item64				0.52
Item69				0.50
Item11					0.56
Item26					0.59
Item41					0.65
Item54					0.67

). The internal consistency of the overall model is 0.90 and ranges from 0.74 to 0.81 across five factors.

Research Question 2: How well does the self-regulated learning strategy model fit Chinese adult learners?

The revised model of the learning strategy scale was subjected to a maximum-likelihood CFA, and the factors were set to correlate freely. According to the pre-determined cutoff criteria, CFA results revealed a satisfactory model fit, with χ²(99) = 783.933, p < 0.001, CFI = 0.923, RMSEA = 0.054, and SRMR = 0.043. Factor loads were presented in Figure 1.

Research Question 3: Does the MSLQ-CAL-LS construct differ between female and male Chinese adult learners?

CFA was run separately for each gender group, which yielded a satisfactory model fit for each group. For the male group, the model fit indices are χ²(199) = 472.34, p < 0.001, RMSEA = 0.058, CFI = 0.912, SRMR = 0.052. For the female group, the model fit indices are χ²(199) = 521.26, p < 0.001, RMSEA = 0.054, CFI = 0.924, SRMR = 0.044. All of the factor loadings were positive and statistically significant in both female and male models. Mean, standard deviation, and factor reliability across the female and male groups are listed in

Table 2. Mean, standard deviation, factor reliability estimates for female and male group.

	Time Management		Organization		Peer Learning		Critical Thinking		Study Environment Management
	Female	Male	Female	Male	Female	Male	Female	Male	Female	Male
M	4.47	4.38	4.46	4.21	4.80	4.70	4.67	4.69	5.04	4.86
S.D.	1.09	1.04	1.26	1.32	1.05	1.11	0.92	0.97	1.02	0.99
α	0.824	0.790	0.782	0.789	0.824	0.742	0.780	0.771	0.755	0.728

.

Configural invariance. Given the good fit of the 22-item 5-factor model in both female and male groups, MI was tested starting with configural invariance to examine factor structure across the female and male groups. As is shown in

Table 3. Fit Indices for Measurement Invariance Tests.

Model	χ2 (df)	RMSEA	SRMR	CFI (ΔCFI)	Comparison	Decision
Model 1: Configural invariance	993.606 (398)	0.056	0.048	0.919		Accept
Model 2: Metric invariance	1018.656 (415)	0.055	0.051	0.918 (−0.001)	Model 1 vs. Model 2	Accept
Model 3: Scalar invariance	1059.127 (432)	0.055	0.053	0.915 (−0.003)	Model 2 vs. Model 3	Accept
Model 4: Strict invariance	1107.600 (454)	0.055	0.056	0.911 (−0.004)	Model 3 vs. Model 4	Accept

, all fit indices suggested a satisfactory model fit (CFI = 0.919, RMSEA = 0.056, SRMR = 0.048), indicating a same factor structure in both groups.

Metric invariance. Given that configural invariance was supported, metric invariance was tested where factor loadings were constrained to be equal across female and male groups. ΔCFI between the baseline (i.e., configural invariance) and metric invariance models is less than 0.01 (Table 3), indicating an adequate model fit [55]. We conclude that metric invariance was established.

Scalar invariance. Given that metric invariance was supported, scalar invariance was tested where intercepts of the items were constrained to be equal across female and male groups. The difference in CFI between the scalar invariance and the metric invariance models was less than 0.01, suggesting that the scalar invariance was supported (see Table 3).

Strict invariance. Finally, strict invariance was tested where item uniqueness was set to be equal across female and male groups. The difference in CFI between strict invariance and scalar invariance was less than 0.01. Therefore, strict invariance was supported (see Table 3).

5. Discussion

Self-regulation is considered central to the 21st century learning to foster students to become sustainable lifelong learners [8]. The purpose of the present study was to extend the work of Tong et al. [11] to further validate the subscale of learning strategies (MSLQ-CAL-LS) that could most accurately measure Chinese adult learners’ use of SRL strategies under diverse learning modes and contexts. We included the following factors from the original MSLQ-CAL: metacognitive self-regulation, time management, study environment management, organization, critical thinking, and peer learning. Although metacognition is a common SRL strategy [19], nevertheless, it was not identified in our factor analysis. Given that metacognitive self-regulation strategies might be closely related to the change in environment and life events [59], it is not surprising that we were not able to capture this strategy as students transitioned between online and in-person learning modes caused by the COVID-19 pandemic. Such finding is consistent with previous studies that the metacognitive self-regulation scale was separated into various factors under online [30] and face-to-face [27] learning environments.

Results of CFA suggested a satisfactory fit of the 5-factor, 22-item model of MSLQ-CAL-LS that showed improvement in factor loading estimates as well as item internal consistency as compared to the original MSLQ-CAL [11]. This has also enabled us to test MI between male and female groups. We discovered configural, metric, scalar, and strict invariances, which statistically confirmed that both female and male students understand and respond to the instrument in a similar pattern. Gender difference in employing SRL strategies has been reported in the existing literature [42]. Therefore, we believe our validation study contributes to the field by extending the previous work of MSLQ-CAL and presenting strong psychometric evidence of MSLQ-CAL-LS, a solid instrument that can appropriately and adequately measure SRL strategies among both male and female Chinese adult learners.

6. Conclusions and Future Work

This study offers the first step toward an accurate and deeper understanding of students’ learning strategies that can inform educators to provide more personalized academic tasks and enhanced learning experience, as well as strategically plan their instruction for a sustainable, quality education, which, in turn, supports students to become lifelong learners. In this validation of MSLQ-CAL-LS, we addressed student characteristics of gender, which is an important component in the SRL model [15]. Another component worthy of exploration is the classroom context (e.g., instructional approach and instructor behavior). According to Broadbent and Poon’s [21] systematic review on university students’ SRL in online learning environments, strategies of time management, metacognition, and critical thinking are positively related to academic performance. At the time of data collection, students in our study had just transitioned back to in-person instruction after a semester-long online learning due to the pandemic, and they may have used different SRL strategies in response to the specific classroom context. Although MSLQ-CAL-LS is found to adequately measure SRL, we call for a systematic statistical approach (e.g., measurement invariance) to refine MSLQ-CAL-LS that can be applicable in both online and in-person learning because when the context changes, so does instruction [60,61].

We acknowledge the limitation in this study that participants came from one coastal area, which is not sufficiently representative of the large population of Chinese adult learners from inland regions. As our work continues, we plan to collect data with a more diverse student sample, and to further investigate the predictive validity of MSLQ-CAL-LS on students’ academic performance.