Predictors of the Effectiveness of Different Approaches to Pandemic Distance Learning

Abstract

Significant attention has been devoted to the forced switch to distance learning as a result of the COVID-19 pandemic. However, some aspects of this issue that are very important for practice are still understudied. The aim of this study is to describe the development of an online-available screening tool which could help the teachers to identify the students at risk of lowered effectiveness during the distance learning and also to select an appropriate teaching approach for the given class. A complex survey involving 35 teachers of Czech language, German language, Mathematics, Physics and Geography, and more than 1400 of their students from 70 classes, was carried out. In the first step, we identified which out of the more than 100 potentially relevant variables have predictive value for the effectiveness of distance learning. Subsequently, a series of multilinear regression models enabling to quantify the impact of the individual variables on effectiveness and perceived usefulness of distance learning were developed. Moderation analysis was also used to model how suitable synchronous and asynchronous activities based on active learning are for classes with different characteristics. Based on the results of the models, a simple screening tool helping teachers to tailor their approach and strategy is being developed.

1. Introduction

The COVID-19 pandemic, which we can certainly describe as unprecedented in modern history due to its scale, has had a fundamental impact on the functioning of society throughout the world. Its manifestations are especially significant in the field of education due to forced interruption of traditional face-to-face teaching and the necessity to switch to distance learning. During the pandemic, according to UNESCO’s summary statistics, schools were closed in the vast majority of countries in the world (more than 90%), and this affected approximately 1.6 billion students [1]. Teachers were forced to adapt to distance learning in a short amount of time regardless of their previous experience [2], which had a significant effect on their mental state and well-being [3]. On the other hand, it is possible to understand the pandemic as a great opportunity for the development of distance learning [4], which had never been implemented as vastly as during the pandemic. In connection with this, the question of its effectiveness comes to the forefront.

Considerable attention had already been paid to the effectiveness of distance learning in the period before the pandemic. Pioneering research synthesis of Russell [5] involving the findings from 355 studies found no difference in effectiveness between distance and face-to-face learning. In their comprehensive meta-analysis, Bernard et al. [6] found a small significant negative effect for synchronous distance learning and a small significant positive effect for asynchronous distance learning. A very highly cited meta-analysis focusing on online learning by Means et al. [7] revealed that purely online learning was as effective as face-to-face instruction, whereas blended approaches are more effective than entirely traditional classroom teaching. Finally, the recent meta-analysis of studies carried out before the COVID-19 pandemic by Badti et al. [8] using robust statistical methods such as moderation analysis brought detailed insight on the successful implementation of online learning. It was found that distance learning is effective, especially in the field of natural and technical sciences, with no significant effect of the educational level and intervention duration. However, a general problem of effectiveness research in the period before the pandemic is the so-called self-selection bias [9], because only students and lecturers who were willing to do so and have certain prerequisites for this form of teaching participated. A completely different situation occurred with the arrival of the pandemic, when teachers and students had no other choice than to participate in distance learning and the selection bias was thus removed.

Many empirical studies focusing on the effectiveness of the pandemic distance learning and related issues have been carried out in the last two years [10,11,12,13,14,15,16]. Besides many empirical studies, several reviews and meta-analyses were also presented. For instance, Hammerstein et al. [17] analyzed 11 research studies from spring of 2020, which were focused on the detection of learning losses mainly in the area of the mother language and mathematics. They showed that school closures had had a significant negative effect, especially for younger children and individuals from families with lower socioeconomic status. Stringer and Keys [1] analyzed 20 empirical studies from different countries looking at learning losses due to school closures during the 2nd quarter of 2020. They found that in reading, students are delayed by around 1.5 months, whereas in math it was around 3 months. Newton [18] analyzed 10 empirical studies focusing on learning losses, collecting data in the UK in the autumn of 2020 (thus accounting for UK school closures in the spring of 2020). The results mostly indicate a negative shift of 2–3 months again, with a more negative effect in mathematics and in younger children. Storey and Zhang [19] conducted a meta-analysis of 10 empirical studies on learning loss. From them, they obtained data describing the situation in the USA. They found that significant deterioration occurred in all studies analyzed, with an average decrease of 0.15 standard deviations of the mean. Based on the meta-regression analysis, it was found that the drop in the knowledge of mathematics is probably slightly higher than in that of reading. Patrinos and Donelly [20] presented a review study that included eight empirical studies from 2020 from different countries in Europe and North America and Australia. Deterioration was demonstrated in seven of them, whereas in four of them a deepening of societal inequalities was found. Konig and Frey [21] carried out three-level random-effects meta-analysis of 18 carefully selected studies examining the average effect of the COVID-19-related school closures with respect to several moderation variables. They found an overall significant negative effect with largely insignificant findings from moderation analysis. Yu [22] carried out meta-analysis of the effect of nine different factors on the learning outcomes. It was found that all the factors are significant with self-efficacy in online learning as the most pronounced factor.

Besides empirical studies and their meta-analyses, significant modelling effort was also invested in order to estimate the effect of the pandemic on education. For instance, Kuhfeld et al. [23] estimated, based on a model, that US elementary school students should learn 63–68% of their pre-pandemic reading in the 2019–2020 school year and only 37–50% in math. Kaffenberger [24] then carried out simulations suggesting that a three-month school closure could reduce long-term learning by a full year’s worth of learning. However, a remediation combined with a long-term reorientation of the curriculum could significantly mitigate the long-term learning loss. Tukiran et al. [25] then focused on optimizing education processes based on structural equation modelling and a technology acceptance model. They found that although students have experienced the convenience and benefits of virtual classroom application technology, they do not consider it desirable or show interest in maintaining this situation in the future. The predictive factors of learning effectiveness of pandemic distance learning have been studied using structural equation models by Tsang et al. [11]. They found that the student–student dialogue, course design, and instructor–student dialogue were the key predictive factors of pandemic learning effectiveness. Hongsuchon et al. [26] then used partial least square regression for identification of variables affecting effectiveness of distance learning in a comprehensible model. They found that online learning strategies and motivation have a significant positive effect on learning effectiveness. Elsaher and Sobaih [27] used a structural equation model that opens sources and information in order to demonstrate how remaining motivated, working together and reflection, and knowledge construction were significant predictors of a positive distance learning experience during the pandemic. The E-Learning Success Model in the Context of COVID-19 Pandemic in Higher Educational Institutions was developed by Jaoua et al. [28]. They concluded that effective e-learning is supported by the interactions among four factors: the e-learning system, e-learning readiness, interactivity, and resistance to change.

Despite the significant amount of evidence on the lowered effectiveness of pandemic distance learning compared with conventional face-to-face instruction and its predictors, several questions of high importance for practice remain largely unanswered.

First of all, the emphasis has been placed mainly on the detection of learning losses based on standardized tests and the determination of the basic factors that influence them (age, gender, socio-economic status). This approach brings a lot of interesting findings, but it cannot provide more detailed information on the effect of individual characteristics of students and teachers such as their personality traits, technology acceptance, etc., on the effectiveness of distance learning. Moreover, only data for the most important school subjects such as mathematics and mother language are typically available. Dikaya et al. [29] confirmed a significant effect of communicative (e.g., self-regulation, shyness) and thinking (e.g., right-hemispheric) skills of students on their attitudes towards pandemic distance learning. Osei et al. [30] then tried to integrate personality traits and motivation with the unified theory of acceptance and use of technology to understand e-learning adoption during the pandemic. Using structural equation modelling, they found, among others, that personality is positively related to behavioral intention and actual usage is positively influenced by motivational factors which may be highly relevant, especially for e-learning system designers. However, to the best of our knowledge there is no tool available to regular teachers which could help them identify students put at risk of reduced distance learning effectiveness based on basic information about them and their class and teacher. It was reported that teachers’ data-driven decision making was in decline during the pandemic [31], despite the fact that the teachers wished to receive a wide range of additional academic, social–emotional, and familial data as a means of improving their decision making.

Secondly, distance education is understood very often as a whole, but it actually includes a broad range of diverse approaches that can suit different classes depending on the personality characteristics of students and teachers, their approach to technology, etc. Only a small amount of attention has been paid to stratification between the different approaches, and the findings here were largely inconclusive. For example, Lin [32] confirmed significantly higher effectiveness of pandemic active online learning in comparison with passive listening to lectures in the group of dental students, whereas Vodovozov et al. [33] argued that the active learning approach is not necessarily the best method of teaching and learning when applied to students with great differentiation and discussed the conditions which must be fulfilled for effective active learning. Similarly, Demirtas and Turk [34] demonstrated comparative advantage of the asynchronous pandemic distance learning in a quasi-field experiment at a state university in Turkey, whereas Baxter and Hainley [35] confirmed the higher effectiveness of the synchronous approach at a university in the United Kingdom. Again, we are not aware of any study or research tool which could help teachers select an appropriate approach (for instance, in terms of prevalence of the use of active learning or an optimal synchronous to asynchronous teaching ratio) based on data on their students, school environment and themselves.

Based on the discussion above, the research questions of this study are as follows:

• Which are the predictors of lowered effectiveness during distance learning on the side of students and teachers?
• How to tailor teachers’ approach to distance learning in order to optimize its effectiveness and/or perceived usefulness by teachers and students?

The practical outcome of the complex research described here should be a simple screening tool available online which should help teachers after the input of relevant data to identify students at risk of lower effectiveness and to select appropriate approach to teaching in terms of active learning methods use and the synchronous-to-asynchronous teaching ratio.

2. Materials and Methods

2.1. Development of the Questionnaires for Teachers and Students

The project leading to the development of the questionnaires was started in October 2020, with recruitment of teachers of Czech language, German language, Geography, Mathematics and Physics. For each subject, seven teachers at schools in Pilsen, Karlovy Vary and Ústí nad Labem regions of the Czech Republic (an area approximately covered by the Faculty of Education in Pilsen as an institution organizing the research presented here) were recruited. The inclusion criteria, besides willingness to participate in the project for a reimbursement, were also having taught the given subject in at least two classes during the pandemic distance learning since March 2020 (i.e., in school years 2019/20 and 2020/21) and at least two years of pedagogical experience before the pandemic in order to be able to compare the teaching outcomes before and after the pandemic. The recruitment was based on the snowball method using our previous contacts and experience with the teachers, and a maximum of two teachers (in total for all subjects) from one school were recruited. We aimed to involve teachers with a broad range of experience with and attitudes towards distance learning in order to have a sufficiently heterogeneous sample close to the makeup of the whole population of the Czech teachers of the given subjects.

We carried out focus groups with the teachers involved in December 2020 as a starting point of the development of the questionnaires. The focus groups were held separately for each subject. A subsequent content analysis of the findings from the groups resulted in identification of the main factors potentially affecting the effectiveness of distance learning from the teachers’ point of view. Based on these findings and an extensive literature search by research team members, a pool of more than 250 items which could be included in questionnaires for teachers or students was prepared in March 2021. In the next step, each of 11 members of the research team including experts in educational research of the individual subjects, an educational psychologist, experts in distance learning, as well as a data analyst, were asked to evaluate each of the items in terms of its suitability for the research. A removal of items judged as less relevant then resulted in pre-final versions of the questionnaires. These versions were piloted with ten students and two teachers each, and findings from this step were applied during the finalization of the research tools in April 2021.

An especially important challenge was the measurement of the effectiveness of distance learning. In the Czech Republic, there are no standardized tests available for studied subjects which could enable to directly compare results of the students after pandemic distance learning with outcomes from previous years with standard face-to-face instruction. For this reason, we have taken into consideration many approaches to defining effectiveness of distance learning. Noesgaard et al. [36] presents, based on a literature analysis, 19 different approaches to define the effectiveness of distance learning. Learning outcomes measured on the basis of pre-test and post-test or comparison with test results from surveys in previous years are most often used as a criterion of effectiveness. However, quite often the evaluation of effectiveness is based on the subjective perception of the students, their attitudes towards the given subject, or on the basis of satisfaction with the implemented course. On the contrary, only very little attention is paid to cost-effectiveness approach quantified by the assessment of outputs and resources necessary to achieve them. For the purpose of our research, we decided to focus on cost-effectiveness because of our expectation that the switch to pandemic distance learning could result in a significant change especially in time needed to achieve the given outputs for both teachers and students. This cost-effectiveness criterion described in detail below was then based on the subjective perception of teachers and students.

The final version of the students’ questionnaire consists of (besides some basic socio-demographics items) four parts:

• Part A—68 items related to the students’ experience with and attitudes towards distance learning, perceived support from the family, class and teacher, equipment for distance learning available, etc., for which the students indicate their (dis)agreement on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree).
• Part B—6 statements reflecting perceived outcomes of distance learning. Specifically, demands of the teacher, how much the students learned, time cost of learning and the overall attitude towards the subject were assessed. A 5-point Likert scale was used for each of the items and all of them were taken relatively to standard learning (for instance, 1 = demands of the teachers were much lower than in the standard in-house learning, 5 = demands of the teachers were much higher than in the standard in-house learning). A composite score of the effectiveness of distance learning by students was computed from these six items (ranging from 6 points corresponding to much higher time cost and much lower learning outcomes at much higher demands to 30 points for the opposite end).
• Part C—evaluation of how commonly 20 selected groups of distance learning activities were used in the distance learning. The list of activities was developed on the basis of the so-called iPadagogy Wheel by Carrington [37]. A 5-point Likert scale was used here (1 = activity was not carried out at all, 5 = activity was carried out very often).
• Part D—evaluation of the same 20 groups of distance learning activities as in part C in terms of their perceived usefulness. A 5-point Likert scale was used (1 = not useful at all, 5 = very useful).

The teachers’ questionnaire included also some introductory questions (gender, length of practice, etc.) and seven different parts:

• Part A—10 items related to how the given class or school stands in comparison with other classes/schools in different parameters such as cognitive abilities of students or their socio-economic status. A 5-point Likert scale (1 = significantly below average, 5 = significantly above average) was used.
• Parts B and C—7 items each covering how confident the teachers feel on a 7-point Likert scale (1 = not confident at all, 7 = very confident) in different aspects related to distance learning at the beginning of the pandemic (part B) and one year later (part C).
• Part D—a question on the ratio of synchronous/asynchronous distance learning and one item about demands of the teacher on the students relatively to standard face-to-face instruction (a 5-point Likert scale: 1 = much lower demands, 5 = much higher demands).
• Parts E and F—the same as parts C and D in the students’ questionnaire but from the point of view of teachers.
• Part G—assessment of all the students participating in the research and taught by the teacher in terms of their outcomes and activity during distance learning (relatively to standard face-to-face learning) on a 5-point Likert scale (1 = much worse, 5 = much better).

The composite score of the effectiveness of distance learning by the teachers was then computed from two items from Part G (outcomes, activity during lessons) and one item from Part D (demands on the students). The minimum number of points for a student was 3 (much worse outcomes and activity at much lower demands of the teacher), maximum was 15 (much better outcomes and activity at much higher demands of the teacher). Some additional information on the development content of questionnaires could be found in our previous study [38], in which some preliminary findings of our research were presented.

2.2. Procedure and Characteristics of Respondents

The data collection was carried out in May and June 2021 out after the finalization of the research tools. All of the teachers recruited for the research selected two classes in which they taught the given subject and were provided with material needed for the data collection. From the ethical point of view, the research was carried out in full accordance with the Helsinki Declaration of 1975. All principals of the schools and parents and guardians of the children involved obtained from the teachers an informed consent in which the aims and methods of the study were clearly described in line with the requirements put on the ethical aspects of research. Contact details of the principal investigator were given in the consent and all participants were instructed not to hesitate to contact him with questions and comments they might have. All principals and an overwhelming majority of parents and guardians agreed with the participation in the research.

In a large majority of cases, the questionnaires were distributed using Google forms and the teachers and students completed them during the standard instruction sessions of the given subject in school (the pandemic distance learning was finished, and standard instruction renewed in Czech schools at the beginning of May 2021). The rest of the participants answered the questionnaires by the pencil-paper method. The teachers usually gave the students unique codes and stated these codes in part G of their questionnaire in order to make the mutual coupling of teachers’ and students’ responses possible. Thus, the identification of the particular students was not available to the researchers at all. Remaining data obtained were fully anonymized and prepared for further analysis.

Initial data screening led to the rejection of approximately 30 clearly biased or largely incomplete questionnaires. As a result, 1421 students from 70 classes and 35 teachers participated in the research. For each subject, we have 14 classes with 7 teachers and the following number of students: Czech language 314 (23.1 per class), Geography 292 (20.9 per class), German language 252 (18 per class), Mathematics (18.4 per class) and Physics 299 (21.4 per class).

In total, 993 (70%) students were attending lower secondary school (314 were in the 7th year of study corresponding to age of 12−13 years and 331 in the 8th year), remaining 428 (30%) were in upper secondary schools (181 in the 10th year of study and 208 in the 11th year). Overall, 489 (34%) students have no siblings living in the same household and having also distance learning, 763 (54%) students have one such sibling, and 171 (13%) have two or more. A total of 489 (34%) mothers and 410 (29%) fathers finished secondary education with a high school diploma; 291 (20%) mothers and 251 (18%) of fathers completed tertiary education. Out of the 35 teachers involved in the research, 12 (34%) were men and 23 (66%) were women. They had on average 11.1 years of pedagogical experience (SD = 7.8 years). In total, 19 (54%) of them cared for one or more children of (pre)school age. Regardless of the subject taught, they had taught on average 6.8 (SD = 3.0) different classes during the distance learning. Groups of students and teachers of the individual subjects were largely comparable in the characteristics mentioned above.

2.3. Data Analysis

Data analysis was carried out in MS Excel 2019 with statistical add-in software Analyse-it Ultimate Edition by Analyse-it Software (Leeds, UK). Data from the paper questionnaires were initially transferred to MS Excel and together with data collected online were screened and coded. Missing data were relatively rare after the initial cleaning representing less than 1% from the total data volume. Due to this low proportion, imputation was not carried out and statistical procedures were always conducted only for the respondents with no relevant data missing for the given model or analysis. Descriptive statistics were computed for all items and correlation analysis was used for identification of the variables having predictive value for effectiveness of pandemic distance learning. We computed correlation coefficients of the particular variables with the composite effectiveness score both by the teachers and by the students. Variables having correlation with score of effectiveness either by the students or by the teachers higher than 0.15 were selected for further analysis. Setting of this threshold is discussed and the computed correlation coefficients are presented in detail in our previous paper [38]. Finally, multiple multilinear regression models with different independent and outcome variables were developed and run. Tests of significance of the regression coefficients were carried out and results with a p-value lower than 0.05 were considered statistically significant. Moderation analysis was used for one of the regression models in order to determine the conditions under which the active and synchronous learning activities are appropriate. A detailed description of this model is given below. The quality of concurrent regression models was judged based on the Akaike information criterion (AIC). Optimization of the parameters in the regression model with moderation analysis was carried out by minimization of the Variance inflation factor (VIF) which described how much the variance of the given regression coefficient is increased because of collinearity. Use of the VIF for model optimization is discussed extensively in [39] and references therein.

3. Results

3.1. Identification and Descriptive Statistics of Predictively Relevant and Outcome Variables

Taking into account our focus on the development of the screening tool for the teachers, we omitted variables having no predictive nature even if they have sufficiently high correlation (see the threshold set in the previous section) (for instance, student-reported item I went through the pandemic distance learning without problem). After this reduction, six student-reported and four teacher-reported variables were kept for the further modelling.

Considering our focus on the distance learning approaches highlighted in Introduction, it was also necessary to set a metric for active learning approach use and the ratio of its synchronous to asynchronous form. In the former case, we develop an active learning use score based on averaging of prevalence of 11 active learning activities (such as work group on a presentation, a long-term project or work of student teams in separate online rooms; further examples are given in [38]) reported by the teachers. We decided to use the teacher-reported prevalence for the model mainly due to significant differences in the corresponding student-reported variable across different students from the same class. Noteworthily, the averaged values reported by students were in good agreement with those of teachers. In the latter case, we used simply the teacher-reported prevalence of the synchronous form (as a percentage of the total learning time).

We computed also the perceived usefulness of the 11 active learning and the remaining 9 passive learning activities (such us independent work of students with a text, explaining of study content with a shared screen, etc.) simply by averaging the values for the individual activities. In this case, we use an average of the teacher-reported and student-reported values. As an indicator of effectiveness, we use the teacher-reported composite score computed in the way described in Methodology, which was found as more consistent than the student-reported score of effectiveness.

An overview of the identified predictive and outcome variables is presented together with the descriptive statistics for the individual subjects in

Table 1. Descriptive statistics of predictively relevant and outcome variables.

Statement/Variable (Name of Variable in Models)	Czech Language (n = 314)	Geography (n = 299)	German Language (n = 252)	Mathematics (n = 257)	Physics (n = 299)	Total (n = 1421)
I often forget to complete my homework (homework)	2.93 ± 1.47 a	3.24 ± 1.50	2.84 ± 1.49	3.42 ± 1.49	3.17 ± 1.47	3.12 ± 1.50
My family does not believe I can manage distance learning (family)	1.47 ± 1.10 a	1.59 ± 1.20	1.43 ± 1.06	1.62 ± 1.21	1.57 ± 1.18	1.53 ± 1.15
Distance learning is for me better than regular in-house learning (distance better)	2.57 ± 1.40 a	2.94 ± 1.54	2.59 ± 1.40	2.50 ± 1.52	2.86 ± 1.47	2.70 ± 1.48
I feel better in online communication than face to face (online communication)	3.00 ± 1.45 a	3.05 ± 1.49	2.74 ± 1.46	2.70 ± 1.52	3.04 ± 1.50	2.92 ± 1.49
The feeling of privacy during distance learning makes me happy (privacy)	4.23 ± 1.06 a	4.08 ± 1.12	4.02 ± 1.14	3.90 ± 1.29	4.13 ± 1.06	4.08 ± 1.14
For me, it is difficult to force myself to work on the given tasks/homework (procrastination)	3.13 ± 1.48 a	3.20 ± 1.45	2.96 ± 1.41	3.28 ± 1.46	3.33 ± 1.44	3.20 ± 1.45
Cognitive abilities of the students in the class reported by the teachers (cognitive)	3.05 ± 0.53 b	3.43 ± 0.79	3.32 ± 0.57	3.28 ± 0.88	3.40 ± 0.93	3.30 ± 0.77
Social–economic background of the students reported by teachers (socio-economic)	3.19 ± 0.40 b	3.38 ± 0.47	3.29 ± 0.45	3.19 ± 0.40	3.15 ± 0.49	3.23 ± 0.53
Methodological support of the distance learning at the level of school reported by teachers (school support)	3.40 ± 0.94 b	3.11 ± 0.62	2.97 ± 0.90	2.38 ± 0.88	3.56 ± 1.23	3.11 ± 1.02
Self-confidence of teacher to manage the distance learning in terms of time management at the beginning of the pandemic (self-confidence)	3.61 ± 1.24 c	4.42 ± 1.26	4.64 ± 1.25	4.70 ± 1.33	4.29 ± 1.19	4.30 ± 1.31
Use of an active learning approach reported by teachers (active learning use)	2.47 ± 0.70 d	3.12 ± 0.46	2.31 ± 0.42	2.76 ± 0.78	2.48 ± 0.43	2.60 ± 0.64
Ratio of synchronous to total learning during the pandemic reported by teachers (synchronous)	0.46 ± 0.35	0.65 ± 0.25	0.73 ± 0.20	0.34 ± 0.31	0.79 ± 0.27	0.59 ± 0.35
Perceived usefulness of an active learning approach by teachers and students (active learning usefulness)	3.37 ± 0.60 e	3.42 ± 0.67	3.33 ± 0.65	3.40 ± 0.50	3.46 ± 0.49	3.39 ± 0.59
Perceived usefulness of a passive learning approach by teachers and students (passive learning usefulness)	3.63 ± 0.38 e	3.72 ± 0.38	3.62 ± 0.52	3.80 ± 0.42	3.61 ± 0.47	3.68 ± 0.47
Effectiveness of distance learning by teachers—composite score (effectiveness)	8.43 ± 1.90 f	7.73 ± 2.14	7.82 ± 2.20	8.20 ± 1.79	8.27 ± 1.70	8.09 ± 1.99

^a mean ± SD, 5-point Likert scale (1 = strongly disagree with the statement, 5 = strongly agree); ^b mean ± SD, 5-point Likert scale (1 = significantly below average, 5 = significantly above average); ^c mean ± SD, 7-point Likert scale (1 = not confident at all, 7 = very confident); ^d mean ± SD, an average from the teacher-reported use of 11 active learning activities on 5-point Likert scale (1 = activity was not carried out at all, 5 = activity was carried out very often); ^e mean ± SD, an average from the perceived usefulness of 11 active learning activities or 9 passive learning activities, respectively, on 5-point Likert scale (1 = not useful at all, 5 = very useful); ^f mean ± SD for the composite score of effectiveness of distance learning by teachers computed from three items (outcomes, activity during lessons and demands on students), range 3–15 points, higher value corresponds to higher effectiveness.

. Note that all subjects were represented by seven teachers teaching 14 classes and the numbers of students is given in the first row of Table 1. The names of the variables to be used in further models is given in parentheses in the first column. Details on the individual variables are presented in the footnotes to Table 1. It may be seen that only few differences occurred across the individual subjects in most variables discussed here. Clearly, a higher prevalence of the synchronous form of distance learning occurred in Mathematics and Physics with a lower proportion than 0.5 in the cases of Czech and German language. The highest reported active learning use was found in Geography, and the same was true also for the methodological support of distance learning. Perceived usefulness of both the active and passive learning approach was comparable, and a similar pattern was observed for all student-reported variables. Relative high self-confidence of teachers in terms of time management was observed for teachers of Mathematics and German language, whereas the opposite was true for Czech language teachers. As a whole, the descriptive statistics presented here suggest a similar pattern for all subjects studied here which could make possible to develop universally valid regression models. The development of such models is described in the next sections.

3.2. Regression Models for Perceived Usefulness of Active and Passive Learning Approach during Pandemic Distance Learning

We developed a multilinear regression model with perceived usefulness of the active learning approach as a dependent variable and six student-reported, four teacher-reported and two variables capturing the approach to the pandemic distance learning (active learning use, synchronous) identified in the previous section as independent variables. The estimated regression coefficients are presented together with the findings from the tests of their significance in

Table 2. Results of a regression model for perceived usefulness of the active learning approach as the outcome variable.

	Estimated Regression Coefficient (Point Estimate)	95% Confidence Interval for the Regression Coefficient	p-Value of the Test of Significance of the Regression Coefficient
Homework	0.012	−0.014 to 0.037	0.374
Family	−0.006	−0.034 to 0.022	0.666
Better distance	−0.001	−0.027 to 0.025	0.914
Online communication	−0.007	−0.033 to 0.019	0.589
Privacy	−0.013	−0.045 to 0.019	0.422
Procrastination	0.005	−0.021 to 0.031	0.709
Cognitive	0.234	0.186 to 0.282	<0.001
Socio-economic	0.134	0.064 to 0.204	<0.001
School support	0.020	−0.012 to 0.052	0.222
Self-confidence	0.026	0.001 to 0.051	0.045
Active learning use	0.409	0.356 to 0.462	<0.001
Synchronous	0.169	0.077 to 0.261	<0.001

Note. p-values lower than 0.05 corresponding to the statistical significance at this level are printed in bold.

. It may be seen that all student-reported variables are insignificant, whereas almost all teacher-reported and approach-related variables are clearly significant at the level of 0.05 with p-values typically lower than 0.001. Specifically, higher cognitive abilities and socio-economic status of the students in the class correspond to higher perceived usefulness of active learning. The same is true for higher self-confidence of the teacher in terms of time-management and also for a more intense use of an active learning approach and synchronous learning.

Subsequently, we developed a multilinear regression model with the same independent variables but with the perceived usefulness of the passive learning approach as the dependent variable. It may be seen from

Table 3. Results of a regression model for perceived usefulness of a passive learning approach as the outcome variable.

Variable	Estimated Regression Coefficient (Point Estimate)	95% Confidence Interval for the Regression Coefficient	p-Value of the Test of Significance of the Regression Coefficient
Homework	−0.012	−0.032 to 0.007	0.219
Family	0.005	−0.017 to 0.027	0.643
Better distance	−0.014	−0.033 to 0.006	0.181
Online communication	0.001	−0.020 to 0.021	0.955
Privacy	0.001	−0.024 to 0.026	0.912
Procrastination	0.009	−0.012 to 0.029	0.416
Cognitive	0.139	0.102 to 0.176	<0.001
Socio-economic	−0.163	−0.219 to −0.107	<0.001
School support	−0.033	−0.058 to −0.008	0.009
Self-confidence	0.072	0.052 to 0.092	<0.001
Active learning use	−0.113	−0.154 to −0.072	<0.001
Synchronous	−0.445	−0.516 to −0.374	<0.001

Note. p-values lower than 0.05 corresponding to the statistical significance at this level are printed in bold.

that all the student-reported variables remain insignificant, whereas the teacher-reported and approach-related variables are clearly significant. Specifically, higher cognitive abilities of the class and self-confidence of the teacher in time management contribute (similarly as in the case of an active learning) to the higher perceived usefulness of the passive learning approach. On the other hand, higher socio-economic background, school support, active and synchronous learning use result in a lower perceived usefulness of passive learning activities. We may conclude that while an active learning approach seems to be more positively perceived in classes with a higher socio-economic background and sufficient school support, more extensive use of this approach combined with synchronous learning results in an increase in its perceived usefulness despite the fact that it remains lower than for a passive learning approach (see Table 1 and discussion in [38]).

3.3. Regression Model for the Effectiveness of Pandemic Distance Learning Moderated by Active Learning Use and Prevalence of Synchronous Learning

In the final step of our data analysis, we focused on the development of a regression model with effectiveness of distance learning as the dependent variable, and with 10 teacher- and student-reported variables and two approach-related variables as independent variables. In order to capture a potential combined effect of class and teacher background, and class pedagogy, we use moderation analysis with active learning use and synchronous learning as moderators between 10 variables reported by teachers and students and the outcome effectiveness of distance learning. The regression model was thus described by the following Equation (1):

$$y={{\displaystyle \sum }}_{i=1}^{10}{\alpha }_i·x_i+{\beta }_1·X_1+{\beta }_2·X_2+{{\displaystyle \sum }}_{i=1}^{10}{\gamma }_i·\left(x_i-A_i\right)·X_1+{{\displaystyle \sum }}_{i=1}^{10}{\delta }_i·\left(x_i-B_i\right)·X_2+const$$

(1)

Here, x_i denotes a teacher- or student-reported variable, X₁ the active learning use variable, X₂ the synchronous variable, α_i, β_i, γ_i and δ_i the corresponding regression coefficients (10 + 2 + 10 + 10 = 32 in total), and finally A_i and B_i constants, which were obtained by an optimization process described later. In this model, positive and statistically significant γ_i mean that a more intense active learning use has a positive effect on effectiveness in the case of x_i value higher than the constant A_i, and a negative effect with x_i lower than A_i. Similarly, a positive and statistically significant δ_i means that a higher synchronous to asynchronous learning ratio has a positive effect on effectiveness in the case of x_i value higher than the constant B_i, and a negative effect with x_i lower than B_i. Determination of the constants A_i and B_i was based on a minimization of the variance inflation factor (VIF) for the given variable. In the first step, the model was calculated with A_i and B_i set in the middle of the scale for the given variable (i.e., four in the case of the variable self-confidence and three for the remaining nine variables). A change in A_i or B_i has no effect on the regression coefficients and the model quality described by the Akaike information criterion (AIC). Taking into account the definition of the VIF and our aim to handle moderation variables as maximal independent parts of the model, we needed to find values of A_i or B_i, for which a minimum of the VIF for variable x_i occurred. For simplicity, the optimization was carried out by changing the A_i or B_i coefficient in 0.1 steps and running the model again and again only for variables with statistically significant regression coefficients. For other variables, A_i or B_i were fixed to the original value in the middle of the scale.

The results from the model are presented in

Table 4. Results of the regression model for effectiveness of distance learning as the outcome variable moderated by active learning use and prevalence of synchronous learning.

Variable	Estimated Regression Coefficient (Point Estimate)	95% Confidence Interval for the Regression Coefficient	p-Value of the Test of Significance of the Regression Coefficient	Variance Inflation Factor for Optimized Ai or Bi Constant (%), the Optimized Ai/Bi
Synchronous	0.585	−0.120 to 1.290	0.104	6.227
Active learning use (ALU)	0.021	−0.402 to 0.444	0.923	7.566
Homework	−0.280	−0.643 to 0.083	0.131	28.923
Family	0.028	−0.398 to 0.453	0.899	24.587
Distance better	−0.021	−0.388 to 0.345	0.910	28.431
Online communication	0.010	−0.355 to 0.375	0.957	29.336
Privacy	0.805	0.406 to 1.204	<0.001	26.048
Procrastination	−0.486	−0.758 to 0.214	0.002	28.335
Cognitive	−0.348	−1.037 to 0.340	0.321	28.657
Socio-economic	3.633	2.460 to 4.807	<0.001	41.573
School support	0.728	0.194 to 1.261	0.008	28.514
Self-confidence	0.857	−1.233 to −0.480	<0.001	24.714
Homework*ALU	0.037	−0.086 to 0.160	0.552	24.814
Family*ALU	−0.069	−0.209 to 0.070	0.331	21.023
Distance better*ALU	−0.008	−0.136 to 0.120	0.904	25.632
Online communication*ALU	−0.005	−0.130 to 0.121	0.943	24.836
Privacy*ALU	−0.092	−0.242 to 0.058	0.227	21.626
Procrastination*ALU	0.072	−0.056 to 0.199	0.269	24.855
Cognitive*ALU	−0.017	−0.266 to 0.233	0.897	24.724
Socio-economic*ALU	0.807	0.436 to 1.178	<0.001	27.592, Ai = 3.1
School support*ALU	−0.140	−0.304 to 0.024	0.094	20.879
Self-confidence*ALU	0.298	0.168 to 0.427	<0.001	20.714, Ai = 3.2
Homework*synchronous	0.031	−0.201 to 0.263	0.793	4.581
Family*synchronous	0.042	−0.218 to 0.303	0.751	5.552
Distance better*synchronous	0.094	−0.143 to 0.332	0.436	4.910
Online communication*synchronous	0.038	−0.201 to 0.276	0.757	5.145
Privacy*synchronous	−0.062	−0.358 to 0.233	0.679	5.856
Procrastination*synchronous	0.113	−0.126 to 0.351	0.355	4.621
Cognitive*synchronous	1.043	0.546 to 1.540	<0.001	4.699, Bi = 3.4
Socio-economic*synchronous	2.599	1.714 to 3.485	<0.001	9.606, Bi = 3.1
School support*synchronous	−0.431	−0.748 to −0.114	0.008	3.898, Bi = 3.2
Self-confidence*synchronous	0.388	0.110 to 0.667	0.006	7.271, Bi = 4.0

Note. p-values lower than 0.05 corresponding to the statistical significance at this level are printed in bold. The symbol * represents the product of the values of the given variables.

. It may be seen that student-related variables privacy and procrastination have significant regression coefficients with a higher level of feeling of privacy and a lower level of procrastination corresponding to a higher effectiveness. The same is true for the teacher-reported variables socio-economic, school support and self-confidence. Use of synchronous and active learning themselves have no effect on the outcome effectiveness. From the point of view of the moderation analysis, we observed that a higher self-confidence of the teacher in terms of time management and socio-economic background of the class in connection with more use of active and synchronous learning result in a higher effectiveness. Moreover, higher cognitive abilities of students promoted a higher synchronous-to-asynchronous learning ratio, whereas effectiveness was increased by a combined action of a higher level of school support and a higher use of an asynchronous learning approach. None of the student-reported variables seem to be sensitive in terms of outcome effectiveness on the use of active or synchronous learning.

In order to assure that none of the variables excluded based on correlation analysis described in Section 3.1. has predictive value, we tried to run models including besides the abovementioned also other randomly selected variables (one by one). In all cases, the quality of the model measured by the AIC decreased and the newly added variable was insignificant suggesting that the identification of the independent variables for the model had been carried out properly. Note that the constants A_i or B_i optimized in cases of statistical significance of regression coefficients using the method described above are relatively close to the middle of the scale for all relevant variables.

4. Discussion and Conclusions

This study is focused on identification of the predictors of a lowered effectiveness during the distance learning (in comparison to standard face-to-face instruction) and on possibilities of tailoring of a teaching approach for a class or school with specific characteristics. Regarding variables significantly affecting the effectiveness, preference of privacy in online communication and tendency to procrastination were identified as the relevant student-reported variables in the corresponding regression model. The negative effect of procrastination on effectiveness is probably not surprising and was also reported by Hong et al. [40] or Melgaard et al. [41]. The latter study in this context also highlights challenges associated with student engagement and the importance of the use of the camera during online lessons. The importance of privacy for effective online learning before the pandemic has been demonstrated by Lorenz et al. [42]. The improved effectiveness of introverted students who are not very active in face-to-face instruction but have no problem with online communication during the distance learning was also highlighted by multiple teachers across subjects in the focus groups organized in the first stage of our research at the end of 2020.

From the point of view of teacher-reported variables, a higher socio-economic status of the class, methodological support of distance learning and self-confidence of the teacher in terms of time management were found to be predictors of higher effectiveness. It is in line with the finding that insufficient development of theoretical backgrounds and methodology of distance learning is a challenge which may be more serious than technical issues [43]. The effect of socio-economic status on the effectiveness of distance learning was then demonstrated in several studies [19,44].

Besides effectiveness, we also focused on the perceived usefulness of two kinds of distance learning activities by the teachers and the students. We found that most common activities during the distance learning were also perceived as most useful by both teachers and students [38] and that the passive learning activities are on average perceived as more useful than the active learning. From this point of view, it is interesting that the prevalence of the use of the active learning approach is a significant positive predictor of the perceived usefulness of active learning activities (see Table 2) and simultaneously a negative predictor of passive learning activities (see Table 3). It suggests that classes with more experience with active learning tend to prefer it over the passive approach, but it is necessary to apply it regularly and in combination with synchronous learning whose prevalence was also found to be a positive predictor for active learning and a negative predictor for passive learning (Table 2 and Table 3). It is true especially for classes with above-average cognitive disposition and socio-economic status. The importance of synchronous learning is then in agreement with the findings of Nguyen et al. [45] that students who have synchronous lessons are more engaged and motivated.

Regarding the effect of the choice of appropriate teaching activities on effectiveness of distance learning, we found through using moderation analysis (largely in line with findings regarding perceived usefulness of an active and passive learning approach) that more often, active learning activities contribute to higher effectiveness in classes with a higher socio-economic status and with teachers highly self-confident in time management. Similarly, a higher proportion of synchronous distance learning increases effectiveness in the cases described in the last sentence and also in the classes with the above-average cognitive dispositions of students. The lack of time-management skills was also mentioned as a significant challenge for pandemic distance learning in a qualitative study of Estonian science teachers [46]. The positive effect of higher cognitive dispositions and socio-economic status is then in line with the conclusions of Vodovozov et al. [33] that only strong students can benefit from active learning in its entirety, whereas others largely require direct, teacher-centered instruction instead of or in addition to active learning.

As in all studies, the research presented here is not free of some significant limitations. First, the composite scores for distance learning effectiveness used in our models was based on a subjective evaluation by the teachers. It would be very valuable to also have results from standardized tests have for comparison. However, such tests are not available for the broad spectrum of subjects and years of study in the Czech Republic. Secondly, despite the complexity of our survey, a large amount of data collected and a significant modelling effort, the quality of regression models measured by the coefficient of determination is not very high. It is probably due to many other intervening variables related to students and teachers, as well as the teaching environment which could not be simply captured by the questionnaire data collection technique. It would be useful to have additional data for example from direct observation of teacher–student and student–student interaction in the given classes for triangulation with the findings from our survey. Finally, although the basic trends presented here seem to be valid for all subjects involved in our study, it is to be expected that subject specifics could play a role in the selection of appropriate teaching techniques and activities which remained uncovered in our study. Further detailed research would be needed to clarify this issue.

Despite the limitations stated above, this study brings valuable findings about the predictors of effectiveness during the distance learning. It also shows the way how to tailor a teaching approach in order to increase effectiveness as well as perceived usefulness of the selected activities by both teachers and students. Finally, it constitutes a solid base for a simple screening tool helping the teachers with the identification of students put at risk of low effectiveness during distance learning and with the selection of appropriate teaching strategies. Development of this tool is currently underway, and it should be finalized and made public at the website of the Faculty of Education, the University of West Bohemia, before the end of September 2022.