Learning Outcomes of a Hybrid Online Virtual Classroom and In-Person Traditional Classroom during the COVID-19 Pandemic

Abstract

This paper explores the effects of using a hybrid virtual/traditional classroom, a blended teaching and learning approach, on undergraduate learning outcomes during the initial stage of the COVID-19 pandemic. The study aimed to investigate the prominent problems that appeared in the process of online teaching and learning and to measure the impact of the COVID-19 pandemic on undergraduates’ learning outcomes in public universities in China. The field of specialization of the subjects of this study is under the rubric of social science and higher education. We designed and conducted a pilot survey to identify students’ perspectives on the key issues and experiences of the use of distance-learning through an online virtual classroom in the initial phase of the COVID-19 pandemic. In addition, we applied a binary logistic regression model to real data from two different economics course exams to measure the short-term impacts of using the two different learning environments on the undergraduates’ performance outcomes. The results indicate a statistically significant negative impact of using virtual classrooms on undergraduate learning outcomes. By contrast, in-person traditional classrooms had more desirable learning outcomes. Moreover, the hybrid approach proved to be more effective than the use of online virtual classrooms alone.

1. Introduction

Over two years has passed since the novel coronavirus-infected pneumonia was first identified in Wuhan City, China, in 2019 [1]. According to the World Health Organization, as of 30 March 2022, over 6 million people across the globe have died from the COVID-19 pandemic, and the number of confirmed cases is still on the rise. The world is now feeling the impact of the pandemic with a new surge of infections of the SARS-CoV-2 Omicron variant in many countries. The eradication of SARS-CoV-2 is not a simple prospect, as the world is a global village, and no individual country can move forward without working with other countries. Pandemic eradication takes the ‘weakest-link’ perspective, in which the active participants are the individual countries investing in epidemic eradication. From this point of view, a chain is as strong as its weakest link, and the overall effort concerning pandemic eradication is equal to the lowest amount of effort across different countries [2].

The pandemic has had a profound effect on education at all levels, especially in the early stages of the pandemic. In the early stages of the pandemic, schools worldwide were forced to take various measures, including social distancing measures, in order to reduce the spread of the virus and to ensure student health and safety. The use of the in-person traditional classroom approach always outperforms the online virtual classroom and distance-learning instruction when it comes to attracting and keeping student interest [3]. However, in the spring of 2020, Chinese higher education institutions were abruptly forced to move from in-person classrooms to online virtual classrooms because of the COVID-19 pandemic, despite the fact that the conventional classroom is preferred by both teachers and students for efficient communication and learning outcomes. Thus, a significant shift toward online learning was prompted by the COVID-19 pandemic, and the virtual classroom automatically became the best option to continue teaching and learning. Hence, the first research question is how a change in the learning environment affects educational activities, especially with the students’ new virtual learning experiences. We investigated this question using a pilot survey.

It is important to note that the authors frequently use the terms ‘short-term impacts’ and ‘the initial phase of the COVID-19 pandemic’. The ‘short-term impacts’ refer to students’ learning outcome variations associated with the blended teaching and learning mode in the initial phase of the COVID-19 pandemic. When the COVID-19 pandemic was first discovered in China, the country did not effectively control the spread of the virus in the first quarter of 2020, but social and economic activities were gradually restored beginning in the second quarter of 2020 [4]. Similarly, in Germany and other countries, beginning in mid-March 2020, governments announced major restrictions to fight against the rapid spread of the pandemic [5]. Hence, on the basis of these facts, we assume late-March 2020 to be the baseline, and we call the time before this baseline ‘the early or initial stages’ and the time after this baseline ‘the later stage’ of the pandemic. Not until the fall semester of 2020 did traditional classrooms gradually begin to be restored on Chinese campuses and beyond. Currently, the virtual classroom remains in use, but only as an auxiliary approach compared with its use in the early stages of the pandemic.

Furthermore, the three modes or environments employed in this research refer to the traditional face-to-face mode; the virtual online mode; and the blended mode, which is a combination of the former two modes. In the spring term (from March to mid-July) of 2020, the delivery was only by online mode, but then in the fall semester (from late-August to December), the traditional face-to-face mode was used. The third mode was the combination of the other two modes, also known as the blended mode. The blended mode here can be understood from three perspectives. First, from the perspective of one entire academic year in 2020, we used the online mode in the spring term and the traditional face-to-face mode in the fall term, which was, for the first time, a combination of the other two modes used. Second, from the perspective of one term (autumn), the above-mentioned blended mode was also applicable in some measures in that it differed from the case prior to the pandemic. Third, from the perspective of economics course teaching and learning practices in China, before the pandemic, microeconomics and macroeconomics always went hand-in-hand; namely, they were delivered in sequential order and always employed the conventional mode.

The two closely related undergraduate microeconomics and macroeconomics courses and the corresponding final exams were assessed at the end of each semester to follow up on the students’ distance-learning experiences and the resulting learning outcomes. We used exam scores to denote the learning outcomes and employed the mean value to assess the microeconomics and macroeconomics learning outcomes. The microeconomics course and its final test were completed in a virtual online classroom, while the macroeconomics course and its final test were completed in a conventional in-person classroom. Hence, a second research question arose: do the blended teaching and learning modes used in the initial stage of the pandemic affect the undergraduates’ learning performance? We further evaluated the impact of the COVID-19 pandemic on undergraduate students’ learning outcomes using a binary logistic regression model. The objective was to explore the causal relationship between the use of a hybrid virtual/traditional classroom and undergraduate learning performance.

Therefore, the objectives of this study were twofold. The first objective was to investigate the student experiences and perceptions of the online open learning environment. The second was to measure the impact of different learning modes and the associated experiences on learning outcomes; we then compared and contrasted this impact with that of the traditional in-person classroom approach. We particularly focus on the key problems associated with distance-learning through a mixed study design, i.e., one pilot survey, followed by two course exams during the 2020 academic year, and use regression model analysis to students’ grade data. To the best of our knowledge, this is the first study to investigate the nexus between learning performance and the use of a blended virtual/traditional learning environment from a holistic perspective for the early days of the pandemic. To date, there is little empirical research evaluating and comparing the relationships between the two hybrid virtual/traditional classroom platforms used during the pandemic to investigate the combined impacts on student learning-outcomes, though at the time of writing this article, a few empirical studies are available that focus on direct influences of COVID-19 pandemic on student learning performance.

This study, therefore, contributes to the literature on the impact of COVID-19 on student learning performance in a mixed hybrid teaching and learning environment. The results indicate a statistically significant negative impact of the use of a virtual classroom on the undergraduate learning-outcomes. In contrast, in-person traditional classroom has more desirable learning-outcomes. Moreover, the blended approach proved to be more effective than using only online virtual classroom.

The rest of the paper is organized as follows. Section 2 provides a literature review. Section 3 presents the details of methodology and describes the dataset and data sources. Section 4 discusses the empirical results and Section 5 offers a discussion of the results. Finally, Section 6 concludes this study.

2. Background Literature

The virtual classroom has received much attention in academia. Researchers have conducted a great many studies focused on the student experience and the learning effects associated with synchronous virtual classrooms. However, various mixed conclusions have been reached. Some studies have found that the virtual classroom environment leads to desirable learning effects. For example, Hiltz [6] reviewed and compared various features of both the traditional classroom and virtual classroom. The findings revealed that the virtual classroom can stimulate and enhance collaborative student learning. Moreover, both teachers and students reported high levels of satisfaction as a result of productive shared contributions. Cao et al. [7] stated that the virtual classroom effectively raises student satisfaction. Lietzau and Mann [8] found that students can learn more and achieve higher marks as long as they engage in synchronous virtual classrooms. Parker and Martin [9] compared the perceptions of undergraduate students using a virtual classroom in a fully online and a blended education course and found that online students rated their perceptions of the characteristics of the virtual classroom higher than students on the blended course.

A number of studies have shown that the virtual classroom is effective in terms of enhancing student learning and achievement [10,11,12,13]. Fidalgo et al. [14] found that most students were interested in taking distance-education courses, though they were somewhat apprehensive. Even during the COVID-19 pandemic, online learning tools, such as Tencent Meeting and Smart Classroom, had a significant impact on student learning performance [15,16]. In another study, Yu [17] revealed that gender, education level, and personality traits might affect student online learning outcomes in the context of the COVID-19 pandemic. In particular, postgraduate students outperformed undergraduate students in terms of online learning outcomes. Because the pandemic has not been fully eradicated worldwide, virtual classroom and distance-learning tools remain of importance in certain critical circumstances, such as during the blockade of a city, while learning or working from home. In such cases, it is vital that teachers highlight students’ perceptions of playfulness or enjoyment during online classes in order to enhance their intention to repeat the virtual learning experience [18].

In addition, course design and structure are important to students’ satisfaction and their learning-outcomes [19,20,21,22,23,24]. Betty et al. [19] found that effectiveness in an online environment is contributive to the development of relationships and the overall success of coursework. Meanwhile, the online learning environment, if it is devoid of personal communication, is inundated with approaches that help students feel a sense of partnership while meeting their learning needs. Van Wart [20] emphasizes the importance of course structure and pre-planning, organization, and consistency, and argues that students prefer virtual class organizations that are modular and repetitive. Kauffman [21] argues that course instructional design to be of great importance. Eom, Wen, and Ashill [22] surveyed 397 university students and found instructor feedback in online courses to be statistically significant; and course structure was found to be the most significant determinant of student satisfaction. Jung [23] argues that online courses are more challenging to organize; students tend to be very critical of what they perceive as any confusion or unclear structure in the distance-learning setting. McMurtry [24] states that good online teachers keep students focused by creating a structured environment that is logically organized, delivered in small chunks, and sufficiently repetitive.

Meanwhile, other scholars have insisted that the virtual classroom has certain disadvantages, such as equipment requirements, the textual and technical skills required, the motivation and regular participation required, and the potential “information overload”. In practice, a semi-virtual classroom may be a good alternative for the distance-education environment [6]. In addition to these factors, interaction difficulty may be negatively associated with student satisfaction during internet-based course learning [18]. Learning from a virtual classroom alone may lead to significantly lower grades compared with learning outcomes from a traditional classroom [25].

In the context of the COVID-19 pandemic, Rizun and Strzelecki [26] revealed that the impact of COVID-19 in terms of shifting higher education to virtual learning in Poland was seen in a positive light and was effective. Sprenger and Schwaninger [27] compared e-lectures, the classroom response system, classroom chat, and mobile virtual reality in terms of technology acceptance and found that the classroom response system had the highest level of acceptance, while mobile virtual reality saw a substantial drop in behavioral intention after 3 months of usage. Various practical problems have been commonly encountered in distance-learning during the COVID-19 pandemic. For instance, Sharma and Bumb [28] listed 25 challenges faced in online classes, including a lack of interaction with peers, interruptions in the online classroom, and mental stress resulting from the pandemic. In addition, even though students reported a moderate feeling that virtual learning tools enhanced their effectiveness and productivity, they still preferred to return to a traditional classroom [16,26].

There are a large number of studies focusing on the comparison between online virtual and traditional teaching [29,30,31,32], with mixed results. For instance, Ali Alghazo [29] used students’ final grade to examine the effectiveness of both online and traditional teaching, and concluded that there existed no significant differences in the effectiveness of online education and traditional face-to-face education. Sondoozi [30] showed that the online learning outcomes of students were similar to the performance of those using the traditional approach, while the satisfaction and attitudes of online learners were positive. Feng et al. [31] compared the effects of online teaching during COVID-19 with the pre-pandemic traditional teaching in compulsory education. They found that student’s performance before the pandemic was better than after the pandemic, with many previously high-scoring students now scoring closer to the mean. Zhao et al. [32] compared teaching efficiency between virtual reality and traditional education in medical education. Their results indicated students in the virtual reality group performed better than those in the traditional teaching group, concluding that virtual reality teaching may enhance learning-comes for medical students.

In general, the virtual classroom has gained considerable popularity since 1994. However, despite both teachers and students covering the material set forth in the textbooks and workbooks [33], and later moving from instructor-centered to student-centered curriculum [34], the traditional face-to-face approach remains key in modern education practice. Since both modes have pros and cons, it is best to synthesize the merits of the two modes in higher education. Although the hybrid virtual classroom/traditional classroom mode offers a promising channel for both teaching and learning in various contexts, little effort has been focused on the blended mode and the relationships between learning patterns and performance during the COVID-19 pandemic.

At present, we are more adaptive to the ongoing pandemic environment in terms of reduced fear and anxiety, as well as more protective ways to address this health crisis. More teachers are adapting to blended virtual classrooms in colleges and universities and some are returning to conventional classrooms. However, compared with the currently numerous research regarding the nexus between learning performance and the various learning patterns, studies focusing on the subject in the initial phase of the COVID-19 pandemic are relatively scarce. China was the first country to be shocked by the pandemic, and few empirical studies in the field of higher education were conducted. To the best of our knowledge, there is only one study that is most relevant to this paper. Hence, this research has practical significance in terms of exploring the undergraduate learning performance associated with the initial virtual online teaching and learning experience, as well as the blended learning approach in higher education.

The results of the available literature depend on the models and data used in the studies, and the impacts of the COVID-19 pandemic on higher education in China and beyond remain unclear with mixed conclusions. Additionally, there is no one-size-fits-all model to adopt for examining the effects of online teaching and learning environment on undergraduates’ learning performance. The most relevant models are regression analysis tools [10,15,35]. Chowdhury [10] using binary logistic regression to explore whether virtual classroom can improve students’ learning performance, found most participants have positive opinions about the use of virtual classroom for learning purposes, but this paper does not include the impact of COVID-19 disruption in the model analysis explicitly. In another study, Quadir and Zhou [15] used regression and showed that determinants of technology acceptance models had a positive effect on student learning outcomes in the context of the pandemic. Guse et al. [36] performed the binary logistic regression model on the association of gender, distress, and depression with serious worries in medical students during the pandemic. They investigated mental health outcomes among medical students during the initial phase of the COVID-19 pandemic, and perceptions of the students on how the learning environment had changed in a larger sample of undergraduates. They found that medical students experienced much distress and mental burden during the pandemic.

In this study, we address the following two main research questions, including but not limited to the above mentioned in the pilot survey. The first research question is from the perspective of university students: what are the main environmental determinants affecting teaching and learning activities during the initial stage of COVID-19? The second research question is whether there exists a relationship between teaching and learning environment and student learning performance.

The Null Hypothesis (H₀):

The use of a virtual classroom does not affect student learning outcomes during the COVID-19 pandemic.

The contributions of this study is the use of primary data for the early stage of the COVID-19 pandemic for two successive semesters that can better represent the characteristics of the subjects of the study, and estimates of the impact of COVID-19 in its initial phase on undergraduates’ learning outcomes in a blended mode.

3. Materials and Methods

3.1. Study Design

This study consists of the design of a questionnaire and conduct of a survey to collect primary data, two successive course exams, and a binary logistic regression model. The questionnaire, as shown in

Table 1. Survey design.

Survey Items
Q1. Devices used for distance-learning Smartphone Computer
Q2. Software chosen for distance-learning Tencent Classroom QQ Classroom DingDing Wisdom Tree Rain Classroom Other (Tencent Meeting, Webex Meet, etc.)
Q3. Impact of online class size on learning performance Little impact Somewhat impact Great impact
Q4. Preference for virtual or traditional classroom Traditional classroom Virtual classroom Hybrid virtual/traditional classroom
Q5. Prominent problems of distance-learning Lack of interactions among teacher and students Poor internet connectivity or/and instability Noise distractions during class
Q6. Quality difference between online and traditional teaching Little difference Somewhat difference Great difference

, was developed to identify the outstanding problems experienced by university students when using virtual classroom during the spring semester of 2020 academic year in China. In order to assess the impact of the early stages of the COVID-19 pandemic in higher education, specifically from 21 April to 30 April 2020, an online questionnaire survey was designed and sent to students of more than 10 Chinese mainland universities, including students from Huazhong Agricultural University in Wuhan City, Hubei Province, where the COVID-19 pandemic was first detected, where participants in those regions were consequently better representing the characteristics of the sudden shock of the pandemic. The principle sampling selection of the pilot survey was using a survey platform named wenjuanxing with the website address https://www.wjx.cn/ (accessed on 21 April 2020).

A total of 689 subjects participated in the pilot survey. We had designed six primary questions, as shown in Table 1, relevant to the virtual classroom. Since the COVID-19 crisis is new, no previous model is applicable for the survey. We developed a list of questions focusing on university students, especially undergraduates. The electronic survey was conducted to investigate the main aspects of the virtual classroom being used in the context of COVID-19. For example, the second question was aimed to gather information on the learning software the respondents employed in their virtual classrooms. Key aspects regarding distance-learning were developed to identify outstanding problems, respondents’ preferences regarding virtual and traditional classroom learning, and comparing the impact of class size on learning outcomes.

Following the pilot survey, data were collected from two course exams that were held at the end of the spring semester in the 2019/20 academic year and at the end of the fall semester in the 2020/21 academic year. The microeconomics final exam was conducted in an open-book approach, while the macroeconomics exam was in a closed-book fashion.

Due to the pandemic, students were learning from home in the spring semester of 2020. The microeconomics final exam took place online on 22 July 2020. It took the form of an open-book exam for the first time because of the COVID-19 pandemic, and a total of 794 students enrolled using Tencent Meeting. For the open-book exam (online), students can bring any related printed materials with them to take the exam. Prior to the pandemic, the open-book exam was taken in traditional classrooms. The second exam, macroeconomics, took place on 23 December 2020 in the form of traditional closed-book exam, and a total of 668 students were enrolled. In contrast, a traditional closed book exam requires students not to bring any printed materials with them when entering into the classroom, and complete the exam in a physical classroom. Exam takers receive a security check before entering the classroom. For both classes, the exam format is identical: single choice, multiple choice, true or false, calculation questions, short answers, and longer essays. In total, 1462 students took part in the final microeconomics and macroeconomics exams during the COVID-19 pandemic in 2020. Finally, a binary regression analysis was used to examine the causality between students’ performance and different teaching and learning modes.

3.2. Data Collection

The pilot survey was administered to collect data on student perceptions of the virtual classroom in the early stages of the COVID-19 pandemic. The responses of 689 respondents on the usage of the virtual classroom and outstanding problems associated with distance-learning were analyzed. The exam grades data of students at Qingdao University of Technology were collected at the end of the respective spring and fall semester of 2020. There is an economics course teaching group, which has seven fixed lecturers with the same course syllabus, textbook, test content, as well as the same grading policy to judge the test papers at the end of each semester. Additionally, the seven lecturers grade and mark the exam papers together. Moreover, for this study, the researchers obtained real data, consisting of 1462 observations from students on the economics courses: 794 observations for the microeconomics course in the 2019–2020 academic year and 668 observations for the macroeconomics course in the 2020–2021 academic year. The two courses were taught over two successive semesters during the pandemic. The dataset was set up as panel data, and included variables such as students’ major, gender, and final exam grades. These are available upon request from both liberal arts and STEM major public universities on the Chinese mainland. Economics courses in this university, regardless of current major, are open to all students as long as they apply for the courses to gain credits that count towards their respective degrees. The descriptive statistical results are shown in

Table 2. Statistical descriptions of economics course data.

Variable	Obs.	Mean	Std. Dev.	Min.	Max.	Skew.	Kurt.
Course grade	1462	66.466	15.187	14	99	−0.392	2.739
Microeconomics in 2019–20 academic year	794	63.466	13.136	24	93	−0.257	2.728
Macroeconomics in 2020–21 academic year	668	70.031	16.632	14	99	−0.748	3.021

.

3.3. Data Analysis

The survey data collected from 689 participants were mainly used for the pre-analysis of the impact of COVID-19 on student learning outcomes associated with the virtual classroom. Among the 689 participants, a total of 647 students, or 93.9%, were undergraduate students, and only 6.1% of respondents in the sample were graduate students. As can be seen in Table 2, the mean result of the microeconomics test was about seven points lower than that of the macroeconomics test. Additionally, it is important to note that the microeconomics course was conducted entirely via a virtual classroom and the final test was conducted in the form of an open-book exam, which was held 3 months after the pilot survey. In contrast, the macroeconomics course was held mostly in a traditional classroom, and the final exam was conducted in the form of a closed-book exam. Furthermore, as is often the case, the course grade in the full sample appeared to be asymptotically normally distributed [35]. Specifically, with a respective skewness and kurtosis of −0.392 and 2.739, as is shown in Table 2, the skewness was around zero and the kurtosis centered on 2, which was close to the normal bell-shaped curve. On the basis of the results shown in Table 2, it is obvious that student learning outcomes achieved via a virtual classroom were less desirable than those achieved via a traditional classroom. Thus, the virtual classroom appeared to affect student learning outcomes in the earlier stage of the pandemic. In order to capture and compare the effects of the virtual classroom and traditional classroom on student learning performance, a binary logistic regression model was applied to further measure the nexus between student performance and the use of hybrid virtual/traditional classrooms during the COVID-19 period.

3.4. The Binary Logistic Regression Analysis

On the basis of the course data analysis, as shown in Table 2, it was obvious that the results were asymptotically normally distributed. Since the student learning outcomes associated with the virtual classroom were lower than expected compared with those of the traditional classroom, we took the traditional closed-book test mean value of the macroeconomics test as the baseline, making comparisons of two respective semester grades. That is, if the actual score (from 100 points) was equal to or greater than 70, it was replaced with 1; otherwise, it was replaced with a 0. Alternatively, we treated the microeconomics grade dataset as the treated group, and the macroeconomics grade dataset as the controlled group, as is shown in

Table 3. Transformed dataset of microeconomics and macroeconomics.

Treated group: Microeconomics grade dataset (N1) Full sample: N = 1462; Subsample: N1 = 794. Improved (grade = 1): 266; Unimproved (grade = 0): 528		Controlled group: Macroeconomics grade dataset (N2) Full sample: N = 1462; Subsample: N2 = 668. Improved (grade = 1): 405; Unimproved (grade = 0): 263
Grade	Subtotal		Ratio
0	791		791/1462
1	671		671/1462

.

As a result of the data transformation shown above, we were able to use the logistic model to examine whether the virtual classroom mode affected student learning performance. Given the features of the above data, it was appropriate to take the binary logistic regression model and examine the nexus among the dependent variable denoted by grade, and three explanatory variables: (1) the virtual classroom or traditional classroom denoted by mode; (2) the varieties of specialties (liberal arts and STEM) denoted by major; and (3) gender. In particular, exploring whether the blended pattern affected student learning outcomes was our primary interest. The binary logistic model is as follows:

$$\ln \left(\frac{{\mathrm{P}}_{\mathrm{i}}}{1-{\mathrm{P}}_{\mathrm{i}}}\right)={\mathrm{X}}_{\mathrm{i}}^{\prime }\mathsf{\beta }$$

(1)

where ${\mathrm{X}}_{\mathrm{i}}$ represents the vector form of the explanatory variable, $\mathsf{\beta }$ is the corresponding estimated parameter, and ${\mathrm{P}}_{\mathrm{i}}$ is the function of the grade conditional on mode or/and major or/and gender. Specifically,

$${\mathrm{P}}_{\mathrm{i}}=\mathrm{P}\left({\mathrm{grade}}_{\mathrm{i}}=1{|\mathrm{mode}}_{\mathrm{i}},{\mathrm{major}}_{\mathrm{i}},{\mathrm{gender}}_{\mathrm{i}}\right)$$

(2)

Note that, similarly to the dependent variable grade, mode, major, and gender are all dummy variables by construction. We assumed that the grade variable was 1 if the grade was equal to or was greater than the mean score for macroeconomics, which was 70; and 0 otherwise. Similarly, the mode took the value of 1 if virtual classroom was adopted and 0 if not. As for major variable, 1 represents liberal arts majors and 0 represents STEM majors. Moreover, the gender variable was 1 if the respondent was female and 0 otherwise. The variables are displayed in

Table 4. Descriptions of dependent and independent variables.

Grade	Mode		Total
	Virtual Classroom	Traditional Classroom
0	528	263	791
1	266	405	671
Total	794	668	1462

.

We were able to further derive the marginal effect of the virtual classroom or traditional classroom mode on student learning outcomes either by using the equation below or by computing the corresponding odds ratio (OR):

$$\frac{\partial \mathrm{P}\left({\mathrm{grade}}_{\mathrm{i}}=1{|\mathrm{X}}_{\mathrm{i}}\right)}{\partial {\mathrm{mode}}_{\mathrm{i}}}=\frac{{\mathrm{e}}^{{\mathrm{X}}_{\mathrm{i}}{}^{\prime }\mathsf{\beta }}}{{\left(1+{\mathrm{e}}^{{\mathrm{X}}_{\mathrm{i}}{}^{\prime }\mathsf{\beta }}\right)}^2}{\mathsf{\beta }}_{\mathrm{i}}$$

(3)

The variables in Equation (3) are the same as those in Equation (1).

4. Results

4.1. Results of the Pilot Survey

Table 5. Results of the pilot survey regarding students’ virtual classroom experiences.

Survey Items	Count	Percent
Devices used for distance-learning (Q1)
Smartphone	286	41.51
Computer	403	58.49
Software chosen for distance-learning (Q2, multiple choice)
Tencent Classroom	449	65.17
QQ Classroom	433	62.84
DingDing	354	51.38
Wisdom Tree	232	33.67
Rain Classroom	148	21.48
Other (Tencent Meeting, Webex Meet, etc.)	267	38.75
Impact of online class size on learning performance (Q3)
Little impact	440	63.86
Somewhat impact	218	31.64
Great impact	31	4.5
Preference for virtual or traditional classroom (Q4)
Traditional classroom	302	43.83
Virtual classroom	98	14.22
Hybrid virtual/traditional classroom	289	41.94
Prominent problems of distance-learning (Q5, multiple choice)
Lack of interactions among teacher and students	387	56.17
Poor internet connectivity or/and instability	486	70.54
Noise distractions during class	255	37.01
Quality difference between online and traditional teaching (Q6)
Little difference	178	25.83
Somewhat difference	397	57.62
Great difference	114	16.55

lists key aspects concerning the use of virtual classrooms during the pandemic. For example, regarding online learning software in the early stage of the pandemic on the Chinese mainland, the top two application software was Tencent Classroom and QQ Classroom. As regards to the preference for the distance-learning environment, 43.83% of the respondents preferred the traditional classroom, only 14.22% preferred the virtual classroom, and the remaining 41.94% preferred the hybrid virtual/traditional classroom. Other prominent problems were poor internet connectivity (70.54%), reduced interaction between teacher and students (56.17%), and noise distractions during class (37.01%). The results shown in Table 5 imply that the virtual classroom environment seemed likely to affect student learning performance in the context of the COVID-19 pandemic.

4.2. Results with Controlling Covariates

A binary logistic regression analysis was conducted with the learning outcomes being denoted by the transformed student exam grades as a dependent variable and the mode, which was the virtual classroom or traditional classroom, student major, and gender as the independent variables. The results of the regression analysis are presented in

Table 6. Logistic regression results with controlling covariates.

Model	Coefficient	Odds Ratio
mode	−1.117 ***	0.327 ***
	(0.109)	(0.036)
constant	0.432 ***	1.540 ***
	(0.079)	(0.121)
Observations	1462
Pseudo R square	0.054

Note: *** indicates a 0.01 significance level. Robust standard errors are presented in parentheses.

and

Table 7. Logistic regression results without controlling covariates.

Model	Coefficient	Odds Ratio
mode	−1.117 ***	0.327 ***
	(0.110)	(0.036)
major	−0.499	0.607
	(0.366)	(0.222)
gender	0.323 **	1.381 **
constant	0.682 *	1.977 *
	(0.368)	(0.727)
Observations	1462
Pseudo R square	0.058

Note: ***, **, and * indicate a 0.01, 0.05, and 0.1 significance level, respectively. Robust standard errors are presented in parentheses.

.

Table 6 displays the logistic regression results of grades on mode with controlling covariates, inclusive of major and gender. Furthermore, all regressors shown in Table 6, other than major, are significantly different from 0 at dissimilar levels; we could thus strongly reject the null hypothesis that the virtual classroom does not affect student learning outcomes during the early stage of the COVID-19 pandemic. Furthermore, it was clear that, when the covariates of major and gender were controlled, the mode was significantly negatively correlated with student learning performance. To be specific, when controlling for major and gender, the mode had a significantly negative impact on exam grades, with a parameter of −1.117 and a significance level of 0.01. The coefficient of mode was negative, which implies the use of a virtual classroom may have lowered undergraduate student final exam scores significantly. In essence, the poor internet connectivity and reduced interaction between teachers and students, as mentioned in Table 5, may have jointly contributed to the relatively low mean of student exam grades. In contrast, the constant coefficient was positive (0.432), which means the effect of the traditional classroom was more likely to enhance student learning performance. Intuitively, this also makes sense, as there are more face-to-face communications and peer effects in a physical classroom.

4.3. Results without Controlling Covariates

Table 7 displays the logistic regression results of grade on mode without controlling covariates, inclusive of major and gender. Furthermore, all regressors shown in Table 7, other than major, are significantly different from 0 at dissimilar levels; we could thus strongly reject the null hypothesis that the virtual classroom does not affect student learning outcomes during the early state of the COVID-19 pandemic. Furthermore, it was clear that, when not controlling the covariates of major and gender, the mode was significantly negatively correlated with student learning performance. To be specific, when not controlling for major and gender, the mode had a significantly negative impact on exam grade, with a parameter of −1.117 and a significance level of 0.01. The coefficient of mode was negative, which implies the use of a virtual classroom may have lowered undergraduate student final exam scores significantly. In essence, the poor internet connectivity and reduced interaction between teachers and students, as in Table 5, may have jointly contributed to the relatively low mean of student exam grades. In contrast, the constant coefficient was positive (0.682), which means the effect of the traditional classroom was more likely to enhance student learning performance. Intuitively, this makes sense, as there are more face-to-face communications and peer effects in a physical classroom. Notice that when not controlling the major and gender variables, the mode still significantly negatively impacted the grade, with both the parameter and the significance level remaining unchanged. Additionally, as shown in Table 7, the major variable had a negative impact on learning performance, but this was insignificant. Interestingly, gender had a significantly positive impact on student learning outcomes. This may be due, in part, to the fact that over half of the undergraduates who major in liberal arts such as economics in China are female, and they are generally more diligent and are much better at taking exams.

4.4. The Marginal Effects

Furthermore, in Table 6 and Table 7, we present the aforementioned marginal effect of mode on learning performance in the form of an odds ratio. Regarding the microeconomics course that adopted a virtual teaching and learning pattern, on average, the probability of grade improvement was in the order of 0.355, while the unimproved probability was 0.665. Thus, the odds ratio of microeconomics grade improvement OR₁ was about 0.504. Moreover, the odds ratio of macroeconomics grade improvement OR₂ was about 1.54. In this way, we derived the ultimate OR, i.e., 0.327, which is the ratio of grade improvement of microeconomics relative to that of macroeconomics (OR₁/ OR₂). In other words, during the COVID-19 disruption period, the virtual teaching and learning mode was found to lower the grade improvement with a probability of approximately 67.3%, as compared with that of the traditional classroom.

5. Discussion

As mentioned earlier, we employed an online virtual classroom during the entire 2019–2020 spring term due to the pandemic and resumed a traditional classroom setting in the fall semester of the 2020–2021 academic year. It is important to note that even though we adopted a conventional classroom pattern in the autumn semester of 2020, there was still an online learning section before and after the physical lesson, involving previews and exercises associated with each chapter. Our regression results support the findings from the pilot survey that, given the choice of a traditional classroom, a virtual classroom, and a hybrid virtual/traditional classroom, 44% of the respondents chose the traditional classroom, 42% chose the blended mode, and only 14% chose the virtual classroom. Hence, given the context of the COVID-19 pandemic, with everything else being equal, the virtual classroom learning environment does have an effect on undergraduate student learning performance.

The results of the current study are similar to [32] to some extent, but are quite different from most prior studies [10,15,26,37,38,39]. For example, Chowdhury [10] found that the virtual classroom could produce improvements in student learning and performance. Rizun and Strzelecki [26] concluded that student acceptance of distance-learning during the COVID-19 pandemic in Poland was characterized by enjoyment and efficiency. In another study, Quadir and Zhou [15] identified that a virtual classroom learning environment, such as the Tencent Meeting system, had a significant effect on the student learning performance in terms of the perceived ease of use and perceived usefulness during the COVID-19 pandemic school disruption. Chen and Meng [38] found that task-oriented behaviors accounted for most online learning processes and student engagement levels during virtual classroom learning processes had a statistically significant impact on their learning outcomes. This may be, in part, because of the apprehensions or anxieties of both instructors and students resulting from COVID-19, which, in turn, may have resulted in inactive or somewhat pessimistic learning behaviors and ultimately lower-than-expected performance; or, to put it slightly differently, the emotional state of the students may contribute to their learning process and performance [37]. For instance, our results are in conflict with the findings of [29,39] who found no major differences between the virtual classroom and face-to-face classroom regarding student grades. Our findings are also in disagreement with those of [10,19,27], who indicated that most features of the virtual classroom had a positive impact on student learning. However, ironically COVID-19 that has caused a significant shift toward online learning, has also undoubtedly led to many pedagogical improvements, with instructors gaining more experience with that modality.

6. Conclusions

This study explored the effects of using different modes of learning on undergraduate learning outcomes during the initial stages of the COVID-19 pandemic in China’s mainland. We investigated the short-term impacts of the pandemic on student learning performance when using a virtual classroom, a conventional classroom, and a blended online and traditional classroom. In particular, we assessed the relationship between the use of a hybrid virtual/traditional classroom and undergraduate student learning outcomes, and examined their impact on the undergraduate learning process.

There are three outstanding problems that should be addressed to support the future use of virtual classroom technology to facilitate distance-learning. Namely, 70.54% of the respondents reported that having a better internet connection became the top priority for distance-learning. This was closely followed by the lack of interactions among teachers and students, and then by noise interference or distractions during online classes. Regarding the preference for the three aforementioned formats, the survey results indicated that the responses ranged between positive and negative: in particular, only 14% preferred the virtual classroom, which was the least preferred item among the three options. There was only one promising result in the pilot survey, namely, that 94.5% of respondents perceived that online class size had almost no impact on learning outcomes, which is dissimilar to the physical classroom in terms of a relatively fixed classroom capacity [40].

This study investigated and measured the causal effects between various learning modes and the corresponding outcomes of undergraduate students at a public university in China. Via a virtual classroom, the academic teacher can offer electronic textbooks and handouts and upload PowerPoint presentations and exercises synchronously; however, as mentioned above, as a result of poor network connection or instability and other unpredictable problems, student learning outcomes are less desirable. In comparison, the blended approach proved to be more effective than the virtual classroom alone in terms of learning outcomes. The results from the binary logistic regression model further revealed that, in contrast to many prevailing arguments, there was a statistically negative correlation between the virtual classroom environment and undergraduate student final grades in this study. Additionally, the marginal effect of mode on student learning performance in the form of an odds ratio was 0.327, which was lower than expected as compared to before the COVID-19 pandemic.

Although this research offers valuable findings regarding the causal relationship between undergraduate student learning outcomes and the associated learning environment while using a hybrid virtual/conventional classroom during the COVID-19 pandemic, it has certain limitations. First, the panel data concerning grades was obtained from one public university on the Chinese mainland during the COVID-19 pandemic. As such, the conclusions might not be representative, whether due to sample bias or the unique pandemic period. Second, reliability and validity evaluation of the pre-test survey was not performed due to the multiple-choice nature of two research questions in the pilot survey. For this reason, certain biases may exist within the results. In future studies, these aspects should be looked into.