How Students, in an Air Conditioning and Heating Course, Assess the Impact of the COVID-19 Pandemic on the ABET Student Learning Outcomes

Abstract

The COVID-19 pandemic has forced almost all universities in the world to switch from face-to-face to an online teaching format. In response to this sudden transition, the Mechanical Engineering department at the Al-Zaytoonah University in Jordan had to adapt to this change within two weeks in mid-March 2020. In addition, the department had to comply with the strict ABET accreditation regulations, among which is the indirect assessment of each course by the students each semester. They judge to what degree ABET-mandated student learning outcomes (SO) are actually met in any course designated to deliver the outcomes. Another ABET accreditation regulation is direct assessment, which is conducted by the instructor and relies on an instructor’s own evaluation of tasks designated for certain student learning outcomes. The aim of this study is to determine if, during the COVID-19 pandemic, students’ assessments changed significantly, in accordance with the ABET mandate, by using statistical analyses of students’ responses on an online completed survey. An independent samples t-test was used to evaluate the statistical importance of the observed differences. The SO coverage rates were quite high during the pandemic, although, generally, somewhat lower than their pre-pandemic equivalents. In this study, the Air Conditioning and Heating (ACH) course was selected to compare direct and indirect student learning outcomes before and after the COVID-19 pandemic. Consequently, our findings revealed a positive effect on SO performance during the COVID-19 pandemic and due to e-learning teaching.

1. Introduction

In today’s world, public administrators have to be more effective at learning and training as well as be sensitive to new governance approaches, and capable of responding to new and more complex challenges, because governance is primarily linked to better preparation of public administrators. Therefore, in order to perform at a high level, public administrators should receive the highest standard of skill and preparation. During this process, the institutions that educate and train these people play a vital role and must always strive for excellence.

Katsamunska [1] conducted a study to examine and provide insight into various national approaches to education and training of public authorities in selected countries in the Americas and Europe and concluded that a single dominant pattern of public administration education and training activities could not be identified. In addition, it was concluded that some countries rely heavily on government-dependent and -controlled institutions to provide education and training for public administrators, while certain countries rely heavily on non-governmental or private sector organizations.

Salehi and Largani [2] used descriptive-analytical research methods to examine the present status of e-learning in the higher education system in Iran as well as its structural shortcomings in Iran. The findings indicated that the idea of good governances in Iran has been faced with significant and minor challenges, especially with respect to public awareness, national budget allocation, lack of innovation, and management weakness due to poorly developed education and social infrastructure and political considerations.

The recent COVID-19 pandemic has forced almost every government in the world to take rigorous measures to counteract the spread of this new and deadly disease. In several of the publications there are concerns about the potential adverse impact of the pandemic on student learning. Some authors have focused on policies that have been centrally implemented for their countries.

One research approach is to study the impact of the COVID-19 pandemic on the ABET accreditation regulations, in general, to which all engineering programs have to comply. One ABET accrediting regulation is indirect assessment of each course by the students enrolled in the course each semester; this assessment tool determines whether or not the ABET-mandated student learning outcomes (SO) were met by any course that was designated to deliver them. Another ABET accrediting regulation is instructor-led direct assessment, which is based on the instructor’s personal evaluation of outcomes of tasks assigned for a certain student learning outcome. In this study, an Air Conditioning and Heating (ACH) course was selected to compare direct and indirect student learning outcomes before and after the COVID-19 pandemic, in accordance with the ABET mandate; to achieve this, different hypotheses were set and tested using independent samples t-tests, as indicated in the data analysis section.

2. Literature Review

In an attempt to identify proposals to minimize the negative impact of the COVID-19 pandemic on education, in the last couple of years, intensive research work has been conducted all over the world. Jena [3] listed the government’s actions to ensure seamless education in India during the pandemic, and also realized that some corners of the country were not fully equipped with digital platforms. The author took into account the advantages of online education and proposed that it should continue following the COVID-19 lockdown.

The COVID-19 pandemic was a major challenge for training systems [4]. This perspective provides instructors, officials, and institutions with guidance on how to tackle the crisis. How should institutions prepare, and address, students’ needs by level and field of study over a short time? What preparations should institutions provide? Students’ and parents’ reassurances are essential elements in the institutional response. Daniel [4] concluded that colleges needed to benefit from asynchronous education, which works best in digital formats, by increasing their capacity to teach remotely. The teaching methodology should also include a range of activities and projects, in addition to normal classrooms, which put COVID-19 in a global and historical context. The design of curricula should, first, aid teachers by focusing on student evaluation. This point of view also proposes flexible solutions for the damage to students’ learning pathways after the pandemic comes to an end and provides a list of resources.

Toquero [5] presented how the COVID-19 pandemic had affected higher education and how it could respond to future challenges. He suggested that educational institutions conduct studies on reducing the impact of the pandemic on the educational system and document them. He also stated that it would be increasingly necessary for educational institutions to strengthen curricular practices and to adapt them, even beyond conventional classrooms, to the learning requirements of students.

In Indonesia, Batubara [6] conducted a study using a qualitative approach that included collecting literacy books, as well as report and scientific journals, and the research findings outlined the barriers to implementing this method of learning, namely that students were not prepared for online teaching and learning. He stated that there were weaknesses in adult education on the use of online media to support the students and a lack of online learning facilities. With the provision of Internet networks, considerable costs are required. In the world of education, in the midst of the COVID-19 pandemic, central and local governments must have great concerns.

Marinoni et al. [7] studied the impact of COVID-19 on university education worldwide. There was only one institution (one HEI based in Burundi, out of 424) that indicated their institution was open, as usual, during the COVID-19 pandemic, and 59 percent of the institutions replied that all campus activities had stopped and their institutions had been closed down completely.

In engineering education, studies on online learning have focused on designing and evaluating learning systems and platforms, and on students’ and teachers’ perceptions and feedback [8]. The emerging development of immersive visualizing and interactive technologies and their educational applications has recently been given special attention [9].

Those immersive environments have several reported benefits, including the integration of e-learning, virtual reality, and virtual worlds to improve the concentration and control of learners in the learning environment, developing psychomotor, cognitive, and affective abilities of students, and developing skills in engineering such as problem solving.

In order to study the first students to readily move towards emergency online learning in response to the COVID-19 pandemic, Naji et al. [10] conducted a study using a mixed-method sequential explanatory approach. The study showed a four-factor structure including initial online learning preparation and motivation, self-efficiency beliefs in online learning, self-directed online learning, and online support. The quality results supported the four elements and provided more insight into the diverse and nuanced manifestation of these factors. In view of the perceived effect of readiness factors, significant differences were identified regarding pedagogy; project-based learning (PBL) students reported higher readiness than non-PBL students. The practical consequences of preparing students for future online emergency learning were discussed.

Naji et al. [10] reported on their COVID-19-based transition experience for engineering students in Qatar, listing effective communication, appropriate pedagogical approach, interactive instruction mode, and availability of early help for students striving to solve problems with the transition to online courses.

Battistini and Kitch [11] described the approach by Angelo State University to develop and implement an evaluation plan to successfully satisfy the requirements of the new ABET student learning outcomes 1–7. While examined individually, the authors believe that the documents used in the preparation of virtual visits, especially the ABET student learning outcome binders, would help.

During the COVID-19 pandemic, Sherwin [12] determined the effects of changes on the implementation of a senior design engineering capstone course. In particular, over several semesters, students’ feedback, course evaluations and performances were compared, and analyzed for the delivery mode (face-to-face, hybrid, and online) and changes to the design of the course. Students took part in an online survey to collect their views on standard courses, and the hypothesis that the mode of delivery had no influence on student learning results was tested.

Khandakar et al. [13] discussed feedback on online assessment technologies incorporated with multi-course project-based learning (MPL), due to online teaching during the COVID-19 pandemic (senior electrical engineering (EE) courses at the University of Qatar). These innovative approaches for learning were compared with standard methods of teaching for the same senior course in the previous semester, in order to provide benchmarking and to highlight the results obtained. On the basis of the feedback from previous case studies conducted by teachers and students, the MPL approach supported the students. According to statistical analyses (such as chi-square, two-tailed t-statistics, and z-tests), it was possible to conclude that MPL and online evaluation helped to better achieve the student learning outcomes (SLOs), even in a pandemic situation.

Supernak et al. [14] investigated if the COVID-19 pandemic created any significant changes in students’ assessments of their learning as mandated by the ABET in the Department of Civil, Environment, and Building Engineering (CCEE) at the San Diego State University (SDSU). The method of research was statistical analysis of student scores submitted by an autonomous online survey. They found that the COVID-19 pandemic, in general, did not alter the positive view of students in their classes with regards to the student learning outcome (SO) coverage. However, COVID-19 limitations had a significantly adverse effect on courses and SOs that had to rely on lab experiments and student teamwork.

Alkhalil et al. [15] investigated how engineering students at the Al-Zaytoonah University in Jordan (ZUJ) coped with the e-learning methods adopted during the pandemic by using a questionnaire, which was distributed among engineering students. There was a total of 470 participant who completed the questionnaire. They concluded that the survey responses revealed that students, although there were a few identified challenges, were satisfied with the online teaching provided.

Almanar [16] demonstrated that the learning performances of students at the University of Muhammadiyah (UMT) Tangerang Google Classroom were significantly affected, including computing, autonomous, and critical learners. In addition, the negative impact was due to connectivity, compatible instruments, fundamental computing abilities, and communications between students and teachers.

In West Bengal, India, Kapasia et al. [17] predicted the effects of lockdowns on the learning status of undergraduates and graduate students, with around 70% of students participating in e-learning. Most of the students used android mobile devices to participate in e-learning; they faced different problems relating to depression, poor Internet connectivity, and unfavorable study environments in their home countries. During this pandemic, students from remote and excluded areas have faced enormous challenges most of the time.

Sunoqrot et al. [18] provided an overview of the measures taken by chemistry instructors at the Jordanian pharmacy schools to assure student learning continuity and to accomplish educational results during the COVID-19 pandemic. They aimed to contribute to the collective insights of chemistry instructors around the world in the midst of this common battle by sharing their experiences, so that they may all be better prepared for future disruptive events.

According to the above, it is necessary to explore how students’ learning outcomes (SOs) were affected by the abrupt shifts in the learning process at universities and schools in general. The main goal of the present study is to see how students in an Air Conditioning and Heating course (ACH) evaluate the impact of the COVID-19 pandemic on the ABET student learning outcomes. The findings may provide some insight into the efficacy and perception of e-learning and distance learning strategies for SO assessment.

3. Methodology

One of the ABET accreditation core is the evaluation of the SO, which could be either directly or indirectly assessed. Direct assessment relies on the teacher’s own evaluation results of tasks designated for certain student learning outcomes. Indirect assessment involves students’ own insights into their learning. While not as reliable as the results from direct assessment, indirect assessment information is still a useful instrument in the continuous paradigm of improvement.

A survey is used for each student, in the Mechanical Engineering Department at the Al Zaytoonah University of Jordan (ZUJ), to indirectly assess the degree to which student learning outcomes, in accordance to ABET are achieved each semester. Specifically, the latest version of the ABET regulations require that all student learning should be achieved and evaluated by means of graduation courses, tests, or projects based on seven well-defined student learning outcomes. Additional student learning outcomes are also defined by the respective professional engineering companies for various engineering disciplines. While the ABET regulations do not formally require these additional results, the ABET does apply these additional student learning outcomes.

The ZUJ currently must comply with seven defined student learning outcomes for the mechanical engineering program. For all the engineering programs, the seven student learning outcomes specifically mandated by the ABET have been adopted without amendment.

In the current study, as a case study, we focused on both mechanical engineering and civil engineering and infrastructure students who took the ACH course during the academic years 2019–2020 or 2020–2021. The Accreditation Board of Engineering and Technology (ABET) must accredit each engineering technology program. Because it is a rigorous and challenging certification, the ABET accreditation is highly valued. It evaluates the program’s achievement of eight primary evaluation criteria. Supernak et al. [14] provided more information about the ABET accreditation and the student learning outcomes assessment.

Every semester, the Department of Mechanical Engineering at the ZUJ conducts an indirect assessment of student learning outcomes. The indirect assessment is a survey that asks students to rate how well the course’s student learning outcomes were achieved according to plan. It was decided that each course would need to cover at least one student learning outcome in order to have a logical set of data. During the fall and spring semesters of 2019–2020 and 2020–2021, the data collection process was reliable conducted: the SOs were evaluated by the students in each course, and all the SOs assigned to courses were evaluated on a scale from 1 to 5.

Direct assessment is based on the outcomes of instructors’ grading of assignments and examinations assigned to each SO. As compared with indirect evaluation, direct assessment results are more dependable.

In this study, a comparison between the effect of the COVID-19 pandemic on direct and indirect assessments of an ACH course was conducted using an independent samples t-test. In addition, a check to see if there was a significant difference between students’ class work marks during the semester and students’ final exam marks was conducted by using the same test. The two-tailed independent samples t-test compares the means of two independent groups in order to determine whether there is statistical evidence that the associated population means are significantly different.

4. Analysis and Results

In this study, as a case study, the indirect and direct assessments of the student learning outcomes were collected and analyzed for the fall and spring semesters during the academic years 2019–2020 and 2020–2021. The Air Conditioning and Heating (ACH) course was taken as a sample of the courses offered by the mechanical engineering department; a total number of 324 students were registered in this course and all of them participated in this evaluation.

This course covers Outcome 1 from the SOs criteria 3 of the ABET accreditation regulations. The student learning outcome states: the ability to identify, formulate, and solve complex engineering problems by applying principles of engineering, science, and mathematics. The performance indicator (PI) that is covered in the ACH course is: to apply the Jordanian local codes and the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) codes in air conditioning.

The direct and indirect assessments of Outcome 1 and the performance indicator are presented in Figure 1. It should be noted that the means of students’ marks, in the online and face-to-face examinations, are 69.42% and 57.42%, respectively. In addition, the PI achievement assessments (percentage) for the online and face-to-face classes are 70.26% and 53.57%, respectively. Furthermore, Figure 1 shows the comparison between the indirect and direct assessments of online and face-to-face teaching methodologies. It is noted that the results of direct and indirect assessment were increased by 12.01% and 16.69%, respectively, due to the lockdown and the effect of using e-learning, as well as the fact that this type of course requires solving many mathematical and design problems.

4.1. Independent Samples t-Test for the Mean of Students’ Total Marks

An independent samples t-test was used to compare the means of students’ total marks in the ACH course (direct assessment), which was delivered both online and by face-to-face teaching. The following hypotheses (H1 and H2) were tested for the online and face-to-face teaching methodologies, assuming that:

μ₁ represents the mean students’ mark in the course delivered online;

µ₂ represents the mean students mark in the course delivered by face-to-face teaching;

The difference is μ₁ − µ₂.

H1: 

There are no differences between the means of students’ marks for the courses, which were delivered online and by face-to-face teaching.

H2: 

There are differences between the means of students’ marks for the courses which were delivered online and by face-to-face teaching.

As shown in

Table 1. Testing of hypothesis results between the means of students’ marks for the ACH courses which were delivered online and by face-to-face teaching.

Independent Samples Test
	Levene’s Test for Equality of Variances		t-Test for Equality of Means
	F	Sig.	t	Mean Difference	95% Confidence Interval of the Difference
					Lower	Upper
Equal variances assumed	12.631	0.000	5.339	12.0064	7.5656	16.4472
Equal variances not assumed			5.444	12.0064	7.6438	16.3690

, there is strong evidence (t = 5.339, p = 0.000) that the students score higher marks during the online teaching method, with about a 12-point increase. However, it is possible to get a “mean paired difference” in marks different from 12.0064. Finally, the 95% confidence interval for the difference in the direct assessment is from 7.5656 to 16.4472.

In addition, an independent samples t-test was used to compare the means of the indirect assessment of the ACH courses which were delivered online and by face-to-face teaching. The following hypotheses were tested for the online and face-to-face teaching methodologies, assuming that:

μ₁ represents the mean of the indirect assessments of the course which was delivered online;

µ₂ represents the mean of the indirect assessments of the course delivered by face-to-face teaching;

The difference is μ₁ − µ₂.

H3: 

There are no differences between the means of indirect assessments of the courses which were delivered online and by face-to-face teaching.

H4: 

There are differences between the means of indirect assessments of the courses which were delivered online and by face-to-face teaching.

As shown in

Table 2. Testing of hypothesis results between the means of indirect assessment for the ACH courses which were delivered online and by face-to-face teaching.

Independent Samples Test
	Levene’s Test for Equality of Variances		t-Test for Equality of Means
	F	Sig.	t	Mean Difference	95% Confidence Interval of the Difference
					Lower	Upper
Equal variances assumed	7.309	0.008	4.950	16.6850	10.0281	23.3418
Equal variances not assumed			5.034	16.6850	10.1324	23.2376

, there is strong evidence (t = 4.950, p = 0.008) that online teaching improves marks. In this dataset, online teaching improved marks, on average, by approximately 17 points. Of course, if we were to consider other samples of marks, we could get a “mean paired difference” in marks different from 16.6850. Finally, the 95% confidence interval for the difference in the indirect assessment is from 10.0281 to 23.2376.

4.2. Independent Samples t-Test for the Students’ Class Work Marks

An independent samples t-test was used to compare the students’ evaluation in course work during the COVID-19 pandemic that included assignments, quizzes, and a project, with their evaluation in course work before the COVID-19 pandemic, which was based on two examinations and a project.

The following hypothesis was tested for the online and face-to-face teaching methodologies, assuming that:

μ₁ represents the mean of students’ class work marks for the course delivered online;

µ₂ represents the mean of students’ class work marks for the course delivered by face-to-face teaching;

The difference is μ₁ − µ₂.

H5: 

There are no differences between the means of students’ class work marks for the courses which were delivered online and by face-to-face teaching.

H6: 

There are differences between the means of students’ class work marks for the courses which were delivered online and by face-to-face teaching.

As shown in

Table 3. Testing of hypothesis results between the means of students’ class work marks for the ACH courses which were delivered online and by face-to-face teaching.

Independent Samples Test
	Levene’s Test for Equality of Variances		t-Test for Equality of Means
	F	Sig.	t	Mean Difference	95% Confidence Interval of the Difference
					Lower	Upper
Equal variances assumed	0.046	0.000	7.645	7.9863	5.9231	10.0494
Equal variances not assumed			7.575	7.9863	5.9231	10.0494

, there is strong evidence (t = 7.645, p = 0.000) that online teaching improves students’ class work marks due to changing the class work evaluation during COVID-19. In this dataset, it improved marks, on average, by approximately 8 points. However, if other samples of mark were considered, it is possible to get a “mean paired difference” in marks different from 7.9863. Finally, the 95% confidence interval for the difference in the class work marks is from 5.9231 to 10.0494.

4.3. Independent Samples t-Test for the Students’ Final Exam Marks

An independent samples t-test was used to compare the means of students’ marks on the final exam for the ACH courses, which were delivered online and by face-to-face teaching. The following hypothesis was tested regarding the online and face-to-face teaching methodologies, assuming that:

μ₁ represents the mean of students’ final exam marks for the course delivered online;

µ₂ represents the mean of students’ final exam mark’ for the course delivered by face-to-face teaching;

The difference is μ₁ − µ₂.

H7: 

There are no differences between the means of students’ final exam marks for the courses which were delivered online and by face-to-face teaching.

H8: 

There are differences between the means of students’ final exam marks for the courses which were delivered online and by face-to-face teaching.

As shown in

Table 4. Testing of hypothesis results between the means of students’ final exam marks for the ACH courses which were delivered online and by face-to-face teaching.

Independent Samples Test
	Levene’s Test for Equality of Variances		t-Test for Equality of Means
	F	Sig.	t	Mean Difference	95% Confidence Interval of the Difference
					Lower	Upper
Equal variances assumed	3.654	0.008	1.145	4.0201	1.7571	6.2831
Equal variances not assumed			1.130	4.0201	1.7571	6.2831

, there is strong evidence (t = 1.145, p = 0.008) that online teaching improves final exam marks. In this dataset, it improved exam marks, on average, by approximately 4 points. However, if other samples of marks were considered, it is possible to get a “mean paired difference” in final exam marks different from 4.0201. Finally, the 95% confidence interval for the difference in the final exam marks is from 1.7571 to 6.2831.

4.4. Independent Samples t-Test for the Students’ Class Work and Final Exam during the COVID-19 Pandemic

An independent samples t-test was used to compare the means of students’ marks on class work and the final exam for the ACH course delivered during the COVID-19 pandemic. The following hypotheses were tested regarding the teaching methodology, assuming that:

μ₁ represents the mean of students’ class work marks for the ACH course delivered during the COVID-19 pandemic;

µ₂ represents the mean of students’ final exam marks for the ACH course delivered during the COVID-19 pandemic;

The difference is μ₁ − µ₂.

H9: 

There are no differences between the means of students’ class work and final exam marks for the ACH course delivered during the COVID-19 pandemic.

H10: 

There are differences between the means of students’ class work and final exam marks for the ACH course delivered during the COVID-19 pandemic.

As shown in

Table 5. Testing of hypothesis results between the means of students’ class work and final exam marks for the ACH course delivered during the COVID-19 pandemic.

Independent Samples Test
	Levene’s Test for Equality of Variances		t-Test for Equality of Means
	F	Sig.	t	Mean Difference	95% Confidence Interval of the Difference
					Lower	Upper
Equal variances assumed	0.560	0.008	1.413	1.7308	−0.6882	1.2245
Equal variances not assumed			1.413	1.7308	−0.6882	1.2245

, there is strong evidence (t = 1.413, p = 0.008) that there is a slight difference between the class work and final exams marks during the COVID-19 pandemic. In this dataset, it improved marks, on average, by approximately 2 points. However, if other samples of marks are considered, it is possible to get a “mean paired difference” in marks different from 1.7308. Finally, the 95% confidence interval for the difference between the mean of students’ class work and final exam marks for the ACH course during the COVID-19 pandemic is from −0.6882 to 1.2245.

On the basis of the above results, it can be noted that the assessment of performance indicators and student learning outcomes increased during the semesters when the lockdown was imposed. This may be due to the fact that solving many mathematical and design models in this course are essential. It should be noted that this was in agreement with the findings of Alkhalil et al. [15], who found that the ZUJ students were satisfied with the online teaching and learning instituted by the Al-Zaytoonah University of Jordan, despite a few identified challenges.

5. Conclusions

The primary means of ensuring quality for education is accreditation in colleges, universities, and programs. Students and the public are told that faculty, curricula, student services, and libraries of institutions and programs comply with high standards. As a result, engineering accreditation agencies (such as ABET) and public educational institutions worldwide require programs to assess specific information on learning outcomes for achieving student learning and to establish accountability. One of the outcomes is indirect and direct assessments made by students and instructors, respectively. The main goal of this study was to see how students in an Air Conditioning and Heating (ACH) course evaluated the impact of the COVID-19 pandemic on the ABET student learning outcomes. The findings may provide some insight into the efficacy and perception of e-learning and distance learning strategies for SO assessment.

From this study, the followings may be concluded:

• Online teaching improved the indirect assessment, by 17 points;
• Online teaching improved the scored marks (direct assessment), by approximately 12 points;
• Online teaching improved class work marks, by approximately 8 points;
• Online teaching improved final exam marks, by approximately 4 points;
• There was a slight difference between the class work and final exams marks during the COVID-19 pandemic, the marks improved approximately 2 points.

Consequently, the findings revealed a positive effect on SO performance during the COVID-19 pandemic and due to e-learning teaching.