Evaluation of Students’ Engagement in Integrated Learning Model in A Blended Environment

Abstract

As new informational conditions contribute to the discovery of new ways to improve the quality of the educational process, a new integrated learning model was elaborated. The purpose of the paper is to evaluate the students’ engagement in a newly-introduced integrated learning model, identify the impact of such a model on students’ learning outcomes, and to determine if students’ engagement levels influence their learning outcomes. For our research we used qualitative and quantitative data of students’ records of professional discipline and English testing, surveys and interviews on behavioral engagement, emotional engagement, and cognitive engagement (N = 63). Results on students’ engagement showed that online activity, especially the online international project, involved students more than face-to-face classes, but at the same time some of them noted that without lectures it would be difficult, or even impossible, to participate in a project. Thus the overall engagement level was quite high. Additionally, an integrated approach positively impacted learners’ outcomes. The correlation analysis showed that learners’ engagement played an influential role and highly impacted students’ learning results. In this case we can conclude that our integrated learning model contributes to students’ involvement in the educational process and, as a consequence, allows them to achieve greater results.

1. Introduction

In the state development strategy of Russia, the most important goals of education modernization are the maximum development of the intellectual potential of an individual, analytical skills, critical thinking, the development of self-analysis skills and awareness of one’s own capabilities, creative ability, initiative with a sense of responsibility for one’s actions, and interpersonal skills [1]. There are new directions in the educational processes of modern universities, which is justified by the need for graduates of educational institutions who speak several foreign languages as part of everyday communication [2]. Thus, in the professional sphere [3] one of the priority areas is the introduction and use of information and communication technologies. New informational conditions contribute to the discovery of new ways to improve the quality of the educational process [4]. To achieve these goals, it is necessary to use such technologies of the educational process, which contribute to the development of students’ independence, their ability to work, taking into account the individual ways of developing educational material, the development of communicative creativity of students and the development of personal motivations [5].

Due to such requirements we elaborated an integrated learning model (Figure 1) that is built on a Content and Language Integrated Learning (CLIL) methodology as a framework and flipped classroom activities, project-based learning as pedagogic tools for creating a blended learning environment. We assume that this model might improve the educational process, students’ engagement and their learning outcomes, as well as stimulate students in professional knowledge and skills and also drive them to continuous education, and to raise self-efficacy and motivation [6].

The first stage of the course was elaborated on the basis of the Moodle educational platform, and the access to preparatory materials was open for students the week before the second stage. All information on the assessment was transmitted to the Course Coordinators’ (CCs’) accounts. The second stage was a face-to-face classroom, which was adapted to the students’ knowledge of initial professional discipline and included more or fewer discussions if the level of students’ confidence in the material was higher or lower, respectively. The third stage was based on the online international X-culture project for students who study International Business, International Management or International Marketing. This project has already proved its efficiency in different countries.

Such an educational model was introduced into the curricula of 4th year undergraduate students in Peter the Great St. Petersburg Polytechnic University in the fall semester (International Business course). Research question of this paper is following: is there a significant influence of students’ engagement on students’ learning outcomes in such a course based on an integrated approach?

Theoretical Background

Khalyapina [7] analyzed existing Russian education approaches to teaching foreign languages based on CLIL. This analysis showed that the introduction of integrated learning models is promising, but it is necessary to take into account the peculiarities of each university and curriculum. Another study [8] showed that blended learning has become one of the most important components for the successful implementation of integrated approaches to learning, since it forms the basis of the entire learning process.

An important value for teaching students in a blended environment is engagement. Fredricks and McColskey [9] noted that although some researchers define it in terms of beliefs and values about the importance of learning, others define it as putting forth effort above and beyond the required minimum.

Based on an analysis of publications in blended learning, Halverson et al. [10] found that about half of their publications mentioned the term “engagement”. Their results also showed that, despite the popular use of this term, in very rare cases, research directly involved participation in blended learning.

Some scientists theorized a multidimensional model of engagement [11,12,13]. According to this theoretical model, engagement is multifaceted, which may include behavioral, emotional, and cognitive aspects [9,14]. According to Fredicks et al. [12], behavioral engagement emphasizes participation, perseverance, and involvement in academic activities. Emotional engagement focuses on positive and negative reactions to peers, teachers, and schools, as well as evaluation of learning outcomes. From the point of view of cognitive interaction, it includes the student’s contribution to the effort to understand the topic. “Cognitive engagement draws on the idea of investment; it incorporates thoughtfulness and willingness to exert the effort necessary to comprehend complex ideas and master difficult skills” [12] (p. 73).

Study engagement predicts various long-term positive outcomes, such as a desire for higher education, consistency in educational ways, better employment opportunities, positive self-perception and well-being, and less depressive symptoms [15,16,17,18,19]. Thus, engagement can have positive, far-reaching consequences even outside the educational context. In addition, it was found that engagement in learning activities is significantly related to academic motivation and functioning: students evaluate their studies, get higher marks and report lower levels of academic abstinence and work evasion [18].

Researchers have recently pointed out the need for using diverse tools to measure engagement [20,21], which leads to this mixed methods study that investigated engagement using a variety of data sources to reflect behavioral, emotional, and cognitive engagement [22]. There are various approaches to measuring engagement, including student self-reports, journal files, and interviews. The advantages of self-reporting are that they are easy to implement at low cost [23], they can evaluate constructs that are not directly observed by researchers [24], and prevent disruption of the classroom’s learning process [14]. However, self-assessment measures fail to capture the engagement process [24]. Compared to subjective indicators, log files from electronic learning management systems can provide more objective and contextual information, for example, how many times a student will enter the system. Although researchers focused on log files for various purposes, they rarely used these files to study engagement. Only recently, Gobert, Baker, and Wickson [20] have developed algorithms to detect engagement in an online research environment for researching scientific inquiries. Therefore, other approaches, such as interviews, have their advantages. Researchers can use the interview technique to obtain detailed information about why students do or do not participate in certain activities, why students differ in interaction behavior and contextual factors that can lead to the engagement or disengagement of students [9]. Taking into account the fact that each approach to measurement has its pros and cons, a number of researchers have recommended using several methods to measure engagement [9,20,21].

Suárez et al. [25] provided a study of student homework engagement that is classified as behavioral engagement. Behavioral engagement, in terms of time, effort, amount of homework done, perseverance, and dedication [26] should have an impact on adolescent academic performance [27]. Suárez et al. [25] propose a structural model in which behavioral engagement in homework mediates between the student’s specific motivational conditions—the student’s motivational conditions and their general academic achievements.

Although among other factors, academic performance may depend on the age of students, the quality of assigned homework, and the procedure used to measure performance, research tends to maintain a positive relationship between work performed and academic achievement [28,29,30,31,32,33,34]. Some studies have found positive relationships [29,30,33,35], and some studies have shown that time spent on homework and achievements may not be related or even negatively related [32,36,37].

Another study [37] examined the emotional engagement of first-year students and its relationship with second-year experience. As a result, Ketonen et al. [38] argued that a higher level of engagement in learning at the beginning of studies also reinforced positive intrapersonal relations between perceived value (which is defined as the importance of the activity as a whole) and positive emotions in the second academic year in addition to the additive effect at the level of positive emotions.

In the current study we would like to share the experience and to fill the existing research gap using a mixed methods approach to take into account both students’ engagement and learning outcomes in a blended environment.

2. Materials and Methods

The study involved 3rd-year undergraduate students from Peter the Great St. Petersburg Polytechnic university, who were enrolled in an integrated learning course (N = 63). Students are 20 years old. The group consists of 37 girls and 26 boys. To obtain the data we used both quantitative and qualitative data (

Table 1. Data collection.

Results	Sort of Data Collection	Type of Data
Behavioral engagement	Pre-lecture test submission in Moodle online system	Quantitative
	Attendance in face-to-face lectures	Quantitative
	Completed peer-evaluation test in online project	Quantitative
Emotional engagement	Motivation factors questionnaire	Quantitative
	Interview revealing their interest to each part of the course	Qualitative
Cognitive engagement	Effort questionnaire	Quantitative
	Interview on students’ willingness to put effort to learn	Qualitative
Learning results	English proficiency testing	Quantitative
	Professional discipline knowledge testing	Quantitative

). We measured behavioral engagement through students’ attendance records of traditional lectures, records of online logins to the Moodle platform and records of completed peer-evaluation tests in the online project. Emotional involvement and cognitive engagement were measured both quantitatively during the survey, and qualitatively through interviews. To determine the students’ learning results we used final testing records on English and International Business. Moreover English tests’ (reading, listening, writing, and speaking) results before the course and after the course were examined and compared to reveal the effect of integrated learning approach.

This paper is based on the following research questions:

• Does the integrated learning model played an influential role and contribute to students’ higher engagement and learning results?
• Is there a significant influence of behavioral engagement, emotional engagement and cognitive engagement on students learning results?

For the analysis descriptive statistics, pair-samples Students’ t-test and Pearson correlation test were conducted.

3. Results

3.1. Behavioral Engagement

The whole course of International Business lasted 15 weeks from the 3 September–24 December. We decided to consider the behavioral engagement of students during the participation in an online project (10 weeks from the 1 October). The results are shown in the

Table 2. Descriptive statistics of students’ behavioral engagement.

Data	Avarage Mean (N = 63)	SD
Pre-lecture test submission in Moodle online system	8	1.14
Attendance in face-to-face lectures	7.38	1.18
Completed peer-evaluation test in online project	8.14	1.28

and Figure 2.

According to statistics students visited more frequently online learning website and contributed to online project than attended face-to-face lecture but, generally, all indicators are quite high.

3.2. Emotional Engagement

3.2.1. Motivation Testing

To get a comprehensive assessment of students’ emotional engagement, everybody from considering groups were offered to complete an online motivation questionnaire anonymously before and after the course. We received 50 responses from students before the course and 57 responses after the course (N = 63). Special statements were created to identify the students’ perceptions about teaching and learning offered, defining five standard indicators for CLIL-learners: desire of learning English after university, anxiety, positive attitude to English, self-esteem, and self-demand. For each of the questions, the participants marked one of the five Likert-scale responses. Questionnaire included two questions for each motivation factor. The maximum score in each factor group was 10. Results are presented in Figure 3.

As we can see from Figure 3 students become more motivated after the CLIL course based on integrated approach. The greatest difference was in the following indicators—self-esteem and anxiety. Students felt themselves more confident and more experienced at the end of the semester.

3.2.2. Interview Revealing Students’ Interest to Each Part of the Course

At the end of the experiment, an interview was conducted to determine the interest of students in each part of the course. For the interview a focus group was determined. We selected six students and conducted a semi-structural interview with the focus group. The focus group consisted of three students who were high achievers and three students who were low achievers according to their final exam scores. The interview lasted about 30 minutes. We as researchers received an agreement to record the conversation from each student for a deeper study of the information received.

The questions suggested to students are the following:

• What parts of the course would you highlight?
• What was the most interesting part of the course? Why?
• What was the most difficult part of the course? Why?
• Did the difficulties encountered during the course cause a desire to overcome them or refuse to perform the task?
• What was the least interesting part of the course? Why?
• Would you like to continue studying in this framework?
• Has your interest increased in relation to the discipline being studied?
• Has your interest increased in relation to the learning process?
• Has your interest increased in relation to your future profession and job?
• Do you find it interesting and useful to participate in such program?

All students clearly identified the following parts of the course: preparatory work in an online environment, attendance at face-to-face classrooms, and participation in the X-culture project. This means that all students understand the structure of the learning process. Four students (three high achievers and one low achiever) identified participation in project work as the most interesting part. One of them said: “Project work allows you to try your knowledge in practice, to apply the acquired skills in solving real commercial problems”. The remaining two students mentioned face-to-face classrooms as the most interesting element of education, since “live communication with the teacher makes the theoretical material read at home clearer”.

As the most difficult part of the course, most students (five of six) identified project work, namely teamwork. Students noted, “Teamwork requires well-developed communication skills that were previously not sufficiently developed to participation in such program. Foreign communication requires special efforts.” However, the difficulties that arose did not frighten the students, but on the contrary motivated them to continue learning and perform the assigned tasks, overcoming difficulties.

When choosing the most uninteresting part of the training, opinions were divided as follows: three students (high achievers) chose pre-classroom activities, two students (low achievers) chose project work, and one student (low achievers) chose face-to-face classrooms. High achievers noted, “Pre-classroom activities contained too detailed material, so a lot of time was spent on reading previously studied material." Low achievers agreed that “participation in discussions and problem solving is impossible in a full-fledged format, since there is insufficient level of knowledge of the English language. To study the professional discipline, the traditional teaching method is more appropriate.” However, five out of six students would like to continue their studies on the method being studied, since, according to students, they developed the necessary skills over the past semester and are ready to fully participate in all parts of the educational process.

All students noted an increase in interest in the studied discipline, learning process and future work. Four out of six students said the proposed studying program was useful and interesting, the other two students still prefer the traditional learning model.

3.3. Cognitive Engagement

3.3.1. Effort Questionnaire

To investigate students’ cognitive engagement we conducted a survey consisted of three items. Each of the items was scored on 5-points Likert scale (

Table 3. Descriptive statistics of effort questionnaire results.

Survey Items	Average Mean (N = 63)	SD
I put a lot of effort preparing to classes on Moodle platform.	3.53	1.7
I was engaged with the topics at hand on lectures.	4.15	1.34
I invest much time in implementing project tasks.	4.74	1.51

and Figure 4).

The results showed that students were more willing to work with project tasks than prepare to classes via online platform. Additionally, the engagement with the topics on lectures was high.

3.3.2. Interview on Students’ Willingness to Put Effort into Learning

At the end of the experiment, an interview was conducted to determine the students’ willingness to put effort to learn. The interview was conducted with the same focus group that was used for the first interview. The interview lasted about 30 minutes. We received an agreement to record the conversation from each student for a deeper study of the information received.

Students were asked the following questions:

• Did this discipline require more effort from you than others?
• Do you think that you paid too much attention to this discipline?
• Do you think that the knowledge gained could be obtained with less effort?
• What part of the training required the most effort?
• What part of the training required the least effort?

All students mentioned that they put much more effort into studying the discipline during the semester compared with other educational programs. However, students noted that the educational results exceeded their expectations. Therefore, students believe that they paid enough attention to a particular discipline.

The most difficult part requiring the greatest amount of effort, according to four students (three low achievers and one high achiever) was preparation to classes on the Moodle platform, since “studying the material individually requires a great deal of concentration, developing the skills of foreign reading, learning unfamiliar vocabulary”. The students also said that the success of the classroom depended on the quality of home preparation. Two students (high achievers) chose the project work as the most time-consuming, since they needed to “put a lot of effort into understanding the problem, as well as finding a solution using the obtained skills”.

As the simplest part, requiring the least amount of effort, all students chose face-to-face classrooms, as in the classroom, students clarified obscure points in the theory studied at home, filled in the gaps in their knowledge, learned deeper the material studied.

3.4. Learning Results

3.4.1. English Testing

As the discipline was taught in English and all communications between participants were totally in English we decided to estimate how the course based on integrated approach influenced the students’ English level. Testing was conducted two times: before the course and after it. Before the experiment, we offered the experimental group of students to identify the level of their English proficiency. The test included the assessment of 4 categories: listening, reading, writing, and speaking. When the course finished students were tested again.

In general, the overall quality of students’ English knowledge in four categories improved (Figure 5).

Results’ comparison of the two tests (before and after the course) of all participants in the experiment indicates that the improvements in listening, reading and speaking were significant at the p < 0.001 level. In the writing category students showed less progressive achievements, but due to Student’s t-test they were also significant at the p < 0.05 level (

Table 4. Descriptive results of pre-test and post-test on English.

Category	Test	Results (Average Mean)	SD	t-Value
Listening	Pre-test	14.5	2.14	5.4 ***
	Post-test	17.71	2.08
Reading	Pre-test	16.32	1.87	5.7 ***
	Post-test	19.02	1.98
Writing	Pre-test	17.1	2.58	2.4 *
	Post-test	17.9	2.5
Speaking	Pre-test	15.54	2.04	6.1 ***
	Post-test	21.88	2.30

* p < 0.05; ** p < 0.01; *** p < 0.001.

). Hence, we can confirm the efficiency of such integrated educational model firstly for English learning purposes.

3.4.2. Professional Discipline Testing

Assessment of the professional discipline (International business course) was in a form of final test consisting of 25 closed questions. These tests were performed through the online platform Moodle by all the learners. Since the course was taught in English, students completed the test in English. Average test results are presented below (

Table 5. Descriptive results of professional discipline.

Testing results	Mean	SD
Professional discipline	72.94	5.63

).

As one of our research goals was to determine the impact of students’ engagement on their learning results we conducted the Pearson correlation analysis and calculated the significance of indicators’ influence (

Table 6. Correlation analysis of students’ engagement and their learning results.

	Login (Moodle)	Attendance	Login (Online Project)	Emotional Engagement	Cognitive Engagement	English Testing	Professional Discipline Testing
Login (Moodle)	1
Attendance	0.16	1
Login (online project)	0.41 ***	0.4 ***	1
Emotional engagement	0.15	0.09	0.34 **	1
Cognitive engagement	0.02	0.21 *	0.17	0.11	1
English testing	0.11	0.23 *	0.26 *	0.32 **	0.20 *	1
Professional discipline testing	0.14	0.34 **	0.28 *	0.28 *	0.29 *	0.72 ***	1

* p < 0.05; ** p < 0.01; *** p < 0.001.

).

The correlation analysis revealed a strong relationship between attendance of lectures and participation in the online project, as well as between login to the online project and login to the Moodle platform. Thus, most students who were actively taking part in an online project also attended face-to-face classes and learned materials using the online platform. At the same time the frequency of logins to the online project had a high influence on emotional engagement and students’ learning outcomes. Emotional engagement and cognitive engagement significantly predicted English testing (p < 0.01; p < 0.05) and professional discipline (p < 0.05; p < 0.05) testing results while the relationship between emotional engagement and cognitive engagement was quite weak (R = 0.11). Cognitive engagement also impacted the attendance of face-to-face classes.

4. Discussion

In our study we aimed to identify the level students’ engagement in recently developed and introduced integrated educational model as well as their learning outcomes, notably professional discipline knowledge and English proficiency, as the discipline was taught in English. The measurement of students’ engagements showed the following:

• An analysis of behavioral engagement determined that students more actively take part in online learning—on average 80% of required tasks were completed online (login to the Moodle platform in each course and weekly implemented peer evaluation tests in the online project);
• A survey of emotional engagement showed that students felt themselves to be interested in learning and were more confident and more experienced after the course;
• Interviews revealing students’ interests to each part of the course determined that the considered methodology really increases the interest of students in learning, but not all students are ready to study in this mode. This technique is new and creates an unusual learning environment. It should probably be implemented in earlier courses so that students would be more prepared. It is also necessary in the English language classroom to pay more attention to the development of foreign language communicative competence.
• A survey of cognitive engagement revealed that students spent a great deal of time on working and implementing tasks during the semester, especially on the online project.
• Interviews on students’ willingness to put effort into learning showed that the use of this methodology requires students to put in more effort, but these efforts are justified by the results of studying.

5. Conclusions

As in many studies in this field, our work indicates the influence of students’ engagement on their learning outcomes and motivation. Researchers have recently pointed out the need for using diverse tools to measure engagement [8,13,19,21], which leads to this mixed methods study that investigated engagement using a variety of data sources to reflect behavioral, emotional, and cognitive engagement. As in the work of Fredricks and McColskey [8] we have found correlations between engagement and indicators of participation (i.e., attendance, teacher ratings of participation).

Unlike some other researches [10,11,15,16,17], we consider only students engagement, as students are more conscious, so we can rely on their questionnaire results. It is noteworthy that there is a difference in the ways of determining the engagement of students in school and university, since schoolchildren are more susceptible to the influence of parents. Our study is characterized by a correlation between different factors of engagement and learning outcomes. However, even when researchers use the same methodology (i.e., self-report surveys), there is variation in how engagement is defined and measured. For example, some of the researchers focus primarily on behaviors such as effort, homework, and attendance. In contrast, other studies include items related to emotional dimensions such as relationships with teachers and cognitive dimensions, such as strategy use. In our study we made an attempt to combine and analyze different types of engagement to make our research comprehensive.

Previous studies of university student engagement showed that intrinsic activity motives predict more positive emotions [20,22,25,37]. Thus, one of the possible mechanisms explaining the extent of engagement may be that students with a high degree of engagement perceive the value of the task with greater likelihood as internal (for example, reading is important because I like to learn new things) and, therefore, positive emotions intensify. On the other hand, students with low student engagement may perceive important tasks more likely to be externally motivated or even under pressure (for example, reading is important because I need to pass an exam). Consequently, these students may experience less positive emotions or even anxiety, even when the value of the task is clear. Thus, based on the results, we assume that the perceived value of the assignment clearly stimulates the emotions of the students, but the overall participation in the studies can have a decisive influence on this dynamic.

According to the gained results on learning attainments we can affirm that educational process based on this model is effective and learners’ records are positive. Having made a correlation analysis of all considered indicators it was identified that learners’ engagement played an influential role and impacted their learning results, so we can conclude that, due to the introduced integrated learning model, students were involved in the educational process and had a high level of engagement and, therefore, achieved high results.

Limitations of the Present Study and Suggestions for Future Research

The theoretical work of other scholars in this field has been a useful resource for planning and designing, and we expect that our study will provide something of value for future researchers, too. Of course, there are some limitations in our study, as it does not take into account a novelty effect—students did not have an experience of learning in a blended environment. Additionally, the sample size was relatively small because it was the first time we implemented such an educational model, and the duration of the course was only one semester.

In our further research we are going to evaluate students’ satisfaction of web-based environments in an integrated learning model and upgrade it due to responses.