A Study of Student and Teacher Challenges in Smart Synchronous Hybrid Learning Environments

Abstract

The COVID-19 pandemic has led to the growth of hybrid and online learning environments and the trend of introducing more technology into the classroom. One such change could be the use of smart synchronous hybrid learning environments (SSHLEs), which are settings with both onsite and online students concurrently, where technology plays a key role in sensing, analyzing, and reacting throughout the teaching and learning process. These changing environments and the incorporation of new technologies can place a greater workload on participants and reduce teacher agency. In light of this, this paper aimed to analyze the workload and teacher agency across various SSHLEs. The NASA-TLX model was used to measure the workload in several scenarios. Questionnaires and interviews were used to measure teacher agency. The results obtained indicated that the workload of the teacher tended to be high (between 60 and 70 points out of 100 for the NASA-TLX workload), especially when they lacked experience in synchronous hybrid learning environments, and the workload of the students tended to have average values (between 50 and 60) in the SSHLEs analyzed. Meanwhile, the teacher agency did not appear to be altered but showed potential for improvement.

1. Introduction

The COVID-19 pandemic had a significant impact on global educational environments, especially in higher education [1]. In response to health regulations and social distancing measures, the deployment of hybrid learning environments (HLEs) and online learning environments increased [2]. HLEs have different forms, but synchronous hybrid learning environments (SHLEs) emerged as a particularly popular solution, especially during the COVID-19 pandemic as a trade-off to meet health regulations [3]. These environments allow students to attend classes both online and onsite in real time, providing greater flexibility in learning and better access to educational resources regardless of their physical location [4]. Therefore, the use of SHLEs is not restricted to situations where social distancing restrictions apply but can be beneficial to make access to formal learning more flexible than in traditional educational settings. However, while SHLEs have the potential to support education, they also pose important challenges. For example, the implementation of these environments requires significant investment in technology, infrastructure, and teacher training to ensure an effective learning experience [3]. In the related article by Raes et al. [3], the authors conducted an in-depth study of various SHLEs, contributing further to a better understanding of their complexities and possibilities. Despite the advantages of SHLEs, more research is needed to better understand their impact on student learning and performance as well as to identify best practices in their implementation [5]. This will enable educational institutions to make more informed and effective decisions about how to adapt to the educational challenges posed by the pandemic and to implement more effective long-term learning environments.

SHLEs can be combined with additional technology to collect, process, and provide supplementary information to the teacher, with the aim of enhancing and making learning more flexible. The environments that employ this technology are referred to as smart learning environments (SLEs) [6]. In these environments, technology plays three key roles: sensing—obtaining data such as audio or positioning; analyzing—processing those data; and reacting—using those data to support teachers and students with their pedagogical activities [7]. Key features of SLEs include adaptability, which enables the personalization of learning to meet the individual needs of students; traceability, which allows educators to make informed decisions by monitoring and analyzing data on student performance; and real-time interaction, which enables the real-time completion of tasks and access to educational resources from anywhere at any time [7]. However, the application of SLEs also has disadvantages. For example, their costs can be high due to the need for additional technology and resources. In addition, technical glitches can disrupt learning and create frustration for students and teachers. There are also issues related to the privacy and security of personal data collected and used by SLEs [5]. Overall, the implementation of SLEs can provide significant benefits in the personalization of learning, informed decision making, and access to educational resources. However, these benefits must be balanced with the constraints and considerations of security, privacy, and teacher agency to ensure the effective and sustainable implementation of SLEs.

This study proposes the concept of smart synchronous hybrid learning environments (SSHLEs) by bringing together the advantages of synchronous hybrid learning environments (SHLEs) and smart learning environments (SLEs). SSHLEs enable students to interact synchronously from different locations. Therefore, SSHLEs can offer greater adaptability and support more complex learning experiences [8]. However, the implementation of SSHLEs also presents challenges inherited from SLEs and SHLEs, including the high cost of additional technology and resources required, possible technical issues, and privacy and security concerns [9]. In conclusion, SSHLEs offer a promising approach to enhance the effectiveness of SHLEs, although they are not free from problems related to the methodology used by the teacher, especially when implementing complex strategies such as active learning and collaborative learning [3]. In the context of SSHLEs, enacting collaborative learning situations is particularly challenging because of the complexity involved in coordinating students and ensuring that activities are carried out effectively. Collaborative learning involves a joint intellectual effort by teachers and/or students to carry out activities in a group of two or more [10]. Collaborative learning can be a valuable approach for fostering teamwork and enhancing students’ learning. However, this type of learning requires careful planning and organization on the part of the teacher to ensure its effectiveness [11], and adding technology into the mix may lead to an increased workload. The workload refers to the combination of mental, physical, and temporal demands imposed on an individual by a task, and it concerns the individual’s task effort, frustration, and performance [12]. A high workload can impact the success of collaborative learning and can be affected by various factors, such as the teacher’s level of experience, the type of activity, and the group size [13]. Therefore, teachers and students must receive the appropriate training and support to plan and effectively manage collaborative learning in SSHLEs [14]. Additionally, technology can play a significant role in facilitating this type of learning, providing tools and resources for collaboration and communication between students and teachers [15].

Moreover, it is important to consider that the addition of new elements into the educational environment, particularly the different types of technology needed to implement SSHLEs, can have an impact on teacher agency [16]. Teacher agency refers to the experiences, professional training, resources, culture, social structure, and environment that influence the teacher’s decision-making process [17]. Therefore, any limitations to teacher agency may not only reduce the teacher’s ability to make effective decisions but also negatively affect students’ performance [18]. To mitigate these problems, it is important to implement SSHLEs carefully and strategically, considering not only the technological benefits but also the impacts on the educational process and teacher agency.

The main contribution sought in this study was to find the factors that influence the workload of the teacher and students and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it addresses scenarios not featured in the related literature and is complemented by different analyses of these scenarios. To this end, two research questions were posed:

• RQ1: What factors influence the workload of the teacher and students in SSHLEs that support collaborative learning situations?
• RQ2: What factors influence teacher agency in SSHLEs that support collaborative learning situations?

2. Materials and Methods

This section lays out the foundation of the research process. It provides an explanation of the methodological approaches taken in this study, thus providing the necessary context for understanding the findings and subsequent conclusions.

2.1. Design

Three experiments developed in SSHLEs that included collaborative learning situations were designed. Specifically, a collaborative learning flow pattern (CLFP) called jigsaw [19] was used in two of these experiments, which were adapted from [20], with the objective of measuring the workload and teacher agency in SSHLEs. The jigsaw pattern involves dividing a topic into subtopics, assigning each student a subtopic in which to become an expert, and then grouping experts of each subtopic together to teach each other the various subtopics. To this end, the jigsaw pattern is divided into three jigsaw phases (JPs), as shown in Figure 1.

The first phase of the jigsaw (JP1) is the individual phase. In this phase, the teacher chooses a topic to be addressed and divides it into various subtopics (3 or 4, for example). Subsequently, each student is assigned one of these subtopics, ensuring that approximately the same number of students cover the same subtopic. Once students receive their subtopic, they are given documents with which to learn about the subtopic. This task can be assigned as homework, as it is an individual task.

The second phase of the jigsaw (JP2) is the expert phase. In this phase, students are grouped according to their subtopic, with the potential for more than one group covering the same subtopic. Additionally, all group members are in the same environment, either all online or all onsite, which is one of the major differences compared to a standard jigsaw CLFP. Each group needs to tackle the problems presented that are related to their subtopic.

The third and final phase of the jigsaw (JP3) is the jigsaw phase. In this phase, groups are formed, each of which must include at least one expert on each subtopic. On this occasion, there is a mix of online and onsite students within the same group. In this phase, the groups need to address problems requiring knowledge of all subtopics to be solved.

2.2. Data Collection

Several sources were used for data collection. Logs of the various applications used in each experiment along with the recording and transcription of classes were the first sources of data. These resources showed the number and timing of the teacher interactions with both online and onsite students. Observations of the teachers’ actions also helped triangulate the information on the workload collected through the questionnaires. The flow between teachers, students, and technology was modeled using epistemic network analysis (ENA) [21]. ENA aided in visualizing the structure of connections between codes in the flow data via dynamic network models. The work of Amarasinghe et al. [22] was used as a reference to define these codes due to the great similarity between the design of our experiments and theirs. This in turn allowed a better comparison with other similar works. The activities linked to each code can be seen in

Table 1. Codes of teachers’ actions for the ENA model.

Code	Definition
T.individual.online	The teacher answers a questions posed by an online student.
T.individual.onsite	The teacher answers a questions posed by a onsite student.
T.group.online	The teacher answers a question posed by an online group.
T.group.onsite	The teacher answers a question posed by an onsite group.
T.group.hybrid	The teacher answers a question posed by a hybrid group (some members online and others onsite).
T.class	The teacher addresses all students, expecting a response/reaction from them. Examples: - The teacher requests information from the class; - The teacher gives instructions to the class about the jigsaw phase or about a task that the students have to carry out (switching groups or submitting tasks).
T.announcements	The teacher announces information to the students. Examples: - The remaining time of the activity; - Information about an assignment; - Information needed to complete the task.
T.perception	The teacher checks or monitors the status of the class (both online and onsite).
T.use.tool	The teacher uses some of the features of the tool, such as checking the level of participation and group management.

. Another source used was a questionnaire to measure workload. This questionnaire consisted of the model proposed by NASA-TLX [23], with 6 questions on a scale of 1 to 100, 15 questions of pairwise comparisons among factors to extract variation, a set of demographic questions, and other questions about the activity to facilitate correlation. Another data collection source used herein was the teacher agency questionnaire. The teacher agency questionnaire was based on the work of Hull et al. [24], which is one of the few main articles addressing teacher agency. The main objective of the teacher agency questionnaire was to compare the perception that teachers have of their agency before and after implementing the SSHLE. The teacher agency questionnaire comprised 17 questions concerning certain factors of teacher agency. The teacher was required to respond on a scale of 1 to 5, indicating how much they agreed with each statement. Questions could be framed positively or negatively; thus, a score of 5 on a positive question implied a higher level of agency, whereas a score of 5 on a negative question indicated a lower level of agency. Interviews with teachers were the final source of data collection. Interviews were designed to obtain the data that could not be obtained through the questionnaires and to provide a deeper insight into the teachers’ perceptions of workload and agency. This interview was based on the evaluation concepts proposed by Stake and Jorrín-Abellán [25].

These data sources were used during each experiment. The teacher agency questionnaire was completed before the design of the jigsaw activity and after the activity was carried out. The interview was conducted either before the jigsaw activity or after it concluded. The class recording, transcription, and log collection were conducted during the jigsaw activity. Finally, the workload questionnaire was completed at the end of the jigsaw activity, by both students and the teacher (except in one experiment). The organization and usage of these sources during the experiments is explained in Figure 2.

2.3. Experiments

The three experiments conducted are summarized in

Table 2. Details of the three experiments carried out in the three SSHLEs.

No.	Place	No. of Participants	Time	Motivation	Data Sources	Technologies
1	Belgium	46 (24 onsite and 22 online)	2 h	Studying a setting prepared for SSHLEs, a classroom with the greater incorporation of specific technology to cover hybrid learning, where the teacher and students had more experience in these environments.	- Teacher Agency questionnaires - Teacher workload questionnaire - Teacher interview - Recording activity	- Televisions - Cameras - Speakers and microphone systems - WeConnect software - Participants’ laptops - Teacher’s laptop
2	Spain	17 (9 onsite and 8 online)	1 h	Studying the topics in a classroom with the usual technologies (whiteboard, projector, speakers, and computer) converted into an SSHLE.	- Teacher Agency questionnaires - Teacher workload questionnaire - Student workload questionnaire - Teacher interview - Recording activity	- Whiteboard - Projector - Speakers - Engageli software - Participants’ laptops - Teacher’s laptop
3	Spain	12 (9 onsite and 3 online)	1 h	Studying a scenario involving participants with experience in these environments for a better comparison and to mitigate the lack of data on SSHLEs.	- Teacher agency questionnaires - Teacher workload questionnaire - Student workload questionnaire - Teacher interview - Recording activity

. The first experiment was conducted at the Catholic University of Louvain (KU Leuven, Belgium) due to their experience in SSHLEs and the availability of classrooms with the appropriate technology for these environments [3]. The aim of this experiment was to study a SSHLE, in this case a classroom with the greater incorporation of specific technology to cover hybrid learning, where the teacher and students had more experience in this type of environment. The experiment was carried out during a session of the university course where a collaborative learning situation was to be implemented. There were no restrictions for participating in this session, and so all the students registered in this course were able to participate. This session lasted 2 h and was attended by 22 online students and 24 onsite students. The software WeConnect was used to support this SSHLE [26]. WeConnect includes participation measures, user profile identification, and tools for group management. In this experiment, the teacher completed the teacher agency questionnaire before beginning the design of the experiment. The second part consisted of three collaborative activities, which could not follow the jigsaw pattern because it did not fit into the design of the session created by the teacher. Instead, students solved three problems in groups of four (homogeneous groups: all students were either online or onsite). The information on the activity was recorded. In the end, only the teacher filled in the workload questionnaire, as the university regulations did not allow the collection of student information when it came to an external experiment. After this, the teacher completed the teacher agency questionnaire and the interview.

The second experiment was conducted at Universidad Carlos III de Madrid (UC3M, Spain) and involved participants from Universidad de Valladolid (UVa, Spain); Universitat Pompeu Fabra (UPF, Spain); and UC3M. The aim of this experiment was to study a classroom with typically available technologies (blackboard, projector, speakers, and computers) transformed into a SSHLE with the addition of supporting software. To achieve this, a one-hour workshop was carried out with online (6 students) and onsite (11 students) participants in this classroom. An open call was launched, seeking individuals from collaborating universities to participate in this workshop, with no restrictions placed on participation. The software Engageli was used to support this SSHLE [27]. Engageli supports communication between teachers and students and provides the teacher with different measures, such as student participation (based on speaking time, resource usage, etc.). In addition, Engageli supports collaboration with virtual tables, collaborative work environments, and group resource management. The teacher completed the teacher agency questionnaire before we started the design of the experiment. An interview was conducted with the teacher in the first part of the experiment. The second part consisted in the implementation of a jigsaw on the theme of user-centered design. The information about the activity was collected from the recording of the Engageli session and the transcription of an observer in the classroom. In the end, both students and the teacher completed the workload questionnaire. After this, the teacher completed the teacher agency questionnaire.

The last experiment was conducted at UC3M and involved participants from UVa and UC3M. The aim of this experiment was to study and repeat the approach of the second experiment, a simple classroom transformed into an SSHLE with the addition of software to support it, but with participants and a teacher more familiar with SSHLEs. This experiment was intended to collect information from participants with greater experience in these environments for a more effective comparison. A one-hour workshop was planned with 3 online students and 9 onsite students. An open call was launched, seeking individuals from collaborating universities to participate in this workshop, with no restrictions placed on participation. The software Engageli was also used to support this SSHLE, and the teacher had gained more experience, having already participated in Experiment 2. The teacher completed the teacher agency questionnaire before starting the design process and performed the interview at the end of the experiment. The second part consisted of a jigsaw focused on the study of research paradigms. The activity information was collected from the recording of the Engageli session and a recording in the classroom. In the end, both the students and the teacher completed the workload questionnaire. After this, the teacher completed the teacher agency questionnaire and carried out the interview.

3. Results

3.1. Experiment 1

The results of the NASA-TLX questionnaire completed by the teachers indicated that the factors that most affected the workload were mental demand and temporal demand (presenting a subscale of 70 and 60, respectively). In addition, temporal demand was the factor that varied the most in the pairwise comparisons among factors, being selected in all five comparisons. The rest of the variations and subscales can be seen in

Table 3. Experiment 1—NASA-TLX teacher results (subscales in a range between 0 and 100, and pairwise comparisons in a range between 0 and 5).

	Mental Demand	Physical Demand	Temporal Demand	Performance	Effort	Frustration Level
Subscales	70	1	60	10	50	25
Greatest Variation	4	0	5	2	2	2

. The teacher’s final workload was 50, in a range between 0 and 100. This value fell within the mid-range of workload (40–60) [23].

The ENA model can be seen in Figure 3. It can be observed that announcements to the class and the use of the tool were among the actions that the teacher had to perform most frequently. Moreover, for most of the occasions on which the teacher had to use the tool, she had just made an announcement; this was because the teacher was checking the impact that this announcement had on the students. In contrast, the lines connecting individual or group interaction actions, regardless of the environment where the students were located (online or onsite), were rather thin, which indicated that there were barely any interactions.

The results of the teacher agency questionnaire indicated that 4 (23.53%) out of 17 factors increased, and only 1 (5.88%) decreased, with the rest remaining the same (70.59%) after conducting the experiment. The variation, both the increase and decrease, was by one point on a scale from 1 to 5. The factors that increased were those that dealt with the possibility of using applications for the design and development of classes, as well as the possibility of choosing the content taught. In addition to these results, the teacher indicated in the interview that hybrid classes required the same effort from her as in-person classes. All of this suggested that there was minimal impact on teacher agency, and, if any occurred, only a slight increase would be noted.

3.2. Experiment 2

The results of the NASA-TLX teacher questionnaire indicated that the factor causing the greatest workload was temporal demand (showing a subscale of 70). Additionally, temporal demand was also the most frequently selected factor in the pairwise comparisons, being chosen in all bilateral comparisons. Another detail to highlight is that mental demand was the second factor that most affected workload, just below temporal demand. This was because, in addition to having a subscale of 50, it was selected in four out of five of the pairwise comparisons. All values from the NASA-TLX questionnaire can be seen in

Table 4. Experiment 2—NASA-TLX teacher results (subscales in a range between 0 and 100, and bilateral comparisons in a range between 0 and 5).

	Mental Demand	Physical Demand	Temporal Demand	Performance	Effort	Frustration Level
Subscales	50	20	70	60	60	60
Greatest Variation	4	0	5	1	2	3

. The final workload for the teacher was 60.67, in a range between 0 and 100. This value was within the high workload range (60–80) [23].

The workload values for students were also obtained (see

Table 5. Experiment 2—NASA-TLX student results. The first 6 students were online and are marked in italics.

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	Mean	SD
Mental Demand	50	60	10	75	60	70	60	80	67	60	50	35	70	80	30	70	60	58.06	18.6
Physical Demand	40	40	1	60	5	1	90	70	8	20	20	10	30	10	60	33	20	30.47	26.33
Temporal Demand	85	80	10	40	65	75	10	80	79	80	40	45	60	80	50	70	30	57.59	24.83
Performance	1	10	70	35	30	15	20	30	27	10	10	45	20	30	5	40	20	24.53	17.14
Effort	60	70	10	65	70	70	80	90	58	60	70	55	50	70	55	70	50	61.94	16.98
Frustration Level	35	20	1	40	40	75	1	60	7	20	1	60	10	30	25	65	30	30.53	23.73
Workload	50	51	21	52	49	66	61	65	58	53	39	49	48	68	32	59	43	50.66	12.21

). The students reported higher mental demand due to the difficulty of coordinating with their classmates who were in a different environment. The physical demand was high due to the noise generated during the activity as a result of communication between students in JP3. This was due to the conversations from other groups filtering through the microphones, making communication within each group more difficult. There was a high time demand due to technical issues causing delays. Despite these challenges, overall performance was good, although some students reported lower performance due to a lack of time to complete the tasks. The reported effort corresponded to levels of mental demand, and the level of frustration was generally low, with only a few students reporting higher levels due to stress from the lack of time.

The actions that were most frequently undertaken by the teacher according to the ENA model (see Figure 4) were observing the state of the class, utilizing the tool, and interacting with the class. Moreover, a strong correlation could be noted between class interaction, class announcements, and interaction with the hybrid groups. In contrast, the use of the tool was significantly related to the rest of the actions, being equally connected to almost all of the others. Furthermore, it could be observed that there were very few individual interactions, whether online or onsite.

The results of the teacher agency questionnaire showed no change before and after the experiment. This could have been due to the fact that the design and implementation of the activity were coordinated jointly with the teacher. The teacher supported this idea during the interview.

3.3. Experiment 3

The results of the NASA-TLX teacher questionnaire indicated that the greatest workload was caused by mental demand and effort. Additionally, mental demand had the highest variation in the pairwise comparisons, being selected in all five comparisons. The high values for mental demand, effort, and temporal demand stemmed from the difficulty faced by the teacher in coordinating the students in different environments within the times planned for each phase of the jigsaw. The teacher’s final workload was 76, in a range between 0 and 100 (see

Table 6. Experiment 3—NASA-TLX teacher results (subscales in a range between 0 and 100, and bilateral comparisons in a range between 0 and 5).

	Mental Demand	Physical Demand	Temporal Demand	Performance	Effort	Frustration Level
Subscales	90	40	80	40	90	70
Greatest Variation	5	0	2	3	3	2

). This value fell within the high workload range (60–80) [23].

The values for the student workload can be seen in

Table 7. Experiment 3—NASA-TLX student results. The first 3 students were online and are marked in italics.

	A	B	C	D	E	F	G	H	I	J	K	L	Mean	SD
Mental Demand	70	40	35	70	40	50	60	80	70	70	60	70	59.58	14.84
Physical Demand	10	40	1	0	2	5	20	20	10	10	0	60	14.83	18.31
Temporal Demand	60	40	70	70	50	40	30	90	20	90	80	50	57.50	23.01
Performance	20	30	20	20	10	20	30	20	20	60	20	20	24.17	12.40
Effort	70	50	30	70	50	40	40	80	60	70	60	60	56.67	14.97
Frustration Level	30	70	20	40	10	50	20	60	10	70	30	80	40.83	24.66
Workload	58.67	46.67	40	56	35.33	36	37.33	74	41.33	70	60	56	50.94	13.38

. The students reported higher mental demand than in onsite classes due to the difficulty of coordinating with their peers who were in a different environment. Some students indicated a high physical demand due to the additional noise generated in the classroom from multiple conversations between groups. There was a high time demand, as the activity that took place in phases JP2 and JP3 was debating, and the students would have preferred more time to further develop their ideas. Despite these challenges, the overall performance was good, although some students reported low performance due to a technical problem. The reported effort corresponded to the levels of mental demand, and the level of frustration was generally low, with only a few students reporting higher levels due to stress from the lack of time.

As can be observed in Figure 5, the actions most frequently undertaken by the teacher were observing the state of the class and utilizing the tool. In addition, it can be noted that alongside class observation, there was a strong correlation for interaction with different groups. In contrast, the use of the tool was quite closely related to interaction with the hybrid groups, significantly more so than with the other types of group. Moreover, there was a relationship between the use of the tool and class announcements. Furthermore, it can be observed that when class announcements were made, often an interaction with the class also occurred.

The results from the teacher agency questionnaire indicated that 3 (17.65%) out of 17 factors increased, 5 (29.41%) decreased, and 9 (52.94%) factors maintained the same value after carrying out the experiment. The factors that increased were those dealing with the possibility of using applications, as well as the efficiency of teaching. The factors that decreased were teacher actions and the effect of time on effective teaching. These data suggest that there was a slight change in teacher agency, as almost half of the factors changed. However, it was not possible to conclude that there was an increase or decrease in teacher agency.

4. Discussion

This paper proposed two research questions, and to address them three experiments using SSHLEs to support collaborative learning situations were carried out. The first research question, “What factors influence the workload of the teacher and students in SSHLEs that support collaborative learning situations?”, was answered with the NASA-TLX questionnaire and the ENA model, complemented with teacher interviews.

The NASA-TXL questionnaire was used to obtain values for different factors that affected workload. The workload of the teachers in each experiment was, respectively, 50, 60.67, and 76. The teacher from Experiment 1 was asked in an interview about possible factors that could have affected her workload. She indicated that she had extensive experience in this type of class and did not find it difficult to conduct hybrid classes as long as she had the appropriate technologies. The other two experiments presented a greater workload than the first. From interviews conducted with the teacher and comments made in the NASA-TLX questionnaire, it was deduced that the main problems encountered were the noise generated during JP3 with the hybrid groups, technical problems, the need for more time to carry out the activities, and the lack of experience of the teacher with SSHLEs. No studies were found that have used the NASA-TLX questionnaire to measure teacher workload in SHLEs or collaborative learning. The study most similar to ours was that of Prieto et al. [28], which measured the workload of teachers in technology-enhanced classrooms. In this study, the teachers provided scores of 53.3 (out of 100) in one session and 56.3 (out of 100) in another session, which could serve as a reference to measure workload in environments with a strong presence of technology like SSHLEs. It was also observed that in these experiments, incorporating collaborative learning and conducting it within an SSHLE increased the workload by between 5 and 20 points, but further research is needed for a broader perspective.

The students had an average workload score of 50.66 and 50.94. These values were within the medium range for workload (40–60) [23]. No studies were found that used the NASA-TLX questionnaire to measure the workload of students in hybrid environments carrying out collaborative activities. The closest study was that of Zhang et al. [29], who measured workload in an onsite class using different collaboration strategies. The results of the study by Zhang et al. (38.94) presented lower values than those obtained in our experiments. It was observed in this case that conducting collaborative activities within an SSHLE increased the workload by approximately 12 points, but further research is needed for a more comprehensive view.

Regarding the ENAs, pairwise comparisons were carried out for the easier comprehension of the differences. A stronger relationship between announcements and tool usage could be observed in Experiment 1 after comparing Experiments 1 and 2 (Figure 6). This was due to the fact that in Experiment 1 the students hardly initiated any interaction with the teacher, and she had to monitor the class progress through the tool. In contrast, Experiment 2 showed a strong relationship between class interaction and hybrid groups. This was due to the teacher requesting general information and, if there was a problem, assisting the indicated group. In both experiments, the teacher made extensive use of the tool. This action often became the pivot among other options, that is, after performing one action, the use of the tool was typically involved. This made the use of the tool a key factor from the workload perspective.

In the comparison between Experiment 1 and Experiment 3 (Figure 7), the same difference could be observed as in the previous comparison. Experiment 1 had a stronger relationship between class announcements and tool usage. Experiment 3 had a greater relationship between the perception of the class and interaction with the different groups. This may have been due to the fact that the teacher already had more experience, and with a general overview, she was able to see where her presence was required. In this case, the most frequently used action was perceiving the state of the class; therefore, if this action were performed easily and quickly, it would decrease the workload.

The two experiments conducted by the same teacher, Experiment 2 and Experiment 3, were also compared (Figure 8). In this comparison, as was the case in the first of these comparisons (Figure 6), the relationship between class interaction and hybrid group interaction in Experiment 2 stood out. In contrast, Experiment 3 was distinguished by its individual onsite interactions with the class and the perception of online groups. Individual onsite interactions occurred when the teacher asked the class for information and a student responded with a problem, so the teacher assisted them individually. The interaction with online groups related to perception was due to the teacher assisting the group when she noticed a problem with any group using the tool. In this comparison, the most frequently performed action was interaction with the class. This action is crucial and should be carried out effectively for all students, whether they are online or onsite. Providing the appropriate tools to carry out this action is crucial for conducting activities in SSHLEs. Moreover, ensuring these tools do not increase the workload is a challenge.

An analysis of the NASA-TLX questionnaire, the ENA model, and the interviews revealed several key factors impacting the workload of these experiments. One of these factors was task complexity. This factor was identified in the literature on technology implementation [28] and gains greater importance in SSHLEs. This is due to the requirement of using new technologies together with the need to work with people in different environments (online and onsite). From the point of view of collaborative learning, this factor becomes more important, as collaborative activities usually require extensive communication and the use of resources for collaboration. The NASA-TLX questionnaires for both the teachers and the students, as well as the interviews with the teachers, highlighted this factor. The characteristics that helped to reduce task complexity, indicated by the teachers in the interviews, were the centralization of resources, the adaptability to various changes that arose during the activity, and the support for group management. Another factor was time limitations, which, as with the previous factor, are also found in learning environments where technology is added [28]. Time limitations become more important in SSHLEs because, unlike other environments, if there is any problem with the technology, especially with the communication technology, it is very challenging (at least in a short period of time) to find a solution or alternative. From the perspective of collaboration, calculating the duration of activities is already a challenge in itself [30]. However, this factor is compounded by the need to take more steps to complete an activity due to technology, i.e., not having alternatives when an error or complication arises (for example, problems with a student’s internet connection or microphone failures). The NASA-TLX questionnaires for both the teachers and the students, as well as the interviews with the teachers, highlighted this factor. The characteristics that facilitated reducing the activity time, indicated by the teachers in the interviews, were adaptability to the different changes that arose and support for group management. Another factor that affected the workload was the tools used in the SSHLEs. This factor is inherited from both SLEs and SHLEs [3,7]. From the perspective of collaborative learning, more specifically, computer-supported collaborative learning (CSCL), tools are also a key factor in enhancing development [31]. In addition to being an individually identified factor in SLEs, SHLEs, and CSCL, the ENA models indicated a significant weight for tool use, highlighting it as a key factor for the workload. The prominent features of the tools according to the teacher interviews were the video/chat, real-time interaction, group manager, file manager, and ability to incorporate external resources. The last identified factor was knowledge about the state of the class and the students, which is present as a feature in some SLEs [7] and was also a factor identified in other studies on collaborative learning [22]. In the case of these experiments, this factor was detected in the ENA models and teacher interviews. The features that contributed to this factor, as indicated by the teachers in the interviews, were student participation data, the notifications when a student had a question, and viewing student progress. All these factors can be seen in

Table 8. RQ1: Factors influencing the workload of the teacher and students in SSHLEs that support collaborative learning situations.

Factor	Data Sources	Reasons	Potential Improvements
Complexity of performing the task	- NASA-TLX questionnaire for teachers and students - Teacher interviews	- Problems inherited from the incorporation of technology - Need to work with people in different environments - Great importance for collaborative learning	- Centralizing resources - Adaptability - Group management
Time limitations	- NASA-TLX questionnaire for teachers and students - Teacher interviews	- Problems inherited from the incorporation of technology - Difficulty in finding an alternative when an error occurs - Difficulty of timing in collaborative learning	- Adaptability - Group management
Tools used	- ENA models - Teacher interviews	- Problems inherited from the incorporation of technology - Important factor in SLEs and SHLEs - Important factor in CSCL	- Video/chat - Real-time interaction - Group manager - File manager - Incorporation of external resources
Knowledge about the status of the class and the students	- ENA models - Teacher interviews	- Present as a feature in some SLEs - Identified in collaborative learning	- Data on student participation - Notifications when a student has a question - Student progress

.

The second research question, “What factors influence teacher agency in SSHLEs that support collaborative learning situations?”, was answered with the teacher agency questionnaire, complemented by the teacher interviews.

The different results obtained from the teacher agency questionnaire seemed to indicate that SSHLEs had a minimal impact on teacher agency. However, in two out of three cases, they increased teacher agency in terms of factors related to the use of tools. Factors related to the material teachers had at their disposal carry significant weight in teacher agency [32]. For this reason, and based on the results obtained herein, it is possible that a specific approach to SSHLEs to support these factors could have a positive impact on teacher agency. In contrast, it should be noted that no studies were found that assessed teacher agency with a questionnaire, an issue also encountered by the creators of the model upon which the questionnaire of this paper on teacher agency was based [33].

Although the questionnaire results did not indicate a significant impact on teacher agency, some factors were affected and were identified in the interview. One of these factors was controlling the creation and management of the activity. This factor was identified because the teachers experienced a slight increase at the beginning and end of Experiments 1 and 3 in the factors related to the creation and implementation of activities using tools. Meanwhile, this factor did not change in Experiment 2, where the activity was designed in collaboration with the teacher. The teachers were asked in the interviews and indicated, in Experiment 1, that designing the activity entirely (they were not forced to follow the jigsaw pattern) gave them more security, control, and freedom when taking actions. In Experiment 2, the teacher indicated that there were no changes in teacher agency due to the collaboration in creating the activity. In Experiment 3, the teacher indicated that she felt more comfortable having more control over the activity. The features highlighted in the interview as potentially improving teacher agency were support in the design and management of collaborative activities and adaptability to possible changes that might arise during implementation. Another factor that was affected was the use of available tools (e.g., the software WeConnect in Experiment 1 and Engageli in Experiment 2 and Experiment 3). Although this factor has been identified in the literature as the available resources [32], in the teacher agency questionnaire it was identified not as general resources but as the available tools. In Experiments 1 and 3, scores for questions related to the use of tools slightly increased (1 point more, on a scale of 1 to 5). In addition, in all interviews, teachers indicated how necessary the tools were to conducting the class and making any modifications. The features that could increase this factor, as highlighted by the teachers in the interviews, were the ease of use; adaptability to possible changes that might arise during the activity; and the ease of access to all resources (teaching material, exercises, shared documents, etc.). The last identified factor was the perception of teaching efficacy. The only scores that decreased were related to the perception of teaching efficacy in Experiment 3. In the interview, the teacher indicated that the lack of time due to technical failures and the difficulty of checking the students’ progress in real time complicated the evaluation. The features indicated by the teachers that could improve this factor were more information about the state of the class and easily accessible measures to check student progress. All these factors are summarized in

Table 9. RQ2: Factors influencing teacher agency in SSHLEs that support collaborative learning situations.

Factor	Data Sources	Reasons	Potential Improvements
Controlling the creation and management of the activity	- Teacher agency questionnaire - Teacher interviews	- Feelings of greater freedom in Experiment 1 - Co-design in Experiment 2 - Greater comfort due to having more control over the activity in Experiment 3	- Support in the design and management of collaborative activities - Adaptability
Use of tools	- Teacher agency questionnaire - Teacher interviews	- Identified in the teacher agency literature - Indicated as necessary by teachers - User-friendliness	- Adaptability - Ease of access to resources
Teaching effectiveness	- Teacher agency questionnaire - Teacher interviews	- Lack of time - Difficulty in checking the progress of the students in real time	- Class status information - Student progress values

.

5. Conclusions

This paper identified and analyzed the factors affecting workload among teachers and students and teacher agency in SSHLEs implemented in collaborative learning situations. To this end, the three experiments conducted in this paper presented different collaborative activities in SSHLEs. Several factors that affected the workload of both the students and the teachers, as well as factors that influenced teacher agency, were extracted from these experiences. The factors found to influence workload in these experiments were: task complexity, time constraints, the tools used, and knowledge about the class and student status. These factors were extracted from the NASA-TLX questionnaire, the ENA model, and interviews with the teachers. The factors found to influence teacher agency were: control over the creation and management of the activity, the use of available tools, and the perception of teaching effectiveness. These factors were extracted from the teacher agency questionnaire and teacher interviews. Furthermore, some characteristics of the experiments that helped or could have been improved to decrease the workload were extracted from the teacher interviews. These characteristics included: centralizing resources, adaptability to errors, group management, the ability to incorporate external resources, and information on student participation and student progression. Also, some characteristics of the experiments were discussed that helped or could have been improved to increase teacher agency. These characteristics included: support in the design and management of collaborative activities, adaptability to errors, the ease of access to resources, information about the state of the class, and information about student progress. In general, the main innovation in this study was the introduction and analysis of scenarios that have not been addressed in the related literature.

The main limitation of this study was finding a real scenario within which to conduct the experiments. SHLEs are present in some institutions, but the difficulty of transforming them into SSHLEs and incorporating the jigsaw pattern to implement a complex collaborative learning activity were a significant barrier to conducting more experiments. Experiments 2 and 3 had to be implemented as workshops with a limited duration. Another limitation was the regulations of the different institutions. In the case of Experiment 1, the collection of the students’ workload data was not allowed, and in Experiment 2, the need to complete the necessary steps for consent caused delays in carrying out the activity. Another limitation of this study was the emergence of technical issues. In Experiment 2, due to a lack of experience, technical problems arose, causing delays in the activity. In Experiments 1 and 3, there were some issues related to student disconnections that could not be resolved, but due to the teacher’s experience, they hardly posed a problem. The final limitation was the noise present when carrying out the last phase of the jigsaw (JP3). In Experiment 2, this was a significant problem reported by both students and the teacher. In Experiment 3, although the use of headphones was recommended, due to the small classroom space, there were occasional issues, though far fewer than in Experiment 2.

Future Work

Future steps include conducting further experiments in other SSHLEs with different distributions of the hybrid learning environment, for example, in telepresence classrooms. These classrooms consist of two rooms, one with the teacher and a group of students, and the other with the remainder of the students. On one of the walls of each room, there is a projection of the other room, simulating a connection between the two. Another area for future work is the incorporation of the features recommended by teachers in the SSHLEs and the evaluation of their benefits. The final proposed direction for future work is to replicate these experiments in more realistic scenarios with participants from university courses, as in Experiment 1.