Technology-Enhanced Learning: An Optimal CPS Learning Application

Abstract

Collaborative problem-solving (CPS) is highly valued in the sustainability of learning to foster the key soft power of talent for the future. In this study, a CPS learning application was built to train and assess individuals with the aim of increasing CPS skills. For effective learning to take place, several issues need to be carefully considered, and these were investigated while testing the proposed application. This study examined the impact of collaborative interactions (CIs) (human–computer agent (HCA) and human–human (HH) interactions) on the CPS performance of students. Gender and learning styles, which may have interaction effects with CIs on CPS performance, were also explored. The results show that the students’ CPS performance in HCA was significantly greater than that in HH. The interaction effect between gender and CI was not significant. The impact of learning style on CPS performance in HH was not significant. In contrast, in HCA, students with verbal, global, and reflective learning styles performed significantly better on CPS tasks than did students with visual, sequential, and active learning styles. Finally, we discussed the optimal ways to teach CPS and the practical effects of a CPS learning application.

1. Introduction

Important changes have been made in the way problems are dealt with. Cooperative relationships are established by different social groups in order to handle various issues over time. Therefore, collaborative and problem-oriented approaches may be essential to the implementation of sustainable development and education [1]. Education, therefore, calls for transdisciplinary knowledge and skills in collaborative problem-solving (CPS), whereby students build and integrate their and others’ knowledge across disciplines and apply transdisciplinary knowledge to real community problems [2,3].

Cooperating with others is an indispensable ability that citizens of the 21st century require [4]. In daily work and life, people must engage in CPS [5,6]. CPS has the ability to solve various complex and insufficient information problems on a non-routine basis [7]. As environments, economic situations, and social structures change, complicated problems can arise. People have to deal with such complex interdisciplinary problems in their daily lives and work and may not have the necessary information to solve them [8]. Problems would not be solved only by reasoning or memory in these situations. Instead, people have to work together and use different information collection strategies to tackle problems. Consequently, an individual has to be supported by others to be able to complete sophisticated tasks [3].

With the advent of the “Fourth Industrial Revolution Age”, efforts are being made to maximize the effect of learning by utilizing new technologies, such as artificial intelligence, the Internet of Things, virtual/augmented reality, and Web technologies. Recently, Web technology has developed rapidly, and online collaborative platforms have become an interface through which people can communicate and share information globally [9]. The use of online technologies for virtual collaboration is a key skill that is needed in workplaces of the future [10]. In educational programs, Web-based CPS learning platforms have, therefore, received considerable attention [8]. They have become comprehensive international educational assessments. For instance, the Program for International Student Assessment (PISA) is a computer-based assessment for the evaluation of the CPS ability of students. It is, therefore, important to use CPS in Web-based scenarios [6]. This study was focused on establishing a CPS training application using Web technology for the sustainability of education. The application trains and assesses CPS skills in communicating information, building teams, sharing knowledge, and developing a problem-solving plan [11].

Collaborative learning is an extremely dynamic and interactive process. In order to be effective, students who learn in a collaborative learning process need to be monitored closely. In the development of a CPS learning application, several issues need to be carefully considered [12].

Previous studies [13,14,15] have examined face-to-face collaborative learning modalities by either developing programs for open assignments [13,15], or by using an interactive digital whiteboard to improve interaction and cooperation among learners [14]. Although these studies produced positive results, one study [16] found that students were dissatisfied with their team members and each considered that they had strived more than the others, and that the students’ collaborative skills had increased more by CPS than by conventional learning. That is, while participating in collaborative work that led to cooperation and competitive behavior, the students analyzed the contribution that others made to problem-solving [17]. These findings led scientists to use computer conversational agents to train students in a range of educational fields to solve competitive behavior problems, such as developing an inferential CPS platform that uses computer agents who collaborate with one another [18,19,20]. With the advent of Web 2.0, the use of conversational agents such as Chatbots constitutes a new chapter in the evolution of Web technology [21]. The possibility for communication between a human and a computer via natural language has been demonstrated by integrating computational linguistics techniques with communication over the Web.

However, the aim of a computer-agent-based CPS learning program is to train people to cooperate. Therefore, there is a need to conduct in-depth and direct comparisons of the pros and cons of human–computer agent (HCA) collaborative interaction (CI) with those of human–human (HH) CI in terms of CPS performance. The results of such investigations would be useful for the design and evaluation of training applications [22].

Collaborative learning can produce more abstract representations from different points of view [23], and can lead to the integration of different perspectives and the generation of a compound analysis [24]. Simply introducing students into a group does not, however, mean that all group members will work together and cognitive development will occur [25]. Research has shown that personal traits are important contributors to CPS performance and that gender differences are, in turn, associated with performance [22,26,27]. Partner gender and personal traits are important variables in cooperative learning with regard to learners’ interaction processes and learning outcomes [28,29]. Accordingly, the interaction effects between CI and learning styles and CI and gender on learners’ CPS performance have to be examined.

Previous studies have argued that female learners are socialized to participate in collaborative tasks [30], interact with their peers [31], and seek assistance and support from peers [32] more frequently than male learners. However, previous studies have also reported inconclusive results on the relationship between gender and computer-supported CPS. Studies have shown that male learners are likely to dominate speech when communicating through computers [33] and are more familiar with computer games [34], resulting in male learners having superior performance to female learners in tasks involving collaboration with computer agents [35]. In contrast, female learners were found to perform significantly better than male learners in CPS tasks involving collaboration with a computer agent in a previous study conducted by Rosen [17]. Female and male learners have different communication styles [36], and the relationship between gender and collaborative learning is complex [24].

Learner personalities or thinking styles play roles in the development of the ability to collaborate [25]. Communication abilities have been correlated with learners’ critical personality characteristics [30]. Researchers have applied knowledge of learning styles to the development of education systems for adaptive learning [37]. Learning style, defined as the established pattern by which people obtain and process information in a learning scenario, influences approaches to problem-solving [38]. There are several models of learning style [39]. The most accepted model of learning styles used in e-learning and Web-based learning research is the Feld and Silverman learning style model (FSLSM) [40], which is also appropriate for hypermedia courseware [41]. The FSLSM balances pairs of extremes (active/reflective, sensing/intuitive, visual/verbal, and sequential/global) [40]. In the active/reflective pair, active learners are more interested in communicating with others and learning in groups. In contrast, reflective learners would choose to work alone or in small groups. The second pair includes the sensing and intuitive learning styles. Sensing learners like to use standard approaches to solve problems. In contrast, intuitive learners tend to be more innovative, creative, and able to discover possibilities. The third pair, which includes the visual and verbal learning styles, differentiates learners who prefer to learn from what they have seen from learners who obtain more out of textual representations, regardless of whether they are written or spoken. The fourth pair includes the sequential and global learning styles. Sequential learners are likely to follow logical stepwise paths to find solutions. In contrast, global learners use a holistic thinking process and are able to solve complex problems. The present study used the FSLSM to investigate associations between learning style and CPS performance through the use of two different CIs involving problem-solving activities.

The aims of the present study are to: (1) Establish a learning application for CPS by using computer agents with Web technology; (2) understand the benefits and drawbacks in terms of the CPS performance in the different CIs; (3) examine how personal characteristics (i.e., learning style and gender) influence the use of CIs; and (4) identify the ideal way to teach CPS and the practical consequences of using a CPS application for the sustainability of education.

2. Methods

2.1. Participants and the CPS Learning Application

The participants consisted of 64 college students who were recruited from undergraduate classes at a university. They received extra credit in exchange for taking part in the study. The participants included 28 women and 36 men (aged 18–22 years; M = 20.13, SD = 1.18). The students had no previous experience with using the CPS learning application.

An application was developed to train and evaluate CPS skills through computer agents with Web technology. Figure 1 presents the structure of the application. In developing the application, the elements of the tasks, the problem scenarios, the medium, and team building were carefully considered. In terms of the task elements and problem scenarios, tasks were related to sustainability, and students and computer agents undertook different duties in the problem scenarios. With respect to the medium, sufficient information was provided to students to enable them to work with the computer agents, and the students determined the rationality of the strategies that were provided by the computer agents. With regard to team building, students worked with computer agents, played different roles, and had different responsibilities.

In most of the CPS-related studies, the learning system was developed using hierarchical subject-based problems with default solutions and science-based content, such as physics, biology, mathematics, and chemistry [42]. In these studies, learners seldom used skills that are relevant to daily work and life [9]. Consequently, students did not obtain an opportunity to develop their CPS skills while working on day-to-day problems and problems that typically do not have predetermined solutions [43]. It is self-evident that learners need to develop techniques to solve the problems that they encounter in daily work and life that differ from those that are usually needed to solve less practical problems. We therefore adopt various strategies to teach these techniques.

Consequently, our application includes three elements. The first is the application’s design. Four CPS experts were invited to help with the application’s development and provide guidance on the problem scripts [9]. Eight problem-solving tasks were developed (fastening shelves, using a microwave, placing water, organizing a bedroom, traveling, buying a phone, building a home, and designing furniture). The second element is a platform for interaction between the application and learners. Learners were able to register with and log in to the application’s platform and then choose problems to solve and collaborate with agents. The platform presented critical information about each problem so that learners could discuss, share, and understand problems with computer agents. Learners communicated with computer agents using keyboards and entered responses or selected items that appeared onscreen. The third element is an item bank. The item bank plays an essential role in the CPS learning application, as it collects information from CPS specialists and previous users. Computer agents provide information that can be retrieved from the item bank in response to a learner’s input. All learners’ input data were also recorded to evaluate learners’ CPS performance.

The present study employed four of the eight abovementioned problem-solving tasks (the fastening shelves problem, the using a microwave problem, the placing water problem, and the organizing a bedroom problem). Figure 2 shows the interface for the “using a microwave” problem that enables learners to work together to prepare food using their knowledge of science and sustainability, including knowledge of the heating method, power, and microwave penetration.

2.2. CPS Performance Scores

A series of questions were used to assess students’ CPS performance in terms of shared information flow, taking reasonable steps to solve problems, and developing and organizing a team. The overall scores of students that were obtained from the assessment of these three competences were recorded and are compared in this study [9].

2.3. Materials

The index of learning styles© questionnaire [40] was used to assess the learning styles of students. It has 44 items and measures learning style according to the eight abovementioned preferences (active/reflective, sensing/intuitive, visual/verbal, and sequential/global).

We also used the Program Performance Evaluation questionnaire, which is based on the DeLone–McLean model of information program success [44]. It includes nine statements in six areas, including service quality (sample statement: “the CPS learning application can enhance the efficiency with which the current task is performed”), program quality (sample statement: “the user interface provides a ‘user-friendly’ experience”), user satisfaction (sample statement: “I intend to use the CPS learning application to solve problems”), program use (sample statement: “the materials that the CPS learning application uses can enhance skills to solve problems”), net benefits (sample statement: “the CPS learning application is useful”), and information quality (sample statement: “learners used the CPS learning application with computer agents”). The response options for these statements ranged from “strongly agree” (1 point) to “strongly disagree” (4 points). The Cronbach’s α value for the questionnaire with nine items was 0.86.

2.4. Design and Procedure

We examined the effects of two CIs on students’ CPS performance with the CPS learning application that was developed in this study, and investigated the interaction effects of gender/CI and learning style/CI on students’ CPS performance. Of the 64 participants, 32 participated in the HH CI first and then in the HCA CI, while the remainder participated in the HCA CI first and then in the HH CI. All students were randomly assigned to one of the two groups.

The students received a brief explanation of the study’s aim and procedure. The students performed two tasks to practice the process of collaboration in HH and HCA interactions and become familiar with the CPS learning application’s interface. The students took a rest for two minutes, and then solved the other two problems in either an HH interaction or an HCA interaction. Each student used a personal computer (PC) in the HCA interaction to perform the CPS task independently. In the HH interaction, the same PCs were used by two students who communicated face-to-face. One of the students typed in their joint answers to the questions posed by the CPS learning application. All of the students’ answers were recorded by the CPS learning application. Students finished the CPS tasks and then answered the learning style and program evaluation questionnaires. Each student took approximately 40 min to finish the experiment.

Table 1. Parameters that were used in the experiment for testing the CPS learning application.

Variable.	Description	Type
ID	Sample (N = 64)	Number
Age	18–22 years old (M = 20.13, SD = 1.18)	Number
Gender	Male (N = 36) Female (N = 28)	Category
Learning style	Four pairs of extremes of learning style preferences were evaluated: Active/reflective, sensing/intuitive, visual/verbal, and sequential/global	Category
Level of learning style	High (top 50%) Low (bottom 50%)	Category
Collaborative Interaction (CI)	HH (human–human interaction) HCA (human–computer agent interaction)	Category
Problem scenario	Scenario 1: Using a microwave Scenario 2: Placing water Scenario 3: Fastening shelves Scenario 4: Organizing a bedroom	Category
Trial	128 trials (each student participated in one HH trial and one HCA trial)	Number
CPS performance	CPS performance was scored (1–100%)	Number
Application evaluation	A nine-item questionnaire with a four-point Likert-type scale (“strongly agree” (1 point) to “strongly disagree” (4 points))	Number

shows the parameters that were used in the experiment for testing the CPS learning application. Figure 3 shows the experimental procedure.

3. Results

An analysis of variance (ANOVA) was performed to examine the interaction effects of CI/gender and CI/learning style on CPS performance. The effects of different collaborative interactions on the CPS learning application were compared with t-tests.

3.1. CPS Performance

Table 2. Students’ CPS performance scores in the human–human (HH) interaction and the human–computer agent (HCA) interaction for each independent variable.

	HH		HCA
	%	SD	%	SD
Gender
Male	78.56	6.86	81.44	8.90
Female	76.71	3.30	76.29	8.12
Learning style
Sensing	78.17	5.01	79.17	6.30
Intuitive	76.83	6.31	80.00	12.20
Visual	77.75	6.52	77.83	8.86
Verbal	78.67	2.74	85.67	8.96
Sequential	78.13	5.71	74.25	5.12
Global	77.38	5.61	84.13	9.15
Active	79.00	6.62	76.61	8.35
Reflective	76.00	3.40	82.00	9.08
Total	77.8	5.6	79.2	8.9

shows the CPS performance scores of students for the independent variables CI, gender, and learning style.

Table 3. Analysis of variance of the CPS performance.

Source	Type III Sum of Squares	Mean Square	F	p	partial η2
Collaborative Interaction (CI)	321.22	321.22	24.49	<0.01	0.31
CI × Gender	32.60	32.60	2.49	0.12	0.04
CI × Sensing/Intuitive	20.00	20.00	1.53	0.22	0.03
CI × Visual/Verbal	112.50	112.50	8.58	<0.01	0.14
CI × Sequential/Global	100.00	100.00	7.62	<0.01	0.12
CI × Active/Reflective	75.00	75.00	5.72	0.02	0.10

presents the ANOVA results for the CPS performance scores.

According the ANOVA analysis, the CI significantly affected students’ CPS performance. Students’ CPS performance scores in the HCA interaction were significantly higher than those in the HH interaction, with an effect size of r = 0.10. The interaction effect of gender and CI on students’ CPS performance scores was not significant. The effects of the interaction between learning style and CI on students’ CPS performance scores were as follows. The sensing/intuitive learning style had no significant interaction effect with CI on CPS performance scores. However, the visual/verbal learning style had a significant interaction effect with CI on CPS performance scores. The CPS performance scores of visually and verbally oriented students did not differ in the HH interaction, but the CPS performance scores of verbally oriented students were significantly higher than those of visually oriented students in the HCA interaction (t(11) = −2.75, p = 0.02), with an effect size of r = 0.64. Moreover, the interaction effect between the sequential/global learning style and CI on CPS performance scores was significant. The students’ CPS performance scores with sequential and global learning styles did not significantly differ in the HH interaction, but the CPS performance scores of globally oriented students were significantly higher than those of sequentially oriented students in the HCA interaction (t(31) = −4.97, p < 0.01), with an effect size of r = 0.67. Additionally, the active/reflective learning style had a significant interaction effect with CI on CPS performance scores. The CPS performance scores of students with active and reflective orientations did not significantly differ in the HH interaction, but the CPS performance scores of reflective students were significantly higher than those of active students in the HCA interaction (t(31) = −2.34, p = 0.03), with an effect size of r = 0.39.

3.2. Program Performance Evaluation

The average score for each collaborative interaction was greater than the total average (= 2). The t-test showed a significantly higher mean score for the HH interaction than for the HCA interaction (t(63) = 3.73, p = <0.01), with an effect size of r = 0.30.

Table 4. The program performance evaluation.

	HH		HCA		t-test
	M	SD	M	SD	t	p
Service quality	3.28	0.44	3.03	0.58	3.24	<0.01
Program quality	3.06	0.36	2.88	0.50	2.98	<0.01
User satisfaction	3.13	0.60	2.88	0.49	3.55	<0.01
Program use	3.13	0.70	2.88	0.70	3.00	<0.01
Net benefits	3.25	0.56	3.00	0.62	2.65	<0.01
Information quality	3.31	0.59	3.00	0.36	5.35	<0.01
Total	3.15	0.37	2.97	0.43	3.75	<0.01

presents the descriptive statistics and t-test results for evaluations of the CPS learning application by CI.

4. Discussion and Conclusions

The professional skills that are needed for a sustainable future have changed [4], and countries have paid a significant amount of attention to training young people in CPS skills with new technologies in order to improve their ability to solve daily work and life problems [7]. We created a CPS learning application with Web technology as a platform to achieve sustainability in education. Computer agents or real individuals can be involved in the CPS application’s training activities. This study investigated the differences in CPS skills of students using two different CIs: HH and HCA. We also examined whether gender and learning style interacted with CI to affect the CPS performance.

Regarding HH interactions, previous studies showed that learners might concurrently have cooperative and competitive behaviors when participating in a CPS task, learners always evaluate each other’s contributions while collaborating with each other, and learners are frequently dissatisfied with their team members and feel that they put forth more effort than others do [17]. In this study, students were required to obey instructions and use the provided information to solve problems within a limited time. However, students require more time in order to communicate with each other and build trust as compared to cooperation with computer agents. These results illustrate why students did much better in the HCA interaction than in the HH interaction. Although we found a significant difference in the CPS performance between HCA and HH interaction, the effect was not strong (an effect size of r = 0.10); that is, HH interaction could be used if careful consideration is given to the timing and status of use. For example, HH interaction can be used when students work with teachers rather than other learners. Because teacher–student relationships are not competitive, competition between students can be avoided. Teachers may also have a greater understanding of the contents of the questions and effective cooperation and communication skills. In this way, teachers may help students to share, understand, and communicate information and establish and maintain organization. In contrast, HCA interaction may be more suitable for training students’ CPS abilities. HCA interaction avoids competition between learners, saves time on communication, builds a good understanding of the subject matter, and develops an efficient approach to performing accurately and effectively that involves other students. In addition, HCA interaction enables students to exercise their CPS skills by using the CPS learning application with a computer agent anytime and anywhere. This study, therefore, identifies a better way to train CPS skills.

The results of the study revealed no significant interaction effects between gender and CI on CPS performance. However, a previous study [17] found that female junior high school students performed significantly better than male students on CPS tasks involving collaboration with a computer agent. In contrast, another study [35] found that male students became more familiar with the computer agent and developed more confidence in dealing with computing software [34], leading to superior CPS learning performance compared with female students in a game-based scenario. The results of the present study are not consistent with previous studies’ results with regard to the effect of gender on learners’ CPS performance. Although students of different genders have different communication styles (e.g., females have been found to be more likely to start a conversation with a question, whilst males tend to initiate conversations with a clarification [36]), the review performed by Prinsen et al. [33] on gender-related differences in participation in computer-supported collaborative learning found that the relationship between gender and CPS performance cannot be understood by performing simple comparisons. Ding and Harskamp [24] argued that the relationship between gender and collaborative learning is complex and can be influenced by other factors, such as academic field, age, and gender dyads.

Recent research has focused on habitual patterns of obtaining knowledge for the development of adaptive educational systems [37]. The present study examined the interaction effects of learning style and CI on CPS performance. We did not find a significant effect of learning style on students’ CPS performance scores in the HH interaction, which is similar to the results of a previous study [26]. In contrast, students with verbal, global, and reflective learning styles were found to perform significantly better than students with visual, sequential, and active learning styles in the HCA interaction. According to the analysis of the relationship between the two CIs and CPS performance, HH and HCA each have their distinct advantages. Identifying the optimal context for each CI (e.g., as noted above, using HH interaction with a teacher instead of with other learners) could avoid the disadvantage of this CI, and adapting the methods that are employed in each CI to allow for learners with various learning styles to flexibly adopt different collaborative approaches may yield optimal learning outcomes.

The program performance questionnaire, which was drawn from the study conducted by DeLone and McLean [44], was used to investigate the advantages and disadvantages of the two CIs. Although students’ CPS performance was significantly better in the HCA interaction, the results of the program performance questionnaire reflected that students had significantly more positive evaluations of HH interaction as compared with HCA interaction. That is, the CPS learning application should be improved when used with HCA interaction with regard to all six components (service quality, program quality, user satisfaction, program use, net benefits, and information quality). Such improvements are necessary to improve the program’s performance and increase learners’ motivation and willingness to use the CPS learning application. The CPS learning application is in the early stages of program development, and it is important to use various prototypes to further determine users’ acceptance of the application and to revise system errors as rapidly as possible. The results of the present study provide important baseline information about the CPS learning application and may constitute a basis for future studies related to the development of such a platform.

We introduced the initial development process of a CPS learning application and used it in the present study. The outcomes of the study will help us to improve CPS learning systems for the sustainability of education and inspire other efforts in the development of CPS-related learning systems. For further research, longer-term studies should involve a more diverse range of CPS scenarios (such as adding misinformation), a larger sample size (the sample size of the present study is relatively small), and a broader range of measures (such as including qualitative measurements). Moreover, with regard to the interactions between gender and CPS, we found no significant interaction effect between gender and CI on CPS performance. However, previous studies have noted that gender is a complex variable [24] and have argued that the gender composition of dyads, which was not investigated in this study, is an important gender-related issue that requires further investigation with regard to CPS (e.g., comparisons between mixed-and single-gender dyads). Finally, it is important to investigate the transversal capacities of students to solve poorly structured problems. This scenario would help students to be capable of building and integrating knowledge across disciplines and dealing with real sustainability issues.