Benefits and Challenges of Collaboration between Students and Conversational Generative Artificial Intelligence in Programming Learning: An Empirical Case Study

Abstract

The utilization of conversational generative artificial intelligence (Gen AI) in learning is often seen as a double-edged sword that may lead to superficial learning. We designed and implemented a programming course focusing on collaboration between students and Gen AI. This study explores the dynamics of such collaboration, focusing on students’ communication strategies with Gen AI, perceived benefits, and challenges encountered. Data were collected from class observations, surveys, final reports, dialogues between students and Gen AI, and semi-structured in-depth interviews. The results showed that effective collaboration between students and Gen AI could enhance students’ meta-cognitive and self-regulated learning skills and positively impact human-to-human communication. This study further revealed the difficulties and individual differences in collaborating with Gen AI on complex learning tasks. Overall, collaborating with Gen AI as a learning partner, rather than just a tool, enables sustainable and independent learning, beyond specific learning tasks at a given time.

1. Introduction

Conversational generative artificial intelligence (Gen AI) is becoming more prevalent in society and noticeable across various industries. Technologies such as Open AI’s ChatGPT are gaining popularity rapidly and are performing on par with humans. Yet, the use of Gen AI in education remains a double-edged sword, with threats lurking beneath the evident benefits [1,2,3]. Using Gen AI output in its current state as a learning artifact may inhibit deeper learning and pose a risk of plagiarism [4,5]. Gen AI output may contain falsehoods or biases, and students’ excessive reliance on Gen AI runs the risk of creating superficial or no learning [6,7,8,9]. Thus, many universities forbid students from using Gen AI in their academic work, thus necessitating a Gen AI policy in education [10].

The ongoing debate about the integration of Gen AI in education persists, particularly its full acceptance. Through our year-long engagement with various Gen AI platforms, such as ChatGPT-3.5, ChatGPT-4, Bing AI, and Claude2, we have uncovered a significant potential for these technologies to foster active learning and self-regulated learning through enhanced communication and collaboration between learners and Gen AI. Gen AI distinguishes itself from its predecessors by facilitating more nuanced and effective communication, enabling a richer exchange of ideas and fostering a deeper understanding [11]. This advancement addresses the limitations previously faced by traditional AI in creating effective collaborative relationships with students [12]. However, it would be difficult for students to construct effective communication with Gen AI in the initial phase due to a lack knowledge and the gap between communication with humans and with AI. Therefore, there are numerous videos and blogs that share useful prompts for communicating with Gen AI.

In this empirical case study, we designed a programming class for beginner students to conduct an effective communication and collaborative relationship with Gen AI (ChatGPT, Bing AI) in programming exercises, followed by discussions of the benefits and challenges based on four research questions:

Research Question 1 (RQ1): How do students view their communication with Gen AI?

Research Question 2 (RQ2): How do students develop their collaboration with Gen AI in concept learning and app development?

Research Question 3 (RQ3): How do students evaluate Gen AI in their learning?

Research Question 4 (RQ4): What kind of challenges are faced by students in their collaboration?

1.1. Student–AI Collaboration

Student–AI collaboration, a type of relationship between students and AI, represents how students build active and mutual interactions with AI to achieve better, more efficient learning [12,13,14]. During student–AI collaboration, rather than passively absorb the information from AI, students actively engage with the continuous interaction with AI while the AI collects and analyzes the data from students to form an accurate understanding of students’ learning status [12,15]. Prior to the advent of Gen AI, the biggest difficulty preventing students and AI from forming a collaborative relationship was the inability of AI to provide consistent and effective interaction [12]. As a result, collaborative relationships between students and AI were not widely discussed and clearly defined. Nevertheless, research related to collaborative learning among students provides numerous insights into this topic [16,17,18,19,20,21].

Both student–AI collaborative relationships and student–student collaborative learning are based on Vygotsky and Piaget’s constructivism theory [12,20,22]. Collaborative learning involves students taking an active role in their education through discussions with others, often in small groups, and is commonly used in higher education [22]. As a student-centered pedagogical approach, collaborative learning assumes that knowledge is not merely transmitted from professors to students but is constructed by students through effective interactions [20,21].

Moreover, the characteristics of collaborative learning can be enumerated as maximum peer interactions, equal partnerships, individual accountability, positive interdependence, and shared learning goals [21,23,24,25]. Maximum peer interaction emphasizes the need for both the quantity and quality of interactions among students [24]. Equal partnerships, individual accountability, and positive interdependence are designed to reduce reliance on teachers and other group members, avoiding relationships where one party depends on or only benefits from the others [23,24,25]. These shared values among group members facilitate the resolution of conflicts and the creation of new communal values and knowledge communities through their interconnected experiences [23].

However, we believe student–Gen AI should not be simplistically equated to student–student collaboration. Gen AI, unlike human collaborators, may exhibit either excessively active or passive patterns within the collaboration. With the capacity to process and present large volumes of information rapidly [26], Gen AI might dominate interactions, making it challenging for students to foster an equal partnership. Moreover, since engagement with Gen AI is reliant on student initiation—remaining inactive until prompted by questions—it does not spontaneously contribute to the dialogue. Therefore, when engaging with Gen AI, students should consciously exert initiative and lead the establishment of learning objectives, strategies, and evaluations. Maintaining a high degree of independence is essential for students to ensure they retain ownership of their learning experience without it being overshadowed by Gen AI involvement.

1.2. Gen AI on Student Programming Learning

On November 2022, ChatGPT debuted as the best-performing AI chatbot ever in programming, writing, and examinations; it continues to improve with each update. Despite the challenges and risks to education, it is essential to integrate Gen AI into higher education by equipping graduates with Gen AI proficiency for various sectors, enhancing personalized education, and preparing for future societal roles [10]. A growing body of research has explored the practical impact of Gen AI on student learning [27,28].

Gen AI’s great potential in facilitating students’ programming learning lies in its ability to bridge the gap between non-programmers and the programming world through its conversational language capabilities [29,30,31,32]. Unlike traditional programming resources that often require a foundational understanding of programming concepts and are typically focused on specific programming languages or environments, Gen AI leverages machine learning and natural language processing technologies. This enables it to understand and respond to user queries in everyday language, making programming more accessible to individuals without programming knowledge [29].

Several studies have explored the potential of Gen AI tools to enrich the learning experience in programming education. Yilamz and Karaoglan Yilmaz [30] investigated university students’ views on using the Gen AI tool for programming learning. Through a case study involving programming assignments over 8 weeks, data were collected via questionnaires and open-ended questions. The findings highlighted both benefits like quick responses, debugging support, and thinking skill development and limitations like potential laziness, incorrect answers, and a lack of programming environment. Moreover, through quasi-experimental designs assessing control and experimental groups, Yilamz and Karaoglan Yilmaz [33] and Sun et al. [31] explored the integration of Gen AI into programming education, focusing on a range of outcomes including student engagement, computational thinking, self-efficacy, and motivation. The first study utilized mixed methods to measure the impact of Gen AI tools on computational thinking, programming confidence, and learning motivation, revealing significant enhancements in these areas. The second study compared the effects of ChatGPT-assisted programming with traditional self-directed learning methods on college students’ programming behaviors and performances. While it was found that ChatGPT increased student engagement, as evidenced by more frequent debugging and interactions, it did not significantly improve programming performance compared to traditional methods.

However, beyond its immediate impact, Gen AI can engage in more meaningful and ongoing conversations with students. The interactions between students and Gen AI, as well as the generated conversations, have not been explored in depth or mentioned in detail in previous research. Hartley et al. [32] examined ChatGPT’s effectiveness across four learning domains: instructional content, programming tools, feedback, and planning. One of the important findings they emphasize is that the success of Gen AI in education relies heavily on the user’s competence. Therefore, it is crucial to study and understand the interactions that students have with AI during their programming learning process.

2. Methodology

2.1. Class Design

The entire class design is illustrated in Figure 1. The class was conducted in four lessons per month, each lasting 180 min. In this class, our goal is to guide students to build effective communication with the Gen AI as well as effective collaboration with the Gen AI in understanding and app development. The course content is progressively more difficult.

In the first lesson, students were introduced to Gen AI through the instructor’s practical demonstrations. They learned to communicate with Gen AI and write effective prompts by having Gen AI solve actual report problems in class, understanding its performance, and exploring its use in their own lives and hobbies.

The second lesson focused on collaborating with Gen AI to understand difficult programming concepts and codes. Students independently deepened their understanding of difficult codes or concepts by engaging in a dialogue with Gen AI, asking questions, and receiving explanations, problem solutions, and feedback on their answers.

In the third lesson, students collaborated with AI to create a simple chatbot application under their guidance. The instructor demonstrated their own development process as an example. Thereafter, students created a basic, minimally functioning app as demonstrated by the instructor and then modified it according to their own preferences.

In the fourth lesson, students developed their original applications with Gen AI. In their app development, they were required to create a balance between “execution” and “understanding”. “Execution” means students needed to ensure the codes were working as well as expected, while “understanding” means students needed to understand all the codes and explain them in their own words.

After all the lessons, students could continue to complete the app alongside Gen AI.

2.2. Teaching–Learning Activities

The lessons were founded on two principal considerations. The primary focus was to help students to forge effective communication with Gen AI. The instructor, who has six months of experience using Gen AI in programming learning, taught communication strategies and coding techniques with Gen AI, articulating five essential principles for beginning students:

• Prompt management: the instructor guided students to articulate their goals, roles, and the context of their interaction with Gen AI, ensuring learning was both contextual and progressive.
• Memory management in Gen AI: to preserve Gen AI’s contextual comprehension, it was advised to initiate new conversational threads at suitable junctures. This approach mitigates the risk of diminished response accuracy due to protracted dialogues.
• Developing working code: aiming to boost the self-efficacy and motivation of novice students, the recommended approach was to start with a simple, functional code and gradually refine its features. This method lays a robust foundation for further development.
• Balancing execution and understanding: the utility of Gen AI in code generation was acknowledged, but the instructor stressed the importance of understanding the generated code to effectively address challenges. Striking a balance is crucial, as an excessive focus on understanding every detail can hinder progress.
• Seeking feedback: the importance of obtaining feedback from Gen AI on programming and prompt management was underscored, facilitating a reflective learning process.

The second consideration revolved around integrating collaborative learning principles into the framework of student–Gen AI interaction. Highlighting the importance of student interdependence and individual accountability, rather than relying heavily on detailed educational materials and methodologies, the instructor facilitated an environment where learning was driven by student engagement with Gen AI. Regular group discussions were conducted, enabling students to share their learning experiences and solutions, thereby fostering a collaborative learning atmosphere. The instructor and teaching assistants participated in these discussions, providing insights and guidance to enhance the learning experience.

2.3. Participants of Classes

To ensure comprehensive instruction and personalized support for each student, as well as to closely monitor their learning progress, the class was kept small and comprised nine students, all of whom were third-year undergraduate students from the college of education at a prominent Japanese university. Prior to enrolling in this class, each student had acquired a foundational knowledge of programming.

2.4. Data Collection

This study applied a comprehensive data collection approach to investigate student learning processes and their interaction with Gen AI.

2.4.1. Class Observations and Questionnaire Surveys

Data were gathered through observations of the classroom setting and the administration of questionnaire surveys before and after lessons. These surveys comprised both descriptive questions and 5-point Likert scale items and were conducted at six different times to capture the evolution of students’ experiences and perceptions over time. Specifically, they aimed to explore students’ interactions with Gen AI, their learning outcomes, and their attitudes towards both the Gen AI and the course design. For further details, please refer to Appendix A.

2.4.2. Final Reports

Students’ final reports were submitted after all the lessons and data were collected. The final reports were used to collect the strategies students used when building collaboration with Gen AI and the learning outcomes in programming learning, which included the following three topics: (1) How would you describe your strategies in collaborating with Gen AI to learn about codes and concepts? (2) How would you describe your strategies in collaborating with Gen AI to develop an app? (3) What are the most important learning outcomes of these four lessons?

2.4.3. Records of Dialogue with Gen AI

After the four lessons, all students’ records of dialogues with the Gen AI were required to be submitted. These dialogues were analyzed as data on student–Gen AI collaboration.

2.4.4. Interviews

In-depth semi-structured interviews lasting 40–60 min with three selected students (Emma, Olivia, and Sophia) provided qualitative insights into their learning experiences, class design, knowledge, attitudes toward Gen AI, and comparisons of Gen AI with live teachers.

2.5. Data Analysis

We used qualitative content analysis to analyze documents and interview data [34]. After several careful readings, relevant data on RQ1–RQ4 were drawn. The first author generated initial codes through a repetitive reading of transcripts and conducted an inductive thematic analysis to develop initial themes. Then, two coauthors reviewed all transcripts to thoroughly examine codes and identify any differences in interpretations. In cases of disagreement, the analytic process and findings were discussed until a consensus was reached. After completing the data analysis, all data were translated from Japanese to English and all translations scrutinized to ensure no misunderstandings arose from the translation.

3. Results

3.1. RQ 1: How Do Students View Their Communication with Gen AI?

To answer the first research question, the data from the survey results, final reports, and classroom observation were analyzed. As a result, there were changes in the way they communicated with Gen AI as the class progressed (Figure 2).

At the beginning of the class, students did not have a clear perception of Gen AI; however, as the class progressed, their communication methods changed. The initial survey conducted before the first lesson showed that students perceived Gen AI as a basic information provider, such as Siri or OK Google, and were concerned about how to use it.

However, as the class progressed, the students began learning to communicate effectively with Gen AI and explored different types of communication, from question-and-answer format to making reasonable and clear statements, to thinking based on the response of Gen AI, and to clarifying and confirming their thoughts with Gen AI.

Below is an example of the type of communication used by a student:

“1. Explicitly state my position and the purpose of what I want Gen AI to do. First, I outline what I actually want Gen AI to do and then list more detailed information in bullet points.

2. Ask questions about what I don’t understand about the responses generated by Gen AI and reiterate my intentions precisely if Gen AI’s answers differ from my own.

3. Once I understand, I ask Gen AI to evaluate what I have created according to what I have learned. In doing so, I clearly indicate to Gen AI the evaluation criteria it will use.

4. Revise according to the advice and have Gen AI evaluate it. I’ll keep repeating this until Gen AI says, ‘good’”.

Through this communication process, the students learned how to interact with Gen AI and began to introspect and reflect on their learning styles. They began to think about how to orient Gen AI to cooperate with their learning based on their anxieties, such as not being able to judge the accuracy of the information provided to Gen AI. In this regard, a student made the following comment:

“In the ChatGPT review, I heard some people say that it gives wrong answers or that the answers are inappropriate, but I felt that this is a matter of how we interact with it. I felt that it is important to consider it a companion at this stage, rather than an omniscient being that returns answers to random questions”.

Moreover, a student expressed that evolving communication with Gen AI deepened his structural and logical thinking, which is fundamental to learning programming.

Furthermore, communicating with Gen AI improved students’ technical, questioning, and expression skills. Students recognized that accurate information transfer is beneficial not only for Gen AI but also for human-to-human communication.

Below are comments from two students:

“I learned that I could get effective answers from Gen AI depending on my questioning and scene-setting skills. Specifically, it was effective to present my position and Gen AI’s position and present some conditions in a concise manner. I felt that the process of obtaining an answer was also important, as writing a communicative and concise statement was in itself a learning experience”.

“When conversing with people, I had never thought about defining the other person’s position because I unconsciously select and choose questions based on their background and attributes. However, because the assumptions behind one’s thinking are hidden in this unconscious selection, I realized that it is important to face one’s own thinking when creating a prompt and verbalize the background of the question. I felt that this ability to explain the background is a skill that can be applied in group discussions as well”.

3.2. RQ 2: How Do Students Develop Their Collaboration with Gen AI in Concept Learning and App Development?

To answer the second research question, the data from final reports were analyzed.

In the final reports, based on their experiences in and after the class, students reported a wide variety of their collaboration with Gen AI in code learning and app development (

Table 1. The strategies formulated by students to collaborate with Gen AI in code learning and app development.

Learning Task	Strategies
Understanding codes or concepts	Organize my own state of understanding before asking Gen AI Dialogue with Gen AI to clarify difficult areas Check the reliability of information and expand knowledge while interacting with Gen AI Paraphrase the response from Gen AI in your own words, think critically and analyze
App development	Self-analysis before setting tasks Clarify the issue by interacting with Gen AI Setting strategies from simple to complex Seek solutions in collaboration with Gen AI rather than asking Gen AI to find a solution

).

First, students recognized several differences between asking teachers and asking Gen AI. They stated that teachers could explain the concepts and codes according to the students’ level because they were better able to assess students’ learning status. When students asked Gen AI for explanations, they needed to self-evaluate their learning status before asking and then ask more clearly and understandably. In some cases, they asked questions more often to examine Gen AI’s responses.

In terms of this situation, students described many strategies to interact with Gen AI more effectively.

Rather than just asking Gen AI to explain the codes or concepts, students tried to analyze their own understanding of themselves before asking. For example, one student stated the following:

“To begin with, I think that matters that are difficult to understand on one’s own can be divided into two patterns: ‘I don’t understand X’ and ‘I don’t understand what I don’t understand in the first place’. If the points that I don’t understand are clear, I can pinpoint them and ask for clarification on my own. And, if I don’t even know what I don’t know, I just listen to the explanation and then ask for more details one by one to move from ‘I don’t know what I don’t know’ to ‘I don’t know what X is’”.

Regarding the possibilities of bias or wrong information from Gen AI, students requested information sources and asked Gen AI to explain the codes in different ways.

One student asked Gen AI to provide some practical examples to broaden their knowledge:

“By asking Gen AI to provide real-life examples, I am able to consolidate not only my understanding of the concept I asked, but also other concepts connected to it, and I am able to expand my knowledge”.

Moreover, compared to just passively accepting the explanation from Gen AI, students emphasized the need to analyze the response and paraphrase it in their own words to confirm their understanding.

For example, one student stated:

“By paraphrasing Gen AI’s explanations in my own words, I can organize my understanding. If there is something I cannot put into my own words during the paraphrasing process, I can solve this by asking Gen AI to elaborate on it again”.

Regarding application development, students described several strategies for figuring out the codes that they did not know including the product working codes.

For example, some students highlighted the benefits of progressing in small increments. One student stated:

“I think it is very suitable for me develop the application step by step. If I proceed all at once, [Gen AI may give me too much information all at once], so it would be very difficult to understand all information and to write working codes”.

Moreover, some students also described how to respond to an “error” in programming. Instead of just asking the Gen AI for solutions directly, students stated that identifying the reason by themselves first and then discussing with Gen AI would be an effective way to solve the problem.

“If an error occurs, working with Gen AI to determine the cause and remedy and sharing error messages as I go. If I don’t understand it, I proceed by checking the accuracy of my understanding”.

“I feel that when I get stuck, I like the approach of presenting another way, one that I can also think about”.

3.3. RQ 3: How Do Students Evaluate Gen AI in Their Learning?

To answer the third research question, the data from the surveys and interviews were analyzed.

Regarding the perception of Gen AI, students stated the Gen AI provides many advantages to promote their learning efficiency and impact their learning approaches (

Table 2. The advantages of Gen AI on students’ learning.

Advantages of Gen AI	Examples
The advantages to improve learning efficiency	Immediate response Accessibility Personalized difficulty levels Encouragement of questioning Continuity in learning Efficient information gathering
Positive impacts on learning approach	Diversity of learning styles Integration of input and output learning Real-time programming learning Promotion of student interdependence and self-regulated learning

).

3.3.1. Advantages of Gen AI

Students appreciated the fact that Gen AI answered their questions immediately and said that they felt it was beneficial for beginner learners, especially when understanding Python syntax, as they could ask questions as they went along.

One student stated:

“I can increase the frequency of questions until I understand, and it doesn’t matter what time or space I ask in, and the Gen AI explains the function along with the code”.

Another major advantage of Gen AI is that students can easily ask even trivial questions that are difficult to pose to teachers or instructors. In particular, students made the following comments regarding the possibility of making requests to the AI to learn at their own pace and adjust to specific difficulty levels:

“It was good because with Gen AI, I could ask for a change in difficulty level without any hesitation, which is something I would have had a hard time expressing to a teacher”.

“When I was learning under teachers, it was difficult to ask questions if I forgot the content of the previous lesson. However, with Gen AI, I can persistently ask questions until my trivial questions are resolved, and I can ask them anytime”.

Moreover, students highly valued Gen AI’s ability to help them improve their performance. For example, students stated that even with limited basic coding knowledge, they could write code that functioned at a minimum level with the assistance of Gen AI. Furthermore, students appreciated that conversations with Gen AI are saved, allowing them to easily resume their work at a later time.

In addition, students appreciated the convenience of Gen AI in collecting and filtering information.

“Unlike reference books or website articles, there is no need to retrieve information from many sources. Thus, I could effectively distinguish between necessary and unnecessary information”.

3.3.2. Impact on Learning Approach

In addition to the convenience that Gen AI brings to student learning mentioned above, we also found that the collaborative relationship between students and Gen AI has a profound impact on the way students learn. These effects not only broaden the diversity of learning styles but also promote student autonomy.

Students introduced their new learning styles through interactions with Gen AI, which involved simultaneous input and output of knowledge.

“I used to learn in a cycle of preparation → lecture → review → problem solving, but now I have found a new way to learn that combines lecture and problem-solving using Gen AI that allows me to check my understanding through dialogue”.

“The conventional learning method is to first input knowledge and then acquire an understanding of it through practice and output. However, the use of Gen AI has changed this style to one in which knowledge input and output are performed simultaneously while learning programming in real-time through hands-on experience”.

Moreover, students highly appreciated the usefulness of collaborating with Gen AI in their programming learning. Particularly, through the collaboration with Gen AI, their self-regulated learning was strengthened because of the freedom to set their own learning goals, prioritize the content they wanted to learn, and evaluate the learning process by themselves.

“When learning through Gen AI, I must consider what I want to do, have Gen AI generate the code to realize it, and then learn what functions it uses. I found that learning through Gen AI is unique in that I can logically construct what I want to do and the strategy for achieving it using natural language, and then learn how to use the programming language”.

3.4. RQ4: What Kind of Challenges Are Faced by Students in Their Collaboration?

To answer the last research questions, the data from class observation, questionnaire surveys, and the records of student–AI dialogue and interviews were analyzed.

3.4.1. Difficulties to Collaborate with Gen AI in Complicated Learning Tasks

The app development task is more complicated than understanding difficult concepts because it requires student self-regulated learning skills such as setting learning goals, learning new codes, adjusting strategies, and finding the cause of an “error” with Gen AI. In the questionnaire surveys of the third and fourth lessons, most students expressed difficulties and frustration in building collaborative relationships.

When asking Gen AI to explain its code or provide solutions to errors, the students felt that they did not know enough to understand Gen AI’s answers. Consequently, they struggled when interacting with Gen AI and felt that they lacked basic programming knowledge.

For example, a student reported:

“I was able to understand what I was doing with functions and conditional branching. However, when it came to the details of functions and things that were a bit more complicated, I still didn’t understand. Also, when an error was returned, I completely relied on Gen AI. I felt it was necessary to learn by myself because I could not find out the reason when the error occurred again. I felt it was necessary to learn on my own”.

Overall, the students were unable to share a common understanding with Gen AI, making it difficult for them to propose improvements. Therefore, as Emma said in the interview, it was difficult to strike a balance between what they could do and what they understood.

“I think Gen AI is adequate for the purpose of creating apps. However, it would take a lot of time to connect this to my deeper understanding. Personally, I found it a bit daunting”.

3.4.2. Consideration of Differences between Students

Another challenge in student–Gen AI collaboration is individual ability. After all the lessons, there was one questionnaire related to students’ self-evaluation about their ability to formulate satisfactory prompts. The results showed that two students were unable to create a satisfactory prompt, even after all the classes.

Regarding this point, a student shared his opinion:

“I feel that Gen AI excels in its ability to provide humans with a great deal of knowledge. However, I feel that how well it utilizes that capability is highly dependent on humans’ ability to express itself and its logical thought processes”.

Moreover, individual differences were also reflected in their dialogue records with Gen AI in app development. There was one student leaning too heavily on AI, although confirmation of their understanding could be seen from time to time. Most students demonstrated moderate skills in collaboration with Gen AI and tried to interact with Gen AI actively to deepen their understanding and finish their app development. One student successfully built a collaborative relationship with Gen AI in app development (see Appendix B). This student’s approach to collaborating with AI for app development was exemplary, showcasing a methodical and interactive process that greatly facilitated problem solving (Figure 3).

Initially, the student established a solid foundation for effective communication with the Gen AI by articulating well-defined prompts that included the current learning status, learning objectives, the intended learning process, and the role of Gen AI, as illustrated in Figure 4.

Engaging with AI’s responses through additional suggestions and questions deepens the dialogue, leading to more precise solutions. Confirming personal understanding and addressing any miscommunications promptly help in avoiding errors, while posing deeper inquiries enhances the coding proficiency (Figure 5).

4. Discussion and Conclusions

In this case study, we designed a programming class for instructing students to develop an effective communication and collaborative relationship with Gen AI. We used multiple data collection to investigate students’ learning experiences and discuss the impact of student–Gen AI collaboration on student learning.

The results of RQ1–RQ3 highlight the benefits of student–Gen AI collaboration in student learning. On the other hand, the RQ4 results reveal the challenges and individual differences in student–Gen AI collaboration.

The results of RQ1 reveal the transformation of students’ perception of and communication with Gen AI. In exploring continuous and effective communication with Gen AI, students realized the importance of making clear statements, critically thinking about Gen AI’s response, rephrasing Gen AI’s response, and appropriately evaluating their learning outcomes. Together with this exploration, a change occurred in the student–Gen AI relationship. Before the class, students were typically the recipients of Gen AI, simply judging the effectiveness of AI as a learning tool. After the class, students used “companion” to describe AI–human relationships, underlying their evolution as collaborators and leaders of Gen AI. More importantly, students extended the ideas learned in communication with AI to programming learning and to interpersonal communication, even though they were not directly taught in class. This indicates that continuous and effective communication can improve students’ critical thinking and logical thinking and positively influence human–human communication.

The results of RQ2 illustrate the strategies described by students to develop collaboration with Gen AI in their programming learning. Students’ stated strategies included self-assessment and active engagement with AI, such as seeking clarifications and paraphrasing responses in learning difficult codes. In the context of app development tasks, students described the strategy by breaking down intricate tasks into manageable segments and advancing incrementally. Moreover, they leveraged discussions with Gen AI to gain diverse perspectives, aiding in problem solving and task completion.

The synthesis of findings from RQ1 and RQ2 points to an enhancement in students’ meta-cognitive and self-regulated learning skills through their interactions with Gen AI [35,36]. Meta-cognition, the awareness and regulation of one’s cognitive processes [37,38], is mirrored in students’ critical evaluation of Gen AI responses and the adaptation of communication strategies. Similarly, the self-regulated learning process, characterized by goal setting, self-monitoring, and strategic planning [36,39,40], is reflected in the students’ methodical approach to learning programming with Gen AI’s collaboration. Particularly, drawing upon Zimmerman’s [40] model of self-regulated learning, the transformation observed in students can be contextualized within the three phases of self-regulated learning: forethought, performance, and self-reflection. The students’ initial setting of learning objectives and planning (forethought phase) was enriched by their interactive experiences with Gen AI, leading to more effective strategies and performance adjustments (performance phase). The subsequent reflection on these experiences and outcomes (self-reflection phase) likely contributed to an iterative cycle of improvement in both meta-cognitive knowledge and self-regulation skills.

RQ3’s results showed the advantage of Gen AI in students’ learning such as immediate response access and non-judgmental interaction, which have been demonstrated in many previous studies [41,42,43]. More importantly, there is a significant change in students learning approach from traditional sequential learning to enabling simultaneous knowledge input and output, thereby offering a hands-on experience in real time, leading to more profound learning outcomes.

The results of RQ4 reveal the difficulties and individual differences in student–Gen AI collaboration for complicated learning tasks.

Students faced difficulties in correcting miscommunications with Gen AI, pointing out Gen AI’s mistakes, and seeking solutions together to ensure their learning outcomes were achieved. These difficulties reflected the challenge of setting shared learning goals with equal partnership in building sustainable student–Gen AI collaboration for achieving complicated learning outcomes. Therefore, more empirical research is needed to explore how students can communicate more equally and effectively with Gen AI and how they can lead Gen AI to help themselves through more complex learning processes.

Furthermore, the individual differences in student–Gen AI collaboration raise a great concern. Several studies have suggested that the success of AI applications in education depends on learners’ self-regulated learning skills and how learners utilize them [32,35,44]. Our study demonstrates that the students who possess strong self-regulated learning skills may be able to keep high levels of interdependence in the interaction with Gen AI and establish more successful student–Gen AI collaboration to achieve better learning outcomes. Hence, the learning outcomes’ gap would be significantly increased between active learners with a high level self-regulated learning skills than passive learners.

Above all, this case study infers that student–Gen AI collaboration can effectively improve students’ meta-cognitive and self-regulated learning skills, which would be effective in avoiding surface learning. At the same time, students who have higher meta-cognitive and self-regulated learning skills could build more effective collaborative relationships with Gen AI. As a result, such students are likely to further develop their self-regulated learning skills and become independent learners and are able to continue learning with Gen AI without faculty intervention. Therefore, this study indicates greater effort should be placed on helping students maintain a higher degree of independence and leadership in their interactions with Gen AI.

5. Limitations

This case study faced two limitations. First and foremost, the small sample size of only nine participants from the education department limits the generalizability of our findings. Additionally, the dynamics of student–Gen AI collaboration are likely to be affected by the variability in instructors’ teaching methods and the diversity of subjects in this study [28]. Notably, our study benefitted from the instructors having extensive experience in programming and with Gen AI, which may not be a common scenario.

6. Implications and Future Directions

Compared to previous studies, focus was placed on the student–AI relationship first, and students were instructed to collaborate with Gen AI on learning tasks instead of just using it. As a result, several implications could be concluded.

Concerns have been raised about Gen AI’s potential to disseminate inaccurate or biased information [44]. Our findings suggest that students, by engaging with Gen AI from diverse perspectives and scrutinizing the sources of information, can mitigate this risk.

Moreover, consistent and meaningful interaction with Gen AI appears to positively influence students’ critical thinking skills and interpersonal communication.

Furthermore, contrary to the fear of superficial learning or the absence of learning due to overreliance on Gen AI, our study reveals that collaborations between students and Gen AI can deepen student engagement in the learning process and enhance their meta-cognitive and self-regulated learning skills.

However, the varied challenges and individual differences in cultivating effective student–Gen AI collaborations highlight the need for further research. Future studies should explore students’ AI competencies and their practical interactions with Gen AI to better understand and enhance this emerging educational dynamic. This will allow education in the Gen AI era to move in the direction of nurturing people who are independent and continue to learn with AI as a companion.