Engineering Education in the Age of AI: Analysis of the Impact of Chatbots on Learning in Engineering

Abstract

The purpose of this paper is to explore the influence of using AI chatbots on learning within the context of engineering education. We framed this study on the principles of how learning works in order to describe the contributions and challenges of AI chatbots in five categories: (1) facilitating the acquisition, completion, or activation of prior knowledge and helping organize knowledge and making connections; (2) enhancing student motivation to learn; (3) fostering self-directed learning and the acquisition, practice, and application of the skills and knowledge they acquire; (4) supporting goal-directed practice and feedback; and (5) addressing student diversity and creating a positive classroom environment. To elicit the uses, benefits, and drawbacks of using AI chatbots in students’ learning, we conducted a thematic analysis of qualitative data gathered from surveying 38 student volunteers from 5 different electronic and mechatronic engineering courses at a South American university. Based on a literature review and an evidence-based discussion, we offer practical suggestions for instructors who want to promote the use of AI to enhance their students’ learning.

1. Introduction

AI-powered chatbots, also known as artificial intelligence (AI) chatbots, are software applications designed to interact with humans through spoken or written language [1]. The rapid evolution of chatbot technology has been fueled by significant advancements in natural language processing (NLP) and machine learning techniques [2,3]. As a branch of AI, NLP empowers machines to effectively analyze and comprehend human language [4,5]. Particularly, the emergence of large language models (LLMs) has contributed to the growing popularity of NLPs, as they exhibit an extraordinary capability to analyze and process extensive text data [6,7]. LLMs are advanced neuronal models that have a considerable number of parameters. These structures integrate billions of parameters and generally train with large volumes of textual data collected from the web through supervised and reinforced learning techniques [8]. Examples of LLMs include Lamda (Google), Palm (Google), Bloom (BigScience), LLaMA (Meta AI), Bert (Google), Ernie (Baidu), PanGu (Huawei), Xlnet (Carnegie Mellon University and Google), and GPT (OpenAI) [9].

The generative pre-trained transformer (GPT) family, introduced by OpenAI, stands as one of the pioneer sets of generative LLMs. Its evolution has significantly influenced the fields of deep learning and NLP research [7]. With each successive iteration, these models have shown increasingly sophisticated language generation capabilities. At the time of writing this article, the latest installment of this series, GPT-4 (launched in 2023), has introduced notable improvements, including the ability to process textual indications and images, marking a significant leap forward in multimodal understanding and generation. Moreover, GPT-4 has substantially increased its number of parameters, exceeding 100 billion, allowing it to achieve unprecedented performance levels. This increase in parameters enables the generation of more natural text and resolves problems with greater precision compared to its predecessors [10].

Recently, there has been a noticeable increase in the release of conversational AI chatbots to the market. Among these, ChatGPT, powered by OpenAI (debuted in November 2022), is one of the most famous and runs on the GPT family of models. Other AI chatbots available on the market include Bing Chat, developed by Microsoft (2023); Bard (Language Model for Dialogue Applications), developed by Google (2023); and Claude, developed by Anthropic (2023). These AI chatbots can generate sophisticated text and engage users in a human-like conversational style [3,11,12]. Also, they are versatile tools that can help with a wide range of tasks, such as finding information, writing documents, brainstorming on various topics, translating texts into different languages, synthesizing content, improving writing quality, troubleshooting, and even writing programming code [13,14]. Although these LLMs are currently in the development phase, they hold enormous potential to revolutionize the way we interact with computers, access information, and enhance overall task efficiency, thereby bringing potential benefits as well as difficulties to the educational processes [3,15,16,17,18].

On one hand, AI chatbots have emerged as a promising and transformative tool, capable of significantly enhancing the learning and teaching processes in engineering education [12,19]. These tools offer continuous and real-time support to engineering students [20], helping them understand complex engineering concepts, solve engineering problems or mathematical challenges [12,21], as well as generate ideas, and queries related to engineering projects [12,17]. The benefits of AI chatbots extend to professors, who experience numerous advantages as well [1,3,17,18]. For example, AI chatbots can save time through efficient lesson planning and easier generation of class materials or support personalized teaching approaches. This facilitates educators in tailoring their instruction to the individual needs of students [22].

On the other hand, AI chatbots in education present a variety of challenges that demand careful attention and resolution [3,18,20,22]. These challenges encompass ethical concerns, the accuracy and reliability of AI chatbot responses, the preservation of academic integrity, and the potential impact on critical thinking and problem-solving skills in students [22,23].

Nonetheless, there is a significant gap in the current research concerning the potential of AI chatbots in engineering education. While some studies focus on the influence of AI chatbots in education, others examine their engineering implications, but few specifically deepen engineering learning. This research is motivated by such a gap in the existing literature.

The purpose of this paper is to examine the influence of using AI chatbots on learning within the context of engineering education. To achieve this objective, we conducted exploratory qualitative research. Based on an analysis of the literature, we discussed the potential contributions and challenges of AI chatbots in learning in engineering and surveyed students from five courses of the mechatronic engineering major at a South American university to gain insights into the use, advantages, and challenges of AI chatbots in their learning. Framed by the understanding of how learning works as described in the principles of learning by Lovett et al. [24], we structured and reported the analysis of the literature, results, discussions, and conclusions of our study and offered practical recommendations for instructors who seek to use AI chatbots to enhance their students’ learning experiences.

2. Contributions of AI Chatbots to Learning

In this section, we review the literature to delineate the contributions, advantages, and potential risks of AI chatbots to the learning process. Following the learning principles stated by Lovett et al. [24], we have structured this review to elaborate on the contributions of AI chatbots in (1) facilitating the acquisition, completion, or activation of prior knowledge and helping organize knowledge and making connections; (2) enhancing student motivation to learn; (3) fostering self-directed learning and the acquisition, practice, and application of the skills and knowledge they acquire; (4) supporting goal-directed practice and feedback; and (5) addressing student diversity and creating a positive classroom environment.

We acknowledge that Lovett et al. [24] introduced eight principles of learning; however, after analyzing the literature on AI chatbots in education, we did not find enough evidence that applied to all the principles; hence, we decided to group our findings into five categories. We used this set of categories as the framework to maintain consistency in our manuscript.

2.1. Contributions to the Acquisition, Completion, or Activation of Prior Knowledge and to Help Organize Knowledge and Make Connections

According to [24], both students’ prior knowledge and the way students organize knowledge and make connections significantly influence their learning. When students have accurate and relevant knowledge, it can help them better understand, interpret, filter, and retain new information. In contrast, prior knowledge that is inaccurate, incomplete, or not activated effectively can impede learning.

When introduced to new material, individuals try to retrieve prior knowledge from their past experiences to help them understand, interpret, and respond to the new input [24]. In that sense, AI chatbots can help students to quickly acquire the background information they need and give access to a wealth of information and relevant resources, including textbooks, documents, research articles, videos, and other online materials [15]. More importantly, students can interact with the AI chatbot using the same natural language as they would when querying tutors or professors [3]. This capability allows for the timely adaptation of previous learning material to the specific needs of each student, significantly speeding up their acquisition of prior knowledge. It could also help them remember previous information, bring it to the forefront of their consciousness, and suggest how to apply it to the new material or task [20].

Similarly, AI chatbots could offer instant and personalized explanations for student queries. Specifically, they can provide students with complete explanations, along with relevant examples of complex concepts and theories, making them more accessible and relatable to students. For example, in a study [21] on integrating ChatGPT in math education, students highly valued its effectiveness in facilitating a clear and comprehensive understanding of diverse and complex mathematical concepts.

Additionally, the way students organize knowledge and make connections influences how they learn and apply what they know because this organization and structure foster a deeper and more contextualized understanding of the content [24]. In this regard, AI chatbots facilitate the connection between new information with prior knowledge, and structure students’ ideas and thoughts [3,18]. In turn, this improves the overall coherence and clarity of their written work [18]. For example, students can use AI chatbots to create outlines for reports, articles, essays, or other forms of writing, making them a valuable tool for organizing ideas during the research and writing process [25,26]. Moreover, AI chatbots can also be utilized to facilitate group discussions and debates by offering personalized guidance to learners throughout the process. For example, chatbots can suggest a discussion structure and offer relevant discussion topics [3,27], or they can break down complex concepts into simpler elements, making them easier to assimilate [21].

In contrast, there are disadvantages to using AI chatbots to activate and organize student’s prior knowledge. Students encounter challenges in assessing the accuracy of information, potentially resulting in the acceptance of erroneous content. AI chatbots learn from vast amounts of data, but their sources may contain biases or inaccuracies [17,20,27]. This is problematic for students who can unknowingly absorb and internalize this misinformation, leading to misconceptions and errors in their understanding [12,18,28]. For example, Jalil et.al., [29] found that although ChatGPT can respond to 77.5% of the examined questions, it provided correct or partially correct answers in only 56.6% of cases and correct or partially correct explanations in only 53% of cases. Other studies have revealed that ChatGPT can exhibit inaccuracies in mathematical calculations and may also generate errors in algorithms or programming codes [27,30]. Also, the quality of the AI chatbot responses is heavily influenced by the clarity and quality of students’ questions and prompts, as clear and well-structured inquiries tend to yield better responses from AI chatbots [11]. In short, AI chatbot responses may be inaccurate or unreliable, which in turn makes the understanding and integration of new knowledge difficult.

2.2. Contributions to Motivate Students to Learn

Students’ motivation is the engine that determines, directs, and sustains the actions students take to learn and significantly influences how students interact and approach their learning experiences [24]. When students recognize the value of their learning goals, have confidence in their potential for success, and experience a supportive learning environment, their motivation to learn is enhanced [31].

AI chatbots have the potential to enrich students’ learning experiences and improve their engagement and motivation [32,33,34]. First, AI chatbots can create more engaging and personalized learning content for students based on their individual preferences and needs, giving them a better sense of autonomy, control, and interest in their learning experience [17]. Second, chatbots can improve student interest and engagement by stimulating their curiosity [35]. By offering interactive and personalized options or granting access to resources at any time, chatbots can motivate students to make the most of their time outside the classroom for flexible and convenient studying, thereby deepening their knowledge. Moreover, using chatbots can result in significant time savings [17]. For example, AI chatbots can efficiently search across multiple sources and search engines, quickly providing relevant information to students [11]. Third, the students’ sense of success is reinforced by the timely assessment of and feedback on their performance [20].

2.3. Contributions to Promote Self-Directed Learning and to the Acquisition, Practice, and Application of Skills and Knowledge

According to [24], promoting mastery and self-directed learning enhances the overall learning experience. It enables students to seamlessly integrate and apply acquired knowledge and skills, fostering the development of metacognitive processes, deeper understanding, and autonomy. Additionally, cultivating mastery in students empowers them to integrate and apply their acquired knowledge and skills seamlessly and effectively. Furthermore, developing metacognitive processes, such as evaluating tasks, self-assessing strengths and weaknesses, and monitoring and adjusting their learning approaches, can improve students’ performance and effectiveness as learners.

Chatbots can contribute to the acquisition of skills and concepts while promoting practice and the application of knowledge [34]. On one hand, they provide personalized practice by offering exercises and activities that can be tailored to each student’s skill level by generating a wide range of exercises, problems, and projects, including programming, and data analysis [36]. On the other hand, they can aid in solving these exercises and problems by offering step-by-step guidance and comprehensive explanations of the solutions, which in turn improves students’ understanding [21].

Moreover, AI chatbots can enhance students’ programming skills [3]. They can help students understand programming concepts, including data structures, algorithms, programming languages, syntax, and error detection. For example, these chatbots can either generate sample programming codes or solution codes based on problem descriptions, or they can provide step-by-step code generation with detailed explanations for syntax and programming logic. Furthermore, they can contribute to code optimization by providing suggestions to reduce memory usage and improve time complexity, enhancing the efficiency and performance of the programs [3]. Students can also take advantage of AI chatbots to detect code errors, helping them to identify and solve problems in their programming projects [3].

Similarly, these chatbots can also support professional skill development. Particularly, AI chatbots can help students practice and improve their language skills. First, these tools can generate interactive dialogs that simulate conversations with native speakers [15,37], or they can be used to create custom vocabulary and grammar exercises, as well as to generate text in specialized domains, such as technical, medical, or business English [3,38]. Second, AI chatbots can greatly enhance reading comprehension by providing instant definitions, explanations, and examples of unfamiliar words or phrases [37]. Third, they can help students improve their writing skills by providing suggestions to improve sentence structure, grammar, and vocabulary, or by pointing out spelling mistakes, generating examples, and providing explanations of language use [20,37]. Fourth, they can assist students with technical report writing, such as medical, legal, and IT reports [18]. Finally, AI chatbots can function as research assistants, offering valuable support for a variety of research-related tasks. They help students locate relevant research papers, summarize them, assist in data analysis, and answer questions based on the knowledge they have gained from related literature and data analysis [3,20].

However, if students rely on AI chatbots for tasks such as writing, problem solving, and developing algorithms or code on their behalf, they might fail to develop these essential skills for their academic and professional success [18]. By depending on chatbots, students can limit their opportunities to practice and apply these skills [12,20]. This may result in a lack of innovation and hinder their decision-making abilities, potentially affecting their overall growth and development [18]. Additionally, chatbots can sometimes provide mechanical practice that focuses primarily on task execution, neglecting the promotion of a deep understanding of concepts [39].

Finally, although engaging in metacognitive processes does not come naturally to students, AI chatbots can facilitate it. For example, AI chatbots can help students prepare for exams by generating tests and exams, allowing students to self-assess their knowledge and providing clarification, troubleshooting guides, and access to relevant course content [17,27].

2.4. Contributions of AI Chatbots to Help Goal-Directed Practice and Feedback

Goal-directed practice, coupled with targeted feedback, enhances the quality of students’ learning. This principle emphasizes that to enhance learning, students should practice with a clear goal, an appropriate level of challenge, and sufficient repetition, along with timely feedback that provides goal guidance and helps make rapid progress [24].

AI chatbots can support goal-directed practice and provide timely feedback, which reinforces the effectiveness of independent study. First, they can act as virtual tutors, offering ongoing assistance to students outside of regular school hours and making learning more accessible and convenient [18,27]. Second, AI chatbots offer quick responses to queries and provide immediate feedback on varied concerns, questions, assignments, or projects [20,33]. This timely feedback helps students to rapidly make necessary improvements, identify misconceptions, or recognize inaccurate or incomplete ideas about a concept [20].

Also, these chatbots can provide students with valuable support in the development of tasks and projects. For example, they support students in researching and defining project problems, setting clear objectives, refining requirements, facilitating the brainstorming process, recommending suitable methodologies, suggesting relevant resources, and addressing specific challenges that arise during project development [18].

On the negative side, there is a risk that students will rely too much on the automatic feedback provided by these systems instead of developing critical skills, such as the ability to independently evaluate their work. Chatbots can provide inaccurate or superficial feedback that lacks depth and precision, and they may not be able to provide adequate or valuable feedback on complex tasks or in a very specific field [27].

2.5. Contributions of AI Chatbots to Address Student Diversity and Create a Positive Classroom Environment

The learning environment, encompassing both the classroom climate and the diversity among students, plays a pivotal role in shaping the learning experience. According to [24], students differ from each other in multiple dimensions, including their identities, stages of development, and personal histories. These differences influence how they experience the world and, in turn, their learning and performance.

Chatbots can provide content considering the characteristics and needs of individual learners, including accommodations for non-English speakers or students with disabilities. First, AI chatbots can generate customized exercises, quizzes, and study materials tailored to meet specific individual requirements [20,22,37]. Second, they can help non-English-speaking students overcome language barriers [20]. Chatbots can offer language translation, grammar correction, and writing suggestions [17,28]. Third, chatbots can enhance the accessibility of educational content for learners with disabilities by generating transcripts or subtitles for videos [15] or offering real-time speech-to-text and text-to-speech [3]. These accommodations address some needs of students with hearing or visual impairments [3]. However, not all students may have equal access to AI chatbots due to technological or financial constraints, such as limited internet access, difficulties in accessing a computer or mobile device, or the inability to afford subscription plans [28,40]. As a result, there can be potential inequities in learning opportunities among students [12,20].

Moreover, AI chatbots may face limitations in understanding and appropriately responding to emotions and contexts, which can result in inappropriate responses. Likewise, there is a risk of creating a cold and mechanical environment if interaction with chatbots becomes exclusive, which could negatively affect the warmth and human dimension in the class environment [12,39]. If students become overly dependent on AI chatbots for consultation and academic support, their engagement in direct interactions with professors may decrease. As a result, professors may have fewer opportunities to closely monitor student’s progress and guide them in their learning process [12].

This literature review also reveals a gap in the current research regarding the potential of AI chatbots in engineering education. Although many studies examine the benefits and challenges of AI chatbots in teaching specific STEM topics (see for example [21,29,41]), or emphasized the advantages and challenges of AI chatbots in education in general (see [17,20,27,28]), few analyze how chatbots affect the learning process (see [12,32,35,36]), and very few specifically delve into learning in engineering (see [36]). Our study aims to address this gap by examining students’ perceptions regarding the impact of AI chatbots on their learning experiences within the field of engineering. To frame our entire study, we also used recognized research on how learning works.

3. Materials and Methods

3.1. Type of Study

This study employs an exploratory qualitative methodology, gathering data from surveys aimed at comprehending the use, advantages, and constraints of AI chatbots in the student learning process across five purposefully selected engineering courses. Furthermore, this study uses thematic analysis to classify and interpret the qualitative data [42,43].

3.2. Selection of Engineering Courses for Study

To better understand the transformative potential of AI chatbots in practice, we encouraged the use of AI chatbots (mainly ChatGPT^®) in five junior and senior courses within the mechatronic engineering program at a public rural university in South America. We purposefully selected these courses based on the ease of naturally integrating AI chatbots as pedagogical tools both inside and outside the classroom. This approach, as well as the professors’ willingness to adopt AI chatbots in their teaching, guaranteed that the selected courses offered a conducive environment to explore the potential of chatbots in education.

The selected courses were Basic Mechatronic Technologies, Electronics III, Microcontrollers, Interdisciplinary Projects Workshop I, and Interdisciplinary Projects Workshop II. In the Basic Mechatronic Technologies course, junior-year students receive an introduction to electronic circuits and microcontrollers through lectures and hands-on practice labs with electronic equipment. Electronics III, a senior-year course, focuses on digital circuits, including the core tenets of logic gates, and combinational and sequential circuits. Microcontrollers, a senior-level course, familiarizes students with the fundamental principles and applications of microcontroller architecture and programming. In Interdisciplinary Projects Workshop I, a senior-year course, students define a real-world problem and generate possible solutions. After selecting the most outstanding solution, students in Interdisciplinary Projects Workshop II formulate an engineering project including a comprehensive analysis of the business ideas and feasibility evaluations of the technical, environmental, social, and financial aspects of such a project. In all these courses, the professor promoted the use of AI chatbots as a learning resource.

Table 1. Promoted uses of AI chatbots in engineering courses.

Course	Uses That Were Promoted by AI Chatbots
Mechatronic Technologies	− Assist students with prior experience in programming and syntax in the C language. − Assist in learning the syntax to program the Arduino UNO board®. − Quickly generating example codes for programming the Arduino UNO board during class.
Electronics III	− Facilitate the understanding of how the conversion between number systems is carried out. − Guide students in understanding the rules of Boolean algebra. − Help students learn syntax and programming in Verilog, a new programming language for students.
Microcontrollers	− Assist students with prior knowledge of programming and syntax in the C language. − Support students who are not fluent in English and have difficulty understanding the microcontroller manual. − Resolve doubts about the configuration and programming of microcontrollers. − Generate example codes for microcontroller programming.
Interdisciplinary Projects Workshop I	− Help identify possible contexts of interest for the development of the course project. − Enrich and improve the wording of questions in surveys for context analysis. − Generate ideas for functional and non-functional requirements of possible solutions. − Brainstorm possible solutions during the creative process.
Interdisciplinary Projects Workshop II	− Generate ideas to formulate functional and non-functional requirements for the solution to be developed. − Improve the wording of project objectives for greater clarity and alignment. − Support students in the analysis of business ideas.

outlines the professor-promoted uses of chatbots in learning for each course. During the first classes, the professor discussed their overall potential benefits and risks. In later classes, the professor presented examples of how to use AI chatbots to enhance their learning experience, including performing practical demonstrations.

3.3. Research Question

Our research question was: What are the contributions and challenges of using AI chatbots in the learning process of students across the five selected engineering courses?

3.4. Instruments

We used a set of two closed-ended questions and four open-ended questions whose wording was adjusted according to the specific course.

Table 2. Student survey questions.

Closed-Ended Questions

Open-Ended Questions

− How often did you use ChatGPT and other AI chatbots during the Microcontrollers course? − In what ways did you utilize AI chatbots during the Microcontrollers course?

− Do you believe that AI chatbots, such as ChatGPT, served as a useful tool to enhance your understanding of the course subjects? Why? − Please share your overall experience using the AI chatbot in the Microcontrollers course. − Did you find that AI chatbots such as ChatGPT contributed to the improvement of your microcontroller programming skills? If so, in what specific ways did it help you? − What negative aspects did you experience when using the AI chatbots, as a support tool in the Microcontrollers course? And how did they impact your learning experience?

presents the survey questions tailored for the Microcontrollers course. Data were gathered in Spanish, the native language of the students, and translated into English for incorporation into this paper.

The survey also requested the students’ consent for their data to be used for research purposes. In addition, the students were informed that all the data collected would be anonymized before analysis and that subsequent dissemination would be presented in aggregated form to minimize potential risks of identification.

3.5. Participants

The surveys were distributed to approximately 100 students enrolled in the selected courses as part of a class activity, and participation was voluntary. A total of 38 students completed the survey, distributed as follows: 6 participants from Mechatronic Technologies, 5 from Electronics III, 10 from Microcontrollers, 12 from Interdisciplinary Projects Workshop I, and 5 from Interdisciplinary Projects Workshop II. Class sizes ranged between 12 and 20 students.

3.6. Data Analysis

3.6.1. Thematic Analysis

The data analysis consisted of two rounds. In the first round, we coded the students’ responses with 8 a-priory codes based on the principles of learning proposed by Lovett et al. [24] These codes were: (1) acquisition, completion, or activation of prior knowledge; (2) organization of knowledge; (3) motivation to learn; (4) self-directed learning; (5) acquisition, practice, and application of new skills and knowledge; (6) goal-directed practice and feedback; (7) student diversity; and (8) classroom environment. The coding was a collaborative effort between the two researchers.

After this initial coding process, for each category, we did a second round of coding to identify recurring themes and patterns in the uses, benefits, and challenges of the chatbots. In this second round, the first author conducted the initial iteration, whereas the second author reviewed it to ensure validity. Our thematic analysis resulted in a classification of the students’ responses according to which course they belonged to; the uses, benefits, and challenges of chatbots; and the principles of learning they were referring to.

3.6.2. Integration with Literature

We synthesized these findings and contrasted them with the existing literature to enrich the analysis and discussion and provide broader insights into the influence of chatbots in learning. We compared our findings with the literature to identify areas where they align with existing research and where they diverge. To report the results and discussion, we draw on the framework described in the literature review section and present the findings within the five categories of the contributions and challenges of chatbots in learning.

It is important to note that the first author of this study was the professor of the five courses investigated, was responsible for the administration of the survey, and led the data analysis. The second author reviewed the clarity and accuracy of the English translation of the quotes reported in this document and supported the organization of the findings and their alignment with both the principles of learning of Lovett et al. [24] and previous research. During frequent online team meetings, the authors discussed and interpreted the findings and collaboratively wrote this document.

3.7. Limitations

It is important to acknowledge that this study has inherent limitations due to its exploratory qualitative approach. The sample is not expected to be representative of all students in similar courses, and the findings may not be generalized to all students in the college. However, we provided detailed descriptions of the results to enhance their transferability to other institutions with similar contexts. Nonetheless, the results will provide a detailed and valuable insight into students’ perceptions of using AI chatbots in the learning process. These insights serve as inputs for future research.

4. Results

Table 3. Reported use of AI chatbots in the five courses.

Course	Reported Uses of AI Chatbots
Mechatronic Technologies	Seek help in solving programming errors. Solve specific doubts in the programming of the Arduino UNO board. Receive advice and practical suggestions for the development of projects or exercises. Get practical examples and sample code on Arduino UNO programming. Enhance understanding of how the Arduino UNO works. Solve specific doubts about syntax in C language. Resolve specific doubts about other topics of the course. Save time.
Electronics III	Clarify doubts about the content of the assignments. Find information and sample codes for microcontroller programming. Search for information on the course contents. Complete the exercises from the study guides of the course. Save time.
Microcontrollers	Solve specific doubts about syntax in C language. Seek help in solving programming errors. Obtain additional information related to course content. Solve specific doubts in the programming of the microcontroller. Receive advice and practical suggestions for project development or exercises. Get practical examples and sample code on microcontroller programming. Expand understanding of how the microcontroller works. Obtain programming tips and recommendations. Save time.
Interdisciplinary Projects Workshop I	Identify potential problems, needs, or opportunities to address in the project. Perform an in-depth analysis of the project problem’s contextual background. Design survey questions. Help identify the causes and effects using the problem tree. Investigate existing technologies or solutions related to the problem. Gather ideas about functional, non-functional requirements, and constraints of the project. Generate ideas for feasible alternative solutions to the problem. Save time.
Interdisciplinary Projects Workshop II	Refine text. Generate, restructure, and expand ideas. Quickly and efficiently identify doubts or concerns. Request for suggestions and guidance on specific topics. Translate. Save time.

below outlines the applications of AI chatbots in the courses reported by the participants.

We organized the uses listed in this table by their contributions to learning based on the principles of learning of Lovett et al. [24] The following results are an initial exploration of the students’ perceptions and are not intended to be generalizable.

4.1. Contributions of AI Chatbots to the Acquisition, Completion, or Activation of Prior Knowledge and to Help Organize Knowledge and Make Connections

The students highlighted the valuable support that these chatbots offer to acquire the prior programming knowledge required in these courses. Referring to concepts studied in past courses, a Basic Mechatronic Technologies student mentioned: “In my case, it turned out to be very informative, since it helped me to solve the doubts that I had while working on the C language codes”. Similarly, a student from the Microcontrollers course stated, “It helped me as a guide for some parts of the code that I had doubts about”. These comments suggest that students used the chatbot to address and clarify specific doubts they had from previous classes. Such clarifications helped them to acquire new concepts, as a student from the Microcontrollers course highlighted: “In general, the GPT chatbot proved to be a very useful tool in the development of the course, clarifying specific doubts and offering syntax examples, programming structures…”

However, students recognized that the quality of the chatbot responses depends heavily on the clarity, quality, wording, and grammar of their questions. As a student from the Interdisciplinary Projects Workshop I course explained: “Sometimes a lot of attention to grammar and writing is necessary to guide the chatbot precisely to the desired information and to ensure that the answers provided are well written”.

In contrast, students from the Basic Mechatronic Technologies course and Electronics III pointed out that the responses generated by AI chatbots, particularly in the case of programming codes, were too advanced for their understanding. One student mentioned, “Sometimes understanding the answers provided by ChatGPT required advanced knowledge or experience in the field”. Another student seemed to agree: “Many times the chatbot generates complex code and includes functions that I have never seen or barely know about”. Nonetheless, students managed this problem by requesting more details from the chatbot responses. As another student noted, “There are cases where you need to repeatedly ask the chatbot for more explanations or remind it to define certain concepts, which can slow down the process”. Another challenge is that students may have difficulty in critically evaluating the reliability and accuracy of the information provided by the AI chatbots. For example, a student from the Interdisciplinary Projects Workshop I course highlighted: “I have frequently observed that the chatbot lacks control or understanding of the veracity of all the information it presents… Numerous times, I found cases where the data provided by the chatbot were inaccurate compared to other sources, but unfortunately, the platform did not allow sharing the source from which the information was derived”.

Nonetheless, some students were conscious that they needed to be skeptical of the information provided by the chatbot. In this regard, a student from the Electronic III course mentioned: “The information provided by the GPT chatbot can always be verified with other sources, simulations, code execution, or practical applications. While the accuracy of the information was not always perfect, it was easier to confirm or refute using other references…” Other students in the interdisciplinary projects workshop supported the need to be skeptical of the information from the chatbot: “It is important for us to evaluate and understand the content provided by the AI instead of blindly accepting it as valid”. “…Many times, the chatbot gives answers that are not entirely correct, so … it is essential to analyze the information provided and, if something seems inconsistent or illogical, check it against other sources”. These comments suggest that the inaccuracies of AI chatbots have contributed to the development of critical thinking. For example, a student from the Electronics III course highlighted that despite its errors, the AI chatbot encourages students to detect errors in the generated code and to be critical of the generated responses, thus contributing to their learning process: “[ChatGPT] often makes mistakes, so while it is a useful tool, we always need our own knowledge to make it work. This really helps us learn because we understand and master the code when we identify the bugs”.

4.2. Contributions of AI Chatbots to Motivate Students to Learn

In our study, we found that students used AI chatbots to optimize their study time. Students of the Electronics III class expressed that AI allows them to address specific questions more efficiently, saving them time. Three students commented on this regard: “I find this to be an incredibly useful tool when it comes to saving time searching for information that is often hard to find”. “It greatly speeds up the process of finding answers to very specific questions, which significantly optimizes study time”. “It helped me optimize the time spent searching, generating, and structuring ideas”. This finding aligns with motivation theory [24,44], indicating that increased utility value enhances students’ motivation to engage in a task, ultimately promoting learning. For example, students in Interdisciplinary Projects Workshop I and II noted the usefulness of the chatbot and its positive impact on their learning: “The tool significantly accelerated my work, offering an integral perspective on relevant issues”. “It’s quite useful and provides information that is sometimes overlooked”. “It showed me solutions that I had not taken into account and this allowed me to further expand my ideas”.

4.3. Contributions of AI Chatbots to Promote Self-Directed Learning and to the Acquisition, Practice, and Application of the Skills and Knowledge They Learn

Students perceived the chatbot as helpful to their learning. Students from the Microcontrollers course noted the potential benefits of AI chatbots in strengthening their programming mastery: “They can be valuable for reviewing your code and discovering more efficient ways to write the same code”. “It helped me improve a little in handling programming syntax”. Although some students perceived that the AI chatbot did not improve their skills, they acknowledged its assistance in solving questions that still contributed to their learning. As one student mentioned, “Not much in terms of improving skills, but it helped me program questions and gave me a basic understanding of how to approach an exercise”.

Some students reflected on how to learn using AI chatbots. A Microcontrollers student shared how she used it to improve her understanding: “It was a nourishing experience since I did not use it to solve problems, but on the contrary only for advice and to be able to understand specific things that otherwise would have been more difficult to understand”. Other students reflected that they would not achieve meaningful learning if they used these tools solely to get complete answers without understanding the process to reach those answers. As one student stated, “I think one of the drawbacks of using ChatGPT is that when asking for a recommendation on an exercise, the AI would directly solve the entire exercise. However, I consider this to be more of a misuse of the tool”.

4.4. Contributions of AI Chatbots to Help Goal-Directed Practice and Feedback

The students noted that the AI chatbot guided them in developing learning experiences in their classes. It provided them with feedback on their work, such as detecting errors or suggestions for improvement. A student from the Microcontrollers course supported this idea: “The chatbot usually generates code bases with simple functions such as setting timers or ADCs. It also helps diagnose and resolve problems”. Moreover, students from the Interdisciplinary Projects Workshop I course underscored the assistance of the chatbot to improve the quality of their work in class. As one student elaborated: “Honestly, I think it helps a lot when it comes to wanting to locate or contextualize the person in certain aspects. In my case, I wanted to know how to start various activities in the class or I wanted to improve the content we already had, telling ChatGPT what do you think of my information? How do you think I can improve this?”

Conversely, students from the Electronics III course expressed several weaknesses of the chatbot. It generated errors in the codes, produced complex programming codes that were difficult to understand or with advanced programming syntax that they did not master, or provided answers that had errors in mathematical operations. In this regard, two students shared their experience with the chatbot: “When it comes to math, the chatbot often has problems, so the calculations it provides need to be checked. It also tends to have small errors in the code it generates, such as referencing non-existent bits or occasionally generating syntax errors”. “It was not very productive because ChatGPT still does not understand electronics or programming very well, which generates advice or recommendations that are of no use”. Nonetheless, some students emphasized that identifying these errors makes them learn. A student from the Microcontrollers course elaborated on this idea: “Sometimes I tried the examples [AI chatbot] provided and they didn’t work, it confused me because I started to doubt whether what it was saying was actually right, but I started to analyze the examples and read over and over again the explanation it gave to that example, and I realized the flaws of the example”.

5. Discussion

The purpose of this paper is to examine the influence of using AI chatbots on learning in five different engineering education courses. This section discusses our findings regarding the advantages and challenges associated with using these tools in learning through the lens of the learning principles of Lovett et al. [24].

In summary, AI chatbots have the potential to transform the way students access information and engage with learning tools [20,32]. Consistent with the literature [1,45], our study found that AI chatbots play positive roles in the learning process. Specifically, they can assume the roles of tutor, peer, and supportive assistant mostly because they offer valuable feedback in complex tasks or specific fields. Nonetheless, AI chatbots present considerable challenges; their responses rely heavily on the clarity and wording of students’ prompts, sometimes exceeding the student’s comprehension level or providing inaccurate and unreliable information. Also, there is a risk of students becoming overly dependent on chatbots, raising controversial academic integrity concerns.

5.1. Contributions of AI Chatbots to the Acquisition, Completion, or Activation of Prior Knowledge and to Help Organize Knowledge and Make Connections

Our results suggest that students use AI chatbots as tutors because they impart educational guidance, help students elucidate complex concepts with illustrative examples, pose queries, and offer instantaneous feedback. Moreover, AI chatbots enriched the students’ learning experiences by adapting instruction to their individual needs and learning approaches. Notably, the students utilized the chatbots to address and clarify specific doubts stemming from previous classes, and this proactive engagement facilitated the acquisition of new concepts.

However, in concordance with [11], the quality of student queries heavily influences the efficacy of AI chatbot responses. The clarity, quality, wording, and grammatical precision of student questions had a strong significance, as they directly influenced the accuracy and coherence of the chatbot’s instructional output. This insight provides an opening for educators to instruct students in the utilization of AI by strengthening their written communication abilities, thereby potentially enhancing the efficacy of the interface between students and AI tutors.

Consistent with the study in [20], our findings also suggest another challenge associated with the utilization of AI chatbots: the need for students to critically evaluate the veracity and precision of the information offered by AI. Our findings, particularly from students of Electronics III, indicated that students’ awareness of the possibility of these inaccuracies catalyzed the development of critical thinking skills. While navigating the errors present in the chatbot’s responses, certain students honed their ability to detect errors within generated code and practiced discernment in scrutinizing the responses. This unexpected yet invaluable pedagogical dimension underscores how instructors can leverage AI chatbots to cultivate a discerning and skeptical mindset in students.

Similarly, students from the Basic Mechatronic Technologies and Electronics III courses faced another instructional challenge: the responses generated by AI chatbots, particularly the programming codes, seemed to surpass the limits of their understanding and did not consider the students’ prior knowledge. This disjuncture, however, prompted a proactive response among some students, who addressed it by soliciting additional explanations. This not only strengthened students’ academic curiosity but also provided a means to harmonize the use of the chatbot as a tutor. Instructors could use this feature as an opportunity to encourage students to use AI chatbots to foster a deep approach to learning [3].

5.2. Contributions of AI Chatbots to Motivating Students to Learn

Our results suggest that students used the AI chatbot as a study assistant because it supported students in tasks such as searching for information, summarizing content, organizing study materials, performing language translations, and outlining documents. These uses are consistent with the findings of [39].

On the positive side, consistent with the insights of [11], we found students using chatbots to optimize their study time. Students use it to increase their time efficiency in trying new ideas, addressing specific questions, and finding information that fits the content they are studying. Nonetheless, the use of AI seemed to transform how the students accessed information, mostly because interacting with the chatbot was faster than getting information from books or other references. Nonetheless, this brings a challenge to instructors because, in the long term, this benefit could become a hindrance to the students’ motivation to study. As suggested by [12,18], students could become too dependent on the quick access to information and answers that chatbots provide, thereby reducing their sense of success and autonomy when they cannot access the chatbot, for example, during exams or evaluation activities.

5.3. Contributions of AI Chatbots to Promote Self-Directed Learning and to the Acquisition, Practice, and Application of the Skills and Knowledge They Acquire

Our data suggest that the chatbot’s ability to understand and provide responses to natural language input facilitates students to search for information and receive personalized guidance in a conversational style. This feature underscores the role of the chatbot as an assistant that contributes to acquiring knowledge and improving the quality of their work [20].

In our study, we found evidence that AI chatbots might have transformed the process of learning coding. Students employed chatbots to seek explanations for programming concepts and syntax, requested examples that illustrate programming principles, and obtained step-by-step guidance for solving programming problems. The students also considered that the chatbot facilitated more efficient coding and helped them easily identify errors in their programming code.

However, the students expressed that the code generated by current AI chatbots could occasionally contain errors. Paradoxically, this challenge served as an opportunity, prompting them to deepen their programming knowledge to discern and rectify errors within AI-generated code. Some students recognized that they did not experience meaningful learning when using AI chatbots to get complete answers without understanding how to get those answers by themselves. This brings another opportunity for instructors: teaching students how to use AI to provide recommendations that guide them instead of asking for direct solutions to the exercises.

Moreover, the role of AI chatbots extends beyond being mere tools; it seems like they actively contribute to a collaborative dynamic between students and technology, significantly enhancing the learning experience. Collaborative engagement involves problem solving, brainstorming, and idea generation. Furthermore, our data suggest a transformative impact on students’ engineering design processes, where AI chatbots became integral collaborators. The students worked collaboratively with the AI tool to understand problems, establish project requirements, and formulate alternative solutions, signaling a shift in their engineering project methodologies.

5.4. Contributions of AI Chatbots to Help Goal-Directed Practice and Feedback

Broadly, the students recognized the role of the chatbot in guiding them through both in-class and homework learning activities. Aligned with the research findings of [20,33], the chatbot provided timely and valuable feedback on assignments, encompassing error detection and suggestions for improvement.

Conversely, as mentioned before, participants in the Electronics III course presented a distinct perspective, expressing reservations about the chatbot’s tendency to generate inaccurate or incorrect responses. This observation aligns with the findings of [41], which posited that chatbots may not consistently offer valuable feedback in complex tasks or specific fields. Nevertheless, a subset of students underscored that the process of identifying and rectifying these errors contributed significantly to their learning experience.

Although it seems that the AI chatbots served better in general programming education, their implementation in more specialized contexts, such as Electronics III, presented challenges. The intricate nature of these topics demands a deep and precise knowledge base, raising concerns about AI’s ability to provide accurate and useful answers. Despite these challenges, the adaptive nature of AI chatbots holds promise for addressing these concerns in the future.

5.5. Contributions of AI Chatbots to Address Student Diversity and Create a Positive Classroom Environment

Although research suggests that developing strategies to ensure responsible and equitable implementation of AI plays a pivotal role in driving the success of these efforts [32], our study did not identify instances where chatbots were utilized to address issues related to student diversity and the classroom environment. Moreover, only a small number of students acknowledged any contributions made by AI chatbots to enhance accessibility to class content, such as translations, text-to-speech capabilities, or other individual needs. For example, in Interdisciplinary Projects Workshop II, only one student mentioned utilizing these tools for translations, and a few Electronics III students highlighted the evident dependence on an internet connection to access these tools, a condition not universally met in many students’ homes.

5.6. Broader Limitations and Potential Challenges of Using AI Chatbots in Education

Although our study did not directly inquire about students engaging in unethical practices with the chatbot, such as plagiarism or cheating, it is imperative to acknowledge the inherent challenges associated with academic integrity that may prove difficult to identify [3,7,18]. Another major challenge is the ethical issues surrounding the use of AI chatbots in education. One of the main concerns is the possibility of copyright infringement if the content used by the AI chatbot is not properly cited [16,22].

On a positive note, none of the responses from our participants indicated a consideration of using the chatbot as a means of cheating or plagiarism. However, this lack of acknowledgment could potentially obscure a different reality: when students fail to disclose their use of AI assistance in their work, there is a risk that they may falsely claim authorship of ideas generated by AI [17]. More concerning is the possibility that students may believe these AI-generated ideas to be their original work. However, as suggested by [46], students who are knowledgeable about AI and its complexities are better equipped to interact with AI chatbots and understand both their capabilities and risks. This understanding fosters a sense of responsibility and accountability among students, motivating them to become more aware users and learners of these AI tools.

As highlighted by [27], the facilitation of plagiarism by such tools not only compromises academic integrity but also undermines the fundamental purpose of assessment—to impartially evaluate student learning. Educators may find it difficult to accurately assess student performance when using AI chatbots for assignments and assessments, thus preventing effective tracking of student learning issues [27,34]. Lastly, AI chatbots learn from vast data sets, which may contain biases or inaccuracies that could lead to the spread of incorrect information and biased responses [3,17,20]. This issue can be particularly problematic for students who may inadvertently absorb and internalize misinformation, resulting in misconceptions and errors in their understanding of a topic [16,20].

Considering these caveats, we recommend that instructors engage in open conversations with students regarding academic integrity when utilizing the chatbot. This proactive approach may help students recognize that misusing the chatbot can imply academic dishonesty, potentially leading to adverse consequences for both their learning and professional development.

5.7. Future Directions

Insights from the literature and the findings of this study suggest that there is a need to explore and explain the impact of AI chatbots on student development. Future research efforts could delve deeper into this topic, specifically focusing on how chatbots influence the development of cognitive, communication, and problem-solving skills among students. Future quantitative research could use quasi-experiments and survey designs to increase the external validity of these findings.

Additionally, not enough attention has been paid to the role of AI chatbots in promoting student diversity and fostering a positive classroom environment. Future research efforts could study how chatbot functionalities can effectively meet the diverse needs of students, considering factors such as cultural background, learning approaches, and accessibility requirements. Future research could suggest ways to promote inclusiveness within educational settings using AI chatbot technology.

6. Conclusions

Our study illustrated the contributions and challenges of the use of AI chatbots for learning in five engineering courses at a rural South American university. In the age of AI, academic institutions are tasked with adapting to this technological paradigm shift. This means not only embracing AI chatbots within teaching and learning processes but also equipping students with an understanding of their intrinsic limitations, including an ethical and responsible engagement with AI technologies [15]. The following paragraphs summarize the contributions and challenges of AI chatbots for learning found in our study.

6.1. To the Acquisition, Completion, or Activation of Prior Knowledge and to Help Organize Knowledge and Make Connections

• Contributions:

• The students found valuable support in chatbots for acquiring prior programming knowledge and addressing doubts from previous classes, facilitating the acquisition of new concepts.
• These AI chatbots acted as tutors, offering educational guidance, elucidating complex concepts, and enriching the learning experience by adapting instruction to their individual needs.

• Challenges:

• The students acknowledged that the effectiveness of chatbot responses relied heavily on the clarity, quality, wording, and grammar of their questions. The precision of their queries impacted the accuracy and coherence of the chatbot’s instructional output, which may hinder the completion or activation of prior knowledge.
• The students found AI chatbot responses, especially regarding programming codes, to be too advanced for their comprehension and lacking consideration of their prior knowledge. Despite this challenge, the students addressed it by requesting additional details from the chatbot responses.
• One challenge is that the students found it difficult to critically evaluate the reliability of information from AI chatbots, although some were aware of the need to be skeptical. Our data suggest that this awareness might foster critical thinking skills as students refine their ability to detect errors and scrutinize responses. Despite this challenge, proactive students seek additional explanations, enhancing their academic curiosity and facilitating a balanced use of chatbots as tutors.

6.2. To Students’ Motivation to Learn

• Contributions:

• The students used AI chatbots to optimize their study time, allowing them to address specific questions more efficiently and saving them time.
• AI chatbots were seen as useful study assistants, supporting the students in various tasks, such as searching for information, summarizing content, organizing study materials, translating information, and outlining documents.
• AI chatbots facilitated increased time efficiency for trying new ideas, addressing specific questions, finding relevant information, and transforming the way the students accessed information.

• Challenges:

• The ease and speed of accessing information through chatbots could potentially hinder students’ motivation to study in the long term, as they may become overly dependent on this quick access and feel less successful or autonomous when they cannot use the chatbots.

6.3. To the Promotion of Self-Directed Learning and the Acquisition, Practice, and Application of the Skills and Knowledge They Acquire

• Contributions:

• The students perceived AI chatbots as helpful assistants in acquiring knowledge and enhancing the quality of their work.
• The chatbots’ capability to understand and respond to natural language input helps students search for information and receive personalized guidance.
• AI chatbots facilitated the learning process of coding, enabling the students to seek explanations, examples, and step-by-step guidance for programming problems.
• Collaborative engagement with chatbots significantly enhanced engineering design processes by aiding in understanding the underlying problem to solve, establishing project requirements, and brainstorming and formulating alternative solutions.

• Challenges:

• Although AI chatbots assist students in solving questions, they may not improve their skills. This is mostly due to the risk of students solely relying on AI chatbots to get complete answers without understanding the underlying process, potentially hindering meaningful learning.
• The code generated by AI chatbots occasionally contained errors; however, this challenge helped the students learn by prompting them to deepen their programming knowledge to discern and rectify these errors.

6.4. To the Goal-Directed Practice and Feedback

• Contributions:

• The students recognized the AI chatbot’s role in shaping their learning experiences by offering timely and valuable feedback on assignments, including error detection, improvement recommendations, and guidance through both in-class and homework learning activities.

• Challenges:

• The students raised concerns about AI’s ability to generate accurate or useful answers to their queries, particularly in specialized topics.
• The students observed errors in the codes generated by the chatbot and encountered complex programming syntax that was difficult to understand. However, they also emphasized that identifying and rectifying these errors contributed significantly to their learning experience.

6.5. To Student Diversity and a Positive Classroom Environment

• Contributions:

• Our study did not identify instances where chatbots were utilized to address issues related to student diversity and classroom environment. Also, only a small number of students acknowledged any contributions made by AI chatbots to enhance accessibility to class content.

• Challenges:

• The students highlighted the dependence on an internet connection to access the chatbots, a condition not universally met in many students’ homes.
• None of the responses from our participants suggested that they used chatbots for cheating or plagiarism. However, this absence of acknowledgment raises concerns that students may fail to disclose their use of AI assistance, potentially leading them to believe that the AI-generated ideas are their original work.

Finally, we summarize our primary recommendations for instructors to facilitate students’ meaningful engagement with AI.

• Strengthen students’ written communication skills to potentially enhance the effectiveness of the interaction between students and AI chatbots. This would increase the usefulness, accuracy, and coherence of the AI responses.
• Foster open conversations about preventing students from becoming overly reliant on quick access to information and answers provided by chatbots. Emphasize the potential impact on their sense of success and autonomy when such access is unavailable, especially during exams or assessment activities.
• Encourage students to use AI chatbots for a deeper approach to learning. Instruct them on how to seek recommendations from AI to guide their understanding, rather than relying on direct solutions to exercises. Highlight that the process of identifying and rectifying errors significantly contributes to their learning experience.
• Engage in open dialogues with students about academic integrity when incorporating the chatbot into their learning. This proactive approach may help students recognize that misusing chatbots can lead to academic dishonesty and have adverse consequences for both their learning and professional development.