Exploring Computer Science Students’ Perception of ChatGPT in Higher Education: A Descriptive and Correlation Study

Abstract

ChatGPT is an emerging tool that can be employed in many activities including in learning/teaching in universities. Like many other tools, it has its benefits and its drawbacks. If used properly, it can improve learning, and if used irresponsibly, it can have a negative impact on learning. The aim of this research is to study how ChatGPT can be used in academia to improve teaching/learning activities. In this paper, we study students’ opinions about how the tool can be used positively in learning activities. A survey is conducted among 430 students of an MSc degree in computer science at the University of Hertfordshire, UK, and their opinions about the tool are studied. The survey tries to capture different aspects in which the tool can be employed in academia and the ways in which it can harm or help students in learning activities. The findings suggest that many students are familiar with the tool but do not regularly use it for academic purposes. Moreover, students are skeptical of its positive impacts on learning and think that universities should provide more vivid guidelines and better education on how and where the tool can be used for learning activities. The students’ feedback responses are analyzed and discussed and the authors’ opinions regarding the subject are presented. This study shows that ChatGPT can be helpful in learning/teaching activities, but better guidelines should be provided for the students in using the tool.

1. Introduction

The survey consists of 12 questions that try to capture the familiarity of the students with ChatGPT, how often they use it, how they think it can be used to improve learning, how they think academic misconduct via the tool is detectable, how the universities can improve awareness regarding the tool, how it can be used in designing software applications, etc.

AIeD (Artificial Intelligence in eDucation) has become a hot topic currently with the launch of an online chatbot tool called ChatGPT [1]. As per [2], “Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces”. ChatGPT is based on a pre-trained Large Language Model (LLM) called GPT-3.5 [3]. AIeD is not new and has been around for more than a decade now. However, the new advancement in the field of Artificial Intelligence has made it possible to create tools like ChatGPT, which can convincingly imitate humans when it comes to writing skills [4]. ChatGPT hit 1 million users across the globe within a week [5]. The strengths of ChatGPT include language understanding, human-like conversations, flexibility to tune to a specific language task, speedy responses, and cost-effectiveness [6]. Particularly, for a student in higher education, LLM can assist in writing tasks, and help promote critical thinking and problem-solving skills [7]. The use of AI is inevitable in academia, so we also need to ensure that the students have a good understanding of how the tool can be used such that the students remain competitive when it comes to employability. Particularly, for the student in computer science, it is expected of them that they are aware of recent technology developments and are expected by employers to have a good understanding of their use. More importantly, for computer science students, ChatGPT can generate algorithms and translate them into any programming language of their choice. It can also help with debugging the code. However, it cannot provide code to obtain output for a given input, as it is not a compiler [8]. Furthermore, its performance on natural language-type questions is far superior to questions containing code snippets. It can assist in completing partially written code blocks, but lacks the execution capabilities required to produce the correct output; after all, it is a language model [8]. Likewise, ChatGPT is unable to evaluate a complex math function for a given input [9]. Thus, it is important that the user is aware of its limitations and does not entirely trust the generated output for all types of quarries.

It is imperative to explore the impact of ChatGPT on international computer science students; First, there is an increase in the number of international students enrolled in higher institutions across the UK’s universities, and, foremost, the majority of them are non-native English speakers [10]. Considering its importance and valuable application in academia, this study conducts research on students’ familiarity with the tool and their opinions about how the tool can be employed to improve learning and teaching activities.

2. Background

ChatGPT is developed by OpenAI, a non-profit AI research and development organization dedicated to benefiting humanity through advancement in digital intelligence [11]. OpenAI is striving to create safe Artificial General Intelligence (AGI) that can benefit all of humanity and wants to ensure responsible development, deployment, and use of their models [12]. Along with ChatGPT, OpenAI also offers a state-of-the-art artificial image generator model named Dall.E 2, which takes a text prompt and turns it into photo-realistic images that have never existed before [13]. Dall.E 2 can assist users to imitate any artist by creating original, realistic art or can combine concepts, attributes, and styles in the generated images. Since the company is free from financial obligations, it can better focus on a positive impact by developing highly autonomous systems that outperform humans at the most economically valuable work [14].

ChatGPT is trained to provide detailed responses to instructions in a prompt. ChatGPT is based on the GPT-3.5 series and is regularly updated as more users interact with the tool and provide the system with their feedback [1]. The capabilities of ChatGPT are showcased in [15], a paper co-authored by ChatGPT itself, which has outlined the two important benefits of using this tool from the perspective of journalism. First, it helps in creating personalized content for readers based on their interests and preferences, and second, it summarizes a long news article to quickly grasp the key points. Later, a detailed survey is performed by [4] around the capabilities of ChatGPT in content generation, which is reported to be indistinguishable from the contents produced by humans. As discussed by the writers, the tool can aid the writers to tackle writer’s block and save them time on repetitive tasks. A good example of these capabilities is in the two recent papers published by the same author, in which the author has used the tool to write the paper, and where the author prompts ChatGPT with questions around the topic and presents the responses as part of the publication [16,17]. Although ChatGPT is capable of producing acceptable research articles in a short time, the originality in the arguments is lacking in these papers. But, the writing structure of a typical research article is well-captured and arguments are presented in an articulated manner, which makes it very convincing for academic writing.

The capabilities of the tool in generating texts have now alarmed research journals. The editorial board of a journal published in Knee Surgery, Sports Traumatology, Arthroscopy (KSSTA), which is a peer-reviewed medical journal, mentions in their guidelines that the authors should not use ChatGPT in their articles submitted to KSSTA. Their guidelines specify that it is currently a preliminary decision but state that if ChatGPT is used in writing a manuscript submitted to a journal then it should be acknowledged [18]. This certainly has opened a conversation around the use of AI chatbots in research, which needs to be defined while considering good scientific and ethical practices. The International Journal of Research in Marketing (IJRM) has taken a clear stand that they will not accept any submission with ChatGPT or any Generative Artificial Intelligence (GenAI) tools as a co-author [19] on the grounds of lack of accountability, and will interpret ChatGPT 1:1 response in any manuscript as plagiarism. It is emphasized in [19] that researchers should be more transparent regarding the use of AI tools in their study.

Many research papers have been written on ChatGPT in this short span centering around its functionality, capabilities, applications, threats, and suggestions around its use. To limit the scope of this paper, only its applications as an aid in student learning and teaching, and its impact on the assessments and guidelines in higher education are explored.

2.1. ChatGPT Utilization in Academia

In academia, ChatGPT can be utilized for a diverse range of applications beyond what has been mentioned here. To begin with, it provides valuable support for writing reports, essays, and scientific articles. It can also proofread the provided text for structural, punctuation, and grammatical errors [20]. Another useful application is acting as a virtual tutor; it can break down a complex concept into an easier-to-understand language [21,22]. For research projects, ChatGPT can not only aid in literature review but can also generate innovative ideas in brainstorming sessions [23,24]. In computer science, it can aid students by debugging their code and suggesting programming solutions [25]. Despite this, if fine-tuned within a specific field, it has the potential to revolutionize automated grading and immediate feedback for essay questions. This is a challenging and time-consuming task, particularly in large modules where grading is handled by multiple markers [26]. Lastly, if trained on University UPRs (University Policies and Regulations), ChatGPT could thoroughly address student inquiries without human intervention [27].

2.1.1. Promoting Student Learning

ChatGPT can be useful for students in higher education in several key aspects of their learning. A study performed by [4] found that ChatGPT can increase student engagement, collaboration, and accessibility, along with facilitating asynchronous communication, feedback, and remote learning. According to [7], tools like ChatGPT can assist in various subjects by simplifying and contextualizing content, promoting problem-solving skills, and developing analytical and critical thinking. They can also facilitate group and remote learning, empowering learners with disability and assisting in professional training [6].

All of these applications lean towards the use of ChatGPT for helping students in their learning and it is important for educators to consider how to include ChatGPT in the university curriculum in a way that does not harm and, at the same time, improves learning. The discussion around how to optimally use ChatGPT is an ongoing question and, according to [6], “the technology behind ChatGPT could potentially be utilized to improve the performance of personalized adaptive learning”. The authors [6] argue that it is ironic that a tool which can facilitate innovative teaching and learning has caused anxiety among academics.

2.1.2. Enhancing Teaching Strategies

ChatGPT can not only aid students in their learning, but it can also assist tutors and educators in various ways. In [7], various ways in which educators can leverage this tool in teaching have been outlined. For example, the tool can be positively used in personalized learning, lesson planning, language learning, assessment, and evaluation. It can also assist the student in aspects like professional development, research, writing for seminars, papers, etc. [6].

Apart from several advantages of employing ChatGPT in teaching, the tool also has various shortcomings. For example, ChatGPT is not as competent in producing text in low-resource languages. As documented in [28], when they are asked to write about medical problems in the Korean language, the texts that are produced by ChatGPT essays are not comparable to the ones produced by medical students. Another example of the problem that ChatGPT faces in generating the texts is referencing. In these texts, the source of the references is not accurate; in some cases it is made up [3], or sometimes missing. Because referencing is very important in academic writing, the students still have to find original references to support their arguments [4]. Another problem with using ChatGPT is, as argued in [7], that the learners sometimes heavily rely on the AI models to do their tasks, which can impact their critical thinking and problem-solving skills, amplifying laziness and apathy towards their own investigation.

In [4], several concerns with ChatGPT use regarding academic honesty and plagiarism are raised, and it is suggested that the universities should ensure ethical and responsible use of the tool by developing policies, as well as providing training to staff and students. Their key suggestion is to drop the essay-style assessments in favor of interactive activities (group discussions, presentations, etc.) where students can demonstrate the application of knowledge and skills, or else use real-time invigilated assessments.

To mitigate the issue of academic misconduct in the wake of ChatGPT, [6] suggests against essay-style assessments. To manage this problem, the authors of the study suggest that the teachers should conduct physical closed-book exams or use surveillance software for online exams. Although, it is usually argued that mastering the skills required for closed-book exams is irrelevant to employability. It is recommended to include digital literacy where AI technology should be part of the curriculum while emphasizing faculty training. It is also suggested that the academic integrity policies should include the use of AI and provide student training on academic integrity [6].

It is recommended in [7] that teachers should use LLM-based tools as a complementary supplement for instruction generation, to promote critical thinking and problem-solving skills by incorporating it into the curriculum. They also suggest incentivizing tutors in generating teaching strategies using LLMs, to engage students in the problem-solving process while monitoring and evaluating the use of these tools for any negative impact on learners.

One question regarding the problems that using ChatGPT may cause is why not use something like a plagiarism detection tool to detect the texts generated by ChatGPT? This is a valid question; however, the answer is not straightforward. Turnitin [29,30] and some other software have been developed to detect plagiarism. The difficulty here is that the texts produced by ChatGPT are original, so it is very hard for the plagiarism detection software to detect them. Although new software may be developed to detect AI-generated texts, because ChatGPT is being improved and learns from experience to generate better and more realistic texts, it is a matter of time before ChatGPT outcompetes these detection tools. As the tools built to detect AI-generated text learn to differentiate between the text generated by humans and those of machines, ChatGPT learns to be more human-like to avoid detection. It is shown in [31] that the successor LLM model of GPT3.5, called GPT4, outcompeted all of the existing LLMs in many evaluation tasks and it can generate longer sequences of data than GPT3.5. GPT 4 is now available to generate text on the ChatGPT platform as a premium service [32].

Many aspects of ChatGPT and its assistance in an educational setting have been explored in several pieces of published literature. Also, the literature is purely written from the standpoint of academics and educational institutions. However, to the best of our knowledge, no work has addressed: 1. The use of ChatGPT as an aid in teaching and learning computer science subjects. 2. The views of non-native English speaker students in computer science and around the use of the tool in academia. Therefore, this paper emphasizes the use of ChatGPT in imparting knowledge to computer science students, as well as analyzing the students’ views regarding the use of ChatGPT in academic settings. Students are important stakeholders in an educational institutions and policymakers should also take into account their perspective as well. The rest of the paper is as follows. Section 3 covers leveraging ChatGPT to enhance the study of computer science disciplines. Section 4 covers the material and methods of the study conducted on computer science students regarding their views on ChatGPT. Section 5 discusses the result of the study and Section 6 provides the discussion on the correlation study. The last Section provides the conclusion of the study.

3. Examples of ChatGPT Use

In this section, we review several ways in which ChatGPT can be used in learning/teaching activities and provide examples for each of them in computing.

ChatGPT is not only capable of generating human language text, but it can also generate computer programming languages [33]. This capability particularly is of utmost importance to computer science students. ChatGPT can be asked to generate a code for a particular task in any given programming language. Although, in many cases, the generated code may have errors, nonetheless, the generated code can provide good insights into how to structure your code.

As a demonstration of how ChatGPT can aid computer science students’ learning, we have chosen a year-3 student’s assignment from the software/application development module or advanced programming module. Usually, in these modules’ coursework, students are expected to design a software/application based on a business case study. In these business case studies, the students should be able to identify the steps required for developing such applications.

In a typical application development assignment, the students should perform the following tasks.

• Identifying business requirements for given applications.
• To design and program the application logic.
• Set up the database and server side-scripting.
• Testing the applications for different business use cases.
• Deployment and maintenance of the application.

To demonstrate the capabilities of ChatGPT in performing such tasks, in Figure 1B, ChatGPT has been prompted to identify business requirements regarding an online food distribution company. Note that Figure 1 only contains the selected part of the ChatGPT responses for illustration. Figure 1B shows the prompt that asks ChatGPT regarding which topics to study for the successful development of the given application in a given programming language. The response provides a wholesome list that almost covers all of the important aspects required for generating a successful application in C# language. For example, it recommends using Windows forms as GUI, which is popular among applications built for desktop-based software applications. Interestingly, the tool suggests the use of encryption and decryption techniques, which are very important for many applications when sensitive data are stored. However, there are some skills that would help students in application development which are not identified by ChatGPT. For example, model-view-controller, which is essential architecture for GUI-based application developments, is not included in this list.

In Figure 1C, ChatGPT is asked to give a list of different class entities required to develop the application mentioned above. The response provides a solution with the name of different classes with their descriptions. As seen in this Figure, the tool has been able to suggest a product class that includes all the necessary properties of the class, along with their data types. It is interesting that not only ChatGPT has been able to identify the method “GetProductByID” properly, but also it has identified the method “GetProductByCategory” and the proper return type.

In the next step, ChatGPT has been prompted to write a sample C# code for the classes in Figure 1C. Figure 1A represents the code for the customer class. While the tool generates good C# code, there still are small differences with what it identifies in Figure 1D. For example, the entity “Email” has been identified in Figure 1A and not in Figure 1D. This means that students should double-check the responses provided by ChatGPT as there may be some mistakes that should be fixed.

In Figure 1D the tool is asked to generate SQL commands to generate the tables based on the class entities suggested for developing the Food distribution application. In this Figure, a sample of the tables generated by the tool is represented. Here, ChatGPT not only finds the entities but also generates all of the class properties along with their datatypes required for the given application and provides a dummy table for each class presented in Figure 1C. One important aspect here is that there is a small discrepancy in the naming of the entities in these two Figures (Figure 1C,D). As observed here, the “quantity” variable in Figure 1C has been replaced by the “stock” variable in Figure 1D, which is a small mismatch. If the student does not detect this, they may obtain an error in their implementation. This issue is most probably because the tool uses LLM to generate this information, which is inclined to use synonyms in natural language. This, however, is not accepted in programming languages.

While ChatGPT can be very helpful in providing answers to students’ questions, in some cases, the tool may provide the wrong information regarding a query [19]. So, students need to understand the limitations of ChatGPT, as well as test the code and focus on debugging the code, which will certainly help them to have a deeper understanding of the codes. With ChatGPT, students can generate quizzes to practice their coding knowledge or ask ChatGPT to generate application ideas, which they can then develop to test their programming skills. Application development assignments can be a huge undertaking for students and the experience gained in this module is then helpful in their final dissertation as well. The big advantage of initiating ChatGPT over the internet is that the solution is modified to a problem statement; in our case, ChatGPT gives an example according to the food distribution application, which is an easy start for the student to then build their knowledge, instead of being clueless about where to begin. Later, for more complex programming solutions, they can navigate the web in the usual manner.

A new form of assessment grading criteria can then be introduced demonstrating basic, intermediate, or advanced programming skills, based on the student submission. The percentage score of students in computer science is meaningless for an employer to interpret what programming skills the student possesses.

4. Materials and Methods

In order to study students’ opinions about ChatGPT, its capabilities, the way it can be incorporated into teaching, and students’ opinions about the tool, we perform a study and collect information from them via a questionnaire. The students enrolled in the 2023 January semester for the module Research Methods in Computer Science at the University of Hertfordshire. The data were collected from a cohort of 430 students; all MSc students in computer science.

4.1. Setting

The module “Research Methods in Computer Science” is a 30-credit module taught at the master’s level. The students attend two-hour lectures per week for a 10-week period. The following learning objectives are defined for this module:

• Identify a range of research methods that may be applied to problems in computer science.
• Understand how those methods may be put to use in the context of an advanced master’s project.
• Critically evaluate and employ techniques that will aid them with planning and executing a substantial program of independent project work at an advanced master’s level.

4.2. Evaluation and Intervention

The students are required to complete the questionnaire that asks about the student’s opinions about ChatGPT. The questionnaire, as a multiple choice unmarked quiz designed on Canvas is given to the students.

• The questionnaire consists of twelve questions that ask the students about different aspects of ChatGPT to study their level of familiarity with the tool, how often they use the tool, its positive impact on the language, the threats associated with using the tool, etc. These are multiple-choice (mainly five-choice) questions.
• Because these are international students with different backgrounds, and one aim of this research is to study the effect of the learning background on their opinion about ChatGPT, we also include a question that asks the students about the continent on which they have studied their undergraduate degree. This way we can also study the variation in student opinions based on different backgrounds.

The questionnaire is presented in the Appendix A The questions are designed in a way that they capture different aspects of using ChatGPT in learning. They try to capture students’ familiarity with the tool, how much and for what tasks they use the tool, if they think the text generated by the tool can be identified by plagiarism detection software, if the way ChatGPT can be used should be taught at the universities, and the threats associated with using the tool.

4.3. Data Collection

The questions are put online on Canvas and the students are asked to fill out the forms. The questionnaire is anonymized, and no demography data of the student that can reveal individual identity is collected. This ensures that students can answer the questions without any hesitation. The first author of this paper was in charge of creating the online questionnaire and collecting data. After the data are collected, they are exported into a .csv file and Matlab was used to perform statistical analysis. The analysis focuses on the responses of all 430 students who took part in the study.

5. Results

The aim of this paper is to perform research about the opinion of the students at the MSc level to identify how they believe ChatGPT can be exploited to improve teaching/learning activities and prevent the negative impacts it can have on education.

The demographic distribution of the students is important because students with different backgrounds might think differently about different subjects of study and the way ChatGPT can be used in education. To identify the demographic background of the students, Question 1 asks in which continent the students have studied for their undergraduate degree. This is represented in Figure 2; all of the students are international students, among which 88% are Asian, and 10% are African.

When we compared the answers provided by the African and Asian students using the chi-square test, we did not find any statistically significant difference between the way they answered the questions. This suggests that students with different backgrounds in their educations show similar behavior in the way they answered the questions. However, it may not be the case for a balanced representation of international students.

We proceed with the questionnaire by asking the students about how much they are familiar with ChatGPT. It is important to know what proportion of the students, and to what degree they are familiar with the tools. The level of familiarity affects the way they employ the software, the way they think it can be used to prompt better learning and the ways in which the negative impacts of the tool can be managed.

The level of familiarity of the students follows a rather normal distribution, which is an expected observation (Figure 3). Around 12% of the students indicate that they are very familiar with the tool, whereas 19% of the students indicate that they are not familiar with the tool at all. It is interesting to see that almost one-fifth of the students in computer science are not familiar with the tool. This suggests that there is more need to teach students about emerging topics and tools as we believe that MSc students in computer science should be familiar with state-of-the-art tools.

The main objective of this paper is to study how ChatGPT can be employed to improve educational practice. Thus, in the third question, we ask the students how often they use the tool for academic purposes (Figure 4). The data suggest that 55% of the students have never used the tool for academic purposes. Clearly, 19% of these subjects are the ones who were not familiar with ChatGPT. However, there remain around 36% of the students who are familiar with the tool but still have not used the tool in their learning activities at the university. There are around 5% (6%) of the students who use the tool very frequently (frequently), despite the fact that, in Question 2 (Figure 3), around 31% of the students indicate that they are familiar or very familiar with the tool. The question here is why there is a limited number of students who use the tool in their learning activities.

One explanation for this behavior could be that there does not exist a clear set of policies regarding the way this software can be used by students in their education. Therefore, they might be afraid to use the tool in case they are breaking the university rules or if they are conducting plagiarism. In such a scenario, the students prefer not to risk engaging in an activity that may harm them in the long run. If this is the reason, then there is a need for the universities to start clarifying the regulations and rules so the students can use the tool for the type of activities they are allowed.

The other explanation for why the students do not use the tool for academic purposes could be that they do not know how to positively exploit the tool to prompt better learning. If the reason for not using the tool is the lack of skill, then we believe that the universities should step in and teach the students how to use ChatGPT in a way that improves and, at the same time, does not negatively affect learning. As suggested in [4,6], the universities should step in and find a way of optimally using this tool. We believe that the use of ChatGPT in education is a very subtle area because, on the one hand, the tool can easily be used by students for academic misconduct. For example, the students can use it to generate proposals, reports, essays, etc. This clearly can have a negative impact on learning. On the other hand, if used properly, it can be utilized to help the students to come up with new insights, new ideas, clarification about the subjects they have doubts about, etc. Finding this optimal way of using the tool and setting the regulations about this is the job of universities.

As the majority of our students are international (Figure 2), we asked the students’ opinions about the impact of ChatGPT on international students. The data suggest that around 28% of the students strongly believe that the use of ChatGPT has a negative impact as opposed to 21%, who strongly believe that it has a positive impact on international students (Figure 5). If we add the number of students who chose option 1 or 2, we obtain 38%, which is the number of students who believe that there is a negative impact from ChatGPT on international students. Interestingly, around the same proportion of the students (37%) believe that ChatGPT has a positive impact on international students. Whereas the remaining (24%) believe that the tool has neither a positive nor a negative impact. In general, this graph shows a rather flat behavior, instead of a bell-shaped distribution, which is usually observed in data.

One reason the students may think that the tool has a negative impact on international students may be the possibility that relying on an intelligent tool to prepare some of the materials and perform some of the tasks for them can prevent the students from developing and enhancing their skills. Another reason could be that the students may use the tool to perform tasks that are designed to enhance critical thinking. By giving such tasks to intelligent machines, the students will be involved in fewer thinking activities that result in the underdevelopment of crucial thinking skills. Another explanation could be that the students prefer human–human interaction as a method of learning, rather than a human–machine interaction which may not have the required dynamics of a successful learning experience. The language barrier could be another reason for believing that the tool results in a negative impact. Many of the students improve their language skills via different learning activities and human–human interactions that they have in the university. Possibly, the students may see the tool as a system that prepares them with the material and stops them from digging into the context. This can result in fewer language-related practices resulting in slower skill improvements.

On the other hand, there are some good reasons why students may think that the tool can have a positive impact. One reason could be that using the tool helps the students to save time on trivial tasks and spend more time on complicated tasks that require more enhanced skills. For example, if the students are given a problem to solve, they can use the tool to gain clarification about the subjects about which they have doubts, so they can more focus on solving the problem itself. One other explanation could be that the system can be used as a resident to the students to prepare the raw material for them. For example, students can ask ChatGPT to find different methods by which a particular problem can be solved. Then, the student can decide which method is better for their problem and investigate more.

One way in which ChatGPT can be used by students is to get help with language [4,6,7]. To study this aspect, we ask if the students use the tool for language-related tasks to improve their English writing in Question 5 (Figure 6). The data suggest that more than half of the students have never used the tool in language-related tasks, to improve their language skills. Although many international students have difficulty coping with language barriers and require practice to improve their skills, it is interesting to observe that they do not employ the system. Among the students, around 9% very frequently use the software to improve their language and around 11% use it frequently. This suggests that around one-fifth of the students get help with the system with their language. One explanation for this observation could be that the students are not familiar with the system’s capabilities and the ways in which they can use the system for their benefit. Thus, it is imperative to impart knowledge regarding ChatGPT capabilities to students, as suggested by [6]. Moreover, the students may be afraid that using the system may be against the university regulations and they might prefer to stay on the safe side. This is another reason that universities should start creating regulations and consider the ways in which ChatGPT can be used to improve language-related skills.

One crucial threat from incorporating ChatGPT in education is academic misconduct and plagiarism. Thus, plagiarism detection companies like Turnitin are now developing algorithms to detect if a text has been generated by a machine or by a human. In Question 6, we asked the students how confident they are that plagiarism detection software like Turnitin is capable of detecting AI-created texts. This is represented in Figure 7. Around 20% of the students are highly confident and around 23% are confident that AI-created texts can be detected by existing plagiarism detection software. Although the older versions of plagiarism detection software were not able to detect these texts, research is being conducted to develop the systems. The reason that a high proportion of the students believe that these texts are detectable is probably due to the recent improvements in the capabilities of the detection algorithms [29]. Around 30% of the students are not sure if it is possible to detect AI-created texts. Among the students, around 18% believe with confidence that the texts are detectable via plagiarism detection software. It is particularly challenging to create regulations for the usage of ChatGPT with regard to controlling the academic misconduct that is performed by ChatGPT. For plagiarism, it is much easier to use software to detect if the student is using some text from another source and, when it is found, it is very easy to prove that the misconduct has happened. One only needs to compare the two texts and decide if the text is the exact copy of the other one. When it comes to the usage of ChatGPT, if the software claims that the text is AI-generated, then it is very hard to decide whether the test is written by the student or academic misconduct has happened. A study is conducted in [34], on 50 academic essays that were generated via ChatGPT, and 40 of them were considered original by the plagiarism detection software. Thus, punishing a student based only on the decision of the software is a hard decision. These challenges should be taken into account when creating the regulations.

One other aspect that should be taken into account is that even if the plagiarism detection software becomes developed enough to detect AI-generated texts with good accuracy, it is not the end of the challenge. As suggested in [6], there is a constant race between text generation tools and detection tools. This could be one reason why around a quarter of the students think that it is not possible for plagiarism detection algorithms to detect AI-generated texts.

In the next Question, no. 7, we ask students if they think the universities should explicitly teach how ChatGPT can be used in their studies (Figure 8). Interestingly, around 46% of the students chose the neutral answer to the question. We believe that the reason why a large number of students are not sure about the answer is the complexity of the decision. On the one hand, the software is a very good tool that, if positively used, it can open doors to many new ways of teaching and learning activities. On the other hand, the tool can be misused by many students and can be used to perform tasks that are supposed to be performed to learn crucial skills. Interestingly, the number of students who think that the universities should not be involved in teaching the topic (19%) is very similar to those who believe that the universities should step in and teach the students (18%). This biased opinion is most likely related to the complexity of the decision. Adding the numbers in the first and second columns (19% + 6% = 25%) shows that around a quarter of the students disagree with the idea that ChatGPT should be taught by universities. On the other side, there are 29% of the students who think that the subject should be taught. These data clearly indicate that there is no clear-cut decision on this matter. A similar observation about ChatGPT within society has already been observed [35]. A minority believes it is not an effective tool, while others view it as a valuable resource that should be utilized more extensively. However, the majority are still uncertain about ChatGPT, considering it premature to make definitive judgments about the tool. Another survey of the US public between the ages of 18–65 reported [35] similar findings to what we observe among students, indicating that it is not only the students and academia that have not yet decided about how the tool should be used; society, in general, is also sceptical.

Many people think that AI can be a threat to humans and ChatGPT is not an exception. To see what the students think is the biggest threat associated with using the tool in academia, we developed Question 8 (Figure 9). Here, around 20% of the students think that the biggest threat is plagiarism and dishonesty. Clearly, the tool can easily be used by students for academic misconduct; this could include using the tool to generate reports, find answers to assignments, write a thesis, etc. Perhaps, therefore, the highest number of students chose the second option, where 39% of the students believe that the largest threat comes from reduced critical thinking and originality. This is understandable, because if the students use the tool for tasks that are designed to prompt thinking and curiosity, it can easily dampen these skills. For example, if the students are asked to perform a short literature review and, based on the literature, find the gap in research and propose a research idea, it is crucial that all steps are performed by the students. Such a task requires critical thinking to find the gaps and creativity to propose ideas. If the students rely too much on the tool, they will not practice what they are supposed to and will not develop the skills they require. This clearly is one concern that should be targeted when rules and regulations are created, and also the way assessments are designed. Furthermore, around a quarter of the students believe that over-dependency is the main threat with using ChatGPT. Because ChatGPT is capable of performing many smart tasks, there is always the possibility that the students may use it for trivial tasks and may become over-dependent on the tool.

One threat that is usually mentioned in the literature [6] is that the tool can sometimes make mistakes in the information it provides. Thus, if the students put too much trust in the system, they may be misled by ChatGPT. Around only 11% of the students have chosen this option, which suggests that a few percent of the students think it is a significant threat. This could be because the students believe that if the mistakes happen, they are capable of detecting and resolving them. Another threat that is considered in the literature [4] is that ChatGPT may produce biased responses. This is because the tool is a machine and cannot think as diversely as a human can do. Here, only 6% of the students think that this is a threat that should be taken seriously. The reason might be that the students think that the machine does not offer much-biased information.

Since many assessments are performed at home without invigilation, it is crucial to design them in a format that minimizes ChatGPT misuse. Question 9 is asked to see what the students think about how this should be performed (Figure 10). Among the answers, the maximum votes have gone to the option that says the assessments should be designed in a way that they require original thinking, so the tool cannot be used easily to solve them. This is reasonable, as such assessments would encourage critical thinking as well as prevent misuse and is in line with the study conducted by [6]. The second option says that the teachers should tell the students that they should forbid the usage of the software altogether. This has achieved the minimum number of votes. This is probably because the students believe that it is impossible to stop all of the students from using the tool, and some will cheat one way or another. In option 3 it is suggested that the teacher should provide guidance on how to positively use the tool for assessments. This has reached 36% of the votes. Combined, more than 73% of the students have voted for option 1 or 3, which means that the majority of students think that ChatGPT should be allowed, but the assessments should be designed and students should be guided in a way that the practice does not harm learning.

ChatGPT is an emerging technology that may cause great changes in many different aspects of life. It is possible that, in the near future, many businesses start to use the tool in their day-to-day activities. For example, companies may use the system in their customer services, where customers are answered via automatic machines. Thus, some companies may require their employees to be familiar with the tool. In Question 10, the students are asked if they believe that having knowledge about ChatGPT is a valuable skill and should be considered as a recruiting criterion for employers (Figure 11). Around 23% of the students believe that the skill should be considered by employers in the recruiting process. Among the students, around a third believe that it should not be a criterion. With 43%, the highest number of students believe that it is the nature of the job that determines the requirement. This is understandable because different industries require different tasks to be fulfilled.

One important application of ChatGPT in computer science can be the help it can provide in learning. The students are asked in Question 11 how they believe ChatGPT can help them in learning (Figure 12). One way the tool can be used in learning computer science is to generate codes, as it is capable of generating programming codes [32]. For example, if asked, it is capable of generating C++ codes for sorting. Around 18% of the students think that the tool is useful in helping them to generate codes. Around 12% of the students believe that it is most useful if used for writing reports.

One way the tool can be used is to help the students to gain clarification about the doubts they might have about their assignments. Around 47% of the students have indicated that they think the tool is most useful if it is used to obtain explanations about their doubts. This is a positive use case of ChatGPT. This way, instead of relying on the tool to produce materials, it is used to get help on improving the materials the students are producing themselves. Thus, the students are provided with better-quality raw materials to create their own set of crafts. This not only improves the quality of the students’ output but also equips them with a reliable source of knowledge. The reason why the students think they require more explanation from a tool like ChatGPT could be a lack of enough clarification during lectures. This could be attributed to the increasing student-to-teacher ratio in recent years. As [36] mentions, this increase has resulted in poor student performance.

The lowest proportion of students (around 6%) have chosen the fourth option; the tool can help them with preparing for assessments. Why students think it is not useful is probably because they believe assessments contain tasks that are not easily performed by ChatGPT. Or, they believe that relying on the tool for the assessments is not a good choice as it prevents them from deeply understanding the course materials. Around 16% of the students think that ChatGPT increases motivation. This is the third most popular choice among the students. One reason the students may think that the tool improves their motivation could be because, by providing good raw materials, the tool makes the tasks more understandable and easier to handle. Also, the tool provides them with good knowledge and insights that can promote their understanding of the subject.

One way ChatGPT can be used by students is to provide them with some help in developing applications. Developing an application consists of many different aspects. In the design process, students can benefit from the ChatGPT potential. To see what the students think about how ChatGPT can be used in the design process of applications, Question 12 is devised (Figure 13). The first option is using the application to develop ideas. Ranked second among all the options, around 22% of the students voted for this option. For example, students can ask the tool what type of applications attract more of an audience. Then, among the suggested types (gaming, health, fitness, entertainment, etc.). Then, the student can choose the more attractive one from the option and dig more about that type of application by asking questions about how long it takes to create one, what skills are required to develop it, whether it would sell, and so on.

The second option is that ChatGPT is best for helping them to create source codes. The highest, most popular option among the students is the second option, so the students believe the tool is most useful when it is used to generate codes. As discussed earlier, the tool can generate simple codes, so the students can spend their time on more complicated aspects of application development. This way, the students can work on the aspects of application development that require more critical thinking and originality like design, testing, deployment, and maintenance.

One important requirement in developing an application is the design of a database system, as many applications need to store and retrieve information about users. Option 3 asks the students if ChatGPT can help them with designing databases. The least popular option among the students is this option. It seems that the students believe that the task requires more expert knowledge and cannot simply be performed by a machine. Deciding what information you need to collect from the users and how to use them seems to be considered too complicated from the students’ point of view.

Testing an application is a crucial step in its design process. Around 19% of the students believe that the tool is useful for testing. Students can use ChatGPT in the testing process by asking the tool about the testing process and resources that the students can use to learn more about the process. Among the students, 16% believe that the tool is more helpful in the development and maintenance process. The tool can be specifically helpful in maintenance when it is used to communicate with customers.

6. Discussion

One important question here is whether there is a correlation between the answers the students provide for different questions. To answer this,

Table 1. This shows the correlation between the answers provided by the students for each pair of questions.

		Question 2	Question 3	Question 4	Question 5	Question 6	Question 7
Question 2	Correlation	-	0.54	0.44	0.51	0.24	0.4
	p-value	-	9.04 × ${10}^{-34}$	1.89 × ${10}^{-21}$	5.27 × ${10}^{-29}$	3.84 × ${10}^{-7}$	2.77 × ${10}^{-17}$
Question 3	Correlation	0.54	-	0.56	0.61	0.25	0.43
	p-value	9.04 × ${10}^{-34}$	-	7.40 × ${10}^{-36}$	6.72 × ${10}^{-44}$	1.47 × ${10}^{-7}$	4.30 × ${10}^{-20}$
Question 4	Correlation	0.44	0.56	-	0.5	0.22	0.5
	p-value	1.89 × ${10}^{-21}$	7.40 × ${10}^{-36}$	-	4.19 × ${10}^{-28}$	7.06 × ${10}^{-6}$	2.12 × ${10}^{-27}$
Question 5	Correlation	0.51	0.61	0.5	-	0.22	0.33
	p-value	5.27 × ${10}^{-29}$	6.72 × ${10}^{-44}$	4.19 × ${10}^{-28}$	-	4.10 × ${10}^{-6}$	4.02 × ${10}^{-12}$
Question 6	Correlation	0.24	0.25	0.22	0.22	-	0.19
	p-value	3.84 × ${10}^{-7}$	1.47 × ${10}^{-7}$	7.06 × ${10}^{-6}$	4.10 × ${10}^{-6}$	-	1.26 × ${10}^{-4}$
Question 7	Correlation	0.4	0.43	0.5	0.33	0.19	-
	p-value	2.77 × ${10}^{-17}$	4.30 × ${10}^{-20}$	2.12 × ${10}^{-27}$	4.02 × ${10}^{-12}$	1.26 × ${10}^{-4}$	-

shows the correlation between the answers the students provided for each pair of questions. The Table also includes the p-value. In this Table, only Questions 2 to 7 are included because these are the questions for which the students provide scaled answers. That is, in these questions, the answers are ordered. The answers for other questions are not ordered so the correlation between their answers is not meaningful. To study the correlation between the students’ responses, a Pearson correlation coefficient test is performed and the associated p-value is recorded, which gives the significance of the correlation (Table 1).

Question 2 asks about the familiarity of the students with the tool and Question 3 asks if they have used the tool for academic purposes. The strongest correlation between Question 2 is observed with Question 3. The significant positive correlation between the answers to these two questions simply suggests that the more familiar the students are with the tool, the more they use it for academic purposes, which is easy to understand. The second strongest correlation between Question 2 is observed with Question 5, which asks if the students have used the tool in language-related tasks. Question 2 has the weakest correlation with Question 6, which asks how confident the students are that plagiarism detection software can detect the texts generated by the software. It seems that students’ familiarity with the tool does not affect their opinion about Question 6.

The answers to Question 3 show their strongest correlation with the answer to Question 5. At 0.61, with the p-value $6.7\times {10}^{-44}$, this is the strongest and most significant correlation observed among the data. Question 3 asks if the students have used the tool for academic purposes and Question 5 asks if they have used it for language-related tasks. The fact that the strongest and most significant correlation is observed between these two questions suggests that the students who use the tool for their academic purposes also use it for language-related tasks. After this, Question 3 shows its second strongest and second most significant correlation with Question 4, which asks students how they perceive the impact of the tool on international students. The strong correlation here suggests that the students who use the tool for academic purposes believe that it can be helpful for international students. Similar to what we observed in Question 2, Question 3 also shows its weakest correlation with Question 6. It suggests that the level at which the students use ChatGPT for academic purposes does not much affect the way they think plagiarism detection algorithms are capable of detecting the texts that are generated by the tool.

The answers the students provide for Question 4 show a significant correlation with the answers to Question 5. Question 4 asks how they perceive the impact of the tool on international students and Question 5 asks if they have used the tool to improve their English skills. Since these two questions are relevant, such a correlation is understandable. There is also a significant correlation between Question 4 and Question 7, which asks if the students believe the way ChatGPT is used as a tool for their studies should be taught by the universities. It suggests that how the students perceive the impact of the tool on international students affects whether they think that using the tool should be taught at the universities. Similarly to the previous questions, the answers to Question 5 show their weakest correlation with Question 6.

The answers to Question 5 show very weak correlations with the answers to Questions 6 and 7, suggesting that the level at which the students use the tool for language-related tasks does not much affect whether they believe that the way the tool is used should be taught at the universities. Finally, it is observed that the p-value is very small for all the values in the Table, which suggests a significant correlation between the answers the students provide for the questions (Table 1).

7. Conclusions

In this paper, we studied the students’ opinions about how ChatGPT can be used as an aid in learning, teaching, and assessments for computer science in higher education. We conducted a survey on 430 students via a questionnaire that consisted of 12 different questions. The questions are designed in a way that they capture different aspects of ChatGPT and its applications in education.

As presented in this paper, although the students are familiar with the tool, they do not use it on a regular basis in their learning activities. The data suggest that the students show concern regarding misusing ChatGPT in their academic activities. It seems that they are aware that if it is not used properly, it can have a negative impact on critical thinking and the skills the students require to investigate and draw conclusions in their work. Also, the data suggest that the students lack in-depth skills and knowledge with respect to the tool.

ChatGPT, like any other technology, can have its benefits and drawbacks. The survey in this paper suggests that the students believe that there are many ways in which the tool can be positively or negatively used for education. One application of the tool could be to use it to improve writing skills. If used properly, students can benefit from the system by finding their mistakes, correcting them, and improving their writing skills. Another possible utilization for computer science students is in code generation and debugging. However, the students must be informed about its limitations, specifically that it is a language model without execution capabilities [8,9]. On the other hand, the tool can be easily misused, by relying too much on the system and using it to create the texts. For international students, the tool can be used to improve their communication skills. For example, by using the system as a smart search engine to find out how they can improve their language skills.

One challenge in this area is when the students use the tool to generate reports for their assignments. Such cases should be considered as academic misconduct. However, the problem is that it is very difficult to identify these reports by only reading the texts. Plagiarism detection applications like Turnitin are not capable of detecting this type of academic misconduct. For this, plagiarism detection applications should improve to be able to detect these types of plagiarism. This is challenging, as automatic text generation applications are constantly evolving and detecting them is a constant race between the two sets of applications. Also, even when a text is detected as being generated by a machine, it is very difficult to prove it and make decisions against the students based on that observation.

In this paper, we studied students’ opinions about whether they believe ChatGPT should be explicitly taught by universities. We believe that the tool has many capabilities in education and it is very important that the students are taught how to use the tool in their studies. Also, we studied students’ opinions about the threats that may harm students in their education.

This study suggests that there are many challenges with regard to incorporating ChatGPT in learning activities. If used properly, it can have many positive impacts, and if misused can harm students. Software applications are developed to help us perform tasks more quickly, more easily, and more efficiently. We believe that completely preventing the students from using the tool is not a smart action as it would only increase the risk of misuse. Students use any tool that makes performing tasks easier for them. Thus, the students use ChatGPT anyway, and the best way is to create guidelines on how to positively use the tool in education. This itself requires more study to find ways in which the tool is incorporated into academia to benefit from its advantages and, at the same time, evade the problems. As a line of future work, studies should be conducted to find the best ways in which the tool can be adopted in education. This can be clearly answered if universities step in and provide the students with the rules, regulations, knowledge, and skills about how the tool can be positively used. While the data in this paper are informative, they are not definite and are subject to change if environmental factors change. For example, if clear policies regarding the rules and training are set by universities, many of the results in these graphs would significantly change.

Another line of research for future work is to compare the performance of students based on the level at which they are using ChatGPT. For example, to compare the final marks of the students who are more familiar with the tool and use it more frequently to those who do not. Also, different policies, with respect to the way this tool is incorporated into teaching, can be compared in terms of their effect on the performance of students. This can help to find the optimal set of policies.