Effects of Generative Chatbots in Higher Education
Abstract
:1. Introduction
- Developing a new conceptual framework that facilitates a systematic approach for application of intelligent chatbots in university teaching and learning;
- Gathering a student dataset for their experience with and without chatbot support, including students’ learning characteristics, perceptions, attitudes toward educational chatbots, and specific problems;
- Uncovering hidden relationships in the student data through the proposed methodology;
- Clarifying the difficulties, expectations, and benefits of using chatbots in formal and informal learning environments;
- Assessing the competence of educational chatbots in handling university learning tasks;
- Providing measures for chatbot applications and recommendations for participants in the learning process to improve the university educational practice.
2. State-of-the-Art Review of Intelligent Chatbot Models, Platforms, and Systems
2.1. Large Language Models for NLP and Their Comparison
- LLMs can generate biased, harmful, or inaccurate content and discriminate based on the input data they are trained on and the specific applications they are used for.
- LLMs can be vulnerable to adversarial attacks, in which attackers deliberately input misleading data to manipulate the model’s output.
- Some experts have raised concerns about the environmental impact of training large language models, as it can require massive amounts of computing power and energy.
2.2. Intelligent Chatbots and Their Comparison
- Functionality: The mentioned chatbots are specifically designed for natural language understanding and generation. The majority of them are constructed using LLMs and transformer architecture. However, there are a few exceptions such as IBM Watson Assistant and AWS Lex, which leverage a combination of various AI methods.
- Language support: ERNIE Bot supports multiple languages, including Chinese and English, while ChatGPT primarily focuses on English language tasks.
- Internet connectivity: Only ERNIE, PanGu, Bard, and ChatGPT Plus have Internet access and can receive real-time information.
- Multi-modality: Only ERNIE Bot, Bard, and ChatGPT Plus can process multimodal inputs (text or images).
- Pricing: PanGu, ChatGPT, and GPT Plus are paid software. These chatbot platforms offer flexible pricing models and even provide free plans for individual users and businesses.
- Ease of use: Some of the available chatbot alternatives, like ChatGPT and ChatGPT Plus, offer user-friendly interfaces that make them more accessible to users without coding knowledge. The ease of use varies among different chatbot options.
- Complexity of set-up: It is an important consideration when choosing a chatbot. Some chatbots may require programming expertise to set up and deploy effectively.
- Use cases: While chatbots have a wide range of applications in NLP, ChatGPT and ChatGPT Plus are particularly useful for conversational interactions, making them suitable for chatbot and virtual assistant applications. On the other hand, ERNIE Bot and BLOOMChat with their multi-language support are ideal for tasks involving multiple languages or cross-lingual applications. Google Bard demonstrates enhanced mathematical skills and improved logical reasoning capabilities. Alpaca and Vicuna models can accelerate the progress in the development of reliable AI chatbots. These models offer capabilities comparable to closed-source models like text-davinci-003.
- Often, chatbots lack real-time data generation capabilities, which hinders their ability to instantly monitor customer conversations and promptly identify potential issues.
- Chatbots can sometimes generate inaccurate or “hallucinated” responses, which necessitates extensive and time consuming fact-checking.
- Another problem is the occasional inability of chatbots to understand users’ questions and requests. This problem arises from users’ lack of knowledge about structuring their questions effectively to elicit responses that meet their specific needs.
2.3. Educational AI Chatbots
- Conversational assistance: They understand and address students’ questions in a conversational manner.
- Multi-modality: They can support multiple modes of communication, including text, speech, and visual elements.
- Multilingual support: They offer multilingual capabilities for diverse student communities.
- Cost-effectiveness and scalability: They are capable of handling large student populations while remaining cost-effective.
- Integration with other software systems: They can integrate with learning management systems, library databases, and online search tools.
- Data analytics and insights: They can provide instructors with data analytics and insights to enhance their teaching methods [42].
- Advanced and specialized subjects benefit from personalized assistance;
- Emphasis on self-directed and independent learning;
- Access to scholarly resources, aiding student research activities such as literature reviews and research methodology guidance.
3. Related Work
3.1. Theoretical Frameworks for Application of AI Chatbots in Education
- They do not fully address the operational aspects of teaching and learning in higher education and may not encompass key elements of the university syllabus, such as knowledge acquisition and skills development.
- Existing frameworks often focus on specific academic courses or support specific stakeholders involved in the educational process, such as students, teachers, administrative departments, or governing bodies.
- They tend to assess the implications of using artificial intelligence tools in higher education primarily through assessing satisfaction levels of students and educators, rather than providing algorithms for chatbot usability analysis.
3.2. Measuring Students’ Attitude toward Chatbot Services
3.3. Measuring the Quality of Chatbot Services
4. Framework for Chatbot-Assisted University Teaching and Learning
5. Verification of the Proposed Chatbot-Based Teaching–Learning Framework
5.1. Student Survey on the Capabilities of Generative AI Chatbots—Questionnaire Design, Data Collection, and Data Analysis
- positive—27, average value 0.76;
- neutral—9, average value 0.54;
- negative—18 (actually 17, because one of the negative opinions has score 0.001), average value 0.16.
5.2. Experimental Evaluation of the Reliability of Chatbots as an Educational Tool
6. Conclusions and Future Research
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Bedwell, W.L.; Salas, E. Computer-based training: Capitalizing on lessons learned. Int. J. Train. Dev. 2010, 14, 239–249. [Google Scholar] [CrossRef]
- Elmesalawy, M.M.; Atia, A.; Yousef, A.M.F.; Abd El-Haleem, A.M.; Anany, M.G.; Elmosilhy, N.A.; Salama, A.I.; Hamdy, A.; El Zoghby, H.M.; El Din, E.S. AI-based flexible online laboratory learning system for post-COVID-19 era: Requirements and design. In Proceedings of the 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), Cairo, Egypt, 26–27 May 2021; pp. 1–7. [Google Scholar]
- Winkler, R.; Söllner, M. Unleashing the potential of chatbots in education: A state-of-the-art analysis. J. Acad. Manag. Annu. Meet. 2018, 2018, 15903. [Google Scholar] [CrossRef]
- Chan, C.K.Y.; Hu, W. Students’ Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education. arXiv 2023, arXiv:2305.00290. [Google Scholar] [CrossRef]
- Adamopoulou, E.; Moussiades, L. Chatbots: History, technology, and applications. Machine Learning with Applications. J. Mach. Learn. App. 2020, 2, 100006. [Google Scholar]
- Pillai, R.; Sivathanu, B. Adoption of AI-based chatbots for hospitality and tourism. Int. J. Contemp. Hosp. Manag. 2020, 32, 3199–3226. [Google Scholar] [CrossRef]
- Zhou, X. Chatbot Improves the Customer Service in Four Important Industries. In Highlights in Science, Engineering and Technology; Katrine, E., Li, J., Eds.; Darcy & Roy Press: Hillsboro, OR, USA, 2023; Volume 39, pp. 339–346. [Google Scholar]
- Mangla, D.; Aggarwal, R.; Maurya, M. Measuring perception towards AI-based chatbots in Insurance Sector. In Proceedings of the 2023 IEEE International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE), Bengaluru, India, 27–28 January 2023; pp. 890–895. [Google Scholar]
- Market Research Future. Conversational Artificial Intelligence (AI) Market Overview. Available online: https://www.marketresearchfuture.com/reports/conversational-ai-market-7913 (accessed on 30 June 2023).
- MarketsandMarkets. Conversational (AI) Market Overview. Available online: https://www.marketsandmarkets.com/Market-Reports/conversational-ai-market-49043506.html?gclid=EAIaIQobChMI8JjmoNzb_gIVDNmyCh2fUA0uEAAYASAAEgL10vD_BwE (accessed on 30 June 2023).
- Zhao, W.X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. A survey of large language models. arXiv 2023, arXiv:2303.18223. [Google Scholar]
- Liu, Y.; Han, T.; Ma, S.; Zhang, J.; Yang, Y.; Tian, J.; He, H.; Li, A.; He, M.; Liu, Z.; et al. Summary of ChatGPT/GPT-4 research and perspective towards the future of large language models. arXiv 2023, arXiv:2304.01852. [Google Scholar]
- Deng, X.; Yu, Z. A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability 2023, 15, 2940. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Ravichandiran, S. Getting Started with Google BERT: Build and Train State-of-the-Art Natural Language Processing Models Using BERT; Packt Publishing Ltd.: Birmingham, UK, 2021. [Google Scholar]
- Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.R.; Le, Q.V. XLNnet: Generalized autoregressive pretraining for language understanding. Adv. Neural Inf. Process. Syst. 2019, 32, 5753–5763. [Google Scholar]
- Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; Wu, H. Ernie: Enhanced representation through knowledge integration. arXiv 2019, arXiv:1904.09223. [Google Scholar]
- Wang, S.; Sun, Y.; Xiang, Y.; Wu, Z.; Ding, S.; Gong, W.; Feng, S.; Shang, J.; Zhao, Y.; Pang, C.; et al. Ernie 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation. arXiv 2021, arXiv:2112.12731. [Google Scholar]
- Floridi, L.; Chiriatti, M. GPT-3: Its nature, scope, limits, and consequences. Minds Mach. 2023, 30, 681–694. [Google Scholar] [CrossRef]
- Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 2020, 33, 1877–1901. [Google Scholar]
- Zeng, W.; Ren, X.; Su, T.; Wang, H.; Liao, Y.; Wang, Z.; Jiang, X.; Yang, Z.; Wang, K.; Zhang, X.; et al. PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation. arXiv 2021, arXiv:2104.12369. [Google Scholar]
- Ren, X.; Zhou, P.; Meng, X.; Huang, X.; Wang, Y.; Wang, W.; Li, P.; Zhang, X.; Podolskiy, A.; Arshinov, G.; et al. PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing. arXiv 2023, arXiv:2303.10845. [Google Scholar]
- Yuan, S.; Zhao, H.; Du, Z.; Ding, M.; Liu, X.; Cen, Y.; Zou, X.; Yang, Z.; Tang, J. WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models. AI Open 2021, 2, 65–68. [Google Scholar] [CrossRef]
- Zou, X.; Yin, D.; Zhong, Q.; Ding, M.; Yang, H.; Yang, Z.; Tang, J. Controllable Generation from Pre-Trained Language Models via Inverse Prompting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, 14–18 August 2021; pp. 2450–2460. [Google Scholar]
- Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.T.; Jin, A.; Bos, T.; Baker, L.; Du, Y.; et al. Lamda: Language models for dialog applications. arXiv 2022, arXiv:2201.08239. [Google Scholar]
- Biriukov, A.; Tereshchenko, I. Prospects for the use of natural language processing technology in the framework of the digital transformation of the Russian economy and building a digital society. Inf. Soc. 2023, 2, 105–120. (In Russian) [Google Scholar]
- Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H.W.; Sutton, C.; Gehrmann, S.; et al. Palm: Scaling language modeling with pathways. arXiv 2022, arXiv:2204.02311. [Google Scholar]
- Scao, T.L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A.S.; Yvon, F.; Gallé, M.; et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv 2022, arXiv:2211.05100. [Google Scholar]
- Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; et al. GLM-130B: An open bilingual pre-trained model. arXiv 2022, arXiv:2210.02414. [Google Scholar]
- Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. LLAMA: Open and efficient foundation language models. arXiv 2023, arXiv:2302.13971. [Google Scholar]
- Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y.T.; Li, Y.; Lundberg, S.; et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv 2023, arXiv:2303.12712. [Google Scholar]
- Al-Sinani, A.H.; Al-Saidi, B.S. A Survey of Chatbot creation tools for non-coder. J. Stud. Res. 2019. [Google Scholar] [CrossRef]
- Nuruzzaman, M.; Hussain, O.K. A survey on chatbot implementation in customer service industry through deep neural networks. In Proceedings of the 2018 IEEE 15th International Conference on e-Business Engineering (ICEBE), Xi’an, China, 12–14 October 2018; pp. 54–61. [Google Scholar]
- Mi, F.; Li, Y.; Zeng, Y.; Zhou, J.; Wang, Y.; Xu, C.; Shang, L.; Jiang, X.; Zhao, S.; Liu, Q. PanGu-Bot: Efficient generative dialogue pre-training from pre-trained language model. arXiv 2022, arXiv:2203.17090. [Google Scholar]
- Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; Hashimoto, T.B. Alpaca: A strong, replicable instruction-following model. Stanf. Cent. Res. Found. Models 2023, 3. Available online: https://crfm.stanford.edu/2023/03/13/alpaca.html (accessed on 6 September 2023).
- Chiang, W.L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J.E.; et al. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023. Available online: https://lmsys.org/blog/2023-03-30-vicuna/ (accessed on 6 September 2023).
- Neto, A.J.M.; Fernandes, M.A. Chatbot and conversational analysis to promote collaborative learning in distance education. In Proceedings of the 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Maceió, Brazil, 15–18 July 2019; Volume 2161, pp. 324–326. [Google Scholar]
- Lien, A.K.; Randem, L.M.; Taralrud, H.P.F.; Edalati, M. Implementing a Chatbot So-lution for Learning Management System. arXiv 2022, arXiv:2206.13187. [Google Scholar]
- Kumar, J.A. Educational chatbots for project-based learning: Investigating learning outcomes for a team-based design course. Int. J. Educ. Technol. High. Educ. 2021, 18, 65. [Google Scholar] [CrossRef] [PubMed]
- Lo, C.K. What is the impact of ChatGPT on education? A rapid review of the literature. Educ. Sci. 2023, 13, 410. [Google Scholar] [CrossRef]
- Baidoo-Anu, D.; Owusu Ansah, L. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available online: https://ssrn.com/abstract=4337484 (accessed on 6 September 2023).
- Gimpel, H.; Hall, K.; Decker, S.; Eymann, T.; Lämmermann, L.; Mädche, A.; Röglinger, M.; Ruiner, C.; Schoch, M.; Schoop, M.; et al. Unlocking the Power of Generative AI Models and Systems Such as GPT-4 and ChatGPT for Higher Education: A Guide for Students and Lecturers; Hohenheim Discussion Papers in Business, Economics and Social Sciences; Universität Hohenheim, Fakultät Wirtschafts- und Sozialwissenschaften: Stuttgart, Germany, 2023; Volume 2. [Google Scholar]
- Rasul, T.; Nair, S.; Kalendra, D.; Robin, M.; de Oliveira Santini, F.; Ladeira, W.J.; Sun, M.; Day, I.; Rather, R.A.; Heathcote, L. The role of ChatGPT in higher education: Benefits, challenges, and future research directions. J. Appl. Learn. 2023, 6, 41–56. [Google Scholar]
- Su, J.; Yang, W. Unlocking the power of ChatGPT: A framework for applying generative AI in education. ECNU Rev. Educ. 2023, 6, 20965311231168423. [Google Scholar]
- Chan, C.K.Y. A Comprehensive AI Policy Education Framework for University Teaching and Learning. arXiv 2023, arXiv:2305.00280. [Google Scholar] [CrossRef]
- Sajja, R.; Sermet, Y.; Cwiertny, D.; Demir, I. Platform-Independent and Curriculum-Oriented Intelligent Assistant for Higher Education. arXiv 2023, arXiv:2302.09294. [Google Scholar] [CrossRef]
- Kanwar, A.; Sanjeeva, M. Student satisfaction survey: A key for quality improvement in the higher education institution. J. Innov. Entrep. 2022, 11, 27. [Google Scholar] [CrossRef]
- Sáiz-Manzanares, M.C.; Marticorena-Sánchez, R.; Martín-Antón, L.J.; González Díez, I.; Almeida, L. Perceived satisfaction of University Students with the Use of Chatbots as a Tool for Self-regulated Learning. Heliyon 2023, 9, e12843. [Google Scholar] [CrossRef]
- Moldt, J.A.; Festl-Wietek, T.; Madany Mamlouk, A.; Nieselt, K.; Fuhl, W.; Herrmann-Werner, A. Chatbots for Future Docs: Exploring Medical Students’ Attitudes and Knowledge towards Artificial Intelligence and Medical Chatbots. Med. Educ. Online 2023, 28, 2182659. [Google Scholar] [CrossRef]
- Al-Emran, M.; AlQudah, A.A.; Abbasi, G.A.; Al-Sharafi, M.A.; Iranmanesh, M. Determinants of using AI-based Chatbots for knowledge sharing: Evidence from PLS-SEM and fuzzy sets (fsQCA). IEEE Trans. Eng. Manag. 2023. [Google Scholar] [CrossRef]
- Firat, M. What ChatGPT means for universities: Perceptions of scholars and students. J. Appl. Learn. Teach. 2023, 6, 57–63. [Google Scholar]
- Talan, T.; Kalinkara, Y. The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course. Uluslararası Yönetim Bilişim Sist. Ve Bilgi. Bilim. Derg. 2023, 7, 33–40. [Google Scholar] [CrossRef]
- Tenakwah, E.S.; Boadu, G.; Tenakwah, E.J.; Parzakonis, M.; Brady, M.; Kansiime, P.; Said, S.; Ayilu, R.; Radavoi, C.; Berman, A. Generative AI and Higher Education Assessments: A Competency-Based Analysis. Res. Sq. 2023. [Google Scholar] [CrossRef]
- Qureshi, B. Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv 2023, arXiv:2304.11214. [Google Scholar]
- Santos, R.P.D. Enhancing Chemistry Learning with ChatGPT and Bing Chat as Agents to Think With: A Comparative Case Study. arXiv 2023, arXiv:2305.11890. [Google Scholar] [CrossRef]
- Ibrahim, H.; Liu, F.; Asim, R.; Battu, B.; Benabderrahmane, S.; Alhafni, B.; Adnan, W.; Alhanai, T.; AlShebli, B.; Baghdadi, R.; et al. Perception, performance, and detectability of conversational artificial intelligence across 32 university courses. arXiv 2023, arXiv:2305.13934. [Google Scholar] [CrossRef] [PubMed]
- Okonkwo, C.W.; Ade-Ibijola, A. Chatbots applications in education: A systematic review. Comput. Educ. Artif. Intell. 2021, 2, 100033. [Google Scholar] [CrossRef]
- Olga, A.; Saini, A.; Zapata, G.; Searsmith, D.; Cope, B.; Kalantzis, M.; Castro, V.; Kourkoulou, T.; Jones, J.; da Silva, R.A.; et al. Generative AI: Implications and Applications for Education. arXiv 2023, arXiv:2305.07605. [Google Scholar]
- Venkatesh, V.; Morris, M.G.; Davis, G.B.; Davis, F.D. User acceptance of information technology: Toward a unified view. MIS Q. 2003, 27, 425–478. [Google Scholar] [CrossRef]
- Verhoeven, J.C.; Heerwegh, D.; De Wit, K. Information and communication technologies in the life of university freshmen: An analysis of change. Comput. Educ. 2010, 55, 53–66. [Google Scholar] [CrossRef]
- Chao, C.M. Factors determining the behavioral intention to use mobile learning: An application and extension of the UTAUT model. Front. Psychol. 2019, 10, 1652. [Google Scholar] [CrossRef] [PubMed]
- Ilieva, G.; Yankova, T.; Klisarova-Belcheva, S.; Dimitrov, A.; Bratkov, M.; Angelov, D. Impact of Conversational Chatbots on Learning of University Students. In Mendeley Data; Elsevier: Amsterdam, The Netherlands, 2023; Available online: https://data.mendeley.com/datasets/k8mkczry5h/1 (accessed on 6 September 2023).
- Harrer, S. Attention is not all you need: The complicated case of ethically using large language models in healthcare and medicine. eBioMedicine 2023, 90. [Google Scholar] [CrossRef] [PubMed]
- Santos, R.P.D. Enhancing Physics Learning with ChatGPT, Bing Chat, and Bard as Agents-to-Think-With: A Comparative Case Study. arXiv 2023, arXiv:2306.00724. [Google Scholar] [CrossRef]
- Wilson, L.O. Anderson and Krathwohl Bloom’s Taxonomy Revised Understanding the New Version of Bloom’s Taxonomy. J. Second Princ. 2016, 1, 1–8. [Google Scholar]
- Frieder, S.; Pinchetti, L.; Griffiths, R.R.; Salvatori, T.; Lukasiewicz, T.; Petersen, P.C.; Chevalier, A.; Berner, J. Mathematical capabilities of ChatGPT. arXiv 2023, arXiv:2301.13867. [Google Scholar]
Name | Functionality | Supported Platforms | Access Type |
---|---|---|---|
BERT | Question answering, text summarization, understanding user search intentions and the content indexed by the search engine | Cloud, On-premise | Source code |
XLNet | Question answering, sentiment analysis, search for relevant information in document bases or online | Cloud, On-premise | Source code |
ERNIE | Chinese language understanding, literary creation, business writing, mathematical calculations, multimodal output generation | Cloud, On-premise—for previous versions | Source code (for previous versions), API |
GPT-3 | Wide range of NLP tasks, including question answering, content generation, text summarization, text classification, information extraction | Cloud, On-premise | API |
PanGu | Wide range of NLP tasks, including natural language inference, common sense reasoning, reading comprehension, text classification | Cloud, On-premise | Source code (for previous versions), API |
Wu Dao | Generation of text and images, natural language processing and image recognition | Cloud, On-premise—for previous version | Source code (for the previous version), API |
LaMDA | Language translation, text summarizing, answering information-seeking questions | Cloud, On-premise—for the previous version | Source code (after approval), API |
YaLM | Different NLP tasks, including generating and processing text | Cloud, On-premise | Source code, API |
PaLM | Multiple difficult tasks: Language understanding and generation, reasoning, programming code generation | Cloud, On-premise—for the previous version | Source code, API (only for the last version) |
BLOOM | Different NLP tasks, including question answering, sentiment analysis, text classification | Cloud, On-premise | Source code, API |
GLM-130B | Different language understanding and language generation tasks | Cloud, On-premise | Source code, API |
LLaMA | AI developers interested in a powerful large language model | Cloud, On-premise | Source code, API |
GPT-4 | Can perform different NLP tasks, text generation, image processing and generation | Cloud | API |
Sample Description | No. of Students | Percentage (%) | |
---|---|---|---|
Question #1. Gender | Male | 22 | 16.8 |
Female | 109 | 83.2 | |
Question #2. Major | Economics | 103 | 78.6 |
Management | 17 | 13.0 | |
Other | 11 | 8.4 | |
Question #3. Educational level | Bachelor | 128 | 97.7 |
Master | 3 | 2.3 | |
Question #4. Frequency of chatbot usage | Never | 14 | 10.7 |
Rarely | 34 | 26.0 | |
Sometimes | 53 | 40.5 | |
Often | 24 | 18.3 | |
Always | 6 | 4.6 |
Chatbot | Remembering | Understanding | Applying | Analyzing | Evaluating | Creating | Total |
---|---|---|---|---|---|---|---|
ChatGPT | 4 | 4 | 4 | 4 | - | 4 | 20 |
Bard | 4 | 4 | 2 | 2 | - | 1 | 13 |
Alpaca-13B | 0 | 0 | 0 | 0 | - | 0 | 0 |
Vicuna-13B | 4 | 4 | 2 | 1 | - | 1 | 12 |
Vicuna-33B | 3 | 0 | 0 | 0 | - | 0 | 3 |
ChatGPTPlus | 4 | 4 | 4 | 4 | - | 4 | 20 |
Edge Chat | 4 | 4 | 1 | 2 | - | 4 | 15 |
Chatbot | Remembering | Understanding | Applying | Analyzing | Evaluating | Creating | Total |
---|---|---|---|---|---|---|---|
ChatGPT | 4 | 4 | 3 | 4 | - | 4 | 19 |
Bard | 2 | 2 | 2 | 2 | - | 1 | 9 |
Alpaca-13B | 0 | 2 | 0 | 0 | - | 0 | 2 |
Vicuna-13B | 0 | 0 | 0 | 0 | - | 0 | 0 |
Vicuna-33B | 0 | 2 | 2 | 2 | - | 0 | 6 |
ChatGPTPlus | 4 | 4 | 4 | 4 | - | 3 | 19 |
Edge Chat | 4 | 4 | 2 | 4 | - | 4 | 18 |
Chatbot | Remembering | Understanding | Applying | Analyzing | Evaluating | Creating | Total |
---|---|---|---|---|---|---|---|
ChatGPT | 2 | 2 | 2 | 3 | 3 | 1 | 13 |
Bard | 1 | 1 | 1 | 1 | 0 | 0 | 4 |
Alpaca-13B | 0 | 2 | 1 | 0 | 0 | 0 | 3 |
Vicuna-13B | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Vicuna-33B | 0 | 0 | 0 | 2 | 0 | 0 | 2 |
ChatGPTPlus | 4 | 2 | 3 | 4 | 4 | 4 | 21 |
Edge Chat | 1 | 1 | 1 | 1 | 0 | 0 | 4 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ilieva, G.; Yankova, T.; Klisarova-Belcheva, S.; Dimitrov, A.; Bratkov, M.; Angelov, D. Effects of Generative Chatbots in Higher Education. Information 2023, 14, 492. https://doi.org/10.3390/info14090492
Ilieva G, Yankova T, Klisarova-Belcheva S, Dimitrov A, Bratkov M, Angelov D. Effects of Generative Chatbots in Higher Education. Information. 2023; 14(9):492. https://doi.org/10.3390/info14090492
Chicago/Turabian StyleIlieva, Galina, Tania Yankova, Stanislava Klisarova-Belcheva, Angel Dimitrov, Marin Bratkov, and Delian Angelov. 2023. "Effects of Generative Chatbots in Higher Education" Information 14, no. 9: 492. https://doi.org/10.3390/info14090492