Insights Gained from Using AI to Produce Cases for Problem-Based Learning ^†

Abstract

Ulster University’s School of Medicine embraces a problem-based learning (PBL) approach, yet crafting scenarios for this method poses challenges, requiring collaboration among medical and academic experts who are often difficult to convene. This obstacle can compromise scenario quality and ultimately impede students’ learning experiences. To address this issue, the school trialed the use of AI technology to develop a case scenario focusing on headaches caused by cerebral haemorrhage. The process involved a dialogue between a single “author” and ChatGPT, with their outputs combined into a complete clinical case adhering to the school’s standard template. Six experienced PBL tutors conducted quality checks on the scenario. The tutors did not immediately endorse its use, recommending further enhancements. Suggestions included updating terminology, names, spelling, and protocols to align with current best practices, providing additional explanations such as interventions and improvements post-initial stability, incorporating real scans instead of descriptions, reviewing symptoms and timelines for realism, and addressing comprehension issues by refraining from directly providing answers and including probing questions instead. From this trial, several valuable lessons were learned: AI can assist a single author in crafting medical scenarios, easing the challenges of organizing expert teams. However, the author’s role shifts to reviewing and enhancing depth, guided by a template, with clinician input crucial for authenticity. ChatGPT respects patient data privacy and confidentiality by abstaining from providing scanned images, and while AI can generate discussion questions for tutorials, it may require modification to enhance specificity and provoke critical thought. Furthermore, AI can generate multiple-choice questions and compile reading resources to support self-directed learning. Overall, adopting AI technology can improve efficiency in the case-writing process.

1. Introduction

Ulster University’s School of Medicine employs a problem-based learning (PBL) approach, requiring medical scenarios to be current, interdisciplinary, and realistic. PBL is a pedagogical approach that enables students to learn while engaging actively with meaningful problems. Students are given the opportunities to problem-solve in a collaborative small group setting, create mental models for learning, and form self-directed learning habits through practice and reflection. Hence, the underpinning philosophy of PBL is that learning can be considered a “constructive, self-directed, collaborative and contextual” activity [1].

In a typical PBL setting, learning is triggered by a problem which needs resolution. Traditionally, creating these problems/scenarios involves collaboration among medical and academic experts, which can be challenging and affect the quality of the scenarios. To address this, AI technology was utilised to develop a PBL case on a clinical condition (headaches caused by cerebral haemorrhage). The process of creating a case involves a dialogue between “the author” and ChatGPT. The first prompt, as shown in Figure 1, was to request a full case to be used in a PBL tutorial to cover a 65-year-old woman who presents with a headache and a focal neurological deficit secondary to a haemorrhage in the left cerebellar hemisphere. The output was a short clinical case which included the patient presentation, signs, investigation results, patient progress, treatment, and rehabilitation plus probing questions.

The author then refined the prompt, as shown in Figure 2, to obtain more specific details regarding the expected physical examination findings, neurological findings, necessary investigations, expected investigation results, treatment options, and postoperative prognosis. The author also requested a clinical-based multiple-choice question and learning resources for one of the learning objectives as shown in Figure 3. The outputs from these questions were combined with the first output into a complete clinical case following the school’s standard template. The total amount of staff time was three to four hours.

Quality checks were then completed by a group of six experienced PBL tutors. The majority of the tutors did not endorse the case for immediate use but recommended further work. The chatbot produced a comprehensive output for each prompt, but some sections required additional details, such as “after initial stability”. It did not mention any interventions that led to improvement and stability. Additionally, the terminology and spelling need to be updated to align with current best practice standards, and the protocol should be revised based on relevant references. The output was well-organized and included discussion questions, although some were redundant as the answers were already present in the case. There were multiple-choice questions at the end to assess knowledge, along with suggested resources. The chatbot maintained patient privacy by not disclosing any investigation results or scan images, instead providing general descriptions. Tutors also noted that the symptoms and timeline should be reviewed to ensure the condition’s development appears realistic [2].

2. Practical Consideration

From this experience, we have identified several insights that could be valuable in creating a robust scenario using AI. These insights can guide the development of AI-generated scenarios, ensuring they are effective, accurate, and aligned with the intended educational goals as shown in Figure 4.

2.1. A Jumpstart: Prompt Development and Objectives

2.1.1. Prompt Development

Creating well-defined and precise prompts is crucial when utilising AI tools like ChatGPT. The clarity and specificity of a prompt help AI produce content that is both relevant and accurate, which is particularly important when developing PBL scenarios. Spending time refining prompts can greatly improve the quality of the outcomes and is transferable across case generation.

Customizing the initial prompt with straightforward language and industry-specific terms enhances AI’s ability to understand and respond within the intended context [3,4]. It’s also important to set the appropriate tone and level of formality depending on the audience and to include contextual hints that help guide the chatbot’s responses [5]. Ongoing refinement of the prompt, based on user feedback and interactions, can further improve AI’s accuracy and understanding [6].

Some studies have employed the technique of asking AI to take on specific roles, such as “You are a developer of teaching materials…” or “You are developing a question bank for medical exams…” [7,8,9].

2.1.2. Predefined Objectives

From our experience, we found that establishing clear learning objectives before creating problem-based learning scenarios is vital. Savin-Baden and Major [10] stress that having well-defined objectives helps ensure that the scenario is not only relevant but also appropriately challenging and aligned with the broader curriculum goals. This approach is key to making sure the case scenario effectively supports the intended learning outcomes, including the development of critical thinking and problem-solving skills.

2.2. Structure Boost: Augmenting Your Case Study Framework

The latest version, GPT-4, surpasses GPT-3.5 by providing more advanced reasoning, enhanced text processing capabilities, and improved image analysis. Additionally, it demonstrates a level of “creativity” that was less apparent in earlier versions. GPT-4 is OpenAI’s most advanced system to date, designed to generate safer and more useful responses [11].

Harel, Katz, Marron, and Szekely [12] explore how generative AI can enhance scenario-based modelling, specifically models like GPT-4. They argue that integrating generative AI into scenario-based modelling processes can significantly improve the efficiency and creativity of model generation. AI’s advanced reasoning and text processing capabilities allow it to suggest novel scenarios, identify potential issues, and even automate parts of the modelling process, making it a powerful tool for developers and researchers working in complex systems. This research highlights the potential of AI to augment traditional modelling techniques, leading to more innovative and effective outcomes enhancing the scenario structure can be achieved through several key steps, each contributing to a more robust and effective outcome.

2.2.1. Use of Template

Utilising a standard template for creating PBL scenarios can streamline the development process. Templates provide a consistent structure, ensuring that all necessary elements are included and making it easier to integrate content generated by AI into the final scenario.

2.2.2. Specify the Working Protocol/Region

It’s essential to clearly define the medical protocols and regional guidelines that are relevant to a scenario. Doing so ensures that the case aligns with the specific practices and standards of the region in which it will be applied, which in turn enhances both its realism and relevance.

In the context of algorithmic ethics, validation and evaluation are indispensable. They enable both researchers and practitioners to evaluate the performance, accuracy, and dependability of AI algorithms. Healthcare professionals should rigorously assess AI-generated recommendations for their accuracy, reliability, and effectiveness, ensuring they align with established clinical guidelines and best practices. This evaluation process is critical for maintaining fairness, transparency, and accountability in AI-driven decision-making, ultimately leading to better patient care and outcomes [13,14].

2.2.3. Add Sources for Scans/Radiological Input

ChatGPT respects patient data privacy and confidentiality; it does not furnish scanned images. However, including scans and radiological inputs adds depth and authenticity to the scenarios [15]. To respect AI’s privacy and confidentiality terms, it might be helpful to provide a clear description of the expected results. The author’s task is then to find a perfect match from other sources.

Providing actual investigation data from real-world cases will make the scenarios more realistic, allowing students to engage more deeply and practice interpreting and applying information in a manner that closely mirrors clinical practice. This approach will not only improve the learning experience but also help in developing critical thinking and decision-making skills that are vital for their future professional roles [16].

2.3. Perfect Finish: Quality Assurance and Refinement

Our concerns about the effectiveness of ChatGPT in healthcare centre around two primary issues: accuracy and limitations [17]. Accuracy pertains to ChatGPT’s ability to produce correct and reliable information in healthcare-related tasks. On the other hand, limitations refer to the boundaries and shortcomings of ChatGPT’s capabilities, including potential biases, a lack of contextual understanding, and challenges in handling complex medical scenarios.

First, ChatGPT may generate responses that are logically coherent but incorrect due to its inability to consciously evaluate the accuracy of its output [18]. For example, errors have been observed in the generation of discharge summaries and radiology reports [19]. As such, clinicians should be cautious and avoid overreliance on ChatGPT, ensuring they select clinically appropriate information.

Second, the nature of ChatGPT’s training data introduces concerns about delays and incompleteness, affecting its ability to provide up-to-date and comprehensive insights into the latest medical and professional research [20]. Addressing this issue requires specific training and continuous updates tailored to clinical practice.

Validity in this context encompasses the accuracy, reliability, and appropriateness of the information provided by ChatGPT. Given that the content generated by ChatGPT can have a direct impact on patient health and well-being, it is vital to prioritise accuracy and reliability to prevent potential harm or misinformation [15]. Achieving comprehensive validation of ChatGPT’s output requires meticulous annotation of large datasets by human experts, resulting in truly valuable and reliable data. However, the aggregation of text within ChatGPT and the inaccessibility of source information complicate the process of querying and validating responses [21]. Consequently, the manual validation required for ChatGPT is extremely time-consuming and resource-intensive.

Despite some limited validation efforts, ChatGPT still necessitates further error correction [21]. Currently, the references generated by the bot have not undergone extensive validation, leaving users to rely on their judgment to determine the accuracy of the content [15]. This reliance on subjective assessment poses a significant risk of adverse consequences.

2.3.1. Use of References

When creating a high-quality case study, it is crucial to adhere to specific guidelines for listing references. To reduce bias and minimise hallucination while using AI, Alkaissi and McFarlane [22] proposed methodology involves supplying a reliable reference for the AI to generate cases, rather than relying entirely on its internal database. By following these guidelines, the case study will be well-supported, credible, and useful to its intended audience. References should be carefully selected, ensuring they have been read, understood, and directly support the case study. It is important to limit the number of references to approximately 15 unless there is a clear justification for including more, as outlined by Budgell [23].

2.3.2. Review Panel

The volume of information generated daily by users on social media platforms, news agencies, and the web is rapidly increasing. Consequently, organizations responsible for fact-checking are overwhelmed by the sheer amount of material requiring verification. The existing processes and pipelines used by these organizations are inadequate for handling the current trend, as they are not designed to scale up to the massive influx of information needing scrutiny. The severity of the situation led the World Health Organization (WHO) director-general to coin the term “infodemic” during the 2020 Munich Security Conference to describe the problem of misinformation amid the ongoing COVID-19 pandemic [24].

A crucial aspect of maintaining high standards in PBL scenarios is the involvement of a panel of experienced tutors in the review process. This step ensures the identification and correction of any inaccuracies or gaps in the content, ultimately enhancing the quality and educational value of the final product. Fact-checking, one of the most studied methods for combating misinformation, involves verifying the truthfulness and accuracy of information. Although it has proven effective in debunking fake news, the labour-intensive nature of fact-checking presents challenges in scaling [25]. Additionally, there are concerns about how adaptable and agile fact-checking can be, whether performed by humans or algorithms, in responding to the rapidly changing misinformation landscape [26].

2.3.3. Do Not Underestimate the Importance of Interdisciplinary Collaboration

To ensure ChatGPT is used safely and effectively in healthcare, it must be trained on extensive datasets that are annotated by experts from multiple disciplines and validated by physicians. This comprehensive validation process can improve the reliability of ChatGPT’s responses, which in turn benefits patient care. Choo et al. [27] reported that even in complex cases requiring multidisciplinary team (MDT) discussions, ChatGPT’s diagnostic and therapeutic suggestions have been consistent with expert recommendations. Researchers propose that ChatGPT could analyze medical records, literature, and current clinical guidelines to generate summary texts. These summaries can be reviewed and used by experienced clinicians to aid in clinical decision-making, provide personalized treatment [28,29] and write clinical cases.

2.4. AI-Specific Consideration

2.4.1. Remember That ChatGPT Is Not a Human

Alan Turing famously posed the question, “Can machines think?” in his seminal paper, Computing Machinery and Intelligence [30,31]. Turing argued that to answer this question, we must first define what thinking entails—a challenging task since thinking is a subjective behaviour. To address this, Turing introduced the Turing test, an indirect method to determine whether a machine can demonstrate intelligence indistinguishable from that of a human. A machine that passes this test qualifies as AI.

It’s crucial to remember that while ChatGPT is a powerful tool, it is not a human expert. It generates content based on patterns in the data it has been trained on and may lack the nuanced understanding that a human professional possesses. Therefore, always cross-check AI-generated content for accuracy and relevance [32].

2.4.2. Avoid False Confidence

Users should exercise caution when relying on AI-generated content without verification. AI outputs can sometimes appear highly confident but may contain errors. It is essential to maintain a critical perspective and cross-check information with trusted sources [33].

AI techniques have been promoted as scalable solutions for detecting misinformation, but they are not foolproof and can also be used to generate it. Large Language Models (LLMs) are capable of producing human-like text that sometimes appears more credible than human-written content. When LLMs are used to create misinformation, they can amplify fringe viewpoints by rapidly producing large volumes of misleading text, creating a false sense of consensus. This capability provides malicious actors with new methods to spread false narratives, leading to unprecedented public confusion [34].

For example, Meta had to remove Galactica, an LLM intended for academic purposes, shortly after its release due to its generation of biased and incorrect information. Similarly, ChatGPT, despite its popularity, has faced criticism for producing biased or false outputs. There is still limited research on the differences between AI-generated and human-created misinformation and the effectiveness of existing solutions in countering AI-generated misinformation [35].

A study, analyzing AI-generated misinformation using text and rapid qualitative analysis, found that AI-generated misinformation is linguistically distinct, more emotional and detailed than human-created misinformation. Existing detection models and assessment guidelines are less effective against AI-generated content because they often appear more credible, transparent, and comprehensive [34].

2.4.3. Train Your Chatbot

Continually training and updating AI with new data and feedback can significantly enhance its performance. By tailoring the chatbot to specific needs and regularly refining its responses, more accurate and relevant content can be produced over time.

Training a chatbot involves teaching it to engage in natural and meaningful conversations, much like how humans learn languages through exposure to conversations and texts. The process is similar to a language learner immersing themselves in native conversations. A chatbot learns by analyzing large amounts of text data, such as interaction transcripts, customer service tickets, and product descriptions. By exposing the chatbot to diverse language patterns and scenarios, it begins to understand the nuances of human communication. Machine learning algorithms help the chatbot make sense of this data, identifying patterns and relationships between words and phrases. Over time, the chatbot’s language understanding becomes more sophisticated, enabling it to respond to user queries accurately and helpfully [35]. The training process, where the chatbot learns to recognize input patterns and generate appropriate responses, is the most time-consuming phase. Maintaining and improving the chatbot is an ongoing responsibility; regularly refining its training data and updating the model ensures that it remains relevant and can adapt to new contexts. Additionally, incorporating user feedback and monitoring key metrics like accuracy, precision, and recall is vital for further optimization [36].

2.4.4. Supplementary Tools: Other Plugins: Diagram, Picture, Video

Incorporating additional multimedia elements like diagrams, pictures, and videos into PBL scenarios can greatly enhance the learning experience. These tools serve as visual aids, helping to clarify complex concepts and making the educational process more engaging and effective.

ChatGPT plugins are third-party tools created to enhance the functionality of ChatGPT. These plugins, developed by individuals with coding expertise, can perform diverse tasks, ranging from simplifying processes to expanding the chatbot’s knowledge. They offer a flexible way to boost ChatGPT’s capabilities and address common challenges of large language models, such as reducing “hallucinations,” staying updated on recent events, and accessing proprietary information (with permission). By connecting to external data, these plugins enable models to provide more accurate, evidence-based responses [37,38].

3. Implications

As the use of advanced technologies continues to grow, medical educators are increasingly voicing their concerns about the potential risks involved. A recent report [39], from a physician defense organization, highlighted the dangers of students relying on these technologies for their assignments. Other literature has raised alarms about the possibility of these innovations compromising the integrity of academic publishing [40,41]. Conversely, there is also enthusiasm among educators, as significant studies have showcased the potential of these technologies to improve assessment practices [42] and streamline processes such as admissions [43]. Traditionally, developing Virtual Patient Scenarios (VPSs) and Intelligent Tutoring Systems (ITSs) has been a labor-intensive task requiring the expertise of content specialists. However, recent studies [44,45] have demonstrated the innovative application of natural language processing to automate the creation of extensive cases. This automation has significantly broadened the range of cases available to learners, effectively addressing common challenges in teaching and evaluating clinical reasoning, such as case variability and the unpredictable nature of real-life clinical encounters.

The study by Gordon et al. [46] reviewed 278 publications and revealed that a significant portion (68%) originated from North America and Europe, covering a range of AI applications in medical education, including admissions, teaching, assessment, and clinical reasoning. The review emphasized the diverse roles of AI, from supplementing traditional educational approaches to introducing novel practices, while also stressing the pressing need for ethical guidelines governing AI’s application in this context. It also highlighted the considerable potential of AI-generated patient cases to enhance medical education across various fields, fostering more dynamic and interactive learning experiences that align with the complexities of modern healthcare. Even more, it recommended incorporating real-life case studies into the medical curriculum as a recommended strategy for teaching AI ethics to illuminate the multifaceted ethical challenges posed by health-related AI, such as informed consent, bias, and transparency. This case-based approach enhances not only theoretical understanding but also develops practical decision-making skills. One way to go about it could be the adoption of current case scenarios from clinical practice, involving AI applications, and requesting that students critically assess both technical and ethical implications for AI. This helps bring forth not only critical reflection but also allows interdisciplinary dialogue—the certain prerequisite for appropriate application of AI in patient-centered care.

Despite the drawbacks of AI in medical education, various benefits emanate from the use of AI-generated cases. Another important implication is the incorporation of interactive lectures regarding new developments in patient care, new medications, advances in technology, and recently revised guidelines, among others. This foundational approach lays the ground for students as they get into the in-depth exploration of AI-generated cases. Instructors might walk students through some of the more complicated management challenges to manage comorbidities and adjust insulin regimens, for example by asking them to work in breakout groups to debate detailed AI-generated patient cases. This collaborative problem-solving format enhances critical thinking and comprehension, leading to a more robust understanding of the clinical scenario. As noted by Masters [47], these development initiatives have provided a conceptual and pragmatic framework for the integration of AI in medical education.

The roleplays allow participants to practice different roles in the simulated consultations generated in these AI scenarios, thus enhancing their communication skills and strategies for offering patient-centred care. This is then followed by reflective practice after such interactions. Facilitated reflection on case discussions allows participants to compare their approaches to established best practices and guidelines, hence encouraging the translation of new knowledge into clinical practice. It is in this regard that guided or journaling reflections enhance this reflective process for the students to make their experience more internalized and applicable [48].

Moreover, presenting teams with complex AI-generated patient cases during team-based learning (TBL) encourages the application of their knowledge in diagnosing conditions, formulating management plans, and predicting patient outcomes. These cases are designed to mimic real-life clinical scenarios, cultivating critical thinking and decision-making skills essential for future practice [49]. In high-fidelity simulation labs, AI-generated cases provide a diverse range of clinical scenarios, particularly in emergency medicine training, allowing students to hone their responses to critical, time-sensitive situations in a controlled environment [50].

AI-generated cases also facilitate interprofessional education (IPE) by promoting collaborative learning among students from various healthcare disciplines, such as medicine, nursing, pharmacy, and social work. By presenting cases that necessitate a multidisciplinary approach, these scenarios foster teamwork and a greater understanding of each discipline’s roles in patient care [51]. Furthermore, as telehealth becomes increasingly prevalent, AI-generated cases provide valuable training for healthcare providers, equipping them with the necessary skills for effective remote patient assessment and management, including accurate diagnosis and treatment planning in virtual consultations [52].

Finally, the integration of adaptive learning platforms powered by AI can tailor the pace, content, and difficulty level of instruction to accommodate individual student progress, fostering a more personalized educational experience. Overall, the versatility of AI-generated patient cases presents a transformative opportunity for enhancing medical education, ultimately preparing future healthcare professionals to navigate the complexities of modern patient care [53].

4. Ethical Issues in Artificial Intelligence

While this powerful technology has the potential to enhance patient care and medical education, it also raises significant concerns regarding patient preferences, safety, and privacy, introducing a new array of ethical dilemmas that require careful examination and resolution. Gerke et al. [54] highlight four critical ethical challenges associated with AI in healthcare: informed consent for its use, safety and transparency, algorithmic fairness and biases, and data privacy. Furthermore, a scoping review has emphasized the need for ethics education in several key areas, including algorithmic bias and equity, resource allocation in healthcare, safety and quality assurance, human interaction and compassionate care, data privacy and security, automation bias and skill preservation, and the importance of transparency and informed consent.

Recent concerns regarding the ethical use of AI in Health Professions Education (HPE) highlight a significant gap in guidance for educators on how to implement AI ethically in their teaching practices [55]. This necessitates a greater awareness of the complex ethical considerations involved, such as data collection, privacy, consent, data ownership, security, algorithmic bias, transparency, responsibility, autonomy, and beneficence.

As AI technology continues to evolve toward what has been termed a “singularity”—where machines may possess intelligence surpassing that of humans [56]—it becomes increasingly crucial to recognize that many educational decisions are fundamentally ethical. These ethical standards, being human constructs, can vary widely across cultures and over time, with few universally accepted rights among the global population. Given that ethical reasoning is inherently flawed, and AI may eventually identify shortcomings in these models, it is conceivable that AI could develop its own ethical frameworks. This development suggests a future where AI’s perspectives on ethics may challenge traditional human views, compelling educators to prepare for an educational landscape shaped by AI’s evolving ethical standards [57].

5. Future Research

Future research could build on these findings by exploring additional dimensions not covered in this study. While this work does not directly compare AI-assisted case development with more traditional approaches, it raises important questions about the specific benefits and challenges of incorporating AI into educational practices. Future investigations could focus on comparing outcomes from AI-generated content and conventional methods, helping to identify any unique strengths or limitations that emerged in this initial exploration. This comparative approach would provide deeper insights into how AI can complement or enhance traditional educational techniques.