Identifying Features of LLM-Resistant Exam Questions: Insights from Artificial Intelligence (AI)–Student Performance Comparisons

Abstract

Large language models (LLMs) are rapidly being explored as tools to support learning and assessment in health science education, yet their performance across discipline-specific evaluations remains underexamined. This study evaluated the accuracy of two prominent LLMs on university-level pharmacognosy examinations and compared their performance to that of pharmacy students. Authentic exam papers comprising a range of question formats and content categories were administered to ChatGPT and DeepSeek using a structured prompting approach. Student data were anonymized and LLM responses were graded using the same marking criteria applied to student cohorts, and a Monte Carlo simulation was conducted to determine whether observed performance differences were statistically meaningful. Facility Index (FI) values were calculated to contextualize item difficulty and identify where LLM performance aligned or diverged from student outcomes. The models demonstrated variable accuracy across question types, with a stronger performance in recall-based and definition-style items and comparatively weaker outputs for applied or interpretive questions. Simulated comparisons showed that LLM performance did not uniformly exceed or fall below that of students, indicating dimension-specific strengths and constraints. These findings suggest that while LLM-resistant examination design is contingent on question structure and content, further research should refine their integration into pharmacy education.

1. Introduction

The rapid integration of generative AI and computer vision continues to transform higher education, posing new opportunities and challenges for educators and students in various fields. Pharmacy education has not remained unaffected [1]. While such tools have proved beneficial in exam generation [2,3] and grading [4], their accessibility and potential for creative misuse [5] and the generally increasing acceptance [6] and higher student confidence [7] in AI place emphasis on large language model-resistant designs in the context of the academic integrity of student assessments [8]. Several studies have already investigated the performance of LLMs in an examination setting [9], including in pharmacy exams [10] and assignments [11]. Research into constructing LLM-resistant exams suggests approaches such as leveraging real-world scenarios outside of the available training data, non-textual elements and deliberate distractors to counteract AI model weaknesses [12]; however, this can be difficult to achieve in empirical and applied sciences, especially when utilizing an evidence-based teaching approach, which is reliant on open access to published data [13]. ChatGPT models are reported as being preferred and frequently used by students [14]; however, the landscape of leveraging particular LLMs in academic pursuits is likely to continuously shift as new generative AI emerge and already-existing models improve. DeepSeek is one such model which has already demonstrated adequate performance in various fields, including Gastroenterology [15]. Computer vision like Google Lens appears to be utilized primarily for the identification of botanical subjects [16,17]. Pl@ntNet and other more specialized computer vision tools show variance in performance and, in some cases, high accuracy for macroscopic images [18] and are a form of participatory botanical observation. Materials used in pharmacognostic study, however, are morphologically distinct from typically utilized training data.

Another important consideration is that pharmacy education requires the development of various competencies in diverse, interconnected, specialized domains [19]. These competencies are not perceived as being equally important by students [20], educators and practitioners [21]. Thus, it seems prudent to investigate how LLMs perform in answering questions associated with the assessment of the development of highly ranked competencies. Competency in pharmacognosy is ranked >60% by community pharmacists [21], as the initial scope of the science has expanded past botanically describing subjects and evolved to include a much more in-depth exploration of the quality, safety and efficacy of herbal substances, preparations and medicinal and borderline products [22]. These include but are not limited to knowledge of traditional medicine systems, evidence-based traditional and well-established use, clinically relevant product interactions, contemporary bioprospecting and drug discovery and substance quality control methods [23]. This posits pharmacognosy-related competencies as crucial for pharmacy practitioners and inherently interdisciplinary, but raises questions about the potential long-term risks of student academic misconduct during assessments. Furthermore, some natural resources, products and traditional practices are region specific [24]. This may influence the availability of the published open access data LLMs and computer vision algorithms are trained on. No less important is the propensity of generative AI to hallucinate or put forth biased or false responses [25], which can be further exacerbated by existing knowledge gaps [26]. So far, no studies have been carried out with the express objective of evaluating LLMs and CV performance in pharmacognosy examinations, nor have they compared their performance to that of students in terms of cognitive dimensions.

The utilization of online learning platforms like Moodle [27] further compounds the issue, by providing the opportunity for digital assessment of students’ skills without the direct oversight of an examiner. Moodle provides functionality for examination design with integrated tools. These include, but are not limited to, different question types (such as multiple choice, true or false, cloze, essay, etc.), the ability to group questions into categories, randomize questions selected for each student per category in a single exam and shuffle question order in an exam [28]. While some publications suggest that LLMs perform well with certain multiple-choice questions [29], others disprove their effectiveness, particularly for medical questions [30]. Similarly, the effects of question type on LLMs’ performance in pharmacognosy examinations have not been explored. In addition to exam design functionality, Moodle has built-in quiz report statistics. These are intended to help examiners determine how well the quiz is helping students and identify faulty or overly difficult or easy questions [31]. The predictive power of metrics derived from student attempts like the facility index (the mean score of students on the question or category) or other psychometrics such as the discrimination index (the correlation between the weighted scores on the question and those on the rest of the examination) [32] on LLMs’ performance has not yet been explored. Therefore, this benchmark study seeks to assess the performance of ChatGPT-4o, DeepSeek, Google Lens and Pl@ntNet on pharmacognosy examinations conducted on the Moodle e-learning platform of Sofia University “St. Kliment Ohrdiski” and compare it to that of students, in the context of competencies, cognitive dimensions and question types, which are variable features of each examination, thus highlighting what leads to simultaneous good student and poor LLM performance, i.e., what makes an examination LLM-resistant.

2. Materials and Methods

Pharmacognosy in the Master’s Program in Pharmacy at Sofia University “St. Kliement Ohrdiski”, Faculty of Chemistry and Pharmacy is taught in two consecutive courses (Pharmacognosy 1 and 2). Each course includes three online examinations in Moodle, performed during the semester prior to the final exam. These examinations cover portions of the course material and theoretical concepts required for practicums, including reading, interpreting and application of experimental results. Study materials prepared by faculty members are not publicly available; however, the course content is based on the publicly available published scientific literature [33,34]. The Pharmacognosy 1 course introduces students to the foundations of this science and continues with the exploration of natural products from the perspective of their biologically active constituents and therapeutic applications. Students are taught how to harvest, identify and analyze plant materials, how to adhere to relevant regulatory standards and utilize these sources in an efficacious and safe matter. Further examples of the topics covered in the course are provided in Appendix A,

Table A1. Topics covered in the Pharmacognosy 1 course.

Thematic Area	Topics Covered	Key Concepts/Examples
Foundations of Pharmacognosy	Nature, objectives, tasks	Basic concepts and approaches
	Historical development	Ancient sources and contemporary practices
	Medicinal plants and herbal substances: concepts, classification, nomenclature	Taxonomy Herbal substances nomenclature
	Interdisciplinary role of pharmacognosy across sciences	Connections to pharmacology, pharmacotherapy, technology of pharmaceutical forms and legislation
Discovery, Sources and Products of Natural Origin	Modern approaches to medicinal plant discovery	Ethnopharmacology, ethnobotany, phylogenetics, chemotaxonomy
	Products of natural origin	Herbal substances Preparations (teas, oils, fats and extracts)
Plant Material Handling	Collection, processing, storage, cultivation	Good Agricultura and Collection Practices (GACP)
	Wild plants and biodiversity as sources	Distribution of natural resources Bioaccumulation Cultivation
	Pharmacognostic study design	Planning and key considerations
	Types of preparations and extraction techniques	Specific requirements in overview of constituents
Standards and Regulations	European Pharmacopeia	Structure Quality control methods Classification of preparations Reference standards; marker compounds
	Medicine agencies	Bulgarian Drug Agency European Medicines Agency—Herbal Medicinal Products Committee Definitions Declaration of extracts
Analytical and Diagnostic Methods	Macroscopic, microscopic, pharmacognostic analyses	Morphological and anatomical features Staining and observation techniques
	Physiochemical pharmacognostic analyses	Loss on drying Ash content Swelling index
Herbal Medicines	Definitions, classification, safety and efficacy	Traditional and well-established use Combination herbal medicinal products (species) Use in sensitive populations Therapeutic indications, posology, counterindications, period of use.
Biologically Active Compounds	Isolation and analysis of bioactive compounds	Methods and screening for activity
	Primary and secondary metabolites, biosynthetic pathways	Shikimate, mevalonate, malonate, etc.
	Chemical groups of natural compounds	Carbohydrates Lipids and lipoids Phenols Flavonoids

.

Each examination in Pharmacognosy 1 quizzes students on a different number of topics and comprises a different total number of categories, structured in thematic sections. These categories contain variations of what is, in essence, the same question in terms of the cognitive domain and competency, but may differ in question type and general layout of text or images, or specific example data required to solve the question. For student attempts, only one question is drawn at random from each category during an examination. Thematic section order is fixed, but category order inside thematic sections is randomized for each student attempt. Individual students can attempt the examination only once. Navigation is consecutive, and sections appear on separate pages. Each question is scored using a point system. Points are awarded for each correct answer. The total achievable score in an examination from all questions ranges from 0.00 to 5.00 points. The points from the three examinations constitute 9.09% of the course grade. The complete evaluation scheme for Pharmacognosy 1 is available in Appendix A,

Table A2. Pharmacognosy 1, course evaluation scheme. The total number of points available for the course is 165. A minimum of 83 points is required for a passing grade.

Quiz, Examination or Exam	Points	Points [%]
Taxonomy and nomenclature quiz	18	10.91
Examination 1	5	3.03
Examination 2	5	3.03
Examination 3	5	3.03
Practical exam	33	20.00
Final exam	99	60.00

.

Our first step was structuring the dataset by exporting recorded students’ results for each question, associated question type and question statistics per examination and then deleting all student personal data. This dataset included a record of all scores and psychometrics per question and per category in a table format, as generated by Moodle. Following this step, each question was labeled with the corresponding competency per the questionnaire developed by Atkinson, J. et al., 2016 [21] and cognitive dimension by Bloom’s classification [35] it assesses. The examination structure is elaborated in Appendix A,

Table A3. Thematic section and category structure of Pharmacognosy 1, Examination 1.

Thematic Section	Categories	Point Value	Questions	Question Type	Facility Index [%]	Discrimination Index [%]	Cognitive Dimension	Competencies
European pharmacopeia	C1. General notices	0.2	1.1	Cloze (MC; FIB)	81	33	Remember	7.7 Ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 10.37 legislation and professional ethics;
			1.2		76
			1.3		74
	C2. Ph. Eur. structure	0.4	2.1	Cloze (MC)	72	10	Apply, analyze	7.7 Ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 9.21 ability to communicate in English and/or locally relevant languages; 10.37 legislation and professional ethics; 11.39 current knowledge of good manufacturing practice (GMP) and of good laboratory practice (GLP)
			2.2		74
			2.3		56
			2.4		74
	C3. Monograph structure	0.4	3.1	Cloze (MC)	64	11	Remember, analyze	7.7 Ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 10.37 legislation and professional ethics; 11.39 current knowledge of good manufacturing practice (GMP) and of good laboratory practice (GLP)
			3.2		70
			3.3		63
Pharmacognosy—basic concepts	C4. Basic concepts	0.1	4.1	Cloze (MC; FIB)	79	59	Remember, apply	9.21 Ability to communicate in English and/or locally relevant languages
			4.2		72
			4.3		60
	C5. Chemotaxonomy	0.4	5.1	DD	86	28	Remember, apply	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology; 10.27 organic and medicinal/pharmaceutical chemistry
			5.2		80
			5.3		73
	C6. Morphological plant parts	0.5	6.1	Cloze (MC)	77	43	Remember, analyze	10.24 Plant and animal biology; 11.39 current knowledge of good manufacturing practice (GMP) and of good laboratory practice (GLP)
			6.2		82
			6.3		66
			6.4		70
	C7. Primary processing and GACP	0.1	7.1	MC	91	46	Remember, understand	11.39 Current knowledge of good manufacturing practice (GMP) and of good laboratory practice (GLP)
			7.2		91
			7.3		92
			7.4		71
HMPC|EMA—recommendations	C8. Case studies	0.5	8.1	Cloze (MC)	58	25	Remember, understand, apply, analyze,	9.21 Ability to communicate in English and/or locally relevant languages; 10.30 anatomy and physiology; medical terminology; 10.33 pharmacotherapy and pharmaco-epidemiology; 13.46 retrieval and interpretation of relevant information on the patient’s clinical background; 17.63 provision of informed support for patients in selection and use of non-prescription medicines for minor ailments (e.g., cough remedies …)
			8.2		91		remember, understand, apply
	C9. Therapeutic area, indication, patient population, posology	0.5	9.1	Cloze (MC)	75	22	Remember, apply	10.33 Pharmacotherapy and pharmaco-epidemiology; 17.63 provision of informed support for patients in selection and use of non-prescription medicines for minor ailments (e.g., cough remedies…)
			9.2		50
			9.3		69
Macroscopic analysis	C10. Identification	0.7	10.1	Cloze (MC)	62	25	Remember, analyze	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			10.2		33
			10.3		55
	C11. Adulteration	0.2	11.1	(MC)	81	48	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			11.2		84
			11.3		56
Microscopic analysis	C12. Methodology	0.2	12.1	Cloze (FIB)	59	44	Remember, understand	7.6 Ability to design and conduct research using appropriate methodology; 7.7 ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 9.21 ability to communicate in English and/or locally relevant languages; 10.37 legislation and professional ethics
			12.2		45
			12.3		59
	C13. Diagnostic characteristics	0.1	13.1	Cloze (MC)	83	14	Remember, apply, analyze	10.24 Plant and animal biology
			13.2		60
			13.3		76
			13.4		65
	C14. Differential staining	0.3	14.1	Cloze (MC)	72	29	Remember, understand, apply, analyze	10.24 Plant and animal biology; 10.27 organic and medicinal/pharmaceutical chemistry; 10.29 general and applied biochemistry (medicinal and clinical)
			14.2		71
			14.3		70
			14.4		52
	C15. Quality standards and adulteration	0.4	15.1	Cloze (FIB)	46	25	Remember, understand, apply, analyze, evaluate	10.24 Plant and animal biology; 10.37 legislation and professional ethics
			15.2		39
			15.3		51

,

Table A4. Thematic section and category structure of Pharmacognosy 1, Examination 2.

Thematic Section	Categories	Point Value	Questions	Question Type	Facility Index [%]	Discrimination Index [%]	Cognitive Dimension	Competencies
Traditional herbal medicinal products	C1. Justification of traditional use requirements	0.2	1.1	FIB	88	30	Remember	7.7 Ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 10.37 legislation and professional ethics.
			1.2		93
	C2. Therapeutic indications and use	0.4	2.1	DD	76	42	Remember, apply	17.63 Provision of informed support for patients in selection and use of non-prescription medicines for minor ailments (e.g., cough remedies…)
			2.2		65
			2.3		73
			2.4		68
Extraction processes	C3. Preliminary processing of plant materials	0.1	3.1	TF	100	15	Remember, understand	7.6 Ability to design and conduct research using appropriate methodology
			3.2		38
			3.3		83
			3.4		85
	C4. Extraction types	0.2	4.1	DD	59	10	Remember, understand	10.38 Current knowledge of design, synthesis, isolation, characterization and biological evaluation of active substances
			4.2	MC	86
			4.3		80
			4.4	FIB	79
			4.5		91
			4.6	DD	78
			4.7		78
	C5. Solvents	0.2	5.1	DD	98	27	Remember, understand	10.27 Organic and medicinal/pharmaceutical chemistry
			5.2		68
			5.3		48
			5.4		54
			5.5		75
Metabolic pathways	C6. Primary and secondary metabolites	0.2	6.1	FIB	82	8	Remember	10.29 General and applied biochemistry (medicinal and clinical)
			6.2		87
			6.3		97
	C7. Biosynthetic pathways	0.1	7.1	Cloze (MC)	65	20	Remember	10.29 General and applied biochemistry (medicinal and clinical)
			7.2		81
			7.3		56
			7.4		61
Macroscopic analysis	C8. Identification	0.3	8.1	DD	64	13	Remember, analyze	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			8.2		92
			8.3		77
			8.4		67
			8.5		81
	C9. Diagnostic characteristics	0.3	9.1	MC	77	22	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			9.2		75
			9.3		63
			9.4		97
			9.5		72
			9.6		82
Microscopic analysis	C10. Diagnostic characteristics	0.3	10.1	DD	83	41	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			10.2		83
			10.3		63
			10.4		97
			10.5	FIB	75
			10.6		63
			10.7		84
			10.8		81
			10.9		88
	C11. Micromorphology	0.3	11.1	DD	100	47	Remember	10.24 Plant and animal biology
			11.2		85
			11.3		45
			11.4		77
			11.5		60
			11.6		78
	C12. Identification	0.3	12.1	MC	78	6	Remember, analyze	10.24 Plant and animal biology
			12.2		60
			12.3	FIB	80
			12.4	MC	55
			12.5		67
	C13. Microchemical reactions	0.3	13.1	MC	75	15	Remember	10.28 Analytical chemistry
			13.2		100
			13.3	TF	71
			13.4		43
			13.5		50
			13.6		17
			13.7	DD	92
			13.8		64
	C14. Starches	0.3	14.1	MC	50	17	Remember, analyze	10.24 Plant and animal biology; 10.37 legislation and professional ethics
			14.2		85
			14.3		77
			14.4		25
			14.5		93
Pharmacognostic analyses	C15. Loss on Drying and Ash—general concepts	0.3	15.1	DD	93	9	Remember, apply	10.37 Legislation and professional ethics
			15.2		93
			15.3	FIB	80
			15.4		93
			15.5	DD	79
			15.6		55
			15.7		96
	C16. Loss on Drying and calculation	0.3	16.1	MC	74	6	Remember, apply	10.37 Legislation and professional ethics
			16.2		67
			16.3		84
			16.4		79
	C17. Ash—calculation	0.3	17.1	DD	91	41	Remember, apply	10.37 Legislation and professional ethics
			17.2		78
			17.3	MC	80
			17.4		89
	C18. Swelling index	0.3	18.1	DD	87	26	Remember	7.6 Ability to design and conduct research using appropriate methodology; 10.37 legislation and professional ethics
			18.2		85
			18.3		80
	C19. Cotton wool—absorbency	0.3	19.1	DD	84	14	Remember	7.6 Ability to design and conduct research using appropriate methodology: 10.37 legislation and professional ethics
			19.2	TF	100
			19.3		50
			19.4		44

and

Table A5. Thematic section and category structure of Pharmacognosy 1, Examination 3.

Thematic Section	Categories	Point Value	Questions	Question Type	Facility Index [%]	Discrimination Index [%]	Cognitive Dimension	Competencies
General questions	C1. Natural identity of constituents	0.1	1.1	MC	100	1	Remember	9.21 Ability to communicate in English and/or locally relevant languages
			1.2		67
			1.3		100
			1.4		100
			1.5		100
	C2. General statements on constituents	0.2	2.1	MC	67	0	Remember, understand	10.27 Organic and medicinal/pharmaceutical chemistry; 10.37 legislation and professional ethics
			2.2		79
			2.3		75
	C3. Recognizing constituents, medicinal and borderline products	0.5	3.1	DD	60	47	Remember, analyze	7.7 Ability to maintain current knowledge of relevant legislation and codes of pharmacy practice; 10.37 legislation and professional ethics
			3.2		32
			3.3		67
Carbohydrates and waxes	C4. Honey	0.3	4.1	Cloze (FIB; DD)	84	12	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology; 10.27 organic and medicinal/pharmaceutical chemistry; 10.31 microbiology; 10.37 legislation and professional ethics
			4.2		96
	C5. Mannitol	0.2	5.1	MC	79	0	Remember, apply	10.31 Microbiology; 10.32 pharmacology including pharmacokinetics; 10.33 pharmacotherapy and pharmaco-epidemiology; 10.37 legislation and professional ethics
			5.2		79
	C6. Waxes	0.2	6.1	Cloze (FIB; DD)	100	21	Remember, apply	9.21 Ability to communicate in English and/or locally relevant languages; 10.34 pharmaceutical technology, including analyses of medicinal products
			6.2		65
Fats and vegetable fatty oils	C7. Quality control of fatty oils	0.3	7.1	Cloze (FIB; DD)	87	17	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.34 pharmaceutical technology, including analyses of medicinal products; 10.37 legislation and professional ethics
			7.2		67
	C8. Extraction and analysis of fatty oils	0.3	8.1	Cloze (FIB; DD)	45	26	Remember, analyze	7.4 Capacity to evaluate scientific data in line with current scientific and technological knowledge; 9.21 ability to communicate in English and/or locally relevant languages; 10.37 legislation and professional ethics
			8.2		47
	C9. Structural characteristics of fatty oils	0.2	9.1	Cloze (DD)	100	26	Remember, analyze	10.27 Organic and medicinal/pharmaceutical chemistry
			9.2		70
			9.3		100
			9.4		56
			9.5		85
			9.6		75
	C10. Castor oil	0.3	10.1	Cloze (FIB; DD)	61	15	Remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology; 10.27 organic and medicinal/pharmaceutical chemistry
			10.2		72
Therapeutic indications	C11. HMPC | EMA recommendations	0.35	11.1	Cloze (MC)	81	2	Remember, apply	9.21 Ability to communicate in English and/or locally relevant languages; 10.33 pharmacotherapy and pharmaco-epidemiology; 17.63 provision of informed support for patients in selection and use of non-prescription medicines for minor ailments (e.g., cough remedies…)
			11.2		77
	C12. Risk and adverse effects associated with use	0.35	12.1	Essay	80	32	Understand, analyze, evaluate, create	7.2 Analysis: ability to apply logic to problem solving, evaluating pros and cons and following up on the solution found; 7.3 synthesis: capacity to gather and critically appraise relevant knowledge and to summarize the key points; 7.5 ability to interpret preclinical and clinical evidence-based medical science and apply the knowledge to pharmaceutical practice; 10.32 pharmacology, including pharmacokinetics; 10.33 pharmacotherapy and pharmaco-epidemiology; 13.46 retrieval and interpretation of relevant information on the patient’s clinical background; 17.63 provision of informed support for patients in the selection and use of non-prescription medicines for minor ailments (e.g., cough remedies…)
			12.2		76
			12.3		86
Phenols and flavonoids	C13. Micro-sublimation	0.3	13.1	Cloze (FIB; DD)	75	13	Remember, analyze	10.33 Pharmacotherapy and pharmaco-epidemiology; 17.63 provision of informed support for patients in selection and use of non-prescription medicines for minor ailments (e.g., cough remedies…)
			13.2		92
	C14. Anthocyanidins	0.2	14.1	Cloze (DD)	56	51	Remember	10.27 Organic and medicinal/pharmaceutical chemistry; 10.28 analytical chemistry
			14.2		73
Macroscopic analysis	C15. Identification	0.3	15.1	DD	92	38	Remember, analyze	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology
			15.2		75
			15.3		67
			15.4		33
			15.5		92
	C16. Combination herbal product quality analysis and risk	0.5	16.1	Cloze (FIB; DD)	40	68	Analyze, evaluate, remember	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology; 10.35 toxicology
			16.2		50
Microscopic analysis	C17. Combination herbal product quality analysis	0.4	17.1	Cloze (FIB; DD)	14	46	Remember, analyze, evaluate	9.21 Ability to communicate in English and/or locally relevant languages; 10.24 plant and animal biology; 10.34 pharmaceutical technology, including analyses of medicinal products
			17.2		23

. E1 makes most frequent use of the ability to combine different question types into one question, by utilizing the coding capabilities of the cloze question type. A total of 80% of all examination question types are cloze questions. The reverse is observable for E2, in which only 4.2% of questions are cloze questions. Of the remaining 95.8%, the majority (42.1%) are drag and drop questions. E3 utilizes a more even spread of question types—66% cloze and 34% non-cloze questions. High-cognitive-dimension questions are most abundant in E3 (38.5%), followed by E1 (25.7%) and E2 (11.3%); however, on average, E1 utilizes the most cognitive dimensions per question (2.2), as opposed to E3 (1.9) and E2 (1.4). E3 covered the most competencies ranked above 60% by community pharmacists (87.0%), followed by E1 (63.2%) and E2 (54.0%). Due to all questions assessing competency in pharmacognosy, it has been excluded from our datasets, in which the questions were tagged. A cumulative comparison of the three examinations, which includes the total number of students attempting each examination and total number of questions is provided in Appendix A,

Table A6. Cumulative comparison of Pharmacognosy 1 course examinations, in terms of question types, cognitive dimensions, competencies, student performance and Moodle statistics.

Student Performance and Examination Statistics
Examinations	Total Student Attempts		Examination Duration [min]		Total Questions	Questions per Student Attempt				Student Average Performance [%]				Standard Deviation [%]
1	53		45		49	15				66.29				12.41
2	52		50		95	19				76.21				11.41
3	53		55		47	17				65.88				11.64
Question types
Examinations	TF		MC		DD	FIB				Essay				Cloze
														(MC)		(DD)		(MC; FIB)		(FIB)	(FIB; DD)
1	0		7		3	3				0				27		0		6		3	0
2	11		25		40	15				0				4		0		0		0	0
3	0		10		3	0				3				2		13		0		0	16
Cognitive dimensions
Examinations	Questions assessing low cognitive dimensions					Questions assessing high cognitive dimensions
	Remembering		Understanding		Applying	Analyzing				Evaluating				Creating
1	43		14		24	25				3				0
2	87		16		15	15				0				0
3	44		6		6	25				7				3
Professional competencies
Examinations	Ranked <60% by community pharmacists […]					Ranked >60% by community pharmacists […]
	7.6	10.24	10.28	11.38	11.39	7.2	7.3	7.4	7.5	7.7	9.21	10.27	10.29	10.30	10.31	10.32	10.33	10.34	10.35	10.37	13.46	17.63
1	3	24	0	0	15	0	0	0	0	10	21	7	4	2	0	0	5	0	0	16	2	5
2	11	31	8	7	0	0	0	0	0	2	20	5	7	0	0	0	0	0	0	29	0	4
3	0	13	2	0	0	3	3	2	3	3	28	9	0	0	2	2	9	6	2	20	3	7

.

The web interfaces for ChatGPT and DeepSeek were used for all tests. In addition, the web interface for Google Lens was used for questions relying on the identification of plant materials from images. For Pl@ntNet, the mobile app, as available on the IOS app store, was used to identify the photographic materials included in the examinations. The models used were ChatGPT-4o, DeepSeek-V3 and Google Lens 1.17.240515009. The initial prompt used was “Solve the following questions”; afterwards, all questions were consecutively, manually prompted into the web interface in the form of screenshots. This approach was chosen instead of copy-and-pasting or typing out question text, due to the time restrictions in place for student attempts, as well as the variable and complicated layout of the question graphical user interface (Appendix B, Figure A1). Question text was manually prompted only when models could not extract text from provided images or if relevant context was missing (i.e., for questions using drop-down menus, which are not expanded by default). Google Lens and Pl@ntNet searches were performed directly with the images embedded in the exam questions. All examination images in identification questions were queried, and successful botanical identifications were recorded (Appendix B, Figure A2). LLM responses were graded using the same criteria as for student responses (i.e., the same point system). This examination schema rewards points for each correct response within a given question. If all responses selected are correct (e.g., all drop-down menu options are accurate), the maximum number of points is awarded.

To model the full range of possible outcomes for each examination based on the recorded responses, a Monte Carlo simulation approach was employed. The Monte Carlo method was selected due to the high combinatorial complexity of the examination format. This method allows for a probabilistic estimation of complex outcome distributions when full enumeration is computationally infeasible [36]. One random question per category was selected by using a uniform distribution, assuming equal likelihood for all variants within a question. The point values for each selected question were summed up to compute a total examination score for that run. This process was repeated 100,000 times for each examination to approximate the probability distribution of total scores. This sampling-based approach avoids the need to explicitly enumerate all possible test permutations, which would exceed 10 million combinations for some of our test structures, making it a useful tool to simulate random outcomes for high question variance examinations. No smoothing, transformation or trimming of the data was applied. All simulations were performed using Python 3.11, with the “pandas” and “random” libraries [37]. Python was chosen for its speed, reproducibility and ability to handle large-scale simulations within memory constraints. All data visualizations were performed with SCImago Graphica [38].

To explore the relationship between the facility index and question outcomes, we used linear regression [39]. The discrimination index values per question category are also provided for further contextualization. The two variables were plotted separately for the LLMs, and the coefficient of determination was calculated using Excel (Microsoft Corporation). Linear regression was carried out once for the entire examination, using all available datapoints, and then individually per question category, separating datapoints into small multiples within the same examination. For the entire examination, the analysis was carried out, including all question categories in order to discern any general positive or negative relationship between the variables. In plotting the variables as small multiples, those categories for which it was immediately obvious that no correlation was observed (i.e., the LLM achieved a score of 0 or 100 per cent on all questions inside the category) were retained; however, question categories which contained less than three questions (i.e., no more than two datapoints could be plotted) were disregarded, as no meaningful result interpretation could be carried out.

3. Results

3.1. Score Distribution of LLM Recorded Responses and Student Examination Score Comparison

The total mean scores per examination attempt for the tested LLMs are provided in

Table 1. Calculated average scores obtained after the Monte Carlo analysis of the results obtained by LLMs in the three Pharmacognosy I examinations. Total average scores as well as their standard deviations are provided as point and percent values.

LLM	Examination 1				Examination 2				Examination 3
	Average		Standard Deviation		Average		Standard Deviation		Average		Standard Deviation
	Point	[%]	Point	[%]	Point	[%]	Point	[%]	Point	[%]	Point	[%]
ChatGPT 4o	2.51	50.2	0.26	5.2	3.13	62.6	0.33	6.6	3.45	69.0	0.16	3.2
DeepSeek R1	1.86	37.2	0.16	3.2	2.31	46.2	0.28	5.6	3.08	61.6	0.23	4.6

and their frequency distribution plots are visualized in Figure 1. ChatGPT 4o outperformed DeepSeek R1 in all three examinations by an average of 12.27 per cent. The highest average score achieved was by ChatGPT on Examination 3 and the lowest was obtained by DeepSeek on Examination 1. With the exception of Examination 3, the total score frequency was more spread out for ChatGPT and differed by the widest margin for Examination 2. ChatGPT’s mean score was over 50 per cent in all examinations, as opposed to DeepSeek, which managed to cross this threshold only in Examination 3. Notable differences in the density of result frequencies are observed between LLMs and with the most apparent being in DeepSeek’s performance in Examination 3, where two separate strings of result frequencies form. Weaker but similar effects are noted in DeepSeek’s performance on Examination 2 and ChatGPT’s Examination 1. This effect is absent from the LLM’s performance in Examination 1. Another notable difference between examinations is related to the density of the average scores: there is a much more even total distribution, notably for both LLMs’ performance observed in Examination 1, in contrast to the other examinations. This is especially apparent in the skew of the distribution plot for both models for Examination 2, relative to the mean score.

Compared to student performance, the models achieved lower average total scores, with the exception of ChatGPT’s performance on Examination 3, where its average total score was 3.12 per cent higher. On Examinations 1 and 2, student performance exceeded that of ChatGPT by an average of 14.9 per cent. The average student scores achieved were 21.1 per cent higher compared to those of DeepSeek; however, the difference in performance was significantly smaller for Examination 3—a higher student average score of only 4.28 per cent compared to 29.09 per cent for Examination 1 and 30.01 per cent for Examination two. Another notable difference was the much greater observed standard deviation for student attempts, though that could be explained by the magnitudes lower total number of student attempts compared to the number of simulations. This comparison is visualized in Figure 2.

3.2. Scores Attained per Examination Category, Relative to Student Performance

Exploring the differences between student and LLM performance in the context of examination categories within examinations confirm the already-established results while providing additional detailed context. In Examinations 1 and 2, where students’ average results were higher, they not only outperformed LLMs in a greater number of categories but also by greater margins, compared to in Examination 3. Examination 1 had the greatest number of categories where student performance was better than that of both LLMs—10 out of 15 (66.7 per cent). This comparison between examinations is visualized in Figure 3. This comparison was both for point values and percentage scores achieved.

In this context, the most impactful categories for Examination 1 were C10 and C9, as they contributed to attempts not only in a significant percentile but also in greater point values. Questions in both categories are constructed using the same question type, but C10 relies on visual identification of images and low cognitive dimensions and examines less-valued professional competencies, in contrast to C9, which is a purely text-based question but relies on a high cognitive dimension and examines higher valued professional competencies. For Examination 2, the most impactful categories were C2 and C11. The question type is once again the same for the categories. All questions rely on selecting the correct response from a predetermined set of options in drop-down menus, which are also interdependent. C2 is characterized by requiring a high cognitive dimension and examining highly valued competencies, conversely to C11, where a low cognitive dimension is required and less valued professional competency is quizzed, but attempts are again reliant on analyzing graphical data—specifically, microscopic illustrations of plant powders. For Examination 3, C16 contributes the most to student attempts. It utilizes a combination of fill in the blank fields and drop-down menus, which are again interdependent, and relies on visual analysis of graphical data and high cognitive dimensions but explores a mixture of high- and low-valued competencies. DeepSeek performed poorly in all categories where visual analysis of graphical data was concerned.

For the LLMs, both homo- and heterogenicity of the highest contributing categories is observed, particularly in Examination 2 and 3 for DeepSeek. For Examination 1, both models performed better than students in category C3: questions which require several multiple-choice answers and rely on textual analysis and low cognitive dimensions though examining highly valued competencies. In Examination 2, C9 and C13 contributed to better ChatGPT attempts and C16 to DeepSeek. The common denominators are the utilization of non-interdependent required answers in simple question types, the requirement for low cognitive dimensions in textual analysis and the examination of low-valued professional competencies. C16 stands out only in that it requires simple arithmetic calculations to achieve a high score. For Examination 3, C7, C10 and C14 seemed to contribute to both LLMs’ performances, with better scores from DeepSeek observed in C7 and C14, compared to ChatGPT. The lower average scores here can be explained with the low performance in C2–C4 and C15 by DeepSeek. For C7, C10 and C14, again, low cognitive dimensions, textual analysis and examination of low-valued or a mix of low- and high-valued competencies are again the common denominators, whereas in the categories where DeepSeek is observed to have poorer comparative performance, either high cognitive dimensions, interdependent answers or attempt success is dependent on the analysis of graphical data.

3.3. Exploring the Connection Between the Facility Index and LLM Scores

Only a very weak positive correlation was observed for Examination 1 between the facility index and ChatGPT scores as a result of the linear regression analysis, with a coefficient of determination (r²) of 0.2504, and for Examination 3, 0.2437. As such, no general observation of the potential predictive capabilities of the metric can be posited. Weaker values were obtained for all other examinations and LLM combinations. The results are visualized in Figure 4. Further context was extracted by exploring this relationship on a category-by-category basis. Much stronger correlations were observed. For Examination 1, the highest r² values for ChatGPT and DeepSeek were calculated for C7—0.7229 and 0.9521, respectively. All calculated coefficients of determination are provided in

Table 2. Calculated coefficient of determination (r²) for linear regression analysis of facility indexes and LLM examination scores per category.

LLM	Examination 1			Examination 2			Examination 3
	Disregarded Categories	Categories with no Correlation	Coefficient of Determination (r2) per Category	Disregarded Categories	Categories with no Correlation	Coefficient of Determination (r2) per Category	Disregarded Categories	Categories with no Correlation	Coefficient of Determination (r2) per Category
ChatGPT 4o	C8	C1 C4 C5 C11 C12 C14 C15	C2 (0.6938) C3 (0.4391) C6 (0.3273) C7 (0.7229) C9 (0.5423) C10 (0.1089) C13 (0.5111)	C1	C2 C6 C7 C8 C11 C12 C13 C14 C16	C3 (0.9185) C4 (0.8193) C5 (0.2936) C9 (0.4717) C10 (0.9670) C15 (0.3466) C17 (0.4700) C18 (0.9370) C19 (0.1046)	C4 C5 C6 C7 C8 C10 C11 C13 C14 C16 C17	C1 C12	C2 (0.9494) C3 (0.1120) C9 (02161) C15 (0.1684)
DeepSeek R1	C8	C1 C4 C5 C6 C10 C13 C14 C15	C2 (0.2521) C3 (0.1007) C7 (0.9521) C9 (0.5423) C11 (0.9912) C12 (0.2500)	C1	C2 C5 C6 C8 C9 C11 C12 C13 C17	C3 (0.9185) C4 (0.3530) C7 (0.8781) C10 (0.2370) C14 (0.2365) C15 (0.4014) C16 (0.6792) C18 (0.4918) C19 (09075)	C4 C5 C6 C7 C8 C10 C11 C13 C14 C16 C17	C1 C3 C9 C12 C15	C2 (0.4652)

and category-specific correlations are depicted in Figure 5.

3.4. Google Lens and Pl@ntNet Capabilities in Identification of Herbal Substances Featured on Pharmacognosy Examinations

The majority of Examination 1 and 2’s embedded photographic materials were not successfully identified. Google Lens suggested the correct botanical identity for only one macroscopic photographic material (2.15 per cent) of the fruit of Tribulus terrestris L. from Examination 1 and three macroscopic photographic materials (9.09 per cent) from Examination 2. One of these was of the inflorescence of Achillea millefolium L. and two were of the seed of Aesculus hippocastanum L. Pl@ntNet identified four (8.60 per cent) macroscopic materials from Examination 1 correctly and the same three macroscopic photographic materials from Examination 2, performing just slightly better by also successfully identifying leaves from Tribulus terrestris L and Achillea millefolium L. No micromorphological photographic materials were correctly identified with either computer vision approach. When searching embedded pollen grain photographic materials, Google Lens suggested other pollen grains, but none of the results identified the specific pollen accurately. The pollen grains included in the examinations were of Carthamus tinctorius L. and Crocus sativus. L. Equisetum L. species spores were particularly difficult to identify, as search results came back only with images of various algae. This was not the case for the photographic materials included in Examination 3. Macroscopic photographic materials from this examination featured various seeds from fatty-oil-producing plants, some of which are commonly used both as traditional medicine and foods or spices or are known poisoning hazards. Of the 20 embedded photographic materials, 16 (80 per cent) were correctly identified. These seeds belonged to the species Brassica nigra (L.) W.D.J.Koch, Cannabis sativa L., Chenopodium quinoa Willd, Datura stramonium L., Glycine max (L.) Merr., Papaver somniferum L., Ricinus communis L., Salvia hispanica L. and Sesamum indicum L. A single outlier, which was also correctly identified, was a macroscopic photographic material of the thallus of Cetraria islandica (L.) Ach. Pl@ntNet’s performance in Examination 3 was in stark contrast to that in Examination 1 and 2 and Google Lens, as it failed to identify any of the seed or lichen images.

4. Discussion and Conclusions

The purpose of this study was to examine how effectively LLMs perform in established pharmacognosy examinations in the context of distant digital evaluation and how their performance compares with that of undergraduate pharmacy students. Recent work has highlighted the expanding role of LLMs in medical and pharmaceutical education, particularly in contexts involving knowledge retrieval and formative assessment [3,4], but not in pharmacognosy online examinations. Additionally, including mixed item types and original score settings offers validity that standard synthetic question sets used in such published works usually lack. By using past examinations and corresponding grading criteria, the study offers a realistic insight into the capabilities and limitations of AI-generated responses in a specialized bioscience domain. The models performed relatively well on lower-order cognitive tasks, such as recalling definitions and identifying basic concepts: areas where prompt clarity and factual recall are most influential [29]. Conversely, questions requiring synthesis, contextualization, or the application of knowledge to novel scenarios exposed clearer limitations: a pattern noted in broader evaluations of AI in clinical and biological science assessments [30]. These patterns reflect the broader discourse on LLM use in medical and pharmaceutical education, where accuracy often hinges on the depth and specificity of domain knowledge.

The use of the facility index and Monte Carlo simulations, which are practical solutions to category-randomized examinations, allowed for the unique findings to be interpreted within a meaningful psychometric framework. Some items with higher facility index values corresponded more closely to successful LLM responses; however, this was not a sustained trend across question categories. Simulated comparisons indicated that the LLMs did not consistently outperform students, reinforcing the idea that AI competence is uneven across question items, cognitive dimensions and examined professional competencies. The study also underscores a practical challenge in educational deployment: while LLMs can reproduce fact-based answers with speed and clarity, they may not reliably generate the nuanced reasoning expected in applied pharmaceutical contexts. Complex, interconnected question types, such as combinations of short answers, drop-down menus, etc., posed significantly higher challenges for LLMs.

Another key consideration is the correct attribution of point values to questions. Inappropriate allocation of a high number of points to simple, answer non-interdependent questions, relying on textual analysis, low cognitive dimensions and assessing low-valued professional competencies appear to be crucial for higher attainable scores by LLMs. By extension, this could provide clarity as to why LLMs have achieved higher scores compared to students on Examination 3, despite the fact that it has the highest percentage of highly valued professional competencies and the second highest number of questions requiring high cognitive dimensions.

Computer vision tools such as Google Lens and Pl@ntNet appear to still be lacking in accurate botanical identification of herbal substance origins, which is in stark contrast to some of the published literature [16,17,18]. This is especially relevant for micromorphological graphical materials, regardless of whether they are photomicrographic or illustrative work. In contrast, macroscopic images of common medicinal substances, foodstuffs and well-known poisonous materials are readily identifiable. This suggests that avoiding the inclusion of such herbal substances in examination questions could fortify LLM-resistant designs.

Several limitations warrant consideration. No additional higher-reasoning, frontier LLMs (e.g., GPT-o1/o3, Claude 3.5, Gemini 3) have been included in this study. Some studies show that these achieve significant exam results in medical domains [40]. While such comparisons are important for establishing a baseline, this study was intentionally limited to only include the most popular and available LLMs. The findings derive from a single subject area and scope, within one institution and one curriculum, potentially limiting generalizability and external validity. We posit that performing the same investigation but on a published standardized pedagogical work in pharmacognosy [41] could prove prudent in providing further insights into LLM-resistant examination design. The prompting strategy, though standardized, may also have influenced the quality of responses, particularly for complex tasks. Retrieval-augmented prompting was disregarded as no one suitable corpus of pharmacognostic course literature in the form of files accessible to LLMs is available. While this is considered a standard state-of-the-art approach in such studies, the goal was to test raw chat models, responding only based on available open access data. Additionally, the absence of multimodal questioning and the lack of follow-up prompts mirror traditional assessment conditions but may underestimate the full potential of interactive AI use. Specifically, it is prudent to point out that no guarantee can be provided as to the preference of students towards screenshot input as opposed to typed-out text. This prompting design is inferred from the time-limited nature of online examinations, which leaves students with an average of 174 s per question to prompt, receive a generated response and potentially evaluate it and fill in the perceived correct responses with an overview of the unique user interface question layout. Potential future studies should also incorporate, at the very least, students’ self-reported prompting preferences. Despite these constraints, the results contribute valuable evidence about how LLM performance aligns with student achievement across different thematic areas of pharmacognostic knowledge.

Educators and curriculum designers may benefit from incorporating LLMs to support revision, question development or feedback generation, provided their use is aligned with the cognitive demands of specific tasks. Future research should examine broader curricula, integrate qualitative analysis of response reasoning and explore interactive or scaffolded prompting to assess the upper limits of LLM capability. As AI integration in pharmacy education accelerates, balanced evaluation will remain essential to ensure that innovation enhances, rather than compromises, learning outcomes.