Abstract
Background: The integration of artificial intelligence (AI) into educational environments is accelerating, with growing interest in its potential to enhance learning outcomes. This pilot study aimed to assess student perceptions regarding the quality and educational impact of AI-generated feedback. Methods: Second-professional-year pharmacy students (n = 148) enrolled in a drug literature evaluation course received AI-generated feedback for a draft assignment which required students to critically evaluate a recently published journal article. Feedback was generated by an AI assistant that offered iterative feedback based on the assignment rubric. After the activity, students completed an electronic survey instrument evaluating the AI feedback regarding its alignment with rubric criteria, accuracy, clarity, and tone. Student perceptions of the feedback’s impact on learning were also assessed. Results: A total of 148 students (100%) completed the activity and responded to at least one item of the survey tool. Most students agreed with the AI-generated feedback for each section of the assignment in regard to clarity (61.5% to 76.2%) and tone (66.9% to 74.3%), with lower agreement related to alignment with the rubric (33.8% to 60.4%) and accuracy (34.5% to 54.2%). Students also expressed concern about the reliability of AI feedback (71.2%) and risk for plagiarism (50%). Conclusions: AI-generated feedback was positively received in some areas. However, students expressed apprehension regarding its accuracy, reliability, and potential accusation of plagiarism. These findings suggest further refinement is necessary to enhance AI trustworthiness and educational value in academic settings.
1. Introduction
Artificial intelligence (AI), including large language models (LLMs) such as ChatGPT, is beginning to influence pharmacy education and practice in visible ways [1,2,3]. A recent survey reported that 58% of pharmacy students reported use of AI for academic work [4]. This is a substantial increase from only 20 to 30% of pharmacy students reporting use of AI for academic purposes in 2023 [5,6]. While the literature has reported that pharmacy students are somewhat apprehensive about incorporating AI into learning activities due to concerns about accuracy, bias, ethical use, and loss of critical thinking skills, most students express a positive outlook on the potential applications of AI and are interested in learning how to effectively use AI to support their success [4,6,7,8,9,10]. Pharmacy students report using AI tools to summarize readings, retrieve drug information, clarify difficult concepts, and generate study materials [6]. One area where AI may be particularly useful is in providing automative feedback for written work. Prior research in higher education suggests that artificial intelligence (AI)-generated feedback can provide rapid, scalable, formative feedback for rubric-based written assignments [11,12]. However, it has also been noted that learner perceptions of trustworthiness and accuracy of automated feedback strongly influence whether they choose to engage with or apply the feedback [13,14]. Students learn within increasingly complex digital environments characterized by continuous exposure to online information, social media content, and rapidly evolving artificial intelligence technologies. As a result, issues surrounding information quality, trust, and digital literacy may influence how learners perceive and utilize AI-generated educational support.
Feedback is a significant factor related to positive student learning outcomes [15]. Formative feedback on written work is a primary mechanism for developing students’ skills in drug information, critical appraisal, and scholarly communication, particularly in journal club assignments. However, providing students with detailed and constructive feedback on written work in a timely manner can be a major challenge for most faculty, another factor that may make AI-generated feedback an attractive solution.
Limited research has examined how pharmacy students perceive AI-generated feedback on their own writing, how much they trust it, or how they integrate it into their revision process [14]. Most existing studies focus largely on AI use patterns, attitudes, and content accuracy rather than on students’ experiences receiving and acting on AI-generated formative feedback [4,5,6,7,8,9,10,16]. This gap is especially important as programs explore AI-assisted feedback for tasks such as literature evaluation and journal club critiques, where the quality of feedback directly shapes students’ reasoning and professional identity. Given limited research, and the fact that student perceptions play an important role in their decision to engage with AI-generated feedback, the present pilot study explores the use of AI-generated feedback in a drug literature evaluation. This Brief Report first describes the implementation of an AI-generated feedback activity, followed by an assessment of student perceptions regarding the quality and educational value of the feedback, and concludes with implications for future integration of AI-supported educational technologies.
2. Materials and Methods
2.1. Charlie AI Activity
The instructional technology team at our institution developed an AI writing assistant, known as Charlie AI, designed to provide rubric-based feedback on student assignments. The AI assistant is powered by OpenAI’s GPT-4 model and offers iterative feedback based on the specific assignment rubric, allowing students to refine and resubmit their work while maintaining data privacy. A pilot active learning exercise using Charlie AI was incorporated into a written assignment for a second-professional-year, required drug information and literature evaluation course. The assignment required critical evaluation of a recently published journal article with respect to ten different sections: journal and authors, study subjects, study design, endpoints, statistical analysis, results, overall strengths, overall limitations, generalizability of results, and clinical impact (rubric provided in the Supplementary Materials). The assignment had a two-page limit. Students submitted their draft written assignment one week prior to the due date and Charlie AI provided feedback on each section of the assignment rubric. Students reviewed the AI-generated feedback during a designated class period and were permitted to discuss their feedback with their peers and the course instructors. After the activity students had the opportunity to revise their draft based on Charlie AI’s feedback before submission of the final assignment. The overall objective of this in-class activity was to foster critical thinking skills and help students understand the strengths and limitations of AI. This course includes three journal club assignments distributed through the semester. Historically, students enrolled in this course participate in three in-class peer-review sessions for the journal club assignments, during which time they receive electronic, anonymous feedback on their draft assignment from three of their classmates and provide feedback to three other classmates. Our experience with this peer-review activity has demonstrated enhanced learning behaviors, increased interest in peer-review, and improved self-competence regarding literature evaluation [17,18]. In this pilot, the traditional peer-review activity associated with the third and final journal club assignment was replaced with the Charlie AI feedback activity.
2.2. Assessment of the Activity
Immediately following the in-class Charlie AI activity, students were asked to complete an electronic survey instrument regarding their perceptions of the quality of Charlie AI-generated feedback and perceptions of the overall impact of Charlie AI on learning. The survey instrument consisted of two sections and is provided for review in the Supplementary Materials. The first section of the survey instrument asked students for their level of agreement (yes, partially, or no) with the Charlie AI feedback in relation to its alignment with the assignment rubric, accuracy, clarity, and tone for each of the assignment sections. The second section of the survey instrument collected student perceptions of using Charlie AI-generated feedback and included selected questions from the validated TAME-ChatGPT survey tool for assessing health-care students’ attitudes towards ChatGPT [19]. Students were provided with approximately 15 minutes at the end of the class period to complete the survey tool and were awarded 5 points for participation. Survey responses were anonymous and confidential. The study was determined to be exempt research by the institution’s Investigational Review Board (IRB2024-1674). Data are presented descriptively.
3. Results
One-hundred and forty-eight second-year pharmacy students reviewed the Charlie AI-generated feedback for their written draft assignment during class, with the opportunity to edit their draft before submitting their final assignment. All students enrolled in the course (100%, 148/148) completed at least one of the survey instrument items. Table 1 shows the results of student agreement with Charlie AI’s feedback in relation to its alignment with the assignment rubric, accuracy, clarity, and tone for each section of the assignment. Levels of agreement with Charlie AI-generated feedback varied across journal club rubric domains, with most students agreeing with the AI-generated feedback for each specific section of the assignment in regard to clarity (61.5% to 76.2%) and tone (66.9% to 74.3%), with lower agreement related to alignment with the rubric (33.8% to 60.4%) and accuracy (34.5% to 54.2%). Student agreement was lowest for the accuracy of feedback related to the more nuanced aspects of study evaluation such as the overall limitations (39.9%, 57/143) and generalizability of study results (34.5%, 49/142).
Table 1.
Student agreement with Charlie AI feedback.
Table 2 presents student perceptions regarding use of Charlie AI’s feedback and impact on learning. Concerns regarding the reliability and risks associated with AI-generated feedback were common among respondents. Approximately half of respondents indicated concern about potential plagiarism accusations associated with use of the AI tool. Participants were split with regard to perceived learning benefits from the AI-generated feedback, with 50.7% (74/146) of students indicating that they learned something new from the feedback, and 46.6% (68/146) reporting that the feedback supported their critical thinking. Only one-third of students agreed that the Charlie AI feedback improved their final assignment grade. Student intentions to continue using the Charlie AI feedback tool were also mixed but trended toward reluctance. More than half of respondents reported that they would not choose to use the Charlie AI tool again in future journal club assignments. Similarly, 50.7% (74/146) of students did not support integrating the Charlie AI feedback tool into future courses. Finally, when asked to identify preferred sources of feedback, students most frequently selected their peers (57.5%, 84/146) or a teaching assistant/instructor (39.7%, 58/146). Only 2.7% of students (4/146) indicated a preference for Charlie AI-generated feedback.
Table 2.
Student perceptions regarding use of AI feedback.
4. Discussion
Use of AI-generated feedback for journal club assignments in a drug literature evaluation course received mixed reviews from second-year pharmacy students. While a small majority of students agreed with the specific feedback in relation to alignment with the rubric, clarity, and tone, they expressed apprehension regarding its accuracy, reliability and potential risk for accusation of plagiarism. Students indicated preference for peer feedback over the AI-generated feedback and were hesitant to support further integration of Charlie AI tools into their coursework.
Previous perception studies suggest that pharmacy and other health profession students generally view AI as promising for learning, while simultaneously expressing concerns about accuracy, bias, and ethical use [4,6,7,8,9,10]. In a recent pilot study, Pellesi evaluated pharmacy students’ trust in AI-generated feedback through the use of Microsoft Copilot to provide feedback for a master’s-level evidence-based medicine course for pharmacy students in Denmark [14]. Student groups participated in peer-feedback activities with the additional option of receiving AI feedback. Groups that chose not to use AI feedback indicated lack of trust and concerns about the quality of AI-generated feedback as reasoning for not using AI. Among student groups who opted in to receive AI feedback, the majority rated peer feedback as more valuable than AI feedback. Reasons for this preference included lack of depth, generalized comments that failed to acknowledge discipline-specific nuances, and redundant comments from AI-assisted feedback. Our findings are consistent with these results in that the majority (71%) of our students were concerned about the reliability of AI feedback and 90% preferred to receive feedback from a human (peer, teaching assistant, or instructor) over AI-generated feedback. Notably, only 30% of students agreed that using AI feedback would improve their grade or that the feedback helped their confidence in critiquing journal articles. In contrast, the previous literature evaluating the impact of peer-feedback activities in pharmacy students has reported positive perceptions including improved self-competence [18,20]. This also aligns with the majority of students in this pilot preferring peer feedback over AI-generated feedback. More research and refinement of AI-generated feedback is needed before AI can replace human feedback.
One of the possible reasons why Charlie AI was not viewed positively by the students in this pilot study could be because it was originally designed and trained for longer form written assignments, such as essay papers in the social sciences. One of the key components to the journal club assignment is the two-page limit. Charlie AI was designed to always provide iterative suggestions for improvement. This resulted in some students sharing a frustration that even if a particular section of the assignment was complete and accurate, Charlie AI would still suggest adding more detail. Charlie AI is powered by OpenAI’s GPT-4 model and generates assignment-specific feedback through rubric-guided prompting, rather than domain-specific training on prior student submissions or grading records. Perhaps further revision of the assignment rubric, improved prompt engineering, and work with the development team to train Charlie for this assignment could improve the quality of feedback. An AI tool specifically designed for literature evaluation could be perceived as more effective and useful by pharmacy students.
Student concerns regarding the reliability of AI-generated feedback may reflect broader challenges associated with evaluating information credibility in contemporary digital environments. Apprehension could also be due to the fact that students had become comfortable receiving peer feedback for the first two in-class activities. Perceptions of AI-generated feedback may have been shaped by prior exposure to peer and instructor feedback. With the peer-feedback component, each student provides feedback for three other examples of the journal critique. Regardless of the feedback received from their peers, they have the opportunity to peer-review other students’ drafts and may notice components they omitted from their own draft. The act of reviewing the work of their peers may expose them to new ideas to incorporate into their own assignment and encourage them to think more critically about the critique and the rubric. This component is missing from AI feedback. The present study did not evaluate the relative contributions of reviewing peers’ work versus receiving AI-generated feedback. Student perceptions may differ in a course environment where AI-generated feedback served as the primary or sole source of formative feedback. Future studies should compare AI-only, peer-review, and hybrid feedback models to better understand their relative educational value.
The results of this pilot are limited to a single class and a single activity at one College of Pharmacy. The Charlie AI tool was originally designed for use in classes with longer written assignments, such as essay papers, which may have affected the feedback provided. It should be noted that this pilot did not independently assess the objective accuracy of Charlie AI-generated feedback. Therefore, survey responses related to accuracy reflect student perceptions rather than verified measures of correctness. The survey instrument also did not collect qualitative data regarding the reasons underlying students’ responses. Future studies incorporating interviews, focus groups, or open-ended survey questions may provide greater insight into factors influencing student trust, acceptance, and use of AI-generated feedback. In addition, many confounding variables could have affected the student perceptions of quality of the feedback including the complexity of the specific clinical trial chosen for the activity, students’ confidence in their own work, their ability to independently assess the validity of the feedback received, the prompts used based on the rubric, and the completeness of the student’s draft assignment, with more complete or high-quality drafts perhaps receiving less useful feedback compared to other drafts. Finally, measures related to critical thinking, reliability, and grade improvement reflect student perceptions and beliefs about AI-generated feedback rather than objective measures of educational outcomes. Future studies should evaluate these outcomes using direct measures of learning, assignment performance, and critical thinking skills.
5. Conclusions
While Charlie AI provided rubric-based feedback and was positively received in some areas, students expressed apprehension regarding its reliability, the potential accusation of plagiarism, and lack of improvement in their grade or self-confidence. Most students preferred peer or instructor feedback and were hesitant to support further integration of Charlie AI into their coursework. These findings suggest that while AI has potential as a supplemental feedback tool, further refinement is necessary to enhance its trustworthiness and educational value. Because these findings reflect student perceptions rather than objective measures of learning outcomes or feedback quality, additional research is needed to evaluate the educational effectiveness of AI-generated feedback and to better understand the factors influencing student trust and acceptance of AI-assisted learning.
This article is a revised and expanded version of “Assessing Student Perceptions of Artificial Intelligence Generated Feedback”, which was presented as a trainee poster at the American Association of Colleges of Pharmacy 2025 Pharmacy Education meeting in Chicago, IL, on 21 July 2025.
Supplementary Materials
The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/pharmacy14070147/s1, Figure S1. Journal Article Critique Rubric; Figure S2. Evaluating the Quality of Charlie-AI-Generated Feedback Survey Items.
Author Contributions
Conceptualization, A.H.S., M.A. and J.B.R.; methodology, A.H.S. and M.A.; formal analysis, M.A.; investigation, A.H.S., M.A. and J.B.R.; data curation, M.A. and D.P.; writing—original draft preparation, D.P., J.B.R. and A.H.S.; writing—review and editing, D.P., M.A., J.B.R. and A.H.S.; supervision, A.H.S. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of Purdue University (IRB2024-1674 approved on 30 October 2024).
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors on request.
Conflicts of Interest
The authors declare no conflict of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| AI | Artificial Intelligence |
| TAME-ChatGPT | Technology-Acceptance Model Edited to Assess ChatGPT Adoption |
| LLM | Large Language Model |
References
- International Pharmaceutical Federation (FIP). An Artificial Toolkit for Pharmacy: An Introduction and Resource Guide for Pharmacists; International Pharmaceutical Federation (FIP): The Hague, The Netherlands, 2025. [Google Scholar]
- Abdel Aziz, M.H.; Rowe, C.; Southwood, R.; Nogid, A.; Berman, S.; Gustafson, K. A scoping review of artificial intelligence within pharmacy education. Am. J. Pharm. Educ. 2024, 88, 100615. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Abdalla, M.; Saad, M.; Abazia, D.; Advani, A.; Alhammad, A.M.; Bernknopf, A.; Bitra, V.; Blommel, M.; Conn, K.M.; Goldwire, M.A.; et al. Responsible Use of Artificial Intelligence in Health Care: Evidence, Challenges, and Best Practices: An Opinion of the Drug Information Practice and Research Network of the American College of Clinical Pharmacy. J. Am. Coll. Clin. Pharm. 2025, 8, 1333–1361. [Google Scholar] [CrossRef] [Scilit]
- Alexander, K.M.; Jorgensen, E.O.; Rowe, C.; Nguyen, K. Pharmacy Students’ Perspectives on Integrating Generative AI into Pharmacy Education. Pharmacy 2025, 13, 183. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Knobloch, J.; Cozart, K.; Halford, Z.; Hilaire, M.; Richter, L.M.; Arnoldi, J. Students’ perception of the use of artificial intelligence (AI) in pharmacy school. Curr. Pharm. Teach. Learn. 2024, 16, 102181. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Anderson, H.D.; Kwon, S.; Linnebur, L.A.; Valdez, C.A.; Linnebur, S.A. Pharmacy student use of ChatGPT: A survey of students at a U.S. School of Pharmacy. Curr. Pharm. Teach. Learn. 2024, 16, 102156. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Busch, F.; Hoffmann, L.; Truhn, D.; Palaian, S.; Alomar, M.; Shpati, K.; Makowski, M.R.; Bressem, K.K.; Adams, L.C. International pharmacy students’ perceptions towards artificial intelligence in medicine—A multinational, multicentre cross-sectional study. Br. J. Clin. Pharmacol. 2024, 90, 649–661. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zhang, X.; Tsang, C.C.S.; Ford, D.D.; Wang, J. Student Pharmacists’ Perceptions of Artificial Intelligence and Machine Learning in Pharmacy Practice and Pharmacy Education. Am. J. Pharm. Educ. 2024, 88, 101309. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Das, S.R.; J.V., M. Perceptions of Higher Education Students towards ChatGPT Usage. Int. J. Technol. Educ. 2024, 7, 86–106. [Google Scholar] [CrossRef] [Scilit]
- Gustafson, K.A.; Berman, S.; Gavaza, P.; Mohamed, I.; Devraj, R.; Abdel Aziz, M.H.; Singh, D.; Southwood, R.; Ogunsanya, M.E.; Chu, A.; et al. Pharmacy faculty and students perceptions of artificial intelligence: A National Survey. Curr. Pharm. Teach. Learn. 2025, 17, 102344. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Agarwal, P.; Agarwal, R.; Iezhitsa, I. AI for Assessment in Medical Education in Post LLM Era: A Scoping Review. Med. Sci. Educ. 2026, 36, 1027–1039. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Kıyak, Y.S.; İş-Kara, T.; Emekli, E. Applications and Outcomes of Large-Language-Model-Generated Feedback in Undergraduate Medical Education: A Scoping Review. MedSciEduc 2026, 36, 81–99. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Nazaretsky, T.; Mejia-Domenzain, P.; Swamy, V.; Frej, J.; Käser, T. Who Gives Feedback Matters: Student Biases Towards Human and AI-Generated Formative Feedback. J. Comput. Assist. Learn. 2026, 42, e70153. [Google Scholar] [CrossRef] [Scilit]
- Pellesi, L. Trust in AI Feedback: Challenges in Pharmacy Education. MedSciEduc 2025, 35, 2615–2619. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hattie, J.; Timperley, H. The Power of Feedback. Rev. Educ. Res. 2007, 77, 81–112. [Google Scholar] [CrossRef] [Scilit]
- Culp, M.L.; Mahmoud, S.; Liu, D.; Haworth, I.S. An Artificial Intelligence–Supported Medicinal Chemistry Project: An Example for Incorporating Artificial Intelligence Within the Pharmacy Curriculum. Am. J. Pharm. Educ. 2024, 88, 100696. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Parikh, B.T.; Sheehan, A.H. Student pharmacists’ perceptions of the implementation of peer assessment instruction and activities within a didactic course. Curr. Pharm. Teach. Learn. 2016, 8, 821–826. [Google Scholar] [CrossRef] [Scilit]
- Montagano, K.J.; Sheehan, A.H. Pharmacy students’ intrinsic motivation to participate in electronic peer review of journal article critique assignments. Curr. Pharm. Teach. Learn. 2021, 13, 855–861. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Sallam, M.; Salim, N.A.; Barakat, M.; Al-Mahzoum, K.; Al-Tammemi, A.B.; Malaeb, D.; Hallit, R.; Hallit, S. Assessing Health Students’ Attitudes and Usage of ChatGPT in Jordan: Validation Study. JMIR Med. Educ. 2023, 9, e48254. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Theising, K.; Wu, K.; Heck Sheehan, A. Impact of peer assessment on student pharmacists’ behaviors and self-confidence. Curr. Pharm. Teach. Learn. 2014, 6, 10–14. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.