Development and Evaluation of a Next-Generation Medication Safety Support System Based on AI and Mixed Reality: A Study from South Korea
Abstract
1. Introduction
2. Materials and Methods
2.1. System Architecture
2.2. Clinical Workflow of the System
2.3. User Evaluation Procedure
2.3.1. Simulation Test
2.3.2. Instruments Used for User Test
- What aspects of the system did you find most helpful?
- Was there any feature that stood out as particularly useful or innovative?
- How did the system improve the workflow or decision-making process?
- Is there anything else you would like to see added to or improved on the system?
2.4. Ethical Considerations
3. Results
3.1. System Performance Evaluation: Recognition Accuracy
3.2. Results of User Evaluation
3.2.1. Cohort Characteristics
3.2.2. Multi-Dimensional User Experience Assessment
3.2.3. Qualitative Results—Open Questions
- Most Helpful Aspects
- Innovative or Useful Features
- Workflow and Decision-Making
- Suggestions for Improvement
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
| API | Application Programming Interface |
| BCMA | Bar-Code Medication Administration |
| GDPR | General Data Protection Regulation |
| HIPAA | Health Insurance Portability and Accountability Act |
| HTTP | Hypertext Transfer Protocol |
| IV Fluids | Intravenous Fluids |
| MR | Mixed Reality |
| MRTK3 | Mixed Reality Toolkit 3 |
| NASA-TLX | NASA Task Load Index |
| OCR | Optical Character Recognition |
| QR code | Quick Response code |
| RIMMS | Reduced Instructional Materials Motivation Survey |
| SUS | System Usability Scale |
| VRSQ | Virtual Reality Sickness Questionnaire |
Appendix A
| Survey Item | Median Score [Q1–Q3] |
|---|---|
| I would like to use this system frequently. | 4.0 [3.0–4.0] |
| The system was unnecessarily complex. | 2.0 [2.0–2.0] |
| The system was easy to use. | 4.0 [4.0–4.0] |
| I need technical support to use this system | 2.0 [1.0–4.0] |
| Functions were well integrated. | 5.0 [4.0–5.0] |
| There was too much inconsistency. | 1.0 [1.0–2.0] |
| Most people would learn to use this quickly. | 4.0 [4.0–5.0] |
| The system was cumbersome to use. | 1.0 [1.0–2.0] |
| I felt confident using the system. | 4.0 [4.0–4.5] |
| I needed to learn a lot before using it. | 1.0 [1.0–2.0] |
| SUS Score (1–100) | 82.5 [80-82.5] |
| Survey Content (Component) | Median Score [Q1–Q3] |
|---|---|
| Relevance (R) | 4.0 [3.7–4.7] |
| It is clear to me how the content is related to things I already know. The content will be useful to me. The content conveys the impression that its contents are worth knowing. | 5.0 [4.0–5.0] 3.0 [3.0–4.0] 4.0 [4.0–5.0] |
| Attention (A) | 2.7 [2.3–3.0] |
| The quality helped to hold my attention. The way the information is arranged helped keep my attention. The variety of digital content helped keep my attention on the lesson. | 4.0 [2.0–4.0] 2.0 [2.0–5.0] 3.0 [3.0–4.0] |
| Confidence (C) | 4.0 [3.7–4.3] |
| I was confident that I could understand the content. The good organization helped me be confident that I would learn the material. After a while, I was confident that I would be able to complete the tasks. | 4.0 [4.0–4.0] 4.0 [4.0–4.0] 5.0 [2.0–5.0] |
| Satisfaction (S) | 4.3 [3.3–4.3] |
| It was a pleasure to work with such a well-designed software. I really enjoyed working with this system. I enjoyed working so much that I think I will continue using it. | 4.0 [4.0–5.0] 4.0 [4.0–4.0] 4.0 [4.0–4.0] |
| Overall Mean Score (1–5) | 3.7 [3.7–4.1] |
| Symptom | Median Severity [Q1–Q3] |
|---|---|
| General Body Symptoms | |
| General discomfort | 0.0 [0.0–2.0] |
| Fatigue | 1.0 [0.0–2.0] |
| Boredom | 0.0 [0.0–0.0] |
| Drowsiness | 0.0 [0.0–0.0] |
| Dizziness | 1.0 [0.0–2.0] |
| Difficulty concentrating | 0.0 [0.0–1.0] |
| Nausea | 0.0 [0.0–0.0] |
| Eye-Related Symptoms | |
| Tired eyes | 0.0 [0.0–0.0] |
| Sore/aching eyes | 0.0 [0.0–0.0] |
| Eyestrain | 0.0 [0.0–0.0] |
| Blurred vision | 0.0 [0.0–1.0] |
| Difficulty focusing | 0.0 [0.0–2.0] |
| Overall VRSQ Score (0–6) | 0.4 [0.0–0.0] |
| Criteria | Median [Q1–Q3] |
|---|---|
| Mental demand | 20.0 [10.0–30.0] |
| Physical demand | 40.0 [10.0–40.0] |
| Temporal demand | 30.0 [30.0–30.0] |
| Effort | 70.0 [30.0–70.0] |
| Performance | 30.0 [10.0–30.0] |
| Frustration | 60.0 [40.0–70.0] |
| Overall Workload Score | 31.7 [31.7–46.7] |
References
- Tariq, R.A.; Vashisht, R.; Sinha, A.; Scherbak, Y. Medication dispensing errors and prevention. In StatPearls; StatPearls Publishing: Treasure Island, FL, USA, 2025. Available online: https://www.ncbi.nlm.nih.gov/books/NBK519065/ (accessed on 2 July 2025).
- Kohn, L.T.; Corrigan, J.M.; Donaldson, M.S. To Err Is Human: Building a Safer Health System; National Academies Press: Washington, DC, USA, 2000. [Google Scholar] [CrossRef]
- World Health Organization. Medication Without Harm. Available online: https://www.who.int/initiatives/medication-without-harm (accessed on 2 July 2025).
- Ministry of Health and Welfare; Korea Institute for Healthcare Accreditation. Korean Patient Safety Incident Report 2023; Korea Institute for Healthcare Accreditation: Seoul, Republic of Korea, 2024; Available online: http://www.kops.or.kr (accessed on 2 July 2025).
- Poon, E.G.; Keohane, C.A.; Yoon, C.S.; Ditmore, M.; Bane, A.; Levtzion-Korach, O.; Moniz, T.; Rothschild, J.M.; Kachalia, A.B.; Hayes, J.; et al. Effect of Bar-Code Technology on the Safety of Medication Administration. N. Engl. J. Med. 2010, 362, 1698–1707. [Google Scholar] [CrossRef] [PubMed]
- Son, E.S.; Kim, A.Y.; Lee, Y.W.; Kim, J.S.; Song, H.S.; Kim, S.E.; Lee, A.Y.; Seok, H.J.; Hwang, K.J.; Park, S.B. Medication Errors and Considerations Related to Injectable Drugs. J. Korean Soc. Health-Syst. Pharm. 2003, 20, 220–224. [Google Scholar]
- Tiu, M.G.; Ballarta, P.J.; De Leon, K.A.; Isulat, T.; Narvaez, R.A.; Salas, M. Impact on Use of Barcode Scanners in Medication Administration: An Integrative Review. Can. J. Nurs. Inform. 2025, 20, 1. Available online: https://cjni.net/journal/?p=14288 (accessed on 4 July 2025).
- Koppel, R.; Wetterneck, T.; Telles, J.L.; Karsh, B.T. Workarounds to Barcode Medication Administration Systems: Their Occurrences, Causes, and Threats to Patient Safety. J. Am. Med. Inform. Assoc. 2008, 15, 408–423. [Google Scholar] [CrossRef] [PubMed]
- Bates, D.W.; Levine, D.; Syrowatka, A.; Kuznetsova, M.; Craig, K.J.T.; Rui, A.; Jackson, G.P.; Rhee, K. The potential of artificial intelligence to improve patient safety: A scoping review. npj Digit. Med. 2021, 4, 54. [Google Scholar] [CrossRef]
- Eckert, M.; Volmerg, J.S.; Friedrich, C.M. Augmented Reality in Medicine: Systematic and Bibliographic Review. JMIR Mhealth Uhealth 2019, 7, e10967. [Google Scholar] [CrossRef] [PubMed]
- Alowais, S.A.; Alghamdi, S.S.; Alsuhebany, N.; Alqahtani, T.; Alshaya, A.I.; Almohareb, S.N.; Aldairem, A.; Alrashed, M.; Bin Saleh, K.; Badreldin, H.A.; et al. Revolutionizing Healthcare: The Role of Artificial Intelligence in Clinical Practice. BMC Med. Educ. 2023, 23, 689. [Google Scholar] [CrossRef] [PubMed]
- Huo, Y.; Gang, S.; Guan, C. FCIHMRT: Feature Cross-Layer Interaction Hybrid Method Based on Res2Net and Transformer for Remote Sensing Scene Classification. Electronics 2023, 12, 4362. [Google Scholar] [CrossRef]
- Ankam, V.S.S.; Hong, G.Y.; Fong, A.C. Design of a Mixed-Reality Application to Reduce Pediatric Medication Errors in Prehospital Emergency Care. Appl. Sci. 2024, 14, 8426. [Google Scholar] [CrossRef]
- Palumbo, A. Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects. Sensors 2022, 22, 7709. [Google Scholar] [CrossRef] [PubMed]
- Microsoft. Mixed Reality Toolkit 3 (MRTK3) Overview. Microsoft Learn: Updated 10 February 2025. Available online: https://learn.microsoft.com/ko-kr/windows/mixed-reality/mrtk-unity/mrtk3-overview/ (accessed on 10 July 2025).
- Strecker, J.; García, K.; Bektaş, K.; Mayer, S.; Ramanathan, G. SOCRAR: Semantic OCR through Augmented Reality. In Proceedings of the ACM International Conference, Lisbon, Portugal, 6–8 September 2023; pp. 25–32. [Google Scholar] [CrossRef]
- NAVER Cloud Corporation. CLOVA OCR Overview. NAVER Cloud Platform Guide. Available online: https://guide.ncloud-docs.com/docs/clovaocr-overview (accessed on 2 July 2025).
- Ageitgey. Face_Recognition: The World’s Simplest Facial Recognition API for Python and the Command Line. Available online: https://github.com/ageitgey/face_recognition (accessed on 13 March 2025).
- Lunding, R.; Hubenschmid, S.; Feuchtner, T.; Grønbæk, K. ARTHUR: Authoring Human–Robot Collaboration Processes with Augmented Reality Using Hybrid User Interfaces. Virtual Real. 2025, 29, 73. [Google Scholar] [CrossRef]
- Brooke, J. SUS: A quick and dirty usability scale. Usability Eval. Ind. 1995, 189, 4–7. Available online: https://www.researchgate.net/publication/228593520_SUS_A_quick_and_dirty_usability_scale (accessed on 20 August 2025).
- Loorbach, N.; Peters, O.; Karreman, J.; Steehouder, M. Validation of the Instructional Materials Motivation Survey (IMMS) in a Self-Directed Instructional Setting Aimed at Working with Technology. Br. J. Educ. Technol. 2015, 46, 204–218. [Google Scholar] [CrossRef]
- Kim, H.K.; Park, J.; Choi, Y.; Choe, M. Virtual Reality Sickness Questionnaire (VRSQ): Motion Sickness Measurement Index in a Virtual Reality Environment. Appl. Ergon. 2018, 69, 66–73. [Google Scholar] [CrossRef] [PubMed]
- Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Adv. Psychol. 1988, 52, 139–183. [Google Scholar] [CrossRef]
- Bangor, A.; Kortum, P.T.; Miller, J.T. An Empirical Evaluation of the System Usability Scale. Int. J. Hum. Comput. Interact. 2008, 24, 574–594. [Google Scholar] [CrossRef]
- Huang, B.; Hew, K.F. Implementing a Theory-Driven Gamification Model in Higher Education Flipped Courses: Effects on Out-of-Class Activity Completion and Quality of Artifacts. Comput. Educ. 2018, 125, 254–272. [Google Scholar] [CrossRef]
- Johnston, M.; O’Mahony, M.; O’Brien, N.; Connolly, M.; Iohom, G.; Kamal, M.; Shehata, A.; Shorten, G. The Feasibility and Usability of Mixed Reality Teaching in a Hospital Setting Based on Self-Reported Perceptions of Medical Students. BMC Med. Educ. 2024, 24, 701. [Google Scholar] [CrossRef] [PubMed]
- Connolly, M.; Iohom, G.; O’Brien, N.; Volz, J.; O’Muircheartaigh, A.; Serchan, P.; Biculescu, A.; Gadre, K.G.; Soare, C.; Griseto, L.; et al. Delivering Clinical Tutorials to Medical Students Using the Microsoft HoloLens 2: A Mixed-Methods Evaluation. BMC Med. Educ. 2024, 24, 498. [Google Scholar] [CrossRef]
- Cecotti, H.; Huisinga, L.; Peláez, L.G. Fully Immersive Learning with Virtual Reality for Assessing Students in Art History. Virtual Real. 2024, 28, 33. [Google Scholar] [CrossRef]








Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Vieira, N.L.; Kim, S.J.; Ahn, S.; Yoon, J.S.; Park, S.H.; Hong, J.H.; Kang, M.-J.; Kim, I.; Son, M.H.; Cha, W.C.; et al. Development and Evaluation of a Next-Generation Medication Safety Support System Based on AI and Mixed Reality: A Study from South Korea. Appl. Sci. 2025, 15, 12002. https://doi.org/10.3390/app152212002
Vieira NL, Kim SJ, Ahn S, Yoon JS, Park SH, Hong JH, Kang M-J, Kim I, Son MH, Cha WC, et al. Development and Evaluation of a Next-Generation Medication Safety Support System Based on AI and Mixed Reality: A Study from South Korea. Applied Sciences. 2025; 15(22):12002. https://doi.org/10.3390/app152212002
Chicago/Turabian StyleVieira, Nathan Lucien, Su Jin Kim, Sangah Ahn, Ji Sim Yoon, Sook Hyun Park, Jeong Hee Hong, Min-Jeoung Kang, Il Kim, Meong Hi Son, Won Chul Cha, and et al. 2025. "Development and Evaluation of a Next-Generation Medication Safety Support System Based on AI and Mixed Reality: A Study from South Korea" Applied Sciences 15, no. 22: 12002. https://doi.org/10.3390/app152212002
APA StyleVieira, N. L., Kim, S. J., Ahn, S., Yoon, J. S., Park, S. H., Hong, J. H., Kang, M.-J., Kim, I., Son, M. H., Cha, W. C., & Yoo, J. (2025). Development and Evaluation of a Next-Generation Medication Safety Support System Based on AI and Mixed Reality: A Study from South Korea. Applied Sciences, 15(22), 12002. https://doi.org/10.3390/app152212002

