Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives
Abstract
:1. Introduction
2. Methods
2.1. ChatGPT Input and Data Sources
2.2. Response Generation with ChatGPT
2.3. Assessing ChatGPT’s Response and NL Test Performance
2.4. Statistical Analysis
3. Results
3.1. The Achievement Level of NL Indicators
3.2. Evaluation of Response Quality
3.3. ChatGPT’s Performance on the NL Test
4. Discussion
Limitations
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Wang, F.; Preininger, A. AI in Health: State of the Art, Challenges, and Future Directions. Yearb. Med. Inform. 2019, 28, 16–26. [Google Scholar] [CrossRef]
- Ghadban, Y.A.; Lu, H.; Adavi, U.; Sharma, A.; Gara, S.; Das, N.; Kumar, B.; John, R.; Devarsetty, P.; Hirst, J.E. Transforming Healthcare Education: Harnessing Large Language Models for Frontline Health Worker Capacity Building using Retrieval-Augmented Generation. medRxiv 2023. [Google Scholar] [CrossRef]
- Gangavarapu, A. LLMs: A Promising New Tool for Improving Healthcare in Low-Resource Nations. In Proceedings of the 2023 IEEE Global Humanitarian Technology Conference (GHTC), Villanova, PA, USA, 12–15 October 2023; pp. 252–255. [Google Scholar]
- Hopkins, A.; Logan, J.; Kichenadasse, G.; Sorich, M. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023, 7, pkad010. [Google Scholar] [CrossRef]
- Zhai, X. ChatGPT User Experience: Implications for Education. 2022. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4312418 (accessed on 1 April 2024).
- Shao, C.-Y.; Li, H.; Liu, X.-L.; Li, C.; Yang, L.-Q.; Zhang, Y.-J.; Luo, J.; Zhao, J. Appropriateness and Comprehensiveness of Using ChatGPT for Perioperative Patient Education in Thoracic Surgery in Different Language Contexts: Survey Study. Interact. J. Med. Res. 2023, 12, e46900. [Google Scholar] [CrossRef]
- Mondal, H.; Mondal, S.; Podder, I. Using ChatGPT for Writing Articles for Patients’ Education for Dermatological Diseases: A Pilot Study. Indian Dermatol. Online J. 2023, 14, 482–486. [Google Scholar] [CrossRef]
- Nakhleh, A.; Spitzer, S.; Shehadeh, N. ChatGPT’s Response to the Diabetes Knowledge Questionnaire: Implications for Diabetes Education. Diabetes Technol. Ther. 2023, 25, 571–573. [Google Scholar] [CrossRef]
- Van Bulck, L.; Moons, P. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value and danger of ChatGPT-generated responses to health questions. Eur. J. Cardiovasc. Nurs. 2023, 23, 95–98. [Google Scholar] [CrossRef]
- Hung, Y.-C.; Chaker, S.C.; Sigel, M.; Saad, M.; Slater, E.D. Comparison of Patient Education Materials Generated by Chat Generative Pre-Trained Transformer Versus Experts: An Innovative Way to Increase Readability of Patient Education Materials. Ann. Plast. Surg. 2023, 91, 409–412. [Google Scholar] [CrossRef]
- Shah, Y.B.; Ghosh, A.; Hochberg, A.R.; Rapoport, E.; Lallas, C.D.; Shah, M.S.; Cohen, S.D. Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health. Urol. Pract. 2024, 11, 87–94. [Google Scholar] [CrossRef]
- Ayoub, N.F.; Lee, Y.-J.; Grimm, D.; Divi, V. Head-to-Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngol.–Head Neck Surg. 2023, 170, 1484–1491. [Google Scholar] [CrossRef]
- Qarajeh, A.; Tangpanithandee, S.; Thongprayoon, C.; Suppadungsuk, S.; Krisanapan, P.; Aiumtrakul, N.; Garcia Valencia, O.A.; Miao, J.; Qureshi, F.; Cheungpasitporn, W. AI-Powered Renal Diet Support: Performance of ChatGPT, Bard AI, and Bing Chat. Clin. Pract. 2023, 13, 1160–1172. [Google Scholar] [CrossRef]
- Göktaş, L.S. The role of ChatGPT in vegetarian menus. Tour. Recreat. 2023, 5, 79–86. [Google Scholar] [CrossRef]
- Niszczota, P.; Rybicka, I. The credibility of dietary advice formulated by ChatGPT: Robo-diets for people with food allergies. Nutrition 2023, 112, 112076. [Google Scholar] [CrossRef]
- Abraham, S.; Noriega, B.R.; Shin, J.Y. College students eating habits and knowledge of nutritional requirements. J. Nutr. Hum. Health 2018, 2, 13–17. [Google Scholar] [CrossRef]
- Yang, S.C.; Luo, Y.F.; Chiang, C.-H. Electronic health literacy and dietary behaviors in Taiwanese college students: Cross-sectional study. J. Med. Internet Res. 2019, 21, e13140. [Google Scholar] [CrossRef]
- Liao, L.-L.; Lai, I.-J.; Chang, L.-C. Nutrition literacy is associated with healthy-eating behaviour among college students in Taiwan. Health Educ. J. 2019, 78, 756–769. [Google Scholar] [CrossRef]
- Racette, S.B.; Deusinger, S.S.; Strube, M.J.; Highstein, G.R.; Deusinger, R.H. Changes in weight and health behaviors from freshman through senior year of college. J. Nutr. Educ. Behav. 2008, 40, 39–42. [Google Scholar] [CrossRef]
- Stok, F.M.; Renner, B.; Clarys, P.; Lien, N.; Lakerveld, J.; Deliens, T. Understanding eating behavior during the transition from adolescence to young adulthood: A literature review and perspective on future research directions. Nutrients 2018, 10, 667. [Google Scholar] [CrossRef]
- Contento, I.R. Nutrition education: Linking research, theory, and practice. Asia Pac. J. Clin. Nutr. 2008, 17, 176–179. [Google Scholar]
- Ngo, T.T.A. The perception by university students of the use of ChatGPT in education. Int. J. Emerg. Technol. Learn. 2023, 18, 4. [Google Scholar] [CrossRef]
- Liao, L.L.; Lai, I.J. Construction of Nutrition Literacy Indicators for College Students in Taiwan: A Delphi Consensus Study. J. Nutr. Educ. Behav. 2017, 49, 734–742. [Google Scholar] [CrossRef]
- Lai, I.J.; Chang, L.C.; Lee, C.K.; Liao, L.L. Preliminary evaluation of a scenario-based nutrition literacy online programme for college students: A pilot study. Public Health Nutr. 2023, 26, 3190–3201. [Google Scholar] [CrossRef]
- Sun, Y.; Zhang, Y.; Gwizdka, J.; Trace, C.B. Consumer Evaluation of the Quality of Online Health Information: Systematic Literature Review of Relevant Criteria and Indicators. J. Med. Internet Res. 2019, 21, e12522. [Google Scholar] [CrossRef]
- Liao, L.-L.; Lai, I.-J.; Shih, S.-F.; Chang, L.-C. Development and validation of the nutrition literacy measure for Taiwanese college students. Taiwan J. Public Health 2018, 37, 582. [Google Scholar] [CrossRef]
- Velpini, B.; Vaccaro, G.; Vettori, V.; Lorini, C.; Bonaccorsi, G. What Is the Impact of Nutrition Literacy Interventions on Children’s Food Habits and Nutrition Security? A Scoping Review of the Literature. Int. J. Environ. Res. Public Health 2022, 19, 3839. [Google Scholar] [CrossRef]
- Doustmohammadian, A.; Omidvar, N.; Shakibazadeh, E. School-based interventions for promoting food and nutrition literacy (FNLIT) in elementary school children: A systematic review protocol. Syst. Rev. 2020, 9, 87. [Google Scholar] [CrossRef]
- Azevedo, J.; Padrão, P.; Gregório, M.J.; Almeida, C.; Moutinho, N.; Lien, N.; Barros, R. A web-based gamification program to improve nutrition literacy in families of 3-to 5-year-old children: The Nutriscience Project. J. Nutr. Educ. Behav. 2019, 51, 326–334. [Google Scholar] [CrossRef]
- Achterberg, C.; Miller, C. Is One Theory Better than Another in Nutrition Education? A Viewpoint: More Is Better. J. Nutr. Educ. Behav. 2004, 36, 40–42. [Google Scholar] [CrossRef]
- Moons, P.; Van Bulck, L. Using ChatGPT and Google Bard to improve the readability of written patient information: A proof of concept. Eur. J. Cardiovasc. Nurs. 2024, 23, 122–126. [Google Scholar] [CrossRef]
- Marcason, W. Dietitian, Dietician, or Nutritionist? J. Acad. Nutr. Diet. 2015, 115, 484. [Google Scholar] [CrossRef]
- Chatelan, A.; Clerc, A.; Fonta, P.-A. ChatGPT and Future Artificial Intelligence Chatbots: What may be the Influence on Credentialed Nutrition and Dietetics Practitioners? J. Acad. Nutr. Diet. 2023, 123, 1525–1531. [Google Scholar] [CrossRef]
- Ponzo, V.; Goitre, I.; Favaro, E.; Merlo, F.D.; Mancino, M.V.; Riso, S.; Bo, S. Is ChatGPT an Effective Tool for Providing Dietary Advice? Nutrients 2024, 16, 469. [Google Scholar] [CrossRef]
- Garcia, M.B. ChatGPT as a virtual dietitian: Exploring its potential as a tool for improving nutrition knowledge. Appl. Syst. Innov. 2023, 6, 96. [Google Scholar] [CrossRef]
- Liu, C.; Bao, X.; Zhang, H.; Zhang, N.; Hu, H.; Zhang, X.; Yan, M. Improving chatgpt prompt for code generation. arXiv 2023, arXiv:2305.08360. [Google Scholar] [CrossRef]
- Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med. Educ. 2023, 9, e45312. [Google Scholar] [CrossRef]
- Nikdel, M.; Ghadimi, H.; Tavakoli, M.; Suh, D.W. Assessment of the Responses of the Artificial Intelligence–based Chatbot ChatGPT-4 to Frequently Asked Questions About Amblyopia and Childhood Myopia. J. Pediatr. Ophthalmol. Strabismus 2024, 61, 86–89. [Google Scholar] [CrossRef]
- Wang, Y.-M.; Shen, H.-W.; Chen, T.-J. Performance of ChatGPT on the pharmacist licensing examination in Taiwan. J. Chin. Med. Assoc. 2023, 86, 653–658. [Google Scholar] [CrossRef]
- Hewko, S.; Oyesegun, A.; Clow, S.; VanLeeuwen, C. High turnover in clinical dietetics: A qualitative analysis. BMC Health Serv. Res. 2021, 21, 25. [Google Scholar] [CrossRef]
Not Achieved at All | Partially Achieved | Fully Achieved | |
---|---|---|---|
Scenario (No. of Indicators) | n (%) | n (%) | n (%) |
S1: Balanced Eating (8) | 137 (57.08) | 77 (32.08) | 26 (10.38) |
S2: Fit Weight (8) | 151 (62.92) | 71 (29.58) | 18 (7.50) |
S3: Dining Out Well (5) | 65 (43.33) | 46 (30.67) | 39 (26.00) |
S4: Fewer Processed Foods (3) | 18 (20.00) | 40 (44.44) | 32 (35.56) |
S5: Limit Sugary Drinks (4) | 76 (63.33) | 34 (28.33) | 10 (8.33) |
S1: Balanced Eating | S2: Fit Weight | S3: Dining Out Well | S4: Fewer Processed Foods | S5: Limit Sugary Drinks | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Criteria | Range | M (SD) | Median | Range | M (SD) | Median | Range | M (SD) | Median | Range | M (SD) | Median | Range | M (SD) | Median |
Accuracy | 4–10 | 7.90 (1.73) | 8 | 3–10 | 7.53 (1.94) | 8 | 3–10 | 7.47 (2.00) | 8 | 2–10 | 7.90 (1.85) | 8 | 2–10 | 6.60 (2.14) | 7 |
Currency | 3–10 | 7.70 (1.73) | 8 | 3–10 | 7.27 (1.91) | 7 | 2–10 | 7.27 (2.29) | 8 | 2–10 | 7.73 (1.93) | 8 | 3–10 | 6.97 (1.69) | 7 |
Completeness | 3–10 | 7.27 (1.89) | 8 | 3–10 | 6.70 (1.93) | 6.5 | 3–10 | 6.73 (2.29) | 7 | 3–10 | 7.90 (1.73) | 8 | 3–10 | 6.80 (1.96) | 7 |
Understandability | 1–10 | 7.00 (2.20) | 7 | 1–10 | 6.23 (1.98) | 6.5 | 1–10 | 6.37 (2.36) | 7 | 3–10 | 7.40 (1.96) | 8 | 2–10 | 6.70 (2.25) | 7.5 |
Readability | 4–10 | 8.03 (1.85) | 8 | 4–10 | 8.03 (1.71) | 8 | 3–10 | 7.97 (2.04) | 8 | 3–10 | 8.27 (1.72) | 8 | 3–10 | 8.00 (1.84) | 8 |
Relevance | 1–10 | 7.43 (2.27) | 8 | 1–10 | 6.57 (2.49) | 7 | 3–10 | 6.83 (2.17) | 7 | 3–10 | 7.83 (1.90) | 8 | 3–10 | 6.83 (1.95) | 7 |
Practicality | 1–10 | 6.77 (2.37) | 7.5 | 1–10 | 6.30 (2.14) | 7 | 3–10 | 6.63 (2.19) | 7 | 3–10 | 7.67 (1.88) | 8 | 3–10 | 7.23 (1.91) | 7.5 |
Positive Opinions (Occurrences = 87, 35.95%) | Concerns (Occurrences = 155, 64.05%) |
---|---|
|
|
Items and Correct Answer | ChatGPT Responses | The Correct Rate of Taiwanese College Students [26] |
---|---|---|
10. According to Rui’s diet list today †, what do you think about his dietary choices: A. Vegetable servings are just right. B. Too many sugary drinks. C. Calories from fats are too high (T). | B | 73.6% |
11. “Research shows that being overweight or obese (i.e., BMI ≥ ___) is a major risk factor for chronic diseases such as diabetes, cardiovascular disease, and malignant tumors”. The number in the blank should be †: A. 24 (T) B. 27 C. 30 | C | 54.2% |
14. Rui spends most of his time either attending classes or returning to the dormitory to study. He hasn’t participated in any extracurricular activities and does not have a regular exercise routine. Based on Table †, Rui’s daily calorie requirement should be: A. 2450 kcal B. 2100 kcal C. 1400~1750 kcal (T) | B | 54.2% |
18. Which of the following products † has the highest total calorie content? A. Product: Lemon-filled cookies, Weight: 80 g (T) B. Product: Braised beef noodles, Weight: 100 g C. Product: Chocolate milk, Volume: 300 milliliters | B | 62.3% |
29. Since Rui has never had a habit of exercising in the past, he is just starting to develop an exercise routine. Which of the following exercise plans is more suitable for Rui? A. Take a 40-min walk around the campus after dinner every day with the principle of slight breathlessness and slightly increased heart rate (T). B. Take a 30-min walk around the campus after dinner every day with the principle of no breathlessness and no increased heart rate. C. Engage in more strenuous and breathless exercise at the gym for at least 60 min every day. | B | 70.1% |
Correct rates of 32 items | 84.38% | 77.40% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liao, L.-L.; Chang, L.-C.; Lai, I.-J. Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives. Nutrients 2024, 16, 1939. https://doi.org/10.3390/nu16121939
Liao L-L, Chang L-C, Lai I-J. Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives. Nutrients. 2024; 16(12):1939. https://doi.org/10.3390/nu16121939
Chicago/Turabian StyleLiao, Li-Ling, Li-Chun Chang, and I-Ju Lai. 2024. "Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives" Nutrients 16, no. 12: 1939. https://doi.org/10.3390/nu16121939
APA StyleLiao, L. -L., Chang, L. -C., & Lai, I. -J. (2024). Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives. Nutrients, 16(12), 1939. https://doi.org/10.3390/nu16121939