Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric
Abstract
:1. Introduction
- Explore the interrater reliability of the rubric;
- Explore the test re-test reliability of the rubric;
- Recommend modifications to the rubric based on the reliability analyses.
2. Materials and Methods
2.1. Rubric
2.2. Data Analysis
2.2.1. Interrater Reliability Analyses
- Overall percentage agreement of markers, for each item across all combined cases.
- Intra-class Correlation Coefficient (ICC) test using one-way random effects model, absolute agreement, multiple raters, to analyze the IRR of overall scores (i.e., marks out of 24) of the three markers (tutor, consumer, and student),
- Fleiss’ Kappa was used to analyze the IRR of pass/fail categories across the three markers, as it is an appropriate measure of IRR for categorical data. Fleiss’ Kappa was also appropriate as this study involved more than two markers, and because this statistic “does not assume that the same raters have assessed all items” [34], as in the current study, different tutors, consumers, and students participated in the interactions, but each role-play was marked by one tutor, one consumer, and one student.
2.2.2. Test Re-Test Reliability Analyses
3. Results
4. Discussion
Strengths and Limitations
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Case Number (Year) | Case 1 (2016) | Case 2 (2016) | Case 3 (2017) | Case 4 (2017) | Case 5 (2018) | Case 6 (2018) |
---|---|---|---|---|---|---|
Topic | Depressive symptoms without suicidal thoughts. | Depressive symptoms with suicidal thoughts. | Depressive symptoms with suicidal thoughts. | Anxiety/depressive symptoms without suicidal thoughts. | Depressive symptoms with suicidal thoughts. | Anxiety/depressive symptoms with suicidal thoughts. |
Reason for pharmacy visit | Feeling depressed for the past two months. | Request to fill antidepressant repeat. | Request multivitamin for energy. | Request sleeping tablet for insomnia. | Request multivitamin for energy. | Request to fill antidepressant repeat. |
Symptoms | Teary and stressed. Severe sleep deprivation (first baby, three months old, unable to sleep for more than half an hour). | Deteriorating mood for the past two months after taking antidepressant (fluoxetine). Stopped taking fluoxetine a few days ago. Poor appetite. Little motivation for daily activities. | Flat and lethargic since recent divorce. Unable to get out of bed in the morning. Little motivation to work. | Insomnia. | Flat and lethargic. Low energy to get out of bed in the morning. Little motivation to go to work. | Feeling worse over the past few weeks and questioning whether life is worth it. Low motivation for self-care. Poor appetite. |
Medical history | Previous depression diagnosis at 21. History of antidepressant use & psychotherapy. | Recent diagnosis of depression. Doctor prescribed fluoxetine last month to help with the mood. | No significant medical history. | Had panic attacks as a teenager. | History of depression in early 20s. Went through psychological therapy for months. | Citalopram for anxiety disorder for years. |
Social history | New immigrant from America. Social isolation. Little support from partner (at work). | Recent financial struggles due to job cuts three months ago. Social isolation (partner at work; two daughters have moved out). | Divorced three months ago.Close to sister. | Has two young children (12 months and 22 months). Supportive parents and partner (working two jobs). | Recent divorce. Closer to sister. | Recent financial issues (house). Recent loss of employment (isolation). Supportive partner but rarely home (work). |
Suicidal thoughts | No. | Current suicidal thoughts with no current plan. No previous attempt. | Current suicidal thoughts with no current plan. No previous attempt. | No. | Current suicidal thoughts with plan (considers jumping off the balcony). No previous attempt. | Current suicidal thoughts with plan (taking all citalopram at once). |
Alcohol/drug use | ½ bottle of red wine each night. | Taking sleeping tablets (doxylamine) every night. A whole bottle of wine on some nights. | Recent increase of alcohol intake | 1–2 glasses of wine each night. | Recent increase of alcohol intake (one bottle of wine per night). | Taking sleeping tablets (doxylamine) every night. Recent increase in alcohol intake (2–3 of partner’s beers every day). |
Says | “I can’t do this anymore”, “My baby would be better off without me.” | “Everyone would be better off without me. I don’t see a point in living anymore.” | “I just feel like it’s not worth it anymore”, “There’s no point in life.” | “It’s just too hard”, “I don’t know if I can keep doing this.” | “I just feel like it’s not worth it anymore”, “There’s no point in life.” | “Everyone would be better off without me. I don’t see a point in living anymore.” |
Appendix B
- Actions shaded in GREY must be performed by the participant to pass the case.
- The participant must receive a score of at least 10/20 AND perform all actions shaded in GREY to pass the case.
- Phrases and actions that are italicized are merely suggestions and examples, and do not need to be articulated by the participant word-for-word.
- Phrases and actions that are underlined must be performed by the participant, to pass the item.
Item | Action | Full Marks (2) | Partial Marks (1) | NO Marks (0) |
---|---|---|---|---|
1 | Approaches the consumer appropriately
| |||
2 | Provides a comfortable setting for the consumer to talk
| |||
3 | Listens and communicates non-judgmentally
| |||
4 | Asks appropriate open-ended questions
| |||
5 | Asks if the consumer is having suicidal thoughts • Asks directly (e.g., Are you thinking of killing yourself/ending your life/suicide?). If the participant asks using indirect language, only, (e.g., harming/hurting yourself), then only give partial marks (1). * Participant MUST assess for suicide risk to pass the case. | |||
6 | Asks appropriate follow-up questions in relation to consumer’s suicidal thoughts
If the participant asks about the plan, only, give one partial mark. If the participant asks about the plan, and at least one other question, give the full two marks. | |||
7 | Gives reassurance and appropriate information
| |||
8 | Displays empathy
| |||
9 | Takes appropriate action
Additional examples of follow-up actions
| |||
10 | Good non-verbal communication
| |||
TOTAL (out of 20) |
Appendix C
- Actions shaded in GREY must be performed by the participant to pass the case.
- The participant must receive a score of at least 10/20 AND perform all actions shaded in GREY to pass the case.
- Phrases and actions that are italicized are merely suggestions and examples, and do not need to be articulated by the participant word-for-word.
Item | Action | Full Marks (2) | Partial Marks (1) | NO Marks (0) |
---|---|---|---|---|
1 | Approaches the consumer appropriately
| |||
2 | Provides a comfortable setting for the consumer to talk
| |||
3 | Listens and communicates non-judgmentally
| |||
4 | Asks appropriate open-ended questions
| |||
5 | Asks if the consumer is having suicidal thoughts
* Participant MUST assess for suicide risk to pass the case. | |||
6 | Gives reassurance and appropriate information
| |||
7 | Displays empathy
| |||
8 | Takes appropriate action
| |||
9 | Encourages self-help
| |||
10 | Good non-verbal communication
| |||
TOTAL (out of 20) |
References
- Rehm, J.; Shield, K.D. Global Burden of Disease and the Impact of Mental and Addictive Disorders. Curr. Psychiatry Rep. 2019, 21, 10. [Google Scholar] [CrossRef] [PubMed]
- World Health Organisation. Suicide Data. Available online: https://www.who.int/mental_health/prevention/suicide/suicideprevent/en/ (accessed on 19 June 2016).
- World Health Organisation. World Health Report: Mental Disorders Affect One in Four People. Available online: https://www.who.int/whr/2001/media_centre/press_release/en/ (accessed on 19 November 2001).
- Australian Bureau of Statistics. National Survey of Mental Health and Well-Being: Summary of Results; ACT: Canberra, Australia, 2007. [Google Scholar]
- International Pharmaceutical Federation (FIP). Focus on Mental Health: The Contribution of the Pharmacist; International Pharmaceutical Federation: The Hague, The Netherlands, 2015. [Google Scholar]
- Pharmaceutical Society of Australia. Mental Health Care Project: A Framework for Pharmacists as Partners in Mental Health Care; Pharmaceutical Society of Australia: Canberra, Australia, 2013. [Google Scholar]
- Boukouvalas, E.; El-Den, S.; Murphy, A.L.; Salvador-Carulla, L.; O’Reilly, C.L. Exploring Health Care Professionals’ Knowledge of, Attitudes Towards, and Confidence in Caring for People at Risk of Suicide: A Systematic Review. Arch. Suicide Res. 2019, 2019, 1–31. [Google Scholar] [CrossRef] [PubMed]
- El-Den, S.; Moles, R.; Choong, H.J.; O’Reilly, C. Mental Health First Aid training and assessment among university students: A systematic review. J. Am. Pharm. Assoc. 2020. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kitchener, B.A.; Jorm, A.F.; Kelly, C.M. Mental Health First Aid Manual, 4th ed.; Mental Health First Aid Australia: Parkville, VIC, Australia, 2017. [Google Scholar]
- Morgan, A.J.; Ross, A.; Reavley, N.J. Systematic review and meta-analysis of mental health first aid training: Effects on knowledge, stigma, and helping behaviour. PLoS ONE 2018, 13. [Google Scholar] [CrossRef] [PubMed]
- Hadlaczky, G.; Hökby, S.; Mkrtchian, A.; Carli, V.; Wasserman, D. Mental Health First Aid is an effective public health intervention for improving knowledge, attitudes, and behaviour: A meta-analysis. Int. Rev. Psychiatry 2014, 26, 467–475. [Google Scholar] [CrossRef] [PubMed]
- O’Reilly, C.; Bell, J.; Kelly, P.; Chen, T. Impact of mental health first aid training on pharmacy students’ knowledge, attitudes and self-reported behaviour: A controlled trial. Aust. N. Z. J. Psychiatry 2011, 45, 549–557. [Google Scholar] [CrossRef]
- Lipson, S.K.; Speer, N.; Brunwasser, S.; Hahn, E.; Eisenberg, D. Gatekeeper training and access to mental health care at universities and colleges. J. Adolesc. Health 2014, 55, 612–619. [Google Scholar] [CrossRef] [Green Version]
- Reavley, N.J.; Morgan, A.J.; Fischer, J.A.; Kitchener, B.; Bovopoulos, N.; Jorm, A.F. Effectiveness of eLearning and blended modes of delivery of Mental Health First Aid training in the workplace: Randomised controlled trial. BMC Psychiatry 2018, 18, 312. [Google Scholar] [CrossRef] [Green Version]
- Seyed Nematian, S.S.; Palenik, C.J.; Mirmasoudi, S.K.; Hatam, N.; Askarian, M. Comparing knowledge and self-reported hand hygiene practices with direct observation among Iranian hospital nurses. Am. J. Infect. Control 2017, 45, 65–67. [Google Scholar] [CrossRef]
- Prince, S.A.; Adamo, K.B.; Hamel, M.E.; Hardt, J.; Gorber, S.C.; Tremblay, M. A comparison of direct versus self-report measures for assessing physical activity in adults: A systematic review. Int. J. Behav. Nutr. Phys. Act. 2008, 5, 56. [Google Scholar] [CrossRef] [Green Version]
- Luckie, K.; Saini, B.; Galstaun, V.; Kritikos, V.; Collins, J.C.; Moles, R.J. The effectiveness of an online training programme to prepare teachers to provide asthma first aid. J. Paediatr. Child Health 2018, 54, 1348–1352. [Google Scholar] [CrossRef] [PubMed]
- Girard, J.; Cohn, J. A Primer on Observational Measurement. Assessment 2016, 23, 404–413. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Scott, I.M. Beyond ‘driving’: The relationship between assessment, performance and learning. Med. Educ. 2020, 54, 54–59. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Boukouvalas, E.; El-Den, S.; Chen, T.; Moles, R.; Saini, B.; Bell, A.; O’Reilly, C. Confidence and attitudes of pharmacy students towards suicidal crises: Patient simulation using people with a lived experience. Soc. Psychiatry Psychiatr. Epidemiol. 2018, 53, 1185–1195. [Google Scholar] [CrossRef]
- El-Den, S.; Chen, T.F.; Moles, R.J.; O'Reilly, C. Assessing Mental Health First Aid Skills Using Simulated Patients. Am. J. Pharm. Educ. 2018, 82, 6222. [Google Scholar] [CrossRef]
- Rollnick, S.; Butler, C.C.; Kinnersley, P.; Gregory, J.; Mash, B. Motivational interviewing. BMJ 2010, 340, 1900. [Google Scholar] [CrossRef]
- Xu, T.; de Almeida Neto, A.C.; Moles, R.J. Simulated caregivers: Their feasibility in educating pharmacy staff to manage children’s ailments. Int. J. Clin. Pharm. 2012, 34, 587–595. [Google Scholar] [CrossRef]
- Miller, W.R. Motivational Interviewing with Problem Drinkers. Behav. Psychother. 1983, 11, 147–172. [Google Scholar] [CrossRef] [Green Version]
- Stellmack, M.A.; Konheim-Kalkstein, Y.L.; Manor, J.E.; Massey, A.R.; Schmitz, J.A.P. An Assessment of Reliability and Validity of a Rubric for Grading APA-Style Introductions. Teach. Psychol. 2009, 36, 102–107. [Google Scholar] [CrossRef]
- Moskal, B.M.; Leydens, J.A. Scoring rubric development: Validity and reliability. Pract. Assess. Res. Eval. 2000, 7, 6. [Google Scholar]
- Brown, M.C.; Conway, J.; Sorensen, T.D. Development and Implementation of a Scoring Rubric for Aseptic Technique. Am. J. Pharm. Educ. 2006, 70, 133. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Rossetto, A.; Jorm, A.; Reavley, N. Examining predictors of help giving toward people with a mental illness: Results from a national survey of Australian adults. SAGE Open 2014, 4, 2158244014537502. [Google Scholar] [CrossRef]
- O’Reilly, C.; Moles, R.; Boukouvalas, E.; El-Den, S. Assessing students’ mental health crisis skills via consumers with lived experience: A qualitative evaluation. J. Ment. Health Train. Educ. Pract. 2019, 14, 361–371. [Google Scholar] [CrossRef]
- One Door Mental Health. Available online: https://www.onedoor.org.au/ (accessed on 20 December 2019).
- American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders: DSM-V, 5th ed.; American Psychiatric Association: Washington, DC, USA, 2013. [Google Scholar]
- IBM Corp. IBM SPSS Statistics for Windows, Version 24.0; IBM Corp: Armonk, NY, USA, 2016. [Google Scholar]
- Collins, J.C.; Chan, M.Y.; Schneider, C.R.; Yan, L.R.; Moles, R.J. Measurement of the reliability of pharmacy staff and simulated patient reports of non-prescription medicine requests in community pharmacies. Res. Soc. Adm. Pharm. 2020. [Google Scholar] [CrossRef] [PubMed]
- Gisev, N.; Bell, J.S.; Chen, T.F. Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Res. Soc. Adm. Pharm. 2013, 9, 330–338. [Google Scholar] [CrossRef] [PubMed]
- Miller, G.E. The assessment of clinical skills/competence/performance. Acad. Med. 1990, 65, 63–67. [Google Scholar] [CrossRef]
- Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [Green Version]
- Karras, D.J. Statistical methodology: II. Reliability and variability assessment in study design, Part A. Acad. Emerg. Med. Off. J. Soc. Acad. Emerg. Med. 1997, 4, 64–71. [Google Scholar] [CrossRef]
- Jonsson, A.; Svingby, G. The use of scoring rubrics: Reliability, validity and educational consequences. Educ. Res. Rev. 2007, 2, 130–144. [Google Scholar] [CrossRef]
- Lasater, K. Clinical Judgment Development: Using Simulation to Create an Assessment Rubric. J. Nurs. Educ. 2007, 46, 496–503. [Google Scholar] [CrossRef]
Action | Yes (2) | Partial (1) | NO (0) | N/A | |
---|---|---|---|---|---|
1 | Approaches the patient appropriately | ||||
2 | Provides a comfortable setting for the patient to talk | ||||
3 | Listens non-judgmentally | ||||
4 | Asks appropriate open-ended questions (e.g., how long have you been feeling this way? Have you sought any professional help? How are you coping?) | ||||
5 | Asks if the patient is having suicidal thoughts | ||||
6 | Asks appropriate follow-up questions (e.g., do they have a plan? Have they attempted before? Are they taking alcohol/drugs?) | ||||
7 | Gives reassurance and appropriate information (e.g., Tells patient they care and want to help, state thoughts of suicide are often associated with treatable mental illness, tells person thoughts of suicide are common and do not have to be acted on) | ||||
8 | Displays empathy | ||||
9 | Takes appropriate action (does not leave the patient alone, connects with professional help, such as Lifeline or the Suicide Call Back Service, connect with a family member to pick up or immediately see the GP) | ||||
10 | Encourages self-help (e.g., looking after self, support groups for those who lost a partner) | ||||
11 | Good non-verbal communication | ||||
12 | Appropriate follow-up actions (e.g., takes phone number to call for follow-up, continual willingness to help) |
Item Number | Overall Percentage Agreement |
---|---|
1 (Approach) | 74.30% |
2 (Setting) | 60.77% |
3 (Listen) | 75.70% |
4 (Ask) | 57.99% |
5 (Suicide assessment) | 94.45% |
6 (Suicide follow-up) | 64.93% |
7 (Reassurance) | 50.39% |
8 (Empathy) | 56.59% |
9 (Action) | 61.46% |
10 (Self-help) | 52.08% |
11 (Non-verbal) | 61.81% |
12 (Follow-up) | 39.93% |
Total Case N | ICC Average | 95% Confidence Interval | |
---|---|---|---|
Case 1 | 20 | 0.74 | 0.46–0.89 |
Case 2 | 18 | 0.75 | 0.45–0.90 |
Case 3 | 16 | 0.80 | 0.56–0.93 |
Case 4 | 5 | 0.09 | −1.03–0.86 |
Case 5 | 19 | 0.57 | 0.13–0.82 |
Case 6 | 18 | 0.40 | −0.24–0.75 |
Combined cases | 96 | 0.70 | 0.58–0.80 |
Total Case N | Fleiss’ Kappa | p Value | |
---|---|---|---|
Case 1 | 20 | 0.82 | <0.001 |
Case 2 | 18 | 0.38 | 0.005 |
Case 3 | 16 | 0.73 | 0.000 |
Case 4 | 5 | 0.66 | 0.011 |
Case 5 | 19 | −0.06 | 0.675 |
Case 6 | 18 | −0.04 | 0.777 |
Overall | 96 | 0.57 | <0.001 |
Total Case N | Mean | SD | Pearson Correlation | p Value | |
---|---|---|---|---|---|
Case 3 test | 16 | 18.69 | 3.071 | 0.96 | <0.001 |
Case 3 re-test | 16 | 17.88 | 2.941 | ||
Case 4 test | 5 | 18.00 | 1.414 | 0.82 | 0.093 |
Case 4 re-test | 5 | 16.80 | 2.168 | ||
Case 5 test | 19 | 17.68 | 3.019 | 0.87 | 0.000 |
Case 5 re-test | 19 | 17.53 | 2.547 | ||
Case 6 test | 18 | 17.94 | 1.893 | 0.77 | 0.000 |
Case 6 re-test | 18 | 17.94 | 1.765 | ||
Combined cases test | 58 | 18.07 | 2.595 | 0.87 | 0.000 |
Combined cases re-test | 58 | 17.69 | 2.386 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
El-Den, S.; Moles, R.J.; Zhang, R.; O’Reilly, C.L. Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric. Pharmacy 2021, 9, 28. https://doi.org/10.3390/pharmacy9010028
El-Den S, Moles RJ, Zhang R, O’Reilly CL. Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric. Pharmacy. 2021; 9(1):28. https://doi.org/10.3390/pharmacy9010028
Chicago/Turabian StyleEl-Den, Sarira, Rebekah J. Moles, Randi Zhang, and Claire L. O’Reilly. 2021. "Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric" Pharmacy 9, no. 1: 28. https://doi.org/10.3390/pharmacy9010028