Evaluating the Quality of Competency Assessment in Pharmacy: A Framework for Workplace Learning
Abstract
:1. Introduction
2. Experimental Section
2.1. Data Collection
Criterion | Description |
---|---|
Acceptability | The extent to which all stakeholders (e.g., students, teachers, employers) approve of the assessment criteria and the way the CAP is carried out |
Authenticity | The degree of resemblance of a CAP to the future workplace |
Cognitive-Complexity | The extent to which the assessments reflect the presence of the cognitive skills and enable the judgment of thinking processes |
Comparability | The extent to which a CAP is conducted consistently and responsibly |
Costs and Efficiency | The time and resources needed to develop and carry out the CAP, compared to the benefits |
Educational Consequences | The degree to which the CAP yields positive effects on learning and instruction, and the degree to which negative effects are minimized |
Fairness | The extent to which learners get a fair chance to demonstrate their competences, for example by limiting assessor bias |
Fitness for Purpose | The degree to which standards, curriculum, instruction and assessment are aligned |
Fitness for Self-Assessment | The degree to which a CAP stimulates self-regulated learning of students, including fostering self-assessment and giving and receiving feedback |
Meaningfulness | The value of the CAP for all stakeholders involved (e.g., students, teachers, employers) |
Reproducibility of Decisions | The extent to which decisions made from results of CAP are accurate and constant over situations and assessors |
Transparency | The extent to which the CAP is clear and understandable to all stakeholders (e.g., students, teachers, employers) |
2.2. Data Analysis
3. Results and Discussion
3.1. Demographic Results
Characteristic | Residents (n = 23) | Preceptors (n = 28) |
---|---|---|
Residency Experience | ||
PGY1 at the institution | 16 (70) | 11 (39) |
PGY2 at the institution | 20 (87) | 9 (32) |
PGY1 and PGY2 at the institution | 13 (57) | 9 (32) |
Completed a residency elsewhere | 7 (30) | 14 (50) |
Completed residency in 2011 | 4 (17) | N/A |
Completed residency in 2012 | 14 (61) | N/A |
Completed residency in 2013 | 15 (65) | N/A |
Currently practicing at the institution | 14 (61) | N/A |
Preceptor Experience | ||
Precepted elsewhere | N/A | 18 (64) |
Precepted less than 3 years | N/A | 4 (14) |
Precepted 3 to 10 years | N/A | 13 (46) |
Precepted more than 10 years | N/A | 11 (39) |
Assessment Method a,b | ||
Verbal Feedback | 23 (100) | 26 (93) |
Assigning Goals and Objectives to Learning Outcomes | 19 (83) | 23 (82) |
Snapshot Evaluations | 3 (13) | 8 (29) |
Mentor Relationship | 22 (96) | 18 (64) |
Quarterly Self Evaluations | 18 (78) | 13 (46) |
Quarterly Program Director Evaluations | 16 (70) | 11 (39) |
Monthly Summative Self Evaluations | 22 (96) | 20 (71) |
Monthly Summative Preceptor Evaluations | 21 (91) | 21 (75) |
Monthly Evaluation of Preceptor | 21 (91) | 21 (75) |
Learning Experience Evaluation | 10 (43) | 10 (36) |
Midpoint Evaluation | 16 (70) | 23 (82) |
Midpoint Self-Evaluation | 9 (39) | 8 (29) |
Observation in Simulated Situations | 3 (13) | 5 (18) |
Observation in Workplace | 11 (48) | 18 (64) |
Portfolio | 5 (22) | 3 (11) |
Proof of Competency Assessments | 3 (13) | 4 (14) |
Assessment Interview | 1 (4) | 2 (7) |
3.2. CAP Responses
Criterion a | N | Mean, Median (Range) | α |
---|---|---|---|
Acceptability | 51 | 69.28, 69.25 (8.25–100) | 0.90 |
The criteria I was evaluated on as a part of the residency assessment program were appropriate for my training | 51 | 75.25, 76.00 (13.00–100) | |
All methods used to evaluate me during residency were appropriate | 51 | 65.86, 68.00 (0.00–100) | |
Authenticity | 51 | 74.92, 80.13 (41.75–100) | 0.87 |
The working conditions in which I was assessed during residency resemble the conditions of the workplace | 51 | 72.49, 80.00 (0.00–100) | |
The competencies on which I was assessed during my residency program are those I need to be successful in the workplace | 51 | 72.29, 76.00 (0.00–100) | |
Cognitive Complexity | 51 | 73.09, 73.25 (31.25–100) | 0.91 |
The assessment program placed an emphasis on the thought processes involved in my process for providing patient care | 51 | 70.65, 73.00 (25.00–100) | |
My reasoning and thought process was assessed within the assessment program | 51 | 75.24, 75.00 (37.00–100) | |
Comparability | 49 | 78.62, 84.75 (25.00–100) | 0.92 |
The activities I was evaluated on were the same from month to month for a similar type of rotation | 49 | 79.41, 75.00 (37.00–100) | |
The process by which I was evaluated for each criterion was the same from month to month for a similar type of rotation | 49 | 81.67, 87.00 (25.00–100) | |
Costs and Efficiency | 51 | 64.27, 64.25 (37.00–95.25) | 0.35 |
I believe the time required for each assessment method utilized during my residency was appropriate. | 51 | 56.59, 53.00 (11.00–100) | |
I would have been willing to put in more time towards the assessment and feedback aspects of residency if I perceived them to have a greater value. | 51 | 73.84, 79.00 (4.00–100) | |
Educational Consequences | 51 | 45.26, 43.00 (10.75–96.50) | 0.66 |
The assessment process during residency motivated me to learn more during residency | 51 | 44.29, 45.00 (0.00–97.00) | |
My assessment during residency impacted the objectives of subsequent learning experiences during my residency | 51 | 43.68, 50.00 (0.00–100) | |
Fairness | 51 | 76.59, 78.88 (36.50–100) | 0.71 |
The residency assessment program during my residency was fair | 51 | 75.98, 77.00 (0.00–99.00) | |
I was evaluated with various assessment methods (i.e., verbal feedback, midpoint evaluations, portfolio) | 51 | 81.00, 87.00 (31.00–100) | |
Fitness for Purpose | 51 | 72.68, 75.50 (25.50–100) | 0.75 |
The activities I was assessed on matched stated educational goals | 51 | 76.82, 76.00 (31.00–100) | |
The assessment program covered assessment of all competency areas required in my residency training | 51 | 69.67, 75.00 (0.00–100) | |
Fitness for Self-Assessment | 51 | 69.43, 72.75 (10.50–100) | 0.77 |
The assessment methods during my residency stimulated me to self- assess my learning | 51 | 67.19, 70.00 (11.00–100) | |
I received feedback that fostered my personal development | 51 | 66.35, 70.00 (0.00–100) | |
Meaningfulness | 49 | 72.39, 75.13 (3.75–100) | 0.85 |
The assessment methods used during my residency enhanced my learning | 49 | 65.35, 70.00 (4.00–100) | |
The competencies I was assessed on are meaningful to me | 49 | 69.94, 76.00 (5.00–100) | |
Reproducibility of Decisions | 46 | 76.02, 77.88 (37.50–100) | 0.80 |
My evaluations were consistent throughout the year | 46 | 67.20, 72.50 (3.00–100) | |
I am assessed the same way in several different work situations | 46 | 83.72, 82.00 (12.00–100) | |
Transparency | 46 | 77.19, 82.50 (25.50–100) | 0.93 |
I was aware of the assessment criteria used during my residency | 46 | 79.24, 88.50 (22.00–100) | |
I was aware of the assessment methods used during my residency | 46 | 76.35, 89.50 (0.00–100) |
3.3. Discussion
Criterion | Residents (n = 23) | Preceptors (n = 28) | Residency Only at Institution (n = 30) | Residency Training Elsewhere (n = 21) | ||
---|---|---|---|---|---|---|
Median (Range) | Median (Range) | p-value | Median (Range) | Median (Range) | p-value a | |
Acceptability | 76.50 (8.75–100) | 67.50 (52.50–95.00) | 0.08 | 67.50 (8.75–95.00) | 75.00 (52.50–100.00) | 0.12 |
Authenticity | 81.75 (43.75–100) | 78.25 (41.75–95.25) | 0.72 | 77.50 (43.75–95.25) | 83.75 (41.75–100.00) | 0.09 |
Cognitive Complexity | 77.25 (50.75–100) | 71.75 (31.25–94.50) | 0.17 | 74.38 (31.25–94.50) | 72.25 (50.00–100.00) | 0.32 |
Comparability | 76.12 (25.00–100) | 88.75 (59.25–100) | 0.09 | 77.38 (25.00–100.00) | 90.00 (55.00–100.00) | 0.04 |
Costs and Efficiency | 59.62 (37–95.25) | 68.12 (38.25–85.00) | 0.21 | 62.50 (39.75–78.50) | 76.25 (37.00–95.25) | 0.16 |
Educational Consequences | 43.75 (10.75–96.50) | 42.25 (12.25–79.25) | 0.91 | 43.75 (12.25–83.75) | 42.38 (10.75–96.50) | 0.83 |
Fairness | 75.75 (36.50–100) | 80.75 (43.00–100.00) | 0.41 | 78.75 (36.50–100.00) | 83.00 (43.00–100.00) | 0.87 |
Fitness for Purpose | 80.87 (25.50–100) | 74.87 (45.00–92.75) | 0.42 | 75.00 (25.50–92.75) | 76.25 (45.00–100.00) | 0.53 |
Fitness for Self- Assessment | 69.75 (10.50–100) | 74.00 (26.25–93.75) | 0.89 | 68.75 (10.50–93.75) | 75.75 (37.25–100.00) | 0.13 |
Meaningfulness | 75.00 (3.75–100) | 75.25 (32.75–91.00) | 0.91 | 70.88 (3.75–90.00) | 83.13 (32.75–100.00) | 0.01 |
Reproducibility of Decisions | 80.00 (37.50–100) | 76.00 (62.50–99.75) | 0.96 | 75.25 (37.50–99.75) | 80.38 (41.75–100.00) | 0.42 |
Transparency | 86.50 (34.00–100) | 78.00 (25.50–98.75) | 0.08 | 78.00 (25.50–100.00) | 87.78 (51.25–100.00) | 0.15 |
4. Conclusions
Author Contributions
Conflicts of Interest
References
- Kak, N.; Burkhalter, B.; Cooper, M.A. Measuring the competence of healthcare providers. Oper. Res. 2001, 2, 1–23. [Google Scholar]
- Tran, T.T.; El-Ibiary, S.; Tindula, R.; Rai, N.; Nguyen, A.; Le, J. Unique pharmacist competency program at community-based, teaching hospitals. J. Hosp. Adm. 2013, 2, 119. [Google Scholar] [CrossRef]
- McLachlan, J.C. The relationships between assessment and learning. Med. Educ. 2006, 40, 716–777. [Google Scholar] [CrossRef] [PubMed]
- American Society of Health-System Pharmacists. PGY1 Pharmacy Residency Progressive Detail Lists of Educational Outcomes, Goals, Objectives, Instructional Objectives. Available online: https://www.ashp.org/DocLibrary/Accreditation/Residency-Learning-System/RTP-RLSPGY1Goals071207.aspx (accessed on 29 September 2015).
- American Society of Health-System Pharmacists. Resident’s Guide to the Residency Learning System. Available online: https://www.ashp.org/DocLibrary/Accreditation/ResidentsGuidetotheRLS.aspx (accessed on 29 September 2015).
- Baartman, L.; Gulikers, J.; Dijkstra, A. Factors influencing assessment quality in higher vocational education. Assess. Eval. High. Educ. 2013, 38, 978–997. [Google Scholar] [CrossRef]
- Baartman, L.K.; Bastiaens, T.J.; Kirschner, P.A.; van der Vleuten, C.P.M. The wheel of competency assessment: Presenting quality criteria for Competency Assessment Programmes. Stud. Educ. Eval. 2006, 32, 153–170. [Google Scholar] [CrossRef]
- Baartman, L.K.; Prins, F.J.; Kirschner, P.A.; Van der Vleuten, C.P. Self-evaluation of assessment programs: A cross-case analysis. Eval. Progr. Plan. 2011, 34, 201–216. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Baartman, L.K.; Prins, F.J.; Kirschner, P.A.; van der Vleuten, C.P.M. Determining the Quality of Competence Assessment Programs: A Self-evaluation procedure. Stud. Educ. Eval. 2007, 33, 258–281. [Google Scholar] [CrossRef]
- Tavakol, M.; Dennick, R. Making sense of Cronbach's alpha. Int. J. Med. Educ. 2011, 2, 53–55. [Google Scholar] [CrossRef]
© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Shah, S.; McLaughlin, J.E.; Eckel, S.F.; Mangun, J.; Hawes, E. Evaluating the Quality of Competency Assessment in Pharmacy: A Framework for Workplace Learning. Pharmacy 2016, 4, 4. https://doi.org/10.3390/pharmacy4010004
Shah S, McLaughlin JE, Eckel SF, Mangun J, Hawes E. Evaluating the Quality of Competency Assessment in Pharmacy: A Framework for Workplace Learning. Pharmacy. 2016; 4(1):4. https://doi.org/10.3390/pharmacy4010004
Chicago/Turabian StyleShah, Shailly, Jacqueline E. McLaughlin, Stephen F. Eckel, Jesica Mangun, and Emily Hawes. 2016. "Evaluating the Quality of Competency Assessment in Pharmacy: A Framework for Workplace Learning" Pharmacy 4, no. 1: 4. https://doi.org/10.3390/pharmacy4010004