The Use of Rubrics in the Clinical Evaluation of Podiatric Medical Students. Objectification of the Subjective Experience

Abstract

Background: We assessed the differences in podiatric medical students' clinical professionalism objective scores (CPOSs) by comparing a previous nonrubric evaluation tool with a more recently implemented objective-centered rubric evaluation tool. This type of study has never been performed or reported on in the podiatric medical education literature. Methods: We conducted a retrospective analysis of 89 third-year podiatric medical students between academic years 2010-2011 and 2011-2012. A Pearson correlation coefficient analysis was performed to compare CPOSs from the students' first (CPOS1) and second (CPOS2) rotations. A correlation analysis was performed comparing students' grade point averages (GPAs) with each of the individual CPOSs to verify the validity of the rubric evaluation tool. Results: The Pearson correlation coefficients for the relationship between 2012 CPOS1 and CPOS2 and GPA were r = 0.233 (P ≤ .093) and r = 0.290 (P < .035) and for the relationship between 2013 CPOS1 and CPOS2 and GPA were r = 0.525 (P = .001) and r = 0.730 (P < .001). Conclusions: These findings suggest that the use of a rubric in the evaluation of podiatric medical students' CPOSs is correlated with their GPAs, and CPOS2 demonstrated a higher correlation than CPOS1. We believe that implementation of the rubric evaluation tool has increased the accuracy of the evaluation of podiatric medical students with respect to CPOSs.

Clinical evaluation remains a challenge, even for the most experienced faculty [1]. Much of what is assessed in medical education regarding aptitude, professionalism, and clinical skills changes as students progress through their respective programs. In the first 2 years of medical education, most of what is assessed is completed objectively through the use of multiple-choice and standardized examinations. During their clinical years, students are evaluated by instruments that are likely to be far more subjective than what they have previously experienced [2]. In 2002, Plymale et al. [3] discussed subjective evaluation forms as being used almost universally in grading clinical students' performance. In the past, clinical students were graded primarily by subjective means of evaluation based on physician faculty and house officers' observations [2]. The leading approach to evaluating third- and fourth-year medical students' performance in the clinical setting is frequently subjective: written evaluation by the instructors [4]. The subjective method by which medical students are graded in clinical clerkships may be contributing to the inflation of clinical grades during the past couple of decades [1,5,6].

In 2000, Speer et al. [6] defined grade inflation as a disproportionate percentage of higher grades than the student's performances merit. Inflation of any type of grade is worrisome, but even more troubling is the inflation of grades of medical professionals who are responsible for providing care to our communities. With most medical students' clinical grades resulting from faculty and resident physicians' ratings, there is a troubling revelation that resident physicians trend toward giving higher clinical grades than do attending physicians [7,8]. With clinical grades on the rise, there is a belief that a greater issue is looming. The concern is that there is a disproportionate increase in the subjectivity of clinical evaluations [1,5,6]. A possible reason could be the halo effect. Fowell and Bligh [9] showed that ratings are prone to the halo effect, in which if a student excels or performs poorly on one evaluation criterion, this will affect the score given in subsequent criteria, which are graded higher or lower than the real performance merits. There is a need to find a solution to this issue to protect the validity and integrity of our current and future physicians. This protects the public from incompetent physicians who are slipping through the cracks owing to clinical grade inflation.

Owing to the issues of the subjectivity of the current grading process and the link to grade inflation, there has been a move in the medical community to adopt a more objective method for grading clinical students. With this move toward objectifying and unifying the way clinical students are graded, various tools have been created [4,10]. To minimize the extent of subjectivity and avoid its potential negative effects on clinical grading, rubrics have been proposed as a potential solution to this problem [1,11]. As defined by Random House Webster's College Dictionary [12] (p1679), a rubric is “[a] title, heading, or the like, in a manuscript, statue, etc., written or printed in red or otherwise distinguished from the rest of the text; any established rule of conduct or procedure.” A rubric at its most basic is a scoring instrument that lays out the exact standards for an assignment [13]. The study of rubrics as an evaluation tool has been well documented in certain disciplines of medicine, including allopathic and osteopathic medicine, nursing, pharmaceutical medicine, and physical therapy.

To our knowledge, there has not been any research assessing the use of rubrics as a clinical evaluation tool for podiatric medical students. The purpose of this study was to assess whether the newly developed clinical evaluation rubric aligns with the academic performance of the student as determined by grade point average (GPA).

Development of a Rubric Evaluation Tool

Before 2011, a subjective evaluation form was used to grade podiatric medical students' clinical professionalism objective scores (CPOSs) at Des Moines University College of Podiatric Medicine and Surgery (DMU-CPMS) (Des Moines, Iowa). The CPOS evaluation form consisted of 11 professionalism objectives that were graded on a 4-point scale (Fig. 1). Starting in 2011, DMU-CPMS implemented a new objective-centered rubric evaluation tool to assist the grading of CPOSs. The rubric was adapted from that used at Duke University School of Medicine and the University of Minnesota Medical School to evaluate podiatric medical students in the clinical curriculum. The faculty at DMU-CPMS was instructed on the use of the rubric evaluation tool in a faculty development session before its implementation. The rubric identified specific criteria for what constituted each rating score in each of the 11 professionalism objectives. These scores were graded by faculty using a similar 4-point scale, according to the objectives stated in the rubric evaluation form (Fig. 2).

Figure 1. A, Clinical professionalism objectives. B, Grading scale of the clinical professionalism objectives.

Figure 1. A, Clinical professionalism objectives. B, Grading scale of the clinical professionalism objectives.

Figure 2. Rubric for clinical professionalism objectives.

Figure 2. Rubric for clinical professionalism objectives.

Methods

The data consisted of two separate groups of third-year classes. The class of 2012 had 53 students who were evaluated without a rubric, and the class of 2013 had 36 students who were evaluated with a rubric. Each of the students in these two groups rotated with each of the four physicians twice, once earlier in the year and then once later in the year. Thus, CPOS1 is the CPOS from the first rotation and CPOS2 is the CPOS from the second rotation. To evaluate the effectiveness of the prerubric and rubric evaluation tools, students' GPAs were used as an outside objective measure of student academic performance. We compared CPOS1 and CPOS2 of the 2012 and 2013 podiatric medicine classes with the students' GPAs to determine the effectiveness of the recently implemented rubric evaluation tool. A Pearson correlation coefficient analysis, via a statistical software program (IBM SPSS Statistics for Windows, version 19.0; IBM Corp, Armonk, New York), was performed to determine the reliability of the recently implemented rubric evaluation tool.

Results

The mean ± SD GPA in the group that was not evaluated with a rubric was 3.48 ± 0.190. The mean ± SD GPA in the group evaluated with a rubric was 3.47 ± 0.196. There was no significant difference between the two different classes regarding GPA (Fig. 3).

Figure 3. Mean ± SD grade point averages (GPAs) for the prerubric and rubric groups.

Figure 3. Mean ± SD grade point averages (GPAs) for the prerubric and rubric groups.

The mean ± SD CPOS1 in the group evaluated without a rubric was 3.27 ± 0.457. The mean ± SD CPOS2 in the group evaluated without a rubric was 2.91 ± 0.504. The mean ± SD CPOS1 in the group evaluated with the rubric was 3.02 ± 0.439. The mean ± SD CPOS2 in students evaluated with the rubric was 2.79 ± 0.503 (Fig. 4).

Figure 4. Mean ± SD clinical professionalism objective scores (CPOSs) for the prerubric and rubric groups. CPOS1, CPOS from the first rotation; CPOS2, CPOS from the second rotation.

Figure 4. Mean ± SD clinical professionalism objective scores (CPOSs) for the prerubric and rubric groups. CPOS1, CPOS from the first rotation; CPOS2, CPOS from the second rotation.

In the group evaluated without a rubric, the correlation between CPOS1 and CPOS2 was high and significant (r = 0.715; P < .001). The correlation between CPOS2 and GPA, although weak, was also significant (r = 0.290; P < .05). However, the correlation between CPOS1 and GPA was not significant and did not correlate (r = 0.233; P > .05) (Table 1).

Table 1. 2012 Podiatric Medicine Class Prerubric Correlation Analysis (N = 53).

Table 1. 2012 Podiatric Medicine Class Prerubric Correlation Analysis (N = 53).

The pattern of correlation between the variables is somewhat different in the group evaluated with a rubric. The correlation between CPOS1 and CPOS2 remained strong and significant (r = 0.617; P < .001). In addition, the correlation between CPOS1 and GPA also turned out to be moderately strong and significant (r = 0.525; P = .001). Moreover, the correlation between CPOS2 and GPA was strong and significant (r = 0.730; P < .001) (Table 2).

Table 2. 2013 Podiatric Medicine Class Rubric Correlation Analysis (N = 36).

Table 2. 2013 Podiatric Medicine Class Rubric Correlation Analysis (N = 36).

Discussion

The prerubric clinical evaluation of CPOS1 and CPOS2 demonstrated a weak correlation with preclinical GPA. There are a couple of reasons that may explain the weak correlation between CPOS1 and CPOS2 for the prerubric evaluation method. Students' clinical grades can vary greatly from their preclinical GPAs when individual faculty physicians differ widely in their understanding of course objectives and in their ability to size up students' achievements [7]. Another explanation for weak correlation could be the faculty's own perceptions of their teaching. Marienfeld and Reid [14] reported that one reason why faculty and preceptors may be hesitant to rate a student as failing or marginal is because they might feel that such a rating would reflect negatively on their own teaching. Another factor to consider in comparing subjective clinical evaluations with objective-centered preclinical evaluations is the content in which the student is graded. Geertsma and Chapman [15] bring to the surface a fundamental question about whether the criteria that clinical students are graded on, such as likeability, relationships with patients, composure, and ethical standards, are, indeed, different from the grading criteria in the preclinical curriculum and may account for some differences in the preclinical scores compared with the subjective clinical evaluations.

The medical literature continues to elaborate on the concerns of grade inflation, especially in the clinical setting [1,4-6]. Andre [16] indicated that educators have for a long time contemplated that the assigning of a grade in evaluation of student clinical performance was too irregular, too biased, or educationally improper. Walsh and Seldomridge [17] elaborated on the skewing of the average of clinical grades toward higher ratings compared with the average of the didactic classes' grades in nursing. Speer et al. [6] demonstrated that in all of the medical schools studied, the percentage of failing grades did not change over the study years, and that almost 50% of these schools did not assign a failing grade in one of the years of the study. They also found that 82% of the faculty was hesitant to give lower grades [6].

From our review of the literature, the need to find a more valid and unified way to evaluate aspects of clinical education is vital. After the second year of podiatric medical students' curriculum at DMU, they move away from the traditional letter grade system in their didactic curriculum to a pass-fail system in years 3 and 4 of their clinical curriculum. With most of the podiatric medical students' clinical grades riding on their performance in the clinic as well as their CPOSs, there is a need to ensure that the evaluation process is as uniform as possible. Accurate and reliable ways to evaluate clinical skills are necessary [18]. Owing to the importance of preceptors' and clinical faculty's ratings in clinical medical education and residency selection, there is a growing need to ensure that these evaluations are as legitimate and credible as possible [19].

Alternatively, there also has been criticism of a more objective tool for evaluating the clinical performance of medical students [20-22]. The use of objective assessment methods is burdensome at the local level, and there is a risk that independent objective assessments, such as the National Board Examinations, may not predict clinical performance correctly [22]. Even with objective evaluations, it is difficult to remove subjectivity from clinical assessment [1,20,21]. Another issue that occurs with movement to an objective evaluation method is the reduction in clinical grades. Sander and Trible [5] go on to say that faculty indicated a perceived reduction in clinical grades with installation of a new objective assessment tool. This was evident in the present study after implementation of the rubric. The mean scores for CPOS1 and CPOS2 were lower using the rubric.

White et al. [23] demonstrated that the strongest predictor of students' clinical performance is academic grades in the second year of medical school. Roop and Pangaro [24] showed that the most important prognosticator of clinical performance is based on preclinical GPAs. Students' preclinical academic knowledge and skills positively influence performance if we consider their influence on overall clinical performance [25]. The rubric evaluations of CPOS1 and CPOS2 were significantly correlated with preclinical GPA, thus providing strong evidence that the rubrics improve the evaluation of podiatric medical students' CPOSs.

Using the rubric evaluation tool, CPOS2 had a stronger correlation with preclinical GPA than with CPOS1. There could be several reasons why CPOS2 had a higher correlation than CPOS1. One reason could be that at the first evaluation of a student, physicians evaluate more based off the rubric and nothing else compared with the second evaluation when the physician can draw on the information and impressions from the previous evaluation and use that as well as information on the rubric to evaluate the student more accurately. Chambers [26] demonstrated that faculty who had evaluated a student multiple times used both information carried over from the previous semester and the student's current performance. Chambers [26] concluded from his study that faculty members who evaluated students multiple times were more accurate in prognosticating the students' overall grades compared with faculty members who evaluated students only one time. Another reason why CPOS2 was more highly correlated with preclinical GPA could be that students learn from their mistakes and flaws from their first rotation, which was evaluated by CPOS1, and work to improve and correct the mistakes in their second rotation with that particular faculty member. Lasater [27] stated that a rubric evaluation method provides a student with a feedback method and serves as a guide for the student's improvement and development of clinical skills.

In addition to the fact that CPOS2 was more strongly correlated with GPA than CPOS1, the scores for CPOS2 were lower than those for CPOS1 for both evaluation methods. One explanation for this might be that the evaluating physician might hold students to different levels of expectations, outside of the rubric objectives, regarding the initial evaluation compared with subsequent evaluations. For example, the evaluating physician will evaluate a student with lower standards in July compared with higher standards in April regardless of the evaluation tool.

Conclusions

The evaluation of podiatric medical students' clinical attributes is a highly subjective process. These findings suggest that when using a rubric in the evaluation of podiatric medical students, CPOSs are strongly aligned with students' GPAs. These findings also demonstrate that a stronger correlation exists between CPOS2 and students' GPAs compared with CPOS1 and students' GPAs. We believe that implementation of the rubric evaluation tool has increased the accuracy of the evaluation of podiatric medical students with respect to CPOSs.