Reliability and Sources of Variation of Preclinical OSCEs at a Large US Osteopathic Medical School

Abstract

The objective structured clinical examination (OSCE) is a well-established tool for assessing clinical skills, providing reliability, validity, and generalizability for high-stakes examinations. Des Moines University College of Osteopathic Medicine (DMU-COM) adapted the OSCE for formative assessments in undergraduate medical education, focusing on interpersonal aspects in the primary care setting. Students are graded by standardized patients and faculty observers on interpersonal skills, history/physical examination, oral case presentation, and documentation. The purpose of the study is to establish the reliability and to identify sources of variation in the DMU-COM OSCE to aid medical educators in their understanding of the accuracy of clinical skills. We examined student performance data across five OSCE domains. We assessed intra- and inter-OSCE reliability by calculating KR20 values, determined sources of variation by multivariate regression analysis, and described relationships among observed variables through factor analysis. The results indicate that the OSCE captures student performance in three dimensions with low intra-OSCE reliability but acceptable longitudinal inter-OSCE reliability. Variance analysis shows significant measurement error in rubric-graded scores but negligible error in checklist-graded portions. Physical exam scores from patients and faculty showed no correlation, indicating value in having two different observers. We conclude that a series of formative OSCEs is a valid tool for assessing clinical skills in preclinical medical students. However, the low intra-assessment reliability cautions against using a single OSCE for summative clinical skills competency assessments.

1. Introduction

The assessment of clinical skills is a key component of the foundational phase of physician training prior to graduating from medical school (undergraduate medical education, UME). In these clinical performance evaluations, students complete a series of tasks in a patient encounter and are being evaluated by trained observers [1]. An objective structured clinical examination (OSCE) is a time-tested standardized format for the assessment of clinical skills based on a structured encounter with a simulated patient that is designed to evaluate if a candidate knows how to perform a variety of clinical tasks [2,3]. An OSCE consists of several stations, each evaluating a separate, well-defined clinical skill that is graded using predetermined criteria. Objectivity is achieved by ensuring that all examinees complete the same set of tasks and are graded using standards set by a panel of content experts. OSCEs stations are structured to give a comprehensive overview of examinee performance across the clinical performance domains of consultation, examination, and procedures [4]. A well-designed OSCE is characterized by reliability, validity, acceptability, affordability, and educational impact—criteria that require extensive planning, thorough psychometric analysis, and broad stakeholder participation in standard-setting [5].

A major concern for using an OSCE as a basis for defensible grading or pass/fail decisions is the reliability of the assessment, which is determined by the number of stations, total testing time and—to a minor extent—the number of observers [4]. To attain satisfactory reliability and generalizability, OSCEs require a minimum of 10–12 stations and might be conducted over several days [6,7,8]. It deserves to be noted that even if OSCEs are a reliable and generalizable assessment that add value to a comprehensive system of clinical skills assessment [9], they are not perfect as they require enormous resources, occur in an artificial setting without real patients, and rely on the deconstruction of a complex clinical encounter into smaller individual tasks [10,11,12].

At our institution (Des Moines University–College of Osteopathic Medicine, DMU-COM), the OSCE format has been adapted to enable its use as a tool for formative assessment during the preclinical phase of education and as part of a comprehensive clinical skills assessment in the clinical phase. Neither of these uses strictly meets the established criteria for OSCEs in terms of numbers of stations, observers, and total examination time [6], which means that the assessments can only provide supporting evidence for clinical competency and cannot be used as a stand-alone assessment. At DMU-COM, students are introduced to the concept of OSCE as part of the Clinical Medicine courses (clinically focused courses that develop students’ foundational diagnostic and procedural skills through lectures, case-based learning, simulations, and mentored patient encounters, emphasizing history-taking, physical exams, and clinical reasoning). In the preclinical phase of the curriculum, students complete seven formative standardized simulated patient encounters that are modeled after the OSCE format, and an eighth examination follows during a comprehensive clinical assessment after the first clerkship year. The DMU-COM OSCE does not include an evaluation of procedural skills since those are evaluated in parallel clinical medicine laboratories. Students are evaluated by the standardized patient and by a faculty observer using a rubric for the interpersonal aspects of the patient interactions and checklists for the technical criteria of the examination, oral presentation, and clinical documentation. Students are given feedback on their performance and are afforded the opportunity to repeat the encounter if the criteria for a passing grade are not met. Finally, the class meets for a mandatory large group debriefing session. With its focus on the patient experience and its formative nature, the DMU-COM OSCE thus incorporates key principles of osteopathic medical education by emphasizing patient-centered primary care [13]. To highlight the differences between the DMU-COM format and the standardized OSCE framework, the exercise is referred to in curricular materials as objective structured clinical experience or encounter (rather than examination).

Assessment of clinical skills via OSCE is a complex task, and an assessment of the quality of the OSCE requires the careful monitoring of multiple performance measures [14]. The present study was undertaken to determine the reliability and generalizability of the DMU-COM OSCE to justify the utilization of this method for formative assessment in undergraduate medical education. The objective of this research is to determine if the abbreviated four-station two-observer OSCE format provides sufficient reliability to certify clinical competency of students at the end of the preclinical curriculum. We hypothesize that, while the shorter OSCE will show lower reliability than the established full 12-station format, the aggregate of OSCE performance over 2 years of preclinical education will show sufficient reliability to justify decisions about students’ clinical competence. This study thus fills an important gap in our knowledge on clinical competency assessments on the individual school level—information that has become more urgent since clinical skills assessments have been discontinued at the national level and the responsibility of clinical skills competency assurance now falls to individual US Medical Schools [15,16]. We hypothesize that the DMU-COM OSCE curriculum provides sufficient longitudinal reliability to contribute value to a comprehensive clinical skills assessment program.

2. Materials and Methods

2.1. OSCE Format

In DMU-COM preclinical DO curriculum, students are assessed with 7 OSCEs spaced out over 4 semesters. The preclinical phase of the DMU curriculum consists of four semesters of instruction in foundational sciences, clinical sciences and practical clinical skills. The preclinical phase ends with students taking the first step of the US medical licensing exams, and upon successful passing of the exams, student progress to the 2-year clinical clerkship phase of undergraduate medical education. In each of the preclinical semesters, students take a Clinical Medicine course, during which they are taught essential clinical skills in patient interviewing, physical examination, clinical reasoning and documentation of patient encounters. Each of the four Clinical Medicine courses features two OSCE encounters. The first OSCE is an orientation to the format and is not scored, which leaves a total of seven OSCE to be included in this study. Another OSCE follows the first clerkship year, but it is not included in this analysis due to significant differences in grading. The patient presentation of the OSCE is coordinated with the concurrent lecture curriculum (i.e., SP presenting with shortness of breath during the cardiology/respiratory/hematology block of clinical sciences). The assessment features four distinct tasks: elicit a patient history, conduct a physical examination, deliver an oral presentation, and document the encounter in a note outlining subjective and objective information, assessment, and plan (SOAP). The assessment does not include a test of procedural skills, which are instead determined in separate clinical skills laboratory sessions. Student performance is graded by the standardized patient (SP) and a faculty observer (FO). SPs rate students’ interpersonal, professional, and physical examination skills using a 5-point rubric (

Table 1. Standardized patient rubric for evaluation of interpersonal and professional skills.

Performance Aspect	Rater	Grading Scheme
Interpersonal skills	SP	5-point rubric, averaged from ratings of verbal communication, non-verbal communication, empathy, listening skills
Professionalism	SP	5-point rubric
Organization of history and physical exam	SP	5-point rubric, averaged from organization history, organization physical
Overall impression	SP	Answer to question “would you return to this provider?”; no = 0, maybe = 0.5, yes = 1
Physical examination	SP	Checklist
SOAP note	FO	Checklist
History	FO	Checklist
Physical examination	FO	Checklist
Oral Report	FO	Checklist *
Oral Report—Assessment	FO	Checklist *
Oral Report—Plan	FO	Checklist *

Grading mechanism and criteria for the DMU-COM OSCE. SP = standardized patient, FO = faculty observer. Checklists item numbers vary by OSCE. * Oral reports were only graded in 2nd year OSCEs.

; for examples of checklists and rubrics see Supplementary Material). In addition, SPs grade the technical elements of the physical examination with a checklist. The FOs rate physical examination, oral presentation, and SOAP note using a checklist. Since the number of checklist items for physical examination varies widely between OSCEs, non-checklist fraction of the grade related to interpersonal skills ranges between 2.3 and 23% (average 7.3%). The OSCE is completed with a self-assessment of students’ performance not included as part of the grade. OSCEs were designed, administered, and monitored by MB and SP. Students are given faculty feedback on their performance and attend a mandatory debriefing session. A score of greater than 3/5 on the interpersonal skills rubric and greater than 70% on the checklist assessment are required to pass the OSCE; failures are remediated by re-taking the OSCE in question with a different SP.

2.2. Study Population

Data from students of the DO24-DO26 cohorts in the preclinical phase of the curriculum (AY2021/22 to 2023/24) were used for the analysis (inclusion criterion: member of the osteopathic medical student cohorts DO24, DO25 or DO26; exclusion criterion: students with incomplete OSCE records). 103 standardized patients participated in the assessments.

2.3. Analysis of Intra- and Inter-Assessment Reliability and Factors

Intra-OSCE reliability: For each assessment, the reliability of the sub-scores on the domains of interpersonal skills, physical examination, and SOAP note was determined using Cronbach’s alpha [17] (all data analyses conducted by MS; SPSS ver. 29, Chicago, IL, USA). Depending on the operational details of each OSCE, up to 5 FO scores of students’ oral presentation and physical examination were included in the reliability study for 3–9 sub-scores per OSCE. After Kaiser-Mayer-Olkin (KMO) and Bartlett’s tests showed suitability of the data for factor analysis (KMO score of 0.72 and Bartletts test significance of <0.001, respectively), an exploratory factor analysis was performed on the 9 sub-scores of the Year 2 Capstone OSCE (Extraction: Principal components, Eigenvalue > 1, Varimax rotation, cutoff 0.5; SPSS).

Analysis of inter-assessment reliability: Data from the 2021–2024 OSCEs were tabulated to track the longitudinal performance of the DO24-DO26 classes in the categories of 1. Interpersonal skills 2. History and physical examination, 3. Professionalism, 4. SOAP note, 5. the likelihood of the patient returning to the provider and 6. the final grade. After removal of incomplete records, Cronbach’s alpha was calculated on the matrix of 594 students × 7 OSCEs (SPSS).

2.4. Analysis of Variance

To assess the variance in the SP-scored items, SPs and students were assigned a unique identifier (594 students, 103 SPs, 4450 encounters), and the sex of students and SPs were coded. Variance of student grades was analyzed for ratings (as dependent variable) of interpersonal skills, professionalism, history/physical (rubric), the likelihood of return to the provider, and the checklist-graded evaluation of the physical examination with the general linear regression model (SPSS). The analysis examined the influence of the student, the SP, the position of the OSCE in the curriculum, and the sex match between student and SP as random factors.

3. Results

3.1. Participation

Data from 594 students (and 103 standardized patients) were included in the study.

3.2. Grade Trends and Difficulty

The checklist evaluations of the physical examination (FO and SP) and the FO evaluation of history taking showed positive trends and improvement in over 2/3 of students. Class average grades remained steady over the 7 preclinical OSCE in all other grading criteria (

Table 2. Grade trends over the 7 preclinical OSCE.

Performance Aspect	Regression (mx + b)	% Showing Positive Trend	Score Average (Scaled to 1) and Standard Deviation
SP Interpersonal skills	0.0015x + 0.74	52.8	0.746 ± 0.06 (N = 4165)
SP Professionalism	0.0058x + 0.72	58.1	0.741 ± 0.07 (N = 4165)
SP Organization of history/physical exam	−0.0015x + 0.76	46.6	0.758 ± 0.06 (N = 4165)
SP Overall impression	0.0016x + 0.85	48.0	0.84 ± 0.136 (N = 4165)
SP Physical examination	0.03x + 0.70	83.8	0.827 ± 0.06 (N = 4116)
FO SOAP note	−0.006x + 0.92	32.8	0.9 ± 0.04 (N = 4102)
FO History	0.005x + 0.9	74.8	0.928 ± 0.05 (N = 3037)
FO Physical examination	0.02x + 0.76	93.3	0.86 ± 0.08 (N = 3037)
SP Interpersonal skills	0.0015x + 0.74	52.8	0.746 ± 0.06 (N = 4165)
SP Professionalism	0.0058x + 0.72	58.1	0.741 ± 0.07 (N = 4165)
SP Organization of history/physical exam	−0.0015x + 0.76	46.6	0.758 ± 0.06 (N = 4165)
SP Overall impression	0.0016x + 0.85	48.0	0.84 ± 0.136 (N = 4165)
SP Physical examination	0.03x + 0.70	83.8	0.827 ± 0.06 (N = 4116)
FO SOAP note	−0.006x + 0.92	32.8	0.9 ± 0.04 (N = 4102)

Grade trends of all students. Trends were calculated by regression analysis of fractional grades (100% = 1; LINEST function, Microsoft Excel).

).

3.3. Intra- and Inter-OSCE Reliability and Factors

The intra-assessment reliability over 3 years (i.e., correlation of the 3–9 sub-scores within one OSCE) varied between 0.38 and 0.59, with an average of 0.5 (Figure 1). The inter-OSCE reliability (i.e., correlation of the sub-scores over the 7 preclinical OSCEs) of the final grade for the 7-assessements on the preclinical curriculum showed a reliability of 0.7, with the most reliable grading component being the FO-scored SOAP note (Figure 2). The factor analysis of the 9 grading components of the Capstone OSCE identified 3 clearly separable dimensions of the examination which can be defined as patient perception, conducting and summarizing an examination, and formulating an assessment/plan (

Table 3. Factor analysis of Capstone OSCE.

Performance Aspect	1	2	3
SP Interpersonal (rubric)			0.790
SP Physical Examination (checklist)			0.806
FO History (checklist)	0.757
FO Physical Examination (checklist)	0.537
FO Oral Report History (checklist)	0.780
FO Oral Report Physical examination (checklist)	0.547
FO Oral Report Assessment (checklist)		0.807
FO Oral Report Plan (checklist)		0.756
FO SOAP note (checklist)	0.629

Factor analysis of the grading items in the capstone OSCE at the end of year 2. The assessment items are loading onto 3 distinct factors that can be summarized as history/physical examination/documentation of encounter (1), formulation of an assessment/plan (2), and patient experience (3).

).

3.4. Inter-Rater Reliability

On 14 occasions, SP and FO both evaluated students’ physical examination skills using a checklist. The inter-rater correlation of these assessments varied between −0.04 and 0.51 (average 0.14, classified as “small”).

3.5. Sources of Variance in SP-Graded Criteria

Most of the variance in the rubric-graded SP observations remains unaccounted by any of the 4 examined factors (SP, Student, number of OSCE, and SP/Student sex match) and is classified as a random error of measurement. Of the examined factors, the SP is the largest source of variance, eclipsing the variance ascribed to the examinee (Figure 3). The variation in the checklist-graded assessment of the physical examination depends most strongly on the combination of student and OSCE, with no significant variance arising from experimental error (Figure 4). The sex match between SP and student did not influence the scores for any of the examined dependent variables.

4. Discussion

At DMU-COM, simulated patient encounters were first introduced in 1996. These encounters were designed as formative assessments with an emphasis on capturing the patient experience, allowing for individualized faculty feedback, inviting self-reflection, holding a formal debriefing, and providing opportunities for immediate remediation of deficiencies. In designing the assessment, the college emphasized SP feedback on examinees’ interpersonal skills and empathy to further its mission of training physicians for patient-centered, compassionate care. The demonstration of procedural skills was deliberately omitted as these aspects of clinical competency are assessed elsewhere in the curriculum. With the emphasis on interpersonal skills and empathy, the college sought to adapt the OSCE model for the use in osteopathic medical education and the training of primary care providers that has long been the mission of the osteopathic profession [18].

As the assessment tools expanded, the simulation sessions gained key characteristics of the time-tested OSCE format [2] but did not meet the established stringent criteria in terms of station numbers, multiple observers, and total time of examination [8]. The grading elements of history, physical examination, oral presentation, and documentation classify the DMU-COM assessment as a four-station OSCE. With the understanding that this assessment would not strictly fit the criteria for high-stakes, summative OSCEs, the college nevertheless adopted the name OSCE for the exercise (but referring to the exercise as experience or encounter in curricular material). The present study was undertaken to assess reliability and sources of variation of the DMU-COM OSCE, with the ultimate goal of estimating the value of these assessments for the certification of clinical skills. Data collection for the present study began in 2021 after the COVID restrictions on in-person instruction were lifted.

Our analysis shows that the observations of physical examination and history-taking show a clear improvement over the course of the preclinical curriculum, indicating that students are progressing in the more technical skills of encounter. By contrast, students’ interpersonal skills and professionalism ratings show little change over the same time period. This might be due in part to the notorious difficulties in measuring communication skills with an OSCE [19], but also suggests that interpersonal aspects of the patient encounter develop differently than purely technical skills. While it is possible to advance communication skills through teaching in the simulated setting [20], there is also clear evidence that without practice, communication skills deteriorate over time [21]. The observed stagnation might reflect a balance of these forces.

The intra-assessment reliability of the DMU-COM OSCE sub-scores varies between different assessments but in no case exceeds 0.59—well in line with similar 4-station OSCEs conducted at other institutions [6,8] but falling short of the generally accepted threshold of reliability for stand-alone assessments of 0.7 [22]. A factor analysis shows that the DMU-COM OSCE has multiple dimensions. The grading items in the scale load onto three separate factors, indicating that the assessment measures three distinct skills–a structure that can be expected to result in low reliability [22]. Our data also show that some OSCE grading components show remarkable inter-assessment reliability over the course of the preclinical curriculum. The faculty-graded SOAP note component and the overall grade both have alpha values above 0.7, which supports the use of a series of OSCEs as an integral part of a robust clinical competency assessment.

A key element of the OSCE–the physical examination–is evaluated by two independent raters: the standardized patient and a faculty observer who watches a recording of the encounter. Both ratings show low inter-OSCE reliability, and the correlation between these ratings is generally poor. This is likely caused by faculty and patients having profoundly different perspectives on the exam. The fact that observers have different viewing angles (first person experience vs. observer behind video screen) might be just as important as the different criteria of patient and experienced physician (i.e., if it feels correct to a lay person vs. if it looks correct to an expert) [23,24]. This finding highlights the importance of engaging two observers who each provide unique aspects to this critical part of the assessment.

A known weakness of the OSCE is the subjectivity of the examiner that can result in a high variability of the score [25,26]. To examine the observer-dependent variance of the DMU-COM OSCE scores, we coded each encounter with a unique standardized patient identifier and categorized the sex match between student and patient. We found that while the identity of the patient contributed very little to the rubric-scored assessments of interpersonal skills, the patient-dependent variation nevertheless eclipsed the student-factor. This finding is somewhat surprising, as it suggests that the scores of students’ interpersonal skills do not most strongly depend on the student but are largely determined by statistical error and the patient. In contrast, the checklist-graded patient evaluation of students’ physical examination skills has only negligible error and depends most strongly on the student and the task. We explain this discrepancy with the greater subjectivity of rubric-grading compared to checklists [27]. However, since the patient rubrics provide valuable information on the patient experience, the OSCE clearly benefits from cautious inclusion of these scores.

The main limitation of this study design lies in its inability to show the predictive value of the clinical competence measurements. While the study shows the reliability and construct validity of the OSCEs in the preclinical phase of the curriculum (years 1 and 2), it does not show how well the OSCE assessments predict clinical performance in the clerkship phase of UME (years 3 and 4). More research on the correlation of OSCE performance with clerkship preceptor evaluations and other clinical performance measures could shed light on the predictive power of the OSCE assessments of clinical and interpersonal skills.

5. Conclusions

While not a perfect measurement of clinical competency, the OSCE clearly adds value to a system of competency assessments that ideally includes a broad spectrum of workplace- and laboratory-based assessments. The variant of the OSCE practiced at DMU-COM, with its key features of formative feedback and the focus on the patient experience, does not achieve sufficient intra-assessment reliability to serve as a sole criterion for summative evaluation. However, the reliability of the DMU-COM OSCE compares well with the reliability of similar OSCEs at other institutions, and we conclude that final grades of the OSCE protocol described in this study have sufficient inter-assessment reliability to support a clinical competency evaluation.