Can We Predict Podiatric Medical School Grade Point Average Using an Admission Screen?

Abstract

Background: Most medical school admission committees use cognitive and noncognitive measures to inform their final admission decisions. We evaluated using admission data to predict academic success for podiatric medical students using first-semester grade point average (GPA) and cumulative GPA at graduation as outcome measures. Methods: In this study, we used linear multiple regression to examine the predictive power of an admission screen. A cross-validation technique was used to assess how the results of the regression model would generalize to an independent data set. Results: Undergraduate GPA and Medical College Admission Test score accounted for only 22% of the variance in cumulative GPA at graduation. Undergraduate GPA, Medical College Admission Test score, and a time trend variable accounted for only 24% of the variance in first-semester GPA. Conclusions: Seventy-five percent of the individual variation in cumulative GPA at graduation and first-semester GPA remains unaccounted for by admission screens that rely on only cognitive measures, such as undergraduate GPA and Medical College Admission Test score. A reevaluation of admission screens is warranted, and medical educators should consider broadening the criteria used to select the podiatric physicians of the future.

Barry University (Miami Shores, Florida) is a federally designated minority-serving institution in the Southeastern United States with a Carnegie classification of doctoral/research university.[1] In 2011, Barry University School of Podiatric Medicine received almost 600 applications for the 67 seats available in the freshman class. Consequently, making decisions about which potentially suitable applicants to invite for an interview and finally to accept is a daunting, high-stakes task for the admission committee as they strive to evaluate each candidate’s potential for graduating from the program.[1]

The School of Podiatric Medicine’s admission committee, similar to that at many schools, screens prospective candidates, at least in part, based on their Medical College Admission Test (MCAT) scores and their undergraduate grade point average (uGPA),[2] although there is no consensus in the literature about the predictive value of these cognitive measures.[3] Although resource limitations dictate that a preinterview admission screen be used, we have previously shown that using a screening model that relies on age and MCAT scores to predict 4-year graduation is inappropriate because it yields too many falsepositives.[1] That is, it identifies matriculants as being at risk of attrition who would ultimately graduate. The academic medicine community is, therefore, challenged to develop a more valid and reliable screening and selection protocol for podiatric medical school admission that builds on the predictive validity of the MCAT score and uGPA and that is sensitive and specific.

Currently, most medical school admission committees use cognitive and noncognitive measures to inform their final admission decisions.[4] In a growing number of medical schools, noncognitive characteristics (such as empathy, compassion, and work ethic) are measured to select suitable candidates for interview.[5] This is followed by a campus visit to make the final admission selections.[5] Applicants to the Barry University School of Podiatric Medicine are invited to an on-campus interview based on a combination of cognitive measures (MCAT score and uGPA), letters of recommendation, and a candidate-written personal statement. At the on-campus interview, the candidate’s motivation to become a podiatric physician, knowledge of the profession, and a variety of personality and character traits are evaluated by a basic science faculty and a clinical science faculty. The screening process is imperfect, and occasionally students enter medical school only to drop out before graduation. The cost of attrition for the medical school is high in terms of institutional prestige, accreditation, and resource costs. The cost of attrition is also high for the medical student in terms of lowered self-esteem and increased economic burden[1]; therefore, we continue to evaluate the screening process.

Screening protocols that are imperfect are not unique to medical school admission. In fact, the Law School Admission Test, administered by the Law School Admission Council and designed to predict first-year academic performance at law school, has also received some criticism. The common practice of law school admission committees to attribute special weighting to Law School Admission Test scores[6] has been reported to be without merit and validity.[7] Furthermore, use of the Graduate Management Admission Test by business school admission committees continues to be reevaluated, as many business schools implement Graduate Management Admission Test waiver policies without affecting the academic performance of their graduates as measured by GPA at graduation.[8]

We evaluated the use of admission data in predicting 4-year graduation from podiatric medical school,[1] yet there is little recently published literature about the value of such data for predicting academic performance as determined by GPA in podiatric medical school. In this study, we evaluated educational background variables as possible predictors of good academic performance (GPA >3.0) in podiatric medical education. This study is the first, to our knowledge, to assess the value of admission data in predicting academic success for podiatric medical students using first-semester GPA and cumulative GPA at graduation as outcome measures.

Methods

Sample

Students were included in the sample if they were first-time, full-time podiatric medical students who entered the program between August 1995 and August 2011. Students were included in the sample only if there was a record of MCAT scores and uGPA. Foreign-trained physicians were excluded from the sample. The sample of 806 students averaged 26 years of age and was 39% underrepresented minorities, 58% men, and 11% foreign born. Table 1 provides descriptive statistics for uGPA, total MCAT score, first-semester GPA in podiatric medicine, and cumulative GPA in podiatric medicine at graduation. Although first-semester GPA may seem unimportant, the correlation between first-semester GPA and cumulative GPA at graduation was large (r = 0.78, P < .001), suggesting that first-semester GPA is an important early indicator of academic success. Hispanic and black students are overrepresented, composing 28% of the sample but only 14% of the population of podiatric medical students since 2007.[9] In terms of race/ethnicity, this sample is not representative of the population of podiatric medical students in the United States; consequently, caution must be exercised in generalizing the results of this study to the entire population. Institutional review board approval was received from Barry University to use archival data.

Data Analysis

Linear multiple regression is the most appropriate statistical procedure for examining the predictive power of an admission screen as long as the required assumptions are approximated. The distribution of each variable was examined to determine the extent to which the required assumption of normality was satisfied. Although first-semester GPA had a disproportionate number of “straight As,” P-P plots showed that the distribution of each variable was sufficiently close to normal to satisfy the assumption. Scatterplots between the dependent and each independent variable confirmed that their relationship was linear and that the assumption of homoscedasticity (equal variance along the trend line) was satisfied.

Table 1. Values for Admission Screens and GPAs

Table 1. Values for Admission Screens and GPAs

In this study, we used a cross-validation technique to assess how the results of the regression model would generalize to an independent data set. The sample was randomly divided into two parts: 50% in an estimation subsample (n = 403) and 50% in a validation subsample (n = 403). A linear multiple regression was run with uGPA, total MCAT score, and a time trend as predictors. For first-semester GPA and cumulative GPA at graduation, all three predictors were statistically significant using the estimation subsample. Then, this regression model was run on the validation subsample from the same population. Reproducing the results in two independent samples (estimation and validation) is essentially equivalent to replicating the results, giving a higher level of confidence than could be achieved from a model with statistically significant predictors from a single sample alone. Generally, the model will not fit the validation subsample as well as it fits the estimation subsample. Consequently, when the regression model for cumulative GPA was run on the validation subsample, the time trend variable was not statistically significant and was removed from the model. For first-semester GPA, all three predictors remained statistically significant. Although the precise interpretation of the time trend in first-semester GPA is not known, it functions as a proxy for potentially confounding variables, such as grade inflation and pedagogy changes. The final regression models analyzed were as follows: 1) uGPA and MCAT score predict cumulative GPA at graduation and 2) uGPA, MCAT score, and a time trend variable predict first-semester GPA. All of the predictors in the final regression models were statistically significant and had the same sign for the estimation and validation samples.

Results

Cumulative GPA at Graduation

A multiple regression analysis was conducted to evaluate how well an admission screen predicted cumulative GPA at graduation. The predictors in the admission screen were uGPA and total MCAT score. The linear combination of predictors was significantly related to cumulative GPA at graduation (F_{2, 453} = 62.843, P < .001). The partial correlation of both predictors indicates a medium effect size. The sample multiple correlation coefficient was 0.47, indicating that only 22% of the variance in cumulative GPA in the sample could be accounted for by the linear combination of predictors (Table 2).

We established that an admission screen based on uGPA and MCAT score can predict only 22% of the variance in cumulative GPA. In other words, although the admission screen is highly statistically significant, it is not very precise (Figs. 1 and 2). In Figures 1 and 2, there is a significant upward trend; however, the substantial variation of individuals around the trend line indicates that the admission screen does not predict most individual variation.

First-Semester GPA

A multiple regression analysis was conducted to evaluate how well an admission screen predicted first-semester GPA. The predictors in the admission screen were uGPA, total MCAT score, and a time trend variable. The linear combination of predictors was significantly related to first-semester GPA (F_{3, 802} = 29.266, P < .001). Again, the partial correlation of each predictor indicates a medium effect size. The sample multiple correlation coefficient was 0.49, indicating that only 24% of the variance in first-semester GPA in the sample could be accounted for by the linear combination of predictors (Table 3).

Table 2. Regression Analysis Summary for Admission Screen Predicting Cumulative GPA

Table 2. Regression Analysis Summary for Admission Screen Predicting Cumulative GPA

Predictive Success

Because there is no intuitive analogue for the proportion of variance explained, it is difficult to evaluate how well an admission screen has performed based on the finding that it explains less than one-quarter of the variance in academic performance. The measures of sensitivity and specificity from the disease detection model provide a more familiar context for evaluating the extent to which an admission screen can identify applicants who will be poor academic performers. To illustrate, we define poor performance as a GPA less than 3.0. For first-semester GPA and cumulative GPA, classification success was evaluated using the regression model’s predicted GPA standardized as the cutoff point (Z =−0.15 or −0.37, respectively) to distinguish a GPA below B from a GPA above B. This cutoff point was determined based on a receiver operating characteristic curve analysis and was chosen to maximize the separation between poor and satisfactory academic performance, ie, the cutoff point was the point on the receiver operating characteristic curve and where the slope equaled the prevalence of poor performance. The screen for first-semester GPA below B had a sensitivity of 68% (95% confidence interval, 62%–73%) and a specificity of 71% (95% confidence interval, 66%–75%). The screen for cumulative GPA below B at graduation had a sensitivity of 63% (95% confidence interval, 49%–74%) and a specificity of 67% (95% confidence interval, 62%–72%).

Figure 1. Scatterplot of undergraduate grade point average (GPA) and cumulative GPA in podiatric medicine at graduation.

Figure 1. Scatterplot of undergraduate grade point average (GPA) and cumulative GPA in podiatric medicine at graduation.

Figure 2. Scatterplot of total Medical College Admission Test (MCAT) score and cumulative grade point average (GPA) in podiatric medicine at graduation.

Figure 2. Scatterplot of total Medical College Admission Test (MCAT) score and cumulative grade point average (GPA) in podiatric medicine at graduation.

Discussion

Medical schools use some form of admission screen to select students with the qualities necessary to graduate and become successful physicians. Most medical schools now place a heavy emphasis on cognitive indicators, such as uGPA and MCAT scores, in the selection process.[2] In this study, however, we suggest that there is overconfidence in the ability of these cognitive measures to predict success in podiatric medical school and perhaps all medical schools.

Table 3. Regression Analysis Summary for Admission Screen Predicting First-semester GPA

Table 3. Regression Analysis Summary for Admission Screen Predicting First-semester GPA

Herein, we report that an admission screen based on uGPA and total MCAT score predicts only 24% of the variance in first-semester GPA in podiatric medical school and only 22% of the variance in cumulative GPA at graduation. It is surprising that the admission screen for cumulative GPA at graduation performs nearly as well as the screen for first-semester GPA because the first semester is much closer in time to the admission decision. One possible explanation could be that both admission screens are selecting for applicant characteristics that go beyond the strength of their undergraduate education background. These traits remain relevant throughout students’ podiatric medical education, including clinical training. The strong correlation between first-semester GPA and GPA at graduation is consistent with the notion that some elements of success are common throughout the curriculum. The strength of this correlation makes first-semester GPA an important early indicator of long-term academic success. This allows faculty to identify students at risk for poor academic performance at an early stage and to implement academic support plans or remediation as appropriate.

Admission screens based on cognitive criteria predict success less well than commonly imagined. The considerable variation of students around the trend lines provides a visual representation of the finding that more than three-quarters of the variance in GPA is not explained by uGPA or MCAT score (Figs. 1 and 2). These figures show that at every level of MCAT score or uGPA, the variation in individual cumulative GPA is tremendous. Consequently, an admission screen can predict average outcomes for an entering class, but individual student outcomes cannot be predicted at the admission stage. This unexplained variance may be attributed to a variety of noncognitive reasons, including but not limited to motivation, conscientiousness, self-discipline, aging, culture, learning disabilities, financial stress, and burnout, as previously reviewed by others.[10] Possibly the MCAT scores are a proxy for academic preparation whereas uGPA is a proxy for affective as well as cognitive factors because noncognitive factors are significant contributors to success at the undergraduate level. Noncognitive factors are presently undervalued at the medical school admission stage. Perhaps interviews and applicant essays could be proxies for these noncognitive variables and would improve the predictive success of a screen based on MCAT score and uGPA alone. As more medical schools move toward problem-based curricula, selection of candidates using noncognitive criteria is becoming increasingly prevalent. However, non-cognitive criteria must also be tested for reliability and validity.[11]

The MCAT score and uGPA are about equally strong predictors of success in podiatric medical school. On this point, the literature is ambiguous. Some medical school admission committees weigh MCAT scores more highly than uGPA, suggesting that uGPA may be school specific and may be “inflated” by high grades in nontraditional premed courses.[12] Evans and Wen[13] indicated that uGPA has a higher predictive performance than does MCAT score as measured by basic science GPA and GPA at graduation. Others report that MCAT scores along with uGPA used at the admission stage can best predict success in medical school.[14]

Although the replication of these findings with a second independent sample makes the findings more credible, the sample is not random and comes from a single podiatric medical school. Caution must be exercised in generalizing the results to all podiatric medical students, not to mention all medical students, especially given the overrepresentation of minority students in the study sample. Some potentially confounding variables, such as motivation, health, and others, were omitted from the analysis. To the extent that these confounding variables are correlated with podiatric medical GPA, they will bias the estimates of the strength of the MCAT score and uGPA as predictors.

Conclusions

In this study, we conclude that the ability of the MCAT score and uGPA to predict success in podiatric medical school is probably overestimated by medical school faculty. A reevaluation of the medical school screening process is required. Perhaps the most definitive screening should occur in the first semester after matriculation rather than at the admission stage. This reevaluation may help increase the diversity of the physician workforce and avoid the predicted physician shortage in 2020.[15]

Financial Disclosure: None reported.

Conflict of Interest: None reported.