Can We Predict 4-year Graduation in Podiatric Medical School Using Admission Data?

Abstract

Background: This study examined the predictive ability of educational background and demographic variables, available at the admission stage, to identify applicants who will graduate in 4 years from podiatric medical school. Methods: A logistic regression model was used to identify two predictors of 4-year graduation: age at matriculation and total Medical College Admission Test score. The model was cross-validated using a second independent sample from the same population. Cross-validation gives greater confidence that the results could be more generally applied. Results: Total Medical College Admission Test score was the strongest predictor of 4-year graduation, with age at matriculation being a statistically significant but weaker predictor. Conclusions: Despite the model’s capacity to predict 4-year graduation better than random assignment, a sufficient amount of error in prediction remained, suggesting that important predictors are missing from the model. Furthermore, the high rate of false-positives makes it inappropriate to use age and Medical College Admission Test score as admission screens in an attempt to eliminate attrition by not accepting at-risk students. (J Am Podiatr Med Assoc 102(6): 463–470, 2012)

Barry University (Miami Shores, Florida) is a federally designated minority-serving institution in the Southeastern United States with a Carnegie classification of doctoral/research university. More than half of the 9,000 students are underserved minorities, and more than half of the population of the region are immigrants. The Barry University School of Podiatric Medicine has been educating doctors of podiatric medicine for 27 years. Every year, the School of Podiatric Medicine receives more than 200 applications for the 67 seats available in the freshman class. Consequently, making decisions about which applicants to accept is a challenging, high-stakes task for the admission committee as they strive to evaluate each candidate’s potential for successful completion of the program.

Although there is no consensus in the literature as to the usefulness of cognitive measures, such as revised Medical College Admission Test (MCAT) scores and undergraduate grade point average (GPA) in predicting student performance at medical school, [1] at Barry University the admission committee continues to evaluate each candidate on the basis of these traditional measures. The MCAT scores are considered to be a good (although not perfect) indicator of readiness for medical school, [2,3] and in combination with undergraduate GPA, they provide an even better basis for prediction of performance, particularly in the first 2 years of medical school. In addition, MCAT scores are a good indicator of future performance on Board examinations. [3] Our use of MCAT scores to influence admission decisions is supported by the observations of Petek and Todd, [4] who showed that MCAT scores were correlated with preclinical GPA and with performance on Part 1 from the examination of the National Board of Podiatric Medical Examiners.

Use of undergraduate GPA was also supported by the observations of Salvatori, [5] who claimed that overall undergraduate GPA was the best predictor of academic performance in all of the health professions. Furthermore, Evans and Wen [6] suggested that overall MCAT scores were poorer predictors of academic performance in schools of osteopathic medicine. Consequently, most medical schools in the United States continue to emphasize undergraduate GPA and performance on the MCAT in the consideration of applicants. [7]

The importance of selecting suitable, racially/ethnically diverse candidates from a large applicant pool is heightened since the cost of attrition for medical schools is high in terms of institutional prestige, accreditation, and resource costs. The cost of attrition is also extremely high for the medical student in terms of lowered self-esteem and increased economic burden. In 2008 and 2009, the total annual cost of attending a public medical school (including living expenses and health insurance) was $44,390, and the cost of attending a private institution was $62,243. [8] The return on this investment may be good when graduates are paid as physicians, but for students who do not graduate, the burden of educational debt may be onerous.

The Association of American Medical Colleges is committed to increasing the number of medical school students by 30% by 2015 [9] and to increasing diversity in medical education. This is a challenge since underrepresented minority medical students are less likely to graduate in 4 years compared with their peers. [10] In fact, minority students and older students are more likely to experience academic difficulties than are their peers. [11,12] Older students are also more likely to achieve lower scores on United States Medical Licensing Examination steps 1 step 2. [13] Medical schools strive to nurture diversity in medical education in an effort to increase diversity in the physician workforce [14] congruent with national health-care goals.

There is a paucity of recently published data in the literature regarding the value of admission data (demographic and educational) for predicting student success in podiatric medical schools. In this study, we evaluated demographic and educational background variables as possible predictors of success in podiatric medical education and described a model that may be used to predict the probability of a student graduating in 4 years. Although other studies have used United States Medical Licensing Examination Steps 1 and 2 pass rates [13,15] to predict attrition [16,17] and other adverse academic status events, [2,11,18] to our knowledge, this is the first study to evaluate the value of admission data in predicting academic success for podiatric medical students, using 4-year graduation rate as an outcome measure.

Methods

Sample

Students were included in the sample if they were first-time full-time podiatric medical students who entered school between August 1995 and August 2005. The sample of 514 students averaged 27 years of age and consisted of 48% underrepresented minorities, 59% men, and 15% foreign-born students. Since this sample is not representative of the population of podiatric medical students in the United States, caution must be exercised in generalizing the results of this study to the entire population. Data on demographic characteristics, educational background, and graduation from the program were obtained from the program database.

Framework

Classical test theory provides the theoretical framework for this research. This framework forecasts a criterion variable (graduation in 4 years) based on predictors. Theoretically, predictors fall into three categories: educational background, academic performance in the podiatric medicine curriculum, and noncognitive predictors. Predicting the probability of 4-year graduation formalizes the uncertainty in prediction; furthermore, the probability of 4-year graduation is the measure of primary interest to faculty and students.

To evaluate the success of the statistical model, an alternative framework—the clinical disease detection model—was used. The reason for using this alternative framework is that statistical measures of success in prediction, such as odds ratios, are nonintuitive and less familiar to most faculty than are measures of predictive success from the disease detection model.

Data Analysis

Linear regression is the statistical method typically used to identify predictors. When the dependent variable is dichotomous, logistic regression is better suited than linear regression because the assumptions of linear regression are violated. [19] Similar to linear regression, logistic regression evaluates the relationship of several predictors to the criterion variable.

A logistic regression model is evaluated in three steps. First, the extent to which the model fits the observed data is evaluated. Next, the statistical significance and effect size of individual predictors are evaluated. Since the predictors are measured on different scales, they were converted to z scores using the sample mean ± SD. Standardization makes the measure of effect size more comparable across predictors because a change of one unit in any variable always represents a change of 1 SD. In the third step, a classification table is generated that compares predicted outcomes with observed outcomes. Sensitivity, selectivity, and predictive value are calculated from the classification table.

In this study, we used a cross-validation technique to assess how the results of the logistic regression model would generalize to an independent data set. The sample was randomly divided into two parts: 58% in an estimation subsample (n = 298) and 42% in a validation subsample (n = 216). A logistic regression was run with all of the predictors that made theoretical or logical sense. Predictors were then eliminated if they were not statistically significant until all of the remaining predictors were statistically significant. This fitting process optimizes the model to best fit the data in the estimation sample. Then, this logistic regression model was run on the validation sample from the same population. Reproducing the results in two independent samples (estimation and validation) is essentially equivalent to replicating the results, giving a higher level of confidence than could be achieved from a model with statistically significant predictors from a single sample alone. Generally, the model will not fit the validation subsample as well as it fits the estimation subsample. Consequently, when the model was run on the validation subsample, any predictors that were not statistically significant were eliminated. In other words, all of the predictors in the final logistic regression model were statistically significant and had the same sign for the estimation and validation samples.

Results

Predictors of 4-Year Graduation

The MCAT component scores, undergraduate cumulative and science GPAs, age at matriculation, race/ethnicity, gender, marital status, citizenship, and state/country of origin were explored as predictors of 4-year graduation. Of these variables, age, total MCAT score, and race/ethnicity were statistically significant predictors in the estimation sample, but race/ethnicity was not replicated as a predictor in the validation sample. Consequently, our most parsimonious model included two predictors: age at matriculation and total MCAT score. For these two variables,

Table 1. Values for Variables Used in the Analysis

shows the differences between students who graduated in 4 years and those who did not graduate or had graduation delayed. Of those who did not graduate in 4 years, 60% were attrition and 40% were delayed graduation. Note that attrition is not necessarily a negative outcome from the student’s perspective. Thirteen percent of these students left the program with GPAs greater than 3.5 and probably transferred to allopathic or osteopathic medical programs.

For the full sample, regression results indicated that the overall model was statistically reliable in distinguishing between success and failure in achieving 4-year graduation. Model fit statistics revealed a good-fitting model (−2 log likelihood = 641.190, Hosmer-Lemeshow goodness of fit = 5.792, P = .67. A P > .05 indicates no statistically significant difference between predicted and observed values. The generated model was significantly different from the constant-only model [χ²₂ = 31.621, P < .001].).

Table 2. Effect of MCAT Score and Age at Matriculation on 4-Year Graduation (N = 514)

shows the impact of age and MCAT scores on 4-year graduation. Each cell shows the 4-year graduation rate for the full sample used in this study. When the total cohort is examined, it is seen that increasing age diminished the 4-year graduation rate by 18% (from 63% to 45%), whereas higher MCAT scores increased the 4-year graduation rate by 26% (from 38% to 64%). However, a closer examination of the data reveals that only 23% of older students with MCAT scores less than 14 graduated in 4 years. This graduation rate is a little more than half the rate of younger students with similar MCAT scores. The table also shows that when MCAT scores are greater than 14, older and younger students show increased graduation rates.

Regression coefficients are presented in

Table 3. Logistic Regression Analysis Summary for Age and MCAT Score Predicting 4-Year Graduation

. Recall that the variables are standardized, making effect size more comparable across predictors because a change of one unit in any variable always represents a change of 1 SD. Wald statistics indicated that both variables significantly predict 4-year graduation. Odds ratios for these variables indicated that increasing age by 1 SD (5.6 years) decreased the odds of 4-year graduation by 0.69 times, whereas increasing the MCAT score by 1 SD (4.9 points) increased the odds of 4-year graduation by 1.53 times.

Classification Success

Classification success was evaluated using a predicted probability of 4-year graduation of 0.645 as the cutoff point to distinguish predicted success (4-year graduation) from predicted failure (attrition or delayed graduation). This cutoff point was determined based on a receiver operating characteristic curve analysis and was chosen to set sensitivity equal to specificity, reflecting neutrality regarding the preference for higher sensitivity (failures predicted more accurately) as opposed to higher selectivity (successes predicted more accurately). This procedure represents a compromise because increasing either sensitivity or specificity causes a necessary decrease in the other. [20] Using the 0.645 cutoff point, the sensitivity of the predictors closely matched the specificity, and the overall accuracy of prediction was 61%.

Table 4. Classification Success for the Predictive Model of 4-Year Graduation

shows how the actual observed 4-year graduation or not graduating in 4 years matched the predicted success or failure. The columns represent the observed graduation outcomes, and the rows represent the predicted graduation outcomes. For example, the second column shows that of the 328 students who graduated in 4 years, 130 were incorrectly predicted to not graduate in 4 years and 198 were correctly predicted to graduate.

Sensitivity, specificity, and positive and negative predictive values were also calculated to assess the predictive success of the model in light of the actual incidence of 4-year graduation (

Table 5. Sensitivity, Specificity, and Predictive Value for Logistic Regression Models Predicting Not Graduating in 4 Years Compared With Random Assignment and Predicting Prostate Cancer

). These parameters help in interpreting the classification success of the model. Sensitivity is a measure of how well the model identifies students who will not graduate in 4 years. In the logistic regression model used in the present study, a sensitivity of 61% represented the ratio of the 113 students who were correctly predicted not to graduate to the 186 students who actually did not graduate. Specificity is a measure of how well the model identifies students who will graduate in 4 years. In this study, a specificity of 60% represented the ratio of the 198 students who were correctly predicted to graduate in 4 years to the 328 students who actually graduated in 4 years.

Sensitivity and specificity are measures of the model getting the prediction correct. Positive predictive value is a measure of true-positives as a proportion of those predicted not to graduate. The positive predictive value of 47% represents the ratio of the 113 students who were predicted not to graduate and did not graduate to the 243 students who were predicted not to graduate. Negative predictive value is a measure of true-negatives as a proportion of those predicted to graduate in 4 years. The negative predictive value of 73% represents the ratio of the 198 students who were correctly predicted to graduate in 4 years to the 271 students predicted to graduate in 4 years.

To put the model’s predictive success into perspective, it was compared with the success of a model that randomly assigns 4-year graduation with a probability of 0.635 (the proportion of 4-year graduates in the sample). Only to the extent that the logistic regression model predicts better than random assignment (Table 5) can we claim that admission data are useful in predicting 4-year graduation. The model did predict better than random assignment.

The model’s predictive success (Table 5) was also evaluated when students with a “B” average or better who did not graduate in 4 years were removed from the sample. For this variation of the model, classification success was evaluated using a predicted probability of 4-year graduation of 0.763 as the cutoff point to distinguish predicted success (4-year graduation) from predicted failure (attrition or delayed graduation) so that the sensitivity of the predictors closely matched the specificity. For the estimation and confirmation samples, age and total MCAT score remained statistically significant predictors. There was an insignificant increase in the overall accuracy of the model from 61% to 63%. There was an insignificant increase of 3% in the sensitivity (proportion of actual attrition/delay correctly predicted) and specificity (proportion of actual 4-year graduation correctly predicted) of the model.

Discussion

This study was conducted to evaluate the use of available admission data as predictors of 4-year graduation from a podiatric medical program. Of the educational background and demographic variables examined, only two (total MCAT score and age at matriculation) were found to be statistically significant predictors of 4-year graduation in the estimation and validation subsamples. Recall that the variables were standardized, making effect size more comparable across predictors. The results showed that the standardized total MCAT score was the strongest predictor of 4-year graduation. These results are in partial agreement with a variety of previous studies [3,4,21] that reported MCAT scores to be good predictors of readiness for medical school and future performance on Board examinations. In addition, MCAT scores were also reported to be useful as predictors of students who might need assistance during the medical program. [3] This study also found that student age at matriculation was a statistically significant, although weaker, predictor of 4-year graduation. Therefore, young individuals with high MCAT scores were predicted to have a better chance of graduating in 4 years than older individuals with lower MCAT scores. However, the results seem to contradict some literature [5] since undergraduate GPA is not a predictor of 4-year graduation.

The present study makes two important contributions to the literature on podiatric medical education: a focus on student outcomes and the use of a more sophisticated method for predicting those outcomes. The main emphasis of the previous studies was on the value of the MCAT score and undergraduate GPA in predicting academic performance during the program, a formative or process measure. The present study addresses the value of admission data in predicting 4-year graduation, a summative or outcome measure. Thus, it identifies predictors of students achieving the defined outcome of the program rather than identifying predictors of their progression in the program, such as passing the Board examinations. In this regard, the present study differentiates itself from the previous literature.

Another value of the present study lies in the logistic regression method used, which allowed several variables to be simultaneously associated with the dichotomous variable of interest (4-year graduation). The contribution of individual predictors in logistic regression is difficult to assess and interpret, and an explanation is warranted here. The magnitude of a predictor’s effect on an outcome is expressed as an odds ratio. The larger the impact of a predictor, the further the odds ratio will be from 1. Odds and odds ratios are best understood through an example using data from Table 2. If a graduate is aged 29 years or older, the odds of 4-year graduation to attrition are 0.82:1 (45%/55% to 1). If a graduate is younger than 29 years, the odds are 1.70:1. The odds ratio for students 29 years or older is 0.82/1.70, or 0.48. Since the odds ratio is far from 1.00, being 29 years or older is a strong predictor of attrition or delayed graduation.

In this study, classification success was evaluated using a predicted probability of 4-year graduation of 0.645 as the cutoff point. This cutoff point was determined based on a receiver operating characteristic curve analysis and was chosen to set sensitivity equal to specificity, reflecting neutrality regarding the preference for higher sensitivity (failures predicted more accurately) as opposed to higher selectivity (successes predicted more accurately). Choosing a cutoff point requires a trade-off between more accurate overall predictions and predictions that are less accurate but do a better job of identifying students who are at risk for not graduating in 4 years. Lowering the predicted probability that was used as the cutoff point between success and failure would have increased the overall accuracy of the prediction, but fewer and fewer students who actually did not graduate in 4 years would have been correctly predicted to not graduate. Consequently, fewer students would potentially benefit from any remediation interventions.

In summary, this model shows a better-than-random ability to correctly classify students, and a classification success as good as the prostate-specific antigen test, a well-accepted disease detection test (Table 5). [22] The present study also validated the model using a second independent sample from the same population, giving greater confidence that the results can be generalized. To our knowledge, this is the first time that cross-validation has been applied in podiatric medical education research.

Although the logistic regression model correctly classified 61% of the students who would not achieve 4-year graduation, “solving” the attrition problem at the admission stage by implementing screening based on this model would come at the cost of discarding 53% of the at-risk applicants who are false-positives and, if admitted, would successfully complete the program in 4 years. Another way of looking at this is to consider that 40% of the applicants who would successfully graduate in 4 years would be barred from the profession to “screen out” only 60% of the students who do not graduate in 4 years. The trade-off is even more severe when admission screening bars applicants who are delayed graduates but successful in residencies and in practice.

Although the classification success achieved by using the model is higher than that achievable by random classification and is as good as prostate-specific antigen testing (Table 5), it is likely that there are additional unidentified predictors. We speculate that among these as yet unidentified predictors are a variety of noncognitive factors, such as motivation and health. Inability to meet the academic demands of a rigorous medical training program is only one reason for students not graduating in 4 years. Low commitment to the podiatric medical profession and “personal” reasons are also cited as factors. Many educators believe that college grades and standardized test scores address only the possibility of academic difficulty. Academic and nonacademic causes of attrition are not independent in a statistical sense. For example, weak commitment to medicine as a career choice would weaken motivation to perform well on the MCAT and could be reflected in lower scores. Or a student with a weak academic background might compromise his or her health or marriage in a heroic effort to maintain good academic standing, but adverse life circumstances, not academic issues, would be recorded as the proximate cause of withdrawal, although life circumstances may not be the cause in fact. It is also clear that age by itself may be a proxy for an amalgam of many additional noncognitive factors, eg, family demands competing with demands of a rigorous medical education. However, within its limits, the model is equally able to predict not graduating due to academic or personal reasons since removing attrition/delay due to adverse life circumstances made no significant change in the model or its classification success (Table 5).

Perhaps the most significant finding of this study is the identification of total MCAT score and age as predictors of 4-year graduation. If 4-year graduation success was to be the sole goal of admission screening, then recruitment of students for podiatric medical school should be directed at young applicants with high MCAT scores. However, a cautionary note should be added. The prediction of attrition and delayed graduation generates many false-positives and does not equate with actually not graduating in 4 years. This is analogous to screening tests for any disease condition. Many of these older students with weaker educational backgrounds can, and do in fact, succeed. Because there are so many false-positives, the results of this study suggest that MCAT score and age should be used to identify students who need academic support after enrollment rather than to reject older applicants with lower MCAT scores.

Given the high cost of attrition/delayed graduation, [8] identifying potentially at-risk students would seem to be an important first step in the process of preventing it. Having identified such students, medical schools could more efficiently target their resources at improving these students’ chances of success in the programs with the goal of producing higher-than-predicted 4-year graduation rates. Each school could, therefore, create an academic support strategy with the goal of giving academic support to at-risk students before poor academic performance leads to their attrition or delayed graduation. Payne et al [23] and Segal et al [24] reported that such an interventionist approach resulted in an increased graduation rate from medical school.

Conclusions

In this study, the predictive ability of admission data to identify 4-year graduates from medical school was tested using the logistic regression model. Based on an analysis of the historical data used, this model identified predictors of 4-year graduation: age at matriculation and total MCAT score. However, despite the model’s capacity to identify at-risk students better than random classification, a sufficient amount of misclassification remained, suggesting that it would be inappropriate to try to “screen out” at-risk applicants at the admission stage and that the cognitive domain may not be the only influence on graduation. Affective elements have recently begun to receive more attention to improve admission to, retention in, and graduation from medical programs. [23,24]