Skip to Content
You are currently on the new version of our website. Access the old version .
MDPIMDPI

Advanced

JAPMAJournal of the American Podiatric Medical Association
PDF
  • Journal of the American Podiatric Medical Association is published by MDPI from Volume 116 Issue 1 (2026). Previous articles were published by another publisher in Open Access under a CC-BY (or CC-BY-NC-ND) licence, and they are hosted by MDPI on mdpi.com as a courtesy and upon agreement with American Podiatric Medical Association.
  • Article
  • Open Access

1 January 2017

Components of an Evidence-Based Analytic Rubric for Use in Medical School Admissions

and
1
School of Podiatric Medicine, Barry University, 11300 NE 2nd Ave, Miami Shores, FL 33161, USA
2
Saba University, The Bottom, Saba, Dutch Antilles. Dr. Coffman is now with Lake Erie College of Osteopathic Medicine, Bradenton, FL, USA
*
Author to whom correspondence should be addressed.
J. Am. Podiatr. Med. Assoc.2017, 107(1), 65-71;https://doi.org/10.7547/16-008
Version Notes

Abstract

Attrition from medical school remains a serious cause of concern for the medical education community. Thus, there is a need to improve our ability to select only those candidates who will succeed at medical school from many highly qualified and motivated applicants. This can be achieved, in part, by reducing the reliance on cognitive factors and increasing the use of noncognitive character traits in high-stakes admissions decisions. Herein we describe an analytic rubric that combines research-derived predictors of medical school success to generate a composite score for use in admissions decisions. The analytic rubric as described herein represents a significant step toward evidenced-based admissions that will facilitate a more consistent and transparent qualitative evaluation of medical school applicants beyond their grades and Medical College Admissions Test scores and contribute to a redesigned and improved admissions process.

Seeking admission to medical school is a highly competitive process for the applicant and an often inexact process for the medical program. Admissions committees are charged with the difficult task of selecting only those candidates who, in their opinion, will be able to navigate the challenges presented by a rigorous medical school curriculum from among many highly qualified and motivated applicants. Using only preadmission data, admissions committees need to reliably identify applicants who will succeed at medical school, although exactly what preadmission data might be used to accurately predict success is much debated by the medical education community. In the absence of consensus, and reluctant to challenge the status quo, admissions committees take the conservative and traditional approach, basing their high-stakes decisions primarily on an applicant's cognitive abilities, identified by a combination of undergraduate grade point average (GPA), science GPA, and Medical College Admissions Test (MCAT) score, and perhaps to some extent on their own intuition or gut feeling of the applicant's potential.[1] This approach has been proven to no longer be effective. Every year, students with traditionally attractive admissions “numbers” leave medical education programs across the United States for a variety of reasons, including, but not limited to, academic underperformance, illness, family problems, or simply making a wrong career choice.[2]
A recent meta-analysis found the average attrition rate from medical school to be 11.1% (range, 2.4%–26.2%).[3] Attrition is also a problem in podiatric and osteopathic medical education: a 4-year study at Des Moines University reported an average attrition rate of 19.48% (range, 11.48%–24.44%) for podiatric medical students and 8.02% (range, 5.56%–10.7%) for osteopathic medical students, with most of the attrition occurring at the end of the first year.[4] High attrition rates represent a significant problem for the profession, the institution, and, particularly, the individual given the high cost of attendance at public and private medical schools.[5]
It is, thus, incumbent on admissions committees to continually reevaluate and assess the effectiveness of their admissions process so as to better identify and select applicants who will pass licensing examinations, succeed in medical school, and graduate to enter residency training. We have previously shown that admissions screens that rely solely on cognitive measures do not predict success in medical school as measured by GPA or graduation with certainty.[6,7] Thus, there is a need to reduce the historical reliance on cognitive factors and gut feeling in medical school admissions and broaden the scope of admissions criteria to include the noncognitive character traits that have also been linked with medical school success.
The medical education community has long been aware of the need to broaden the scope of admissions criteria to include noncognitive factors in the medical school admissions process.[6,8] In fact, Albanese et al[9] identified 87 compelling personal qualities that could be assessed during the admissions process. However, progress toward formally embracing these noncognitive factors in the admissions process has been slow, and a comparison of current admission practices and those used 25 years ago reveals that little has changed in the past two and a half decades.[10] Several reasons for the slow pace of adoption of these noncognitive factors in the admissions process may include institutional self-interest, inertia, a lack of creativity and imagination, or a variety of historical and philosophical factors.[9]
Medical educators are now moving toward a consensus that continued success throughout the medical school experience to graduation and beyond requires a combination of character traits associated with motivation, attention to detail, altruism, compassion, creativity, and grit, built on a solid foundation of academic excellence and clinical competence. With input from the committee responsible for the fifth comprehensive review of the MCAT, the Innovation Lab Working Group, and the Association of American Medical Colleges (AAMC) Admissions Initiative, the AAMC committee on admissions was recently able to reach agreement on nine noncognitive core personal competencies needed for students entering medical school: ethical responsibility to self and others, reliability and dependability, service orientation, social skills, capacity for improvement, resilience and adaptability, cultural competence, oral communication, and teamwork.[11]
The AAMC holistic review project[12] has facilitated the introduction of innovative admissions policies and practices that reduce reliance on cognitive measures of academic ability and encourage review of the “whole” applicant in making admissions decisions. This project, combined with implementation of the MCAT 2015 with new sections on behavior, sociology, and critical reasoning,[13] is symptomatic of the changes taking place in medical education, and together these initiatives have resulted in a paradigm shift in medical school admissions.
This paradigm shift in medical school admissions encouraged us to evaluate the literature on medical school admissions and our own admissions protocols and challenged us to identify better metrics predictive of medical school success. Based on this literature review and our own professional judgment, we assembled an innovative analytic rubric (Fig. 1) that incorporates the most likely predictors of student success in medical school. In this way, an applicant's knowledge, skills, and professional attitudes can be recorded and, thus, their suitability for interview and acceptance to medical school quantified, effectively removing gut feeling from admissions decisions. The analytic rubric combines multiple cognitive measures that best predict early medical school success[14] and noncognitive personal qualities that best predict success during the clinical years.[15] Statistically, we acknowledge that none of these measures used in isolation is ideal; however, when used together, a more reliable composite score for each applicant can be formulated and used to more accurately predict success.[16] Statistical evidence supports the use of a composite score in predicting success given that all of the predictor variables are oriented in the same direction, ie, a high score on the variable is associated with a better chance of success, and each predictor is intercorrelated positively.[16] The analytic rubric described herein thus represents a major step toward evidenced-based admissions that will facilitate a more consistent and transparent qualitative evaluation of medical school applicants beyond their grades and MCAT scores and contribute to a redesigned and improved admissions process.
Figure 1. Analytic rubric for scoring medical school applicants. GPA, grade point average; MCAT, Medical College Admissions Test.
The admissions rubric is divided into three domains that are aligned with the nine core personal competencies outlined previously herein: academic background, medical motivation, and character (Fig. 1). These domains incorporate both cognitive and noncognitive predictors such that each applicant's life experiences, community involvement, and creative abilities are considered alongside their academic credentials in a more holistic approach.
Academic background and medical motivation predictors can be scored using preadmission data from the applicant's file and used together to screen applicants for a campus visit. Applicants reaching a predetermined threshold can be invited to interview. During the campus visit, the applicant's character can be assessed using scores from the following tools: a structured interview, an Insight Resume, the Big Five Personality Test, a grit analysis, and the applicant's letters of recommendation from faculty and physicians. The data compiled during the campus visit are particularly important in admissions decisions that are ultimately based on a composite score for each applicant using a formulaic approach. In addition, composite scores may also be used to identify students accepted to medical school who may be at risk for attrition and in need of academic support on matriculation and those who are scholarship eligible. Although other researchers have used a composite score derived from the applicant's extracurricular activities, personal statements, and letters of recommendation to make admissions decisions and have reported a correlation with multiple measures of academic success,[17] to our knowledge, this is the first rubric to assimilate both academic variables and character traits using the composite score technique.

Academic Background

The academic background of each applicant is readily assessed, in part, using a compilation of their total undergraduate GPA, science GPA, and MCAT score. These cognitive variables remain the best, although not perfect,[6] predictors of academic success in the preclinical years of medical school, and when used together, their predictive value is increased over that for each variable individually.[18] The predictive value of these academic variables declines with increasing time from medical school admission.[14] This decline parallels the change in the curriculum from a focus on knowledge acquisition in the basic medical sciences to a focus on clinical cases, oral examinations, and patient interaction in the clinical curriculum,[14] hence the need to supplement these putative predictors of academic success with additional measures.
A significant limitation to the use of GPAs in medical school admissions is the variation in this metric among institutions. In reality, it is difficult to make a direct comparison between applicants with the same GPA from different institutions because grading policies and procedures vary. Furthermore, undergraduate GPA may be inflated at some institutions by high grades in nontraditional premedical courses.[6] Consequently, we propose to account for this variation in academic rigor and grading stringency between institutions by adjusting for academic or institutional selectivity combined with GPAs and MCAT scores to increase the predictive value of these academic measures. Although the benefits of the use of institutional selectivity in medical school admissions are much debated and contradictory evidence does exist,[19] the use of institutional selectivity has previously been demonstrated to improve the value of undergraduate and science GPAs[20] and MCAT scores[21] in predicting medical school success. Institutional selectivity is incorporated into the analytic rubric as institutional tier, although the exact nature of the interaction between this measure and undergraduate and science GPAs may be determined only by further study. We also hypothesize that successful completion of a postbaccalaureate master's program would be predictive of success in medical school because, in our opinion, good performance in such programs demonstrates that a student has remained committed to a career in the medical field and is less likely to drop out of school once accepted.

Medical Motivation

Each applicant's medical motivation can be quantified using the preadmission data available in the applicant's file. Each applicant's work experience, history of shadowing a physician, commitment to community service, teamwork, and leadership are all quantified (in hours) and recorded in the analytic rubric as evidence of the applicant's potential to work collaboratively in team-based practice. Hypothetically, each applicant's work experience might be scored using a 5-point scale (range, 0–4 points) as follows: applicants reporting 400 hours or more of work experience would score 4 points toward the composite score, applicants reporting 300 to 399 hours of work experience would score 3 points, applicants reporting 200 to 299 hours would score 2 points, those reporting 100 to 199 hours would score 1 point, and those reporting less than 100 hours of work experience would score no points. Likewise, a greater number of hours reported by the applicant for shadowing a physician and in community service would score more points toward the composite score. Teamwork and leadership are particularly important given that recent health-care reforms necessitate that practicing physicians work as members of an interprofessional clinical team to collectively address patient needs. Hypothetically, scoring teamwork and leadership would necessitate an evaluation of the applicant's roles in clubs and societies, with more points allotted for leadership roles.

Character

Accurately eliciting the character traits of each applicant represents a major challenge to medical school admissions committees. Consequently, in this model, we propose the use of multiple measures to assess an applicant's character and personality and, thus, their suitability for medical school admission. Scores from a structured interview, the Insight Resume, an applicant's letters of recommendation, the International Personality Item Pool (IPIP) Big Five Personality Test, and a grit analysis contribute to the composite score. Except for the letters of recommendation, all of this information can be obtained during the applicant's campus visit, which thus serves as a rich source of information for the admissions committee. Together, these measures provide the admissions committee with insight into each applicant's character.
Criticism of the traditional medical school admissions interview as unstructured, informal, and conversational, lacking objective scoring criteria, continues to grow, yet it still plays a prominent role in the selection process at most medical schools. Data derived from this unstructured interview lacks reproducibility and validity. Furthermore, subjective factors can introduce bias into the selection process. Consequently, we recommend the use of a more structured interview with a standardized question format and a scoring system to assess applicant responses. We suggest using clinical and basic science faculty in the interview to reduce the negative consequences of the halo effect and improve interview reliability. Furthermore, the structured interview provides important insight into the applicant's character that can be complemented by the information in the letters of recommendation, the Insight Resume, and the IPIP Big Five Personality Test. It provides an opportunity for the admissions committee to observe and record how the applicant interacts with others. Candidates could score points in the interview for communication skills, knowledge of the profession, and articulation of their realistic career goals.
During the campus visit, each applicant should complete an Insight Resume, a written experiential assessment used with the permission of Oregon State University. It is a series of six questions that assesses the applicant's leadership/group contributions, knowledge in a field/creativity, ability to deal with adversity, community service contributions, ability to handle systemic challenges, and goals/task commitment. Each question is scored on a 4-point scale (range, 0–3 points), for a possible total of 18 points on the Insight Resume.
Letters of recommendation have long been used as a tool in the medical school admissions process to screen applicants for interview and admission despite its poor predictive validity and reliability and difficulties in interpretation.[22,23] Given that the letter writers are chosen by the applicants, their credibility as a source of additional reliable information is questionable. Furthermore, there always remains the problem of fraudulent letters of recommendation. They may, however, provide some utility when written by physicians who the applicant has shadowed and scored by content analysis.
The scores from the 50-item version of the IPIP Big Five Personality Test reveal each applicant's personality traits: extroversion, conscientiousness, neuroticism, agreeableness, and openness. By adding the scores for multiple personality traits conjointly rather than adding individual traits into the analytic rubric, the predictive power of the MCAT score and GPA is improved.[24] Although historically used to establish connections between personality and job performance,[25] the IPIP Big Five Personality Test has been used more recently to identify predictors of academic achievement as measured by GPA[26] and is, thus, a legitimate candidate for inclusion in the analytic rubric. The extroversion dimension measures excitability, emotional expression, enthusiasm, and sociability. Although it is a desirable trait for those in the health-care profession interacting with patients, extroverts are easily distracted and may spend more time socializing in the early preclinical years and, thus, may receive lower grades than introverts, who are more focused on study. Conscientiousness is a motivational personality trait that reflects orderliness, self-regulation, attention to detail, and impulse control.[27] It facilitates goal-oriented behavior, such as thoughtful planning, and has been consistently and closely linked to academic success.[28] Conscientious individuals have been shown to perform well in medical school,[15] where effort and volition are crucial to success. The neuroticism dimension measures emotional instability, anxiety, and irritability and is, thus, negatively correlated with academic success. This negative trait should be scored in reverse, with low levels of neuroticism being more desirable in medical school applicants, because it is associated with calmness under pressure and fewer negative emotions, such as anger, guilt, and depression. The agreeableness dimension measures altruism, affection, trustworthiness, and kindness. These are all traits that will become increasingly important as the ability to work in interdisciplinary health-care teams becomes a more desirable characteristic. High scores in this domain are likely associated with good clinical performance as patient contact increases. The openness dimension measures insightfulness, intellectual curiosity, imagination, and creativity. This dimension is associated with success in practicing the art of medicine in the clinical years and in clinical practice. Interestingly, it is associated with improved patient care outcomes and high physician ratings by patients.[29]
In our opinion, the results of the Big Five Personality Test contribute substantially to providing a more complete picture of the medical school applicant, particularly when combined with other measures of personality as described herein. These Big Five personality traits may be measured at the admissions stage and used to predict short-term (first semester GPA) and longer-term (graduation or residency placement) success because the component traits have been shown to remain stable in adults over time.[30]
As part of the campus visit, medical school applicants should complete a self-reporting questionnaire called a grit analysis that is scored using the Grit Scale.[31] Grit is defined as the perseverance and passion for long-term goals. It is often cited as being just as important to academic achievement as intellect, making it another legitimate candidate for inclusion, particularly given that it is thought to be an excellent predictor of achievement in self-selected goals, such as medical school graduation.[31,32] The Grit Scale measures endurance and the ability to persevere with high reliability and validity and, thus, provides the admissions committee with supplemental noncognitive data regarding the applicant's character that we believe is useful in predicting success in the rigorous medical school curriculum.[32] Although grit is a lower-level personality trait in the conscientiousness domain, it has demonstrated incremental predictive validity of success measures over conscientiousness; thus, in our opinion, the assessment of both grit and conscientiousness is warranted.[31]

Discussion and Further Work

Medical school attrition remains a significant cause of concern to the medical education community. Therefore, as a matter of considerable urgency, medical school admission policies and procedures need to change to more accurately identify applicants at the admissions stage who will succeed in medical school. In our opinion, this can be achieved, at least in part, by embracing the use of noncognitive measures in the admissions process.
Herein, we described an analytic rubric that combines research-derived predictors of medical school success. Cognitive measures predictive of early success are combined with character traits that are predictive of later success to formulate a composite score that is more valid and reliable than any single component score. The composite score may be used to determine each applicant's eligibility for interview and acceptance and to identify individuals who may be at risk for academic underperformance and attrition after matriculation. As outcome data accumulate from use of the analytic rubric, each of the component measures will be evaluated for validity and reliability; some may be removed or improved, and others may reveal themselves as more information regarding correlation strengths and effect size is gathered and noncognitive assessment tools improve. Longitudinal analysis may reveal interactions between predictors and generate data that may help assign weights to the stronger predictors of success.
From a resource perspective, as the analytic rubric is implemented, a considerable amount of faculty development and training will be required to ensure consistent scoring of each of the predictors. In fact, it is likely that even after faculty and admission's office staff have been trained, implementing the proposed system on a regular basis will be more labor-intensive and time-consuming than current practices, perhaps necessitating the hiring of additional admissions staff. In addition, with implementation, some limitations may be revealed. The inherent limitations associated with self-reporting may represent a problem for admissions committees as they face the dilemma of distinguishing an honest response from a response that the applicant thinks the committee wants to hear/read or a response that has been coached. Despite these limitations, we believe that the use of this analytic rubric represents an important step toward a more thorough, transparent, and ethically defensible assessment of applicants to allopathic, osteopathic, and podiatric medical programs. We, therefore, advocate for the widespread adoption of analytic rubrics of this type in all medical schools, including the nine podiatric medical schools in the United States. Use of this qualitative tool may well result in an improvement in multiple mission critical outcome measures, such as reduced attrition, increased graduation rate, and increased residency placement.

Acknowledgments

David Molnar and Elizabeth Martinez for their helpful comments on earlier drafts of the manuscript.

Financial Disclosure

None reported.

Conflicts of Interest

None reported.

References

  1. Barr DA: Science as superstition: selecting medical students. Lancet 376: 678, 2010.
  2. Maher BM, Hynes H, Sweeney C, et al: Medical school attrition: beyond the statistics: a ten year retrospective study. BMC Med Educ 13: 13, 2013. Available at: http://www.biomedcentral.com/1472-6920/13/13. Accessed January 13, 2016.
  3. O'Neill LD, Wallstedt B, Eika B, et al: Factors associated with dropout in medical education: a literature review. Med Educ 45: 440, 2011.
  4. Yoho RM, Vardaxis V, Comstock K: Admission characteristics and academic performance of podiatric and osteopathic medical students at Des Moines University. JAPMA 100: 276, 2010.
  5. Youngclaus J, Fresne JA: Physician education debt and the cost to attend medical school: 2012 update. Association of American Medical Colleges website. Available at: https://aamc-orange.global.ssl.fastly.net/production/media/filer_public/8d/aa/8daa5c2d-838b-4690-a353-170c7f4a7bab/physician_education_debt_and_the_cost_to_attend_medical_school_2012_update.pdf. Published February 2013. Accessed December 22, 2015.
  6. Shaw GP, Velis E, Molnar D: Can we predict podiatric medical school grade point average using an admission screen? JAPMA 102: 499, 2012.
  7. Sesodia S, Molnar D, Shaw GP: Can we predict 4-year graduation in podiatric medical school using admission data? JAPMA 102: 463, 2012.
  8. Hurtado A: Impressions of the 1956 Institute on the evaluation of the student: the appraisal of applicants to medical school. J Med Educ 32: 847, 1957.
  9. Albanese MA, Snow MH, Skochelak SE, et al: Assessing personal qualities in medical school admissions. Acad Med 78: 313, 2003.
  10. Monroe A, Quinn E, Samuelson W, et al: An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med 88: 672, 2013.
  11. Koenig TW, Parrish SK, Terregino CA, et al: Core personnel competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admissions process? Acad Med 88: 603, 2013.
  12. Adams AN, Bletzinga RB, Sondheimer HM, et al: Roadmap to Diversity: Integrating Holistic Review Practices Into Medical School Admissions Processes, Association of American Medical Colleges, Washington, DC, 2010.
  13. Schwartzstein RM, Rosenfeld GC, Hilborn R, et al: Redesigning the MCAT exam: balancing multiple perspectives. Acad Med 88: 560, 2013.
  14. Siu E, Reiter HI: Overview: what's worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ 14: 759, 2009.
  15. Lievens F, Ones DS, Dilchert S: Personality scale validities increase throughout medical school. J Appl Psychol 94: 1514, 2009.
  16. Wainer H: Estimating coefficients in linear models: it don't make no nevermind. Psychol Bull 83: 213, 1976.
  17. Peskum C, Detsky A, Shandling M: Effectiveness of medical school admissions criteria in predicting residency ranking four years later. Med Educ 41: 57, 2007.
  18. Mitchell KJ: Use of MCAT data in selecting students for admission to medical school. Med Educ 62: 871, 1987.
  19. Blue AV, Gilbert GE, Elam CL, et al: Does institutional selectivity aid in the prediction of medical school performance. Acad Med 75: 31, 2000.
  20. Clapp TT, Reid JC: Institutional selectivity as a predictor of applicant selection in medical school. J Med Educ 51: 850, 1976.
  21. Wiley A, Koenig JA: The validity of the medical college admission test for predicting performance in the first two years of medical school. Acad Med 71: 83, 1996.
  22. Kirchner GL, Holm MB: Prediction of academic and clinical performance of occupational therapy students in an entry-level master's program. Am J Occup Ther 51: 775, 1997.
  23. Ross CA, Leichner P: Criteria for selecting residents: a reassessment. Can J Psychiatry 29: 681, 1984.
  24. Shen H, Comrey AL: Predicting medical students' academic performances by their cognitive abilities and personality characteristics. Acad Med 72: 781, 1997.
  25. Griffin B, Wilson I: Associations between the big five personality factors and multiple mini-interviews. Adv Health Sci Educ 17: 377, 2012.
  26. Komarraju M, Karau SJ, Schmeck RR, et al: The Big Five personality traits, learning styles, and academic achievement. Pers Indiv Diff 51: 472, 2011.
  27. John OP, Naumann IP, Soto J: “Paradigm Shift to the Integrative Big Five Trait Taxonomy: History, Measurement and Conceptual Issues,”in Handbook of Personality: Theory and Research, 3rd Ed, edited by OP John, RW Robbins, LA Pervin, p 114, Guilford Press, New York, 2008.
  28. Poropat AE: A meta-analysis of the five-factor model of personality and academic performance. Psychol Bull 135: 322, 2009.
  29. Duberstein P, Meldrum S, Fiscella K, et al: Influences on patients ratings of physicians: physicians demographics and personality. Patient Educ Couns 65: 270, 2007.
  30. Duckworth AL, Peterson C, Mathews MD, et al: Grit: perseverance and passion for long term goals. J Pers Soc Psychol 92: 1087, 2007.
  31. Duckworth AL, Quinn PD: Development and validation of the Short Grit Scale (GRIT-S). J Pers Assess 91: 166, 2009.
  32. Cobb-Clark DA, Schurer S: The stability of big-five personality traits. Econ Lett 115: 11, 2012.

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Metastatic Melanoma of a Metatarsal
Previous
New Initiatives
Next