Measuring Teaching Effectiveness—or Not

Abstract

Faculty in the present-day academic medicine environment are expected to perform multiple functions, notably, the provision of high-quality teaching to the medical professionals of tomorrow. However, evaluating the effectiveness of this teaching is particularly difficult. Student evaluations of teaching, despite their many flaws, are widely used as a convenient tool to measure teaching effectiveness. Administrators continue to routinely use student evaluation of teaching surveys in faculty retention/promotion and merit pay decisions. This practice should be reevaluated since it may have unintended consequences, such as grade inflation and content debasement, and may contribute to faculty leaving the institution and even the profession. A more valid, reliable, and formative protocol for the evaluation of genuine teaching effectiveness needs to be developed as a matter of some urgency. In this review, alternatives to the student evaluation of teaching are explored to better measure true teaching effectiveness.

In the present-day academic medicine environment, the role of faculty is “multidimensional.”[1] Faculty must actively pursue and secure research funding, provide service to the institution and the community, publish scholarly works in prestigious peer-reviewed journals, and provide high-quality instruction, all while being evaluated and scrutinized by an ever-vigilant administration intent on improving the quality of education and on increasing enrollment.

Although the measurement and evaluation of research productivity and service endeavors are problematic, the valid and reliable evaluation of faculty teaching effectiveness is even more difficult and may, in fact, be one of medical education’s greatest challenges.[2] Measuring teaching effectiveness, if it even exists as a metric, is made even more difficult because of the absence of a clear definition of what is actually good teaching. In many medical schools, the evaluation of teaching effectiveness is attempted using often-vague student evaluation of teaching surveys, despite their many documented limitations[3] and their potential for misuse by administration in decisions of compensation, retention, and tenure. Oftentimes, these surveys do not generate useful data and do not reflect a true measure of teaching effectiveness. Faculty often resent the use of these subjective surveys, dismissing them as “popularity contests” or even “consumer satisfaction” surveys. Faculty committed to academic rigor and to making students think generally receive lower student evaluation of teaching scores,[4] ie, there is a negative correlation between academic rigor (workload, student effort, and involvement) and the student’s evaluation of teaching.[4] Faculty point to the possibility that the current use of these surveys may have unintended consequences, such as grade inflation, since students are known to give better evaluations to lenient-grading faculty.[5] This perception restricts teaching innovations and deters faculty from challenging and questioning students for fear of a negative evaluation. In fact, the perception that better evaluations can be “bought” with grades alters faculty behavior[6] and may even invalidate the student evaluation of teaching instrument.[7] Some faculty may simply resort to entertaining the student population and place too little emphasis on learning, similar to a modern-day Dr. Fox.[8] The students themselves often resent repeatedly completing these so-called surveys of teaching effectiveness and repeatedly question their usefulness since they do not see the consequences of their anonymous actions.

There are many potential sources of error in the use of the student evaluation of teaching survey, ranging from poorly constructed questions, rater bias, timing of survey administration, and environmental factors to the presence of a course laboratory component and academic area. I submit that the anonymous completion of the end-of-semester surveys by students is not an effective paradigm by which to evaluate teaching effectiveness and may even be responsible for content debasement.[9] In fact, this strategy is so flawed that harsh student comments made anonymously and without fear of recourse can lead to faculty demoralization and disillusionment and may contribute to faculty leaving the institution and even the profession. Lowenstein et al[10] previously reported that lack of recognition for teaching and clinical excellence is closely linked to this intent to leave careers in academic medicine.

In this era of increased accountability in higher education following publication of the Miller report,[11] it is appropriate that the evaluation of teaching effectiveness be revisited. In fact, if medical school faculty are to develop to their full potential and provide the highest-quality education, a more valid, reliable, and formative protocol for the evaluation of genuine teaching effectiveness needs to be developed as a matter of some urgency. This protocol would reward evidence-based risk taking in pedagogy, intrinsic motivation, and the development of teaching resources and teaching innovations that demonstrate measurable improvements in student learning outcomes. This protocol would involve the collection of several different types of data, be transparent, provide benchmarks for faculty, and involve mentoring and career guidance as appropriate. It would involve peer observation of classroom instruction with formative, documented feedback to faculty. This process could be undertaken at midterm to allow midpoint corrections to be made, a paradigm that is well perceived by students.[12]

Although student attitudes toward nontraditional teaching paradigms that encourage critical thinking skills remain a significant barrier to their implementation by faculty,[13,14] student input to evaluations of teaching effectiveness remains an important source of raw data. However, these data must be collected in a confidential rather than an anonymous manner, which is unambiguous and nonthreatening, perhaps in the form of an end-of-semester (or midsemester) small focus group led by teaching assistants or faculty colleagues. A more accurate assessment of what a student learned in a course, a true indicator of teaching effectiveness, would be gained by analysis of student performance in subsequent classes controlled for student characteristics and performance in prerequisite classes.[9,15] In the academic medicine environment, this could feasibly be ascertained by analysis of the subject-specific scores on national board examinations. Surveys of graduates and residents would also provide relevant data regarding the retention and relevance of learned material that could be used to inform instruction, although response rates on such surveys may well be lower.

And finally, if faculty are to develop to their full potential in the present-day academic medicine environment and deliver the highest-quality medical education, the limitations of the teaching-effectiveness survey as currently administered need to be acknowledged. Junior faculty should feel able to share the results of their teaching evaluations with more senior colleagues so as to place student comments (both good and bad) in perspective without compromising their tenure bid. Some student comments may have merit; therefore, those of us who strive to be the best medical educators we can be would do well to recall the words of John Cotton Dana: “who dares to teach must never cease to learn.”