Course Evaluation - Criticism of Course Evaluations As Measures of Teaching Effectiveness

Criticism of Course Evaluations As Measures of Teaching Effectiveness

Summative student evaluations of teaching (SETs) have been widely criticized, especially by teachers, for not being accurate measures of teaching effectiveness. Surveys have shown that a majority of teachers believe that a teacher's raising the level of standards and/or content would result in worse SETs for the teacher, and that students in filling out SETs are biased in favor of certain teachers' personalities, looks, disabilities, gender and ethnicity. The evidence that some of these critics cite indicates that factors other than effective teaching are more predictive of favorable ratings. In order to get favorable ratings, teachers are likely to present the content which can be understand by the slowest student. Consequently, the content has been affected. Many of those who are critical of SETs have suggested that they should not be used in decisions regarding faculty hires, retentions, promotions, and tenure. Some have suggested that using them for such purposes leads to the dumbing down of educational standards. Others have said that the typical way SETs are now used at most universities is demeaning to instructors and has a corrupting effect on students' attitudes toward their teachers and higher education in general.

The economics of education literature and the economic education literature is especially critical. For example, Weinberg et al. (2009) finds SET scores in first-year economics courses at Ohio State University are positively related to the grades instructors assign but are unrelated to learning outcomes once grades are controlled for. Others have also found a positive relationship between grades and SET scores but unlike Weinberg et al. (2009) do not directly address the relationship between SET scores and learning outcomes. A paper by Krautmann and Sander (1999) find that the grades students expect to receive in a course are positively related to SET scores. Isely and Singh (2005) find it is the difference between the grades students expect to receive and their cumulative GPA that is the relevant variable for obtaining favourable course evaluations. Another paper by Carrell and West (2010) use a data set from the U.S. Air Force Academy where students are randomly assigned to course sections (reducing selection problems). It found that calculus students got higher marks on common course examinations when they had instructors with high SET scores but did worse when they took later courses requiring calculus. The authors discuss a number of possible explanations for this finding, including that instructors with higher SET scores may have concentrated their teaching on the common examinations in the course rather than giving students a deeper understanding for later courses. Hamermesh and West (2005) find that students at the University of Texas at Austin gave attractive instructors higher SET scores than less attractive instructors. However, the authors conclude that it may not be possible to determine if attractiveness increases the effectiveness of an instructor, possibly resulting in better learning outcomes. It may be the case that students pay more attention to attractive instructors.

The empirical economics literature is in sharp contrast to the educational psychology literature which generally argues that teaching evaluations are a legitimate method of evaluating instructors and are unrelated to grade inflation. However, similar to the economic literature other researchers outside of educational psychology have offered negative findings on course evaluations. For example, some papers have examined online course evaluations and found them to be heavily influenced by the instructor’s attractiveness and willingness to give high grades in return for very little work.

Another criticism of these assessment instruments is that largely the data they produce are difficult to interpret for purposes of self- or course-improvement. Finally, paper based course evaluations can cost a university thousands of dollars over the years, while an electronic survey is offered at minimal cost to the university.