Thursday, January 31, 2008

Reading the Evals

Although we got the statistical reports on evaluations a few weeks ago, the actual forms with narrative comments did not come back to my mailbox until today.

I'm becoming a fairly popular teacher here, and so affirming narrative comments are embarrassingly common here. Not because I think I'm this amazing teacher, but because I'm beginning to think that the way students fill out evaluations is as much a function of their overall learning across many courses, as well as their general critical thinking acumen, more than what happens in one course. And that relationship is not necessarily a direct relationship, either.

Take the 300-level commonwealth course: The statistical scores were a bit more varied than other courses I've taught, with a few lower scores than I've gotten at this institution (though not other places--more about that in a sec). And yet the narrative comments say so much more than a 4.04 statistical mean on one score or the other. For example, the evaluation that answered the question, "Course material challenged students intellectually" with "usually" (3 out of 5), wrote in her narrative comments: "Horace is the absolute best professor I have ever had at BRU. He masters his subject matter, encourages critical thinking, and his enthusiasm makes you want to learn and participate. He is also incredibly helpful and has a great personality."

So wow. That's a dream comment, and I strive to live up to that kind of vision that some students have of me. But her decision to evaluate the questions individually instead of doing a straight-ticket evaluation tells me more about my teaching, and affirms my teaching more than the student who assigns all 5s. Why? because one of the higher levels of Bloom's taxonomy of learning is "evaluation." Which is to say that this student has (or at least demonstrates) a clearer sense of what the critical task of evaluation is than a student who assigns straight 5s, or, for that matter, straight 3s, and then leaves no narrative comments at all. The straight 3s, by the way, come from the only student who anticipated a C in the course.

In fact, I find that this institution generally has a bit of grade inflation when it comes to evaluation scores. Listen, I think many of my colleagues are likely fantastic teachers. But I also thought that my colleagues at my last institution were also amazing teachers as well. And while I don't have access to anyone else's evaluations, I can guess that eval scores and Rate-my-professor scores probably correlate pretty generally. And I know that here, RMP scores for my department are really generally quite positive, and at my last instituion (andexpensive, private, USNews-top-50 school) my colleagues RMP scores (and mine, too) were pretty average, even mediocre.

I fear that what this means is not that here we're better, or here the students are nicer, but rather at the last institution, there was a culture of critical thinking that is not strong here. Now this is a tenuous conclusion, I know, and it privileges all sorts of things (class and geographic location among them) in drawing that conclusion. But I fear that indirectly, at least, the most uniformly good scores that I get on evaluations may be potentially a sign of a secondary failure, by me and by my institution generally, to really make critical thinking (including but not limited to evaluation, but also analysis, creativity, and other higher-order thought) a priority and an automatic response.

Furthermore, I am both selfishly happy and a little disturbed to see how often "enthusiasm," "personality" and "humor" are listed as characteristics of excellent teaching in these evaluations. Not disturbing in and of themselves: I know I tell some decent jokes, and I like to think of myself as a nice person. But the fact that evaluations generally have become so important in terms of faculty evaluation makes me concerned for colleagues who are, say, low key, or not very funny, but brilliant, and who actually provoke a great deal of learning without necessarily being, say, entertaining, or lovable. This kind of customer-service teaching sends us in many of the wrong directions; It's a lucky (and not necessarily skilled) teacher, who can strike a balance between rigor and likability.

In the short term, these evaluations give me some nice data for my annual report. But they also send up red flags about the culture of critical thinking at this institution (and perhaps many others), and more generally, for the reliability of teaching evaluations for determining teaching efficacy and faculty merit.


Sisyphus said...

Very interesting how you're linking our assessment and their ability to do critical thinking. I'll have to ponder this further.

At my place the eval scores are so tied to the carrot and stick of grad student funding that they mean nothing and can't tell you much of anything about students' critical thinking ---- though it can show a lot of nasty parallels with student grade grubbing and grade expectations.

We can't apply for internal fellowships and can potentially not get renewed for TAing if our scores are low --- we _need_ those 1s straight across the board the way our students wail that they can't get anything lower than an A or they're screwed. In fact I've seen some of my fellow TAs explicitly instruct their students that 1s are like As, 2s like Bs, etc. and then give them a sob story about how they will not be chosen to teach again with low scores. (which is true, but I think it's not cool to guilt trip students into high evals. And my scores show that, I admit.)

New Kid on the Hallway said...

I came here to say something else, but I'm kind of floored by Sisyphus' comment about TA funding being tied to teaching evals. Wow, that would suck. Hell, no one ever LOOKED AT our teaching evals when I was a TA.

But originally I was going to respond to your point about the dilemma of the low-key professor - last year Former College ran a search and one of the finalists turned out to have a very low-key, laid-back, somewhat reserved persona. Hir letter writers actually commented on this, that s/he was quiet but effective in the classroom, but the SC was still very dubious that s/he would be successful with Former College students, and it was one reason why s/he didn't ultimately get an offer from us (I realize it's kind of indiscreet to say this, but I do so anyway because s/he got a job elsewhere and the search ultimately failed, so I think, really, FC's strategy backfired).

undine said...

I'm not as worried about what the course evals say about students' critical thinking, for haven't studies shown that students form their impressions in the first 5 minutes you're in the classroom and don't vary them after that? There are also studies contrasting low-key lecturers who are actually saying something with those who babble nonsense with sweeping gestures and entertaining asides--and guess who gets the best evals there?

No, it's the USE of these as the be-all and end-all of assessing teaching that's the problem, and that's in the hands of the administration. As you say, these are red flags, but for me they're more about the culture of the institution than the culture of critical thinking.

calypsojaved said...

I came here to say something else, but I'm kind of floored by Sisyphus' comment about TA funding being tied to teaching evals. Wow, that would suck. Hell, no one ever LOOKED AT our teaching evals when I was a TA.but i like to this
There are a lot of sites out there showing book video. BookVideoTV, BookTelevision and of course CSPAN, but I like how and Reader's Entertainment TV have specific genre channels and original shows. There's just more to see and I can be specific in what genre I'm interested in. Anyone else watch online tv?