Unskilled and unaware of it
I’m still somewhat annoyed about exam complaints. The system is there to fix cases where some kind of mistakes were made, it is not there as a second chance for improving the grade.
Why are we running into problems with this new batch of students?
Upon further reflection my theory is this: They are very good students who are just not used to poor performances. To be accepted into this study program, you have to be pretty good, so before entering the university they were all top of the class.
In math or computer science, where I’m used to teaching, you get a wider spectrum of students. Here, you only get the best of the best.
The problem is, of course, that they are still not equally talented. If you pick the best in the class at high school and put them together in a class at the university, some who used to be at the top are now at the bottom.
Not only that, since we have to grade them, they now get low grades that they would never have gotten in high school.
That must be an unpleasant experience.
That is not the full story of course. They are clearly feeling that they are being treated unfairly. Now that of course could be true, but would require some sort of conspiracy on the teachers’ part since this goes on in all the classes, with different teachers and different examiners.
The exam system could also be broken. They all are, to some extend, but we are doing the best we can to make exams fair.
Another problem could be self assessment.
Unskilled and unaware of it
In a paper from 1999, Kruger and Dunning show that the poorer the performance, the poorer the self assessment:
Justin Kruger and David Dunning
Journal of Personality and Social Psychology. Vol 77(6), Dec 1999, 1121-1134.People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
If you grade peoples’ performance and then let them grade their own performance, you get a picture like the one on the right.
The trend is correct – the bottom quartile judge their own score to be lower than the top quartile – but the worse you score the more you tend to overestimate your own performance.
This is now called the Dunning-Kruger effect and boils down to:
- Incompetent individuals tend to overestimate their own level of skill.
- Incompetent individuals fail to recognize genuine skill in others.
- Incompetent individuals fail to recognize the extremity of their inadequacy.
(quoted from the wikipedia page I linked to above).
Incompetent here, of course, is relative. It doesn’t mean complete moron, just those at the lower end of the curve.
The top quartile tends to underestimate their performance, but this might not be poor self assessment but simply poor assessment of the others. The paper puts it:
Despite the fact that top-quartile participants were far more calibrated than were their less skilled counterparts, they tended to underestimate their performance relative to their peers. [...] That is, top quartile participants did not underestimate themselves because they were wrong about their own performances, but rather because they were wrong about the performances of their peers. [...] In the absence of data to the contrary, they mistakenly assumed that their peers would tend to provide the same (correct) answers as they themselves – an impression that could be immediately corrected by showing them the performances of their peers.
About the bottom quartile, they say:
[...] incompetent individuals fail to gain insight into their own incompetence by observing the behavior of other people. [...] We reasoned that if the incompetent cannot recognize competence in others, then they will be unable to make use of this social comparison opportunity.
To test it, they gave the participants five other exam answers to grade and then asked them to re-evaluate their own performance.
The top quartile significantly improved their self assessment, the bottom quartile did not.
Despite seeing the superior performances of their peers, bottom-quartile participants continued to hold the mistaken impression that they had performed just fine.
It would seem that the only way to improve self assessment would be to improve skills.
They tried that by training the participants. It does work. When it comes to over estimating performance for the bottom quartile they still did it, but gained skills did translate into better calibrated self assessment. Of course, if the only way to be able to assess yourself accurately is to actually be skillful, then that doesn’t really help on the problem for those incompetent.
In the discussion the write a bit about why this problem could come about:
One puzzling aspect of our results is how the incompetent fail, through life experience, to learn that they are unskilled. [...] it is striking that our student participants overestimated their standing on academically oriented tests as familiar to them as grammar and logical reasoning. Although our analysis suggests that incompetent individuals are unable to spot their poor performances themselves, on would have thought negative feedback would have been inevitable at some point in their academic career. So why had they not learned?
[...]
The problem with failure is that it is subject to more attributional ambiguity than success. For success to occur, many things must go right: The person must be skilled, apply effort, and perhaps be a bit lucky. For failure to occur, the lack of one of these components is sufficient. Because of this, even if people receive feedback that points to a lack of skill, they may attribute it to some other factor.
Indeed.
–
183-185=-2
July 3rd, 2009 at 9:27 am
Hmm… This can probably be conferred to physical tasks such as driving or playing football. Most drivers see themselves as good drivers and the rest of the world as lunatics who should be locked up for irresponsible driving; and many of the spectators at sports events scream and shout at players for even the slightest error done, while claiming they could have done that much better.
July 3rd, 2009 at 1:00 pm
Quite right, but on the other hand, with many physical activities the feedback – both positive and negative – is much more obvious. It is much harder to fool yourself into believing that you are in the top quartile in 100m sprint in a competition than it is in a multiple choice exam… but then again, if you sprint against the clock and are asked how well you did without being told your time against the others, perhaps you would be just as bad at your self assessment… I don’t know…
July 8th, 2009 at 2:25 pm
I’ll just focus on one item in your post that seems to encapsulate a couple of points that are worthy of discussion:
The problem is, of course, that they are still not equally talented.
Firstly, do you actually believe that any two people one earth can be equally talented? The more multi-dimensional your view of “talent” or, indeed, “intelligence” is, the more improbable it becomes that you can have equally talented people. And if there is one thing that we know about people’s abilities then it’s that they are multi-dimensional.
Secondly, why anyone would insist on using strictly one-dimensional standards for assessment, in the form of grades, is one of the great mysteries of this world. By any educational standard, it simply doesn’t make any sense whatsoever. (It does make sense, though, in a social sorting kind of way, much like the equally ridiculous one-dimensional IQ numbers. But I would submit that no self-respecting human being would come out arguing that that’s the purpose of higher education.)
And thirdly, whatever your educational goal is, the relative achievement of which you are trying to grade on a scale, it simply doesn’t follow that there must be some people at the top and some at the bottom of that scale. If anything, your job is to make sure that as many as possible reach a position at the top of the scale. That’s what education means: making people smarter. Not simply judging their innate capabilities on an arbitrary scale that is removed from reality as far as logically possible.
Now, nobody expects you to have thought about this for any length of time, and nobody expects you to be a fantastic teacher who loves what he’s doing just because you took one teaching course. In fact, this is probably why you’re struggling with your teaching. How can anyone be expected to love what they’re doing and be good at it if nobody has really bothered to show them how? But there’s hope: obviously, you want to do a good job and are prepared to think about some of the issues.
The first step, I would suggest, is that you need to have your biases challenged. A good place to start is Alfie Kohn’s website, which has loads of well-researched and well-sourced information. Computer Scientists might want to have a look at Roger Schank’s take on education; he also has his own website.
July 8th, 2009 at 4:10 pm
Thanks for your comments, Peter. I appreciate them.
I also mainly agree with them, but let me address them point by point.
First, no of course I don’t believe any two people to be equally “talented” and of course “talent” is a multi dimensional quantity that is not easily measured. Still, during an exam we are asked to grade one dimensionally on the students’ performance on that day, and in relation to the specific class. Talents – whatever that means – in other areas does not come into it.
Secondly, since we are asked to grade we do project talents into a single dimension. Whatever it is we measure, we do try to quantify something not easily quantifiable. What we grade depends on the class (but of course the performance for the various classes are correlated) and although it is a very crude way of measuring performance I think it does – on average over all classes – reflect the relative performance of the students. With respect to the studies of course, and to some extend their general ability to do good on exams. General nervousness and such probably is measured in on the grades, even though it probably shouldn’t be.
As for your third comment, I think I disagree a bit. Not on that the point of education is to move everyone forward on their knowledge and abilities, I agree completely there, but when it does come to exams and grading I think it does follow that not everyone can end up at the top.
While teaching, I think everyone should be challenged. So any class should be hard enough that even the very best will find some parts difficult. It is by overcoming the difficulties we learn, not by solving easy problems. Then, when it comes to the exam, I think this should be reflected there as well. I think it should be possible, within the constraints of the exam to do well or poorly.
The point of education is of course not to grade the students. The point of the exam, however, is.
If everyone ends up with the same grade – whether it is the top grade or the bottom grade – I think that is a failure of the exam. It means it was either too hard or too easy. If the exam reflects the class, which it should, it means that either the class was too hard or too easy.
If we want to grade at all – and I’m not saying we necessarily should – then that grade should actually graduate the student performance.
Personally, I would be happy with exams that you either pass or fail, but otherwise do not grade. I’m not sure what that does to motivation; I think some gets motivated to work harder to get a higher grade, and I am not sure they would work just as hard if they could pass the exam with less work, but that is a different problem. We would just have to find other ways of motivating.
That is not my choice, however. We are told to grade, so we should actually do that.
If my goal really was to get as many as possible to the top end of the grade, I could just adjust the difficulty of the exam to get everyone up there. That would, of course, just mean that the final grade is meaningless.
So while I agree that the point of education is to get everyone to learn as much as possible, I do not agree that we can conclude from that, that we should aim at getting everyone to score as high as possible.
As for your final point re teaching, I couldn’t agree more. I think that it is a different issue than the exam and the grading – which was the topic of this post – but probably a much more important issue.
Teaching teachers to teach is a problem at our university (and many other universities, I would think). We get positions and tenure based on our research and are then just expected to be teachers as well, as if being good at research also makes you good at teaching. There probably is some correlation between the two – I think there is – but it is simplistic simply to assume that one follows from the other.
The class I took on teaching is a new initiative here at AU. A few years ago, teaching didn’t really come into consideration when hiring. I think it is still mainly ignored, but this is slowly changing. A teaching portfolio is now required for anyone applying for a position. I don’t know how much weight it carries, but it is a step in the right direction. The teaching class is mandatory for assistant professors, to improve our teaching. Another step in the right direction.
I took the class a year ago, and have tried to read up on the teaching literature a bit since then. There’s a center for science education two floors below my office, and I’m trying to get some help from the head of that center who used to teach courses similar to mine. I am trying to improve my teaching based on what I read and what advice I get.
One frustrating thing is that, while I can see that it actually works in activating the students (and in how they handle the exams later on), my teaching evaluations have been in free fall since I tried implementing these ideas. I used to get nice evaluations when doing the plain old one-way communication lectures where I would just lecture and present slides. Now I get very poor evaluations and the main complaints relate to my attempts to activate the students.
I guess I still have a lot to work on…
July 8th, 2009 at 4:48 pm
I can see where you’re coming from. And I can’t say I would have reacted entirely differently if my university career had started at a similar point to yours.
For a more detailed account, I’d have to refer you to some external sources (see below), but there’s two things I’d like to point out. First, the main point of education should not be to teach people to do as they’re told. Similarly, I can see no inherent value in doing something just because someone tells us to do it. If somebody tells me that we have to grade performance, my first response should be to ask for a compelling reason, something that serves an educational purpose. Because if it doesn’t so serve, then it doesn’t belong in an educational institution. And I’ve never heard such a reason. Incidentally, the main point of education arguably is to teach students not to do something just because that’s what they’re told to do, but to be critical, to think for themselves, and to be responsible. We’re not exactly setting a great example in insisting on generally rather pointless graded exams.
But even if you feel you don’t really want to rock the boat that hard, there’s still something to be done about grades and exams. The first is conveyed in Ben Zander’s practice of “Giving an A”, detailed in his The Art of Possibility. The second is that whenever we’re evaluating repeatable perfomance, then we’re not challenging our students. If anybody could have come up with exactly the same answer, then we’re asking the wrong questions. Students have a right to be taken seriously as individuals, not just as cogwheels in a huge clockwork, to be replaced if they’re not up to the job.
But as I said, please refer to the articles I have linked below for more details. I think you might find them illuminating as well as inspiring.
http://www.alfiekohn.org/teaching/challenging.htm
http://www.alfiekohn.org/teaching/fdtd-g.htm
July 8th, 2009 at 5:21 pm
I’m not sure the grades are pointless… the point is just not particularly related to the teaching.
It is my impression that the grades are mainly – if not exclusively – used for picking which students get to continue on our PhD program. I have no illusions that the grades are important for anyone who graduates and I doubt that anyone even looks at them. But they are used for picking PhD students. With that purpose in mind, we want as many grades as possible! If we are measuring average performance – and assuming that what we measure is at all relevant for what we use our measurement for – the more measures we have the better we can estimate the average performance.
When I studied we had several classes that lasted for a year and ended up with a single grade. Getting a bad grade there was very bad indeed. Getting lucky or unlucky could decide your chances of getting into the PhD program.
Now we have four terms a year, with typically three classes per term, so since each class is graded a few outliers doesn’t matter all that much for the final average.
So probably not pointless as such, but I agree that when considering any single class they are not that important, and I certainly agree that the grades should not be used as a teaching tool.
As for changing the system and not grading, that is just not something I can do. This is a system implemented at the university wide level – with parts of it at the national level – so whether I like it or not I have to do it. If I consider it pointless and refuse – even if an entire department refuses – nothing will change.
What I can influence is how the evaluation and grading is done – so picking exams that actually align well with the teaching goals – and I am sure that something can be done here to improve my teaching.
As for having students do something just because, I agree completely. I am right now reading Why self discipline is overrated from one of the links you gave before. We certainly do not want people to just do what they are told, but actually think about what they are doing and why (and stop doing it if it is just plain wrong).
I also think we try to capture that in the exams, actually. This is why I hate multiple choice exams and their ilk. I much prefer oral exams where it is possible to discuss why this and that is done in this and that situation, rather than just what they are told to do. I put very little value in knowing what to do, compared to why.
Anyway, thanks for the links. I’m reading them with great interest!