Unskilled and unaware of it
Friday, July 3rd, 2009I’m still somewhat annoyed about exam complaints. The system is there to fix cases where some kind of mistakes were made, it is not there as a second chance for improving the grade.
Why are we running into problems with this new batch of students?
Upon further reflection my theory is this: They are very good students who are just not used to poor performances. To be accepted into this study program, you have to be pretty good, so before entering the university they were all top of the class.
In math or computer science, where I’m used to teaching, you get a wider spectrum of students. Here, you only get the best of the best.
The problem is, of course, that they are still not equally talented. If you pick the best in the class at high school and put them together in a class at the university, some who used to be at the top are now at the bottom.
Not only that, since we have to grade them, they now get low grades that they would never have gotten in high school.
That must be an unpleasant experience.
That is not the full story of course. They are clearly feeling that they are being treated unfairly. Now that of course could be true, but would require some sort of conspiracy on the teachers’ part since this goes on in all the classes, with different teachers and different examiners.
The exam system could also be broken. They all are, to some extend, but we are doing the best we can to make exams fair.
Another problem could be self assessment.
Unskilled and unaware of it
In a paper from 1999, Kruger and Dunning show that the poorer the performance, the poorer the self assessment:
Justin Kruger and David Dunning
Journal of Personality and Social Psychology. Vol 77(6), Dec 1999, 1121-1134.People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
If you grade peoples’ performance and then let them grade their own performance, you get a picture like the one on the right.
The trend is correct – the bottom quartile judge their own score to be lower than the top quartile – but the worse you score the more you tend to overestimate your own performance.
This is now called the Dunning-Kruger effect and boils down to:
- Incompetent individuals tend to overestimate their own level of skill.
- Incompetent individuals fail to recognize genuine skill in others.
- Incompetent individuals fail to recognize the extremity of their inadequacy.
(quoted from the wikipedia page I linked to above).
Incompetent here, of course, is relative. It doesn’t mean complete moron, just those at the lower end of the curve.
The top quartile tends to underestimate their performance, but this might not be poor self assessment but simply poor assessment of the others. The paper puts it:
Despite the fact that top-quartile participants were far more calibrated than were their less skilled counterparts, they tended to underestimate their performance relative to their peers. [...] That is, top quartile participants did not underestimate themselves because they were wrong about their own performances, but rather because they were wrong about the performances of their peers. [...] In the absence of data to the contrary, they mistakenly assumed that their peers would tend to provide the same (correct) answers as they themselves – an impression that could be immediately corrected by showing them the performances of their peers.
About the bottom quartile, they say:
[...] incompetent individuals fail to gain insight into their own incompetence by observing the behavior of other people. [...] We reasoned that if the incompetent cannot recognize competence in others, then they will be unable to make use of this social comparison opportunity.
To test it, they gave the participants five other exam answers to grade and then asked them to re-evaluate their own performance.
The top quartile significantly improved their self assessment, the bottom quartile did not.
Despite seeing the superior performances of their peers, bottom-quartile participants continued to hold the mistaken impression that they had performed just fine.
It would seem that the only way to improve self assessment would be to improve skills.
They tried that by training the participants. It does work. When it comes to over estimating performance for the bottom quartile they still did it, but gained skills did translate into better calibrated self assessment. Of course, if the only way to be able to assess yourself accurately is to actually be skillful, then that doesn’t really help on the problem for those incompetent.
In the discussion the write a bit about why this problem could come about:
One puzzling aspect of our results is how the incompetent fail, through life experience, to learn that they are unskilled. [...] it is striking that our student participants overestimated their standing on academically oriented tests as familiar to them as grammar and logical reasoning. Although our analysis suggests that incompetent individuals are unable to spot their poor performances themselves, on would have thought negative feedback would have been inevitable at some point in their academic career. So why had they not learned?
[...]
The problem with failure is that it is subject to more attributional ambiguity than success. For success to occur, many things must go right: The person must be skilled, apply effort, and perhaps be a bit lucky. For failure to occur, the lack of one of these components is sufficient. Because of this, even if people receive feedback that points to a lack of skill, they may attribute it to some other factor.
Indeed.
–
183-185=-2