I hate to take criticism myself. Of course I do. And while I will complain very vocally about a criticism, if it comes from someone I respect, then once I cool down I will try to address it. I have found that the best (and most serious) criticisms I have had on papers were actually signed. The reviews that I work the hardest to actually address were signed.

I'm not sure that is because those were the most important issues, but I will spend more time on issues raised by people I respect a lot than anonymous people.

I don't buy the black-list idea, and I don't think it is the way to go. I know, as an editor, who my reviewers are, and I already blacklist reviewers I don't trust if they give unreasonable reviews. I have a bunch of people I really trust and I usually send papers to them and value their opinion a lot more than reviewers I don't know.

That makes me non objective, I know, but as an editor I can never be objective. I am always subjective, but at least you will know who is making the judgment. That, I think, is more important than pretending that we are objective.

Nothing wrong with being humans but we shouldn't pretend otherwise.

As for bad feelings about bad reviews, I am sure that is an issue. It can't be that different from giving a bad grad to a close friend at an exam, and we have both done that. I might even have given you a bad grade, although I only remember giving you better grades than you probably deserve just because you are so cute in a kilt ;)

This leads me to thinking that open reviews are necessary. This can also lead to an inflation of grades, where weak accept means reject and only strong accept meaning accept because people are afraid to sign negative reviews.

I like the emerging trend of not only awarding the best paper at conferences, but also the best review. Those are also dangerous, because they reveal the reviewer to the author, and best review is not given for the best paper (after all if it's a good paper, it will get brief reviews saying it is good).

Meta-reviewers help, but for the exact same reason not always enough.

It should be possible to grade reviews and have the PC/editor black-list bad reviewers. To some extent it should already happen, but doesn't in my experience. Being banned as a reviewer should of course also ban you as an author, as you cannot "repay" the reviews you get for your papers. I think that is a better middle-way solution than forcing open reviews.

Before I go and run these tests, figured I'd poke this old thread -- (and might have results to post soon)

john

"Howcome, when it comes to sex, not knowing what the hell you are doing is a virtue?" Because then they don't know what the hell their partner is doing either, and thus their partner can afford to be lousy :-) Which must be why some people like to get in bed with virgins?