Blind reviews? For or against?

I've been away for a long time. RSI really keeps me away from blogging and I'm not planning to get back to blogging full time, but I really miss it so I hope to get back to writing a bit here and there...

Anyway, there is one thing that's been on my mind for a while, and I'd like input on it.

Should reviews be open or blind?

I've discussed this with colleagues for a while and I see all the points for or against, but really haven't made up my mind. Ideally I think that reviews are such an important part of the publication process that we cannot ignore it, but I see a lot of good arguments for blind reviews as well.

First some personal background: about five-ten years ago, I decided to accept all papers I was asked to review. I figured that it was 1) important to do my duty for the scientific community to review papers and 2) I would learn a lot from reviewing as many papers as I could get my hands on.

Okay, that was a very bad idea. The more papers I reviewed the more papers I was asked to review, and I ended up reviewing tens of papers a month. That means that my reviews ended up crappy since I simply did not have the time to give the papers any serious thought. I could spend two-three hours on a review max, and that is just not enough.

I felt really bad the first time I started refusing reviews, but I simply had to.

Recently I have limited the number of papers I can handle as an editor as well.  I just cannot handle too many papers and give them a fair judgment.

I feel bad about this, but is a question of survival. I cannot handle as many papers as I think I should, so I have to limit it.

This brings me back to the title of this post. Should reviews be blind or not?

When I was overwhelmed by papers I dropped signing my reviews. I didn't feel comfortable with the quality of my reviews and I am sorry to say that it meant that I could do the reviews but not be honest and stand behind them.

I can't really live with that. I don't want to give people criticism that I cannot be man enough to stand behind, so I have seriously limited the number of papers I accept to review and now I sign my reviews again.

This now leads me to a serious question: how does me signing the reviews affect my reviews?

I would like to say that my reviews have always been of high quality, but obviously they haven't been of equal quality since I now handle fewer manuscripts when I sign them. I obviously feel more reason to write better reviews when I sign them.

Having to sign my reviews clearly, for me at least, means that I am much more careful about the report I write.

Now I never write a report unless I've been thinking about the paper for a couple of days. And I think my reviews are much better for it.

That sounds like I am all in favour for open reviews (and generally I am) but I also see the problems with this.

At the pub in Cambridge my last visit there we discussed this. It is easy to sign a positive review but you can get in trouble when you sign negative reports, so why bother?

If open reviews means anything it means that you always sign reviews even if they are negative, and that can be a problem since you will make enemies when you are giving negative reviews.

I really don't know.

I know that my reviews are better now that I sign them, but I am also much more selective on the papers I want to review, so that puts a limit on what I will review. I still spend about a day a week on reviewing but I am much more selective on what papers I accept and I only accept papers I am likely to be positive about.

Even if we don't require reviews to be open, would we make better reviews if reviews were published together with papers, perhaps as supplemental material?  I don't know

I would love your thoughts.

2 Responses to “Blind reviews? For or against?”

  1. Michael Says:

    I really like the idea of open reviews. I'm having severe problems with getting reviews we like to qualify as "weak accept, high confidence, one-liners", where the review just restates the title and obviously the reviewer has spent at most 10 minutes on the paper.

    This leads me to thinking that open reviews are necessary. This can also lead to an inflation of grades, where weak accept means reject and only strong accept meaning accept because people are afraid to sign negative reviews.

    I like the emerging trend of not only awarding the best paper at conferences, but also the best review. Those are also dangerous, because they reveal the reviewer to the author, and best review is not given for the best paper (after all if it's a good paper, it will get brief reviews saying it is good).

    Meta-reviewers help, but for the exact same reason not always enough.

    It should be possible to grade reviews and have the PC/editor black-list bad reviewers. To some extent it should already happen, but doesn't in my experience. Being banned as a reviewer should of course also ban you as an author, as you cannot "repay" the reviews you get for your papers. I think that is a better middle-way solution than forcing open reviews.

  2. Thomas Mailund Says:

    Yes, it is a difficult system. I honestly think that the open reviews would be best, but it only works if we think that people can take constructive criticism. Of course we know that they can't :)

    I hate to take criticism myself. Of course I do. And while I will complain very vocally about a criticism, if it comes from someone I respect, then once I cool down I will try to address it. I have found that the best (and most serious) criticisms I have had on papers were actually signed. The reviews that I work the hardest to actually address were signed.

    I'm not sure that is because those were the most important issues, but I will spend more time on issues raised by people I respect a lot than anonymous people.

    I don't buy the black-list idea, and I don't think it is the way to go. I know, as an editor, who my reviewers are, and I already blacklist reviewers I don't trust if they give unreasonable reviews. I have a bunch of people I really trust and I usually send papers to them and value their opinion a lot more than reviewers I don't know.

    That makes me non objective, I know, but as an editor I can never be objective. I am always subjective, but at least you will know who is making the judgment. That, I think, is more important than pretending that we are objective.

    Nothing wrong with being humans but we shouldn't pretend otherwise.

    As for bad feelings about bad reviews, I am sure that is an issue. It can't be that different from giving a bad grad to a close friend at an exam, and we have both done that. I might even have given you a bad grade, although I only remember giving you better grades than you probably deserve just because you are so cute in a kilt ;)

Leave a Reply