Mailund on the Internet

I am reviewing a paper for PLoS One today. At PLoS One they have an option for reviewers that they can 1) waiver anonymity to the paper authors, and 2) allow your review -- or sound bites from it -- to be published together with the paper, so readers of the paper can see what the reviewers thought.

This got me thinking about reviews in general.

Anonymous reviewers?

I realize that there are good reasons to keep reviewers or authors anonymous during the reviewing process.

I've never actually reviewed a paper where the author was not known, but there are good arguments for keeping the authors unknown to the reviewer. Like it or not, we do have prior ideas about our peers and the quality of their work, and that is likely to influence our reviews. We are more likely to believe the results of authors with proven track records than people who's papers we have rejected on several occasions.

That being said, I do not think it is a major problem to know the authors.

What about anonymous reviewers, then?

In my experience -- which admittedly is limited to computer science, biology and bioinformatics -- the common case is that the reviewer is kept anonymous from the authors, and that he has the possibility to write comments that will be shown to the authors and in addition to write comments that will only be seen by the editor(s).

I've never used the "private" comments when reviewing, and I have never written a review that I wouldn't put my name on. Not that all my reviews are positive -- far from it, some would say -- but any honest review should not be something you would be ashamed of admitting to have written.

Of course, there are reasons for anonymity here. A bad review hurts, especially if you feel that it is unfair, and hurt feelings can affect your judgment down the line. "If I get a bad review from you, then you will get a bad review from me".

How much of an issue this is, I don't know. I should hope it is a minor issue and that reviewers are more objective than that. If not, they shouldn't accept the review; it is clearly a case of conflicting interests: revenge vs science.

In any case, let's be honest, reviewers are less anonymous than you would think. Even when they are not named, their suggestions often give you a reasonable good idea as to who it is, so why not just name them?

I think there are very good reasons to name both authors and reviewers in the reviewing process. I've only on one occasion know a reviewer on one of my papers -- not because the policy was to name the reviewers but because he explicitly signed the review -- and being able to discuss the results with the reviewer was helpful.

This shouldn't in itself be an argument for disclosing the reviewers -- I can easily imagine being spammed/flamed in order to change my mind on a review -- I'm just saying that there are some benefits from knowing who your reviewers are.

PLoS's policy of letting the reviewer decide whether to disclose his name or not is a good idea, in my view. As a sign of good faith and honesty in the review, I would say a reviewer in general should choose to disclose his name, but in rare occasions where there are good reasons not to, he should be allowed not to disclose his name (and the editor should probably consider these reasons and judge if they cause a conflict with the objectivity of the review).

Of course, should a reviewer choose to remain anonymous, that should be respected. The journal should not disclose confidential information under any circumstances (under the law) -- and this is the point where I refer you to the editorial in the current issue of Science that addresses exactly this issue.

Public reviews?

So much for anonymous reviewers. What about the actual reviews?

Lately I find that I often search for online reviews of papers I read. Reading blog posts about a paper is no substitute for reading the actual paper -- that goes without saying -- but I find it very helpful to read what other people think about a paper and what other papers they refer to. It is like one global journal club discussion.

I would love to see more of this.

I realize that I am comparing oranges and apples here. It is a very different situation reviewing a paper with the intend of judging whether it is publishable and suggesting ways to improve it, compared to commenting on an already published paper.

The quality control that is peer-reviewing cannot be substituted by the blogosphere. That would turn the whole process into a popularity contest.

Still, once a paper is published, why not publish the reviews together with it, so the reader can learn of the concerns or suggestions of the reviewers? Maybe there are good reasons to keep parts of reviews confidential, but that could be left to the editor's discretion.

In any case, I'd love to see more public discussion on published research.

I am doing my part in this, small as it is. I probably review five times as many papers on my blog than I do for journals. I submit my reviews to Research Blogging and have recently joined CiteULike and keep a list of the papers I read there.

CiteULike, by the way, is a great place to find related literature. Try searching through the lists that overlap your own and you will find lots of papers worth reading.

A Web 2.0 journal club?

PLoS One has a comments section with blog tracebacks on each paper. That is a great way to get other opinions on a paper and to discuss a paper, but it is only one journal. For other journals, you need a bit of google'ing to find discussions.

Wouldn't it be cool with a website that would aggregate paper reviews, discussions and related literature? A mix of Medline (and similar) for related literature, combined with CiteULike and Research Blogging for the "social networking" component.