On review quality…
Following up on the last post, and an older one, I’m going to rant a little bit about the reviews I’ve gotten on two papers recently.
I’m not complaining about the reviews I’ve gotten for the Bioinformatics paper I mentioned in the previous post. Those are detailed, thoughtful, relevant and all reasonable. There my only problem is that I have a page limit that keeps me from adressing all the comments.
What I am a bit miffed about is two papers submitted to BMC Bioinformatics. Do not take it as a critisism of that journal, though, I have also gotten nice reviews there. I have another paper submitted there, that is getting nice reviews (in the sense that there are lots of suggestions to consider, not that they are just positive). Not so for the last two papers.
First of all, the review reports are very short. Maybe 15-20 sentences. Secondly, there aren’t really any constructive criticism. Not surprising with less than 20 sentences, of course. Thirdly, and this is the most annoying, they haven’t made any decision!
The “positive” reviews are just summaries of the paper (essentially paraphrasing the abstract). The “negative” reviews are saying things like: “I do not really like this / I do not find it interesting” or “other people are doing something similar”.
Of course reviewers are permitted to not like a paper and to not find it interesting. They shouldn’t make their decision on this, but on whether the results are novel and sound. If they think that the results are too small an increment on existing work — and there will always be similar work out there, if I submit it to BMC; it it was truly novel I would go for higher impact — in that case they should say so, justify it, and reject the paper!
Telling me that they do not find the results interesting, and then telling me to resubmit is just crazy! How can I make any improvements if that is all the criticism I get?
If I resubmit, the paper will end up with the same reviewers, and they still won’t like it.
The form letter from the editor just asks us to resubmit and include a cover letter “addressing the reviewer concerns”. That is of no help at all! “To make the paper more interesting, we have included a Dilbert strip and a picture of a clown.” Is that going to work? I doubt it.
This is really pissing me off.
If, as a reviewer, you do not have any constructive criticism — good or bad — just make your decision and let us get on with our lives. If the paper is rejected, it would probably also be rejected after a resubmission, but now I know that so I can decide on whether to abandon the paper or try somewhere else.
It is not just the reviewers that are the problem here, though. In a situation like this, I think the editor has a lot of the responsibility. The final decision is his, so he should get involved at some point. By now, he should a good idea about whether the papers will get accepted or rejected. After all, there are no additional experiments or improvements suggested, so the content of the papers are not going to change.
As a side note, BMC isn’t that bad in this regard. We once had a paper at European J of Hum Gen in review for more than a year, where each iteration consisted of very minor changes but the form letter kept telling us that no decision was made yet.
We all want our papers as good as we can get them, so if you have made your decision then let us know! If it gets rejected, we wont waste any more time on it, and if it gets accepted we will still address reasonable comments to improve the final version.
You are no more “unable to make a decision” at this point than you will be after a resubmission, if the reviews do not ask for any actual additional work!
Grrr!
June 16th, 2008 at 7:09 am
Have you asked the editor about your concerns? A polite request for guidance might be helpful – it’s also in the editor’s interest to make sure you understand what is needed.
I’m generally not worried by the “no decision yet”. I interpret it as acceptance with modifications, but the journal wants to b sure that the modifications are done properly. They wouldn’t do this if they were going to reject the paper anyway.
June 16th, 2008 at 9:14 am
Oh, I have had papers rejected after a few iterations… and I think that is ok if the decision really wasn’t made before the iteration.
The “unable to make a decision” part doesn’t worry me if it is by a reviewer who suggest some extra work, but in a case like this…
June 16th, 2008 at 7:30 pm
You describe so well what a reviewer should not do that I am sure you will be a good reviewer. Please, contact me if you are willing to make the scientific publication world better.
June 17th, 2008 at 6:12 pm
This is really what makes the argument for per-article impact factors instead of per-journal. It would help reduce the load at Nature, Science, etc, which would mean reviewers could take a little more time to do a good review.
June 17th, 2008 at 6:54 pm
Not that I’m saying your manuscripts were like this, and I do feel your pain as an author who has received unhelpful reviews myself, but personally, I end up reviewing a lot of technically okay, but scientifically boring papers and it’s not really clear what to say other than “this is not very interesting or original; this is basically Smith and Jones, 2005 with a slightly different data set” etc. Just like with movies, it is easy to review the really good or bad manuscripts; it’s the mediocre ones that are hard. I really don’t what to say in these cases.
June 17th, 2008 at 7:29 pm
Gunn:
I’m not sure the impact has much to do with it. Yeah, there’s probably a higher quality of reviewing going on at really high impact papers — and the editors are more likely to get involved from my (admittedly limited) experience — but I don’t think this is the main problem.
I have received really great review reports on conference papers (for computer science conferences only, though), and those aren’t really “high impact”.
I think it is only a matter of how serious you take it…
June 17th, 2008 at 7:39 pm
Jonathan:
I recognize that situation very well. I also find it much harder to judge such papers. It is easy to accept great papers and easy to reject terrible papers, but most papers are in between. Just as you say.
Now, the two papers I am referring to are in this category. If I thought they were better, I would submit them somewhere else ;-)
My main complaint isn’t really that the reviewers aren’t saying much, it is that they are not making any decision.
There’s no point in asking for a resubmission if there is nothing much to say about the paper.
If I get the feeling, when reviewing, that the paper can go either way, I never feel that minor changes here and there will change my impression. So I make up my mind and decide whether I think it should be accepted or not, and that is that.
Sometimes, I think it just needs a little extra. Like an extra experiment to clear up something I am in doubt about, or some clarification to make sure that I really understand the claims made in the paper. Then I will ask for that.
The point is, if I don’t feel that the suggestions I make will change the way I judge the paper, then I do not ask for a second review.
June 18th, 2008 at 3:46 pm
Hi Thomas,
Editors generally ask reviewers not to make a decision part of their report. Judging how influential the paper might turn out to be, identifying errors, and making constructive criticism is the referee’s job; the decision is the editor’s responsibility. (The editor would rather avoid the he-said/she-said situation of the author saying “but referee #1 said you should accept the paper without revision, why did you reject it?”) In the editor’s decision letter, they should make it clear to you which of the referees’ points they found particularly persuasive in reaching their decision, so you know what to respond to. So I’m with Bob O’H; you ought to take this up with your editor, rather than holding it against your referees.
Sean
June 18th, 2008 at 4:22 pm
Sean:
I see your point that putting the decision in the referee report could be a problem. Still, a decision is made, whether it is forwarded or not in the report. If that decision is “unable to decide”, then I have a problem with it…
That being said, I guess I am just annoyed that the decision letter didn’t contain any information about which points I needed to consider and whether it would be worth the effort to resubmit.
Anyway, I have followed Bob’s advice. I’m not really as annoyed as the post sounds. I was when I wrote it, but I was really just blowing off steam. I’ve calmed down now and find the whole situation a lot more reasonable than I did last week ;-)
June 18th, 2008 at 7:41 pm
[...] On review quality… [...]