Crappy review report
We just got reviews back from a paper we submitted a few months ago on an algorithmic improvement on the neighbour joining method. One of them was this:
For large-scale phylogeny reconstruction parsimony and likelihood are the preferred methods. Both are more accurate than neighbor joining (particularly large datasets). It is not clear to me if RapidDiskNJ is a sufficient advance to publish in this journal.
That’s it. The complete review.
Not a single comment on the actual contents of the paper. Just a blank rejection of the neighbour joining method in general.
And this is an algorithmic improvement. The paper is on speed and on handling really large data sets, with tens or hundred of thousands of taxa. Parsemony and likelihood methods simply do not scale to those data sizes. At least not unless you very large clusters to the problem we solve on a desktop computer.
We improve the speed by several orders of magnitude compared to other implementations of the method, and in comparison with likelihood methods there really is no competition at all.
With reviews like this, peer reviewing just fails!
–
187-190=-3
July 7th, 2009 at 6:46 pm
Nice.
You’ve fallen prey to the always disastrous reviewer who _thinks_ they know enough about the subject to review your paper, but who in truth doesn’t actually know enough to be competent judge.
July 7th, 2009 at 7:12 pm
Actually, I blame the editor more than the reviewer on this one…
True, the review is as bad as it gets, but in the end it is the editor who makes the decision. I am only a young editor myself, and already I am falling behind on my TODO list there, but at least I try very hard to make the decisions on papers I am handling based on both my own impression of a paper and the review reports. Maybe that is why I am behind on a few papers now … anyway, I would never have put any weight on a review that doesn’t address the paper at all!
Of course, I am as biased as it gets on this one, but I am really disappointed on this one.
I have had several NJ papers rejected by people who just reject NJ in the first place, so I am not that surprised, but it still pisses me off no end. My NJ tools are the ones I get the most feedback on by emails from actual uses, so I know that NJ is being used and I know that algorithmic improvements has practical importance.
I know a few other people I’ve talked to at conferences with similar experiences. If you do algorithmics on NJ, you just have to try try try and then try again to get the results published.
As an extra kick in the teeth we found out today that there is a poster at RECOMB with similar results as ours – except that from the time they report we are faster – so getting this particular paper rejected is even more annoying. With our usual luck we won’t manage to publish until long after our competitors… but at least we know that there are other people out there interested in the problems we work on…
If people wants to reject my papers because the result is crap, that is good and fair. That is how the system should work. If they reject it just because they don’t like the underlying methods – without even considering the contribution in the actual paper – I get pretty angry.
The other reviewer had essentially the same comments to the paper, but at least that review report contained two whole paragraphs. The content was the same as the one I cited above – and again didn’t at all address the paper, just the use of the NJ method – but at least he made the effort to read the paper and know what our actual results were before rejecting it for not being about something else, like maximum likelihood methods…
July 7th, 2009 at 7:47 pm
quote the reviewer, “For large-scale phylogeny reconstruction parsimony and likelihood are the preferred methods. Both are more accurate than neighbor joining (particularly large datasets).”
Is this a true statement, assuming scalability is not a problem?
July 7th, 2009 at 7:53 pm
Well, maybe the editor was lazy … OR is a casualty of the phenetics wars, doesn’t philosophically like NJ, sent your paper off to a couple reviewers that are known to share that viewpoint, and waited for you to be shot down…
Or am I too cynical?
July 7th, 2009 at 7:57 pm
Haibao: Yes it is. Statistical methods, at least, are better at inferring the true phylogeny than heuristics such as NJ. I’m not sure about maximum parsimony, though. Those are really also heuristics, but they seem to work better than distance based methods.
So yes, it is a valid complain in the sense that we should not use NJ if we can use a better method. The point is that on such massive data sizes as we attempt to tackle, that is not really an option. So saying that you should use a better method – when it simply does not scale – is not really a valid statement. The point is that it does not scale, so if you have tens of thousands of taxa to build a phylogeny on, you don’t have that much of a choice on which method to use. You have to use whatever works.
Of course, if you do have access to massive computer power, you have less of a problem. You still need to build an algorithm that can handle the data, but you do have the option of using a better method.
If you don’t have that massive computing power, what can you do?
Either use a method that will get the job done – which is what we have developed – or sit on the data until you get access to that computing power.
Good luck on the latter. Moore’s law doesn’t provide any more. Well, it does – you still get more and more transistors – but you do not get a doubling in speed every 18 months any more. Without algorithmic improvements, you are screwed.
July 7th, 2009 at 7:59 pm
rr: I would prefer to attributed to laziness what could otherwise be contributed to malice, to paraphrase Hanlon’s razor
July 7th, 2009 at 8:20 pm
Yet another reason to publish with PLoS One?
http://www.plosone.org/static/reviewerGuidelines.action#criteria
July 7th, 2009 at 8:35 pm
Anders: Not really a solution, I think :)
Except that I am an editor there, so by cheating a lot I might be able to handle the paper ;-)
No, but serious, this paper was submitted to BMC Bioinformatics which is also an open access journal. The difference between ONE and BMC BIoinformatics is just that the “relevance” or “novel” criteria are not valid arguments for ONE – the papers should be judged on scientific rigor not on whether the results are interesting. BMC has Research Notes with the same criteria as ONE – I am also an editor there, btw, and they did suggest that we resubmitted there,.
Anyway, I’m ranting more about the review report than the actual criteria they choose to apply for acceptance. I’m fine with having a paper rejected because it does not meet the journal’s criteria – even if I sometimes disagree with that criteria or whether my paper meet it it (of course).
If there is something wrong with the underlying method, it should be rejected in both BMC Bioinformatics, BMC Research Notes and PLoS ONE.
What I’m really miffed about here is that the review doesn’t even say that. It just say – in so few words – that the topic is not interesting because there are other methods. I think that is simply not true. Yes, there are better methods, but none of them scale to the the data size we are working on here.
Lots of people use NJ, because sometimes that is the only choice they have. What is wrong with greatly improving a method that is still in wide use – even if the state of the art has moved beyond it? The fact is that a lot of people still use the method, so algorithmic improvements are still important.
Ignoring the paper and just rejecting the method – unless the method has long been considered faulty and obsolete – is just poor reviewing, in my not so humble opinion…