In this post John Hawks complains about the important information that is left out of papers and hidden in supplemental material.
I mention this, because Asger Hobolth and I were just discussing this yesterday.
In the olden days, ten years ago, I would simply put the two papers side by side and find the discrepancies. But nooooo, we can’t do that any more. Now, all the relevant parameters from one of the papers (you guessed it, the one published by the Nature Publishing Group) are hidden away in a supplement.
You’d think that might not be so bad, since I have the supplement. But I have to keep tracking the cross references to the paper to find out where the methods apply. It’s a pain in the neck. Nobody else ever seems to complain. But that’s because they simply don’t read the papers! AAARGGGH!
Trust me, we complain!
Asger has just spent the last week trying to reproduce a result from a paper, only to find out that a lot of crucial info was left out of both paper and the supplemental material (that contained the data, but not the filtering that was done on the data). He had to get that info from one of the authors.
Personally, I spent a couple of weeks in December trying to reconstruct a method hidden deep in the supplemental material of a Nature paper — on a project very related to the rest of John’s post, by the way — but never managed to reproduce the results from the paper. I got close, but never quite there.
It might be taking it a bit too far to say that people don’t complain, because they don’t read the papers, but I think very few people read the supplemental material. At least in any kind of details.
I know that I only read the supplemental material for very few of the papers I read; only those where I want to reconstruct a method, or where I don’t really believe the results and want to see how the data really support it.
Sadly, very often the supplemental information doesn’t help much there either.
Is the supplemental material even reviewed?
Post score: 8-4 = 4