6

I've downloaded the source code and data published in a paper. I've followed their methods, in particularly, copied and pasted their sample code. Unfortunately, I've failed to reproduce the figures in the paper with their software. For example, the variance accounted by the first principal component reported in the paper is about 10% more than I can see on my computer.

My generated figure:

enter image description here

Their reported figure in http://www.nature.com/nbt/journal/v32/n9/fig_tab/nbt.2931_F3.html

enter image description here

The difference is small but the medians are different in the paper to my generation. The figure in the paper looks better and more aligned to the medians (a good thing). I did a PCA plot (they also did that for the paper), the eigenvalues are different to reported.

I followed their methods in https://www.bioconductor.org/packages/3.3/bioc/manuals/RUVSeq/man/RUVSeq.pdf

I simply copied and pasted their code and used their R-package.

Can I consider this as an example of data-manipulation (to make the figures look better) and therefore academic misconduct?

EDIT:

The method is non-stochastic, it involves computing singular value decomposition for factor analysis. There is nothing stochastic, no random number is generated. Identical data-set (downloaded from their R-package).

SmallChess
  • 2,397
  • 2
  • 16
  • 25
  • 1
    Would you be more specific on the variance accounted by the first principal component? In particular, variance of what? – Nobody Nov 19 '15 at 09:44
  • 2
    Do you use the exact same version of R and of all packages you include? – Stephan Kolassa Nov 19 '15 at 09:48
  • @StephanKolassa EDITED – SmallChess Nov 19 '15 at 09:51
  • Again, did you do this for all of the data in the package or only a sample subset? – Ébe Isaac Nov 19 '15 at 10:07
  • @ÉbeIsaac Everything, all of the data. Otherwise I wouldn't have posted here. – SmallChess Nov 19 '15 at 10:08
  • Did the authors apply this to a subset or the entire set? – Ébe Isaac Nov 19 '15 at 10:10
  • @ÉbeIsaac They didn't say they applied to a subset, so I have to assume they did it on the entire set. – SmallChess Nov 19 '15 at 10:14
  • That's not how statistics goes. Unless otherwise stated, statistical measures are taken only from a subset of a given population. You might have to consult this with the authors themselves for further clarification. – Ébe Isaac Nov 19 '15 at 10:17
  • 2
    I only see negligible difference, which may even be to float arithmetic mistakes on different OSes or different R versions. Let it go. – Alexandros Nov 19 '15 at 11:05
  • 3
    Do you think they would falsify data and publish the source code to expose them? – Davidmh Nov 19 '15 at 12:18
  • The most noticeable difference is a different color scheme, and different labels on the axes. This tells me you did not use the exact same code to generate your figure. Maybe that also explains other differences? – ObscureOwl Dec 03 '19 at 11:19

3 Answers3

23

Academic misconduct is the last thing you have to consider, not the first.

First, you have to consider that there might be something you are overlooking (e.g. an updated version of the code, or the data set is not really the same, or something related to the machine or etc.). Then, you have to consider that mistakes happen, and as I wrote in this answer, yes, there are plenty of errors out there, that do not come from academic misconduct: a mistake can have leaked somewhere in the paper, code or graphs.

Therefore, if you have reasons to consider that difference significant, you can contact politely the authors trying to set up a scientific discussion -- not an accusation -- to understand where the difference comes from.

Community
  • 1
Massimo Ortolano
  • 55,428
  • 19
  • 166
  • 207
  • 3
    Regarding mistakes: it is possible that the authors made a mistake; it is even more likely that you have made a mistake. Why more likely? Because they probably have more experience with the topic, the methods, and the data, and they almost certainly have spent more time on them. – David Ketcheson Nov 19 '15 at 10:56
  • 5
    Additionally, the differences I see in the plots are very small. If they do not significantly affect the scientific conclusions drawn by the paper, it's very unlikely that the data have been falsely manipulated. Why risk your career by falsifying data, if there is no reward? – Moriarty Nov 19 '15 at 11:07
11

You seem to want to head immediately to the worst case, and a fairly serious allegation. Data manipulation and academic misconduct are very serious things. And starting off from that position is pretty antagonistic and is likely not going to be very productive in actually figuring out what's going on - more likely, it's going to get everyone defensive.

The first thing you should do is email the authors with a clear, detailed account of what you did in your replication attempt, and where the differences you found from their results are. Ask them if they might be able to provide more details on their own methods, or review yours to see where they differ. There's any number of ways a replication based experiment can subtly deviate from a published work.

Even if the answer is "No, we can't account for that", it may very well be human error, rather than misconduct. But before you can even go down that path, you need to make sure it is a genuine replication failure.

Fomite
  • 51,973
  • 5
  • 115
  • 229
3

If the result proposed by the published paper depends on some sampling of data, then a probable sample error could be introduced. This would mean you would have to follow a Test of Hypothesis to disprove their result.

Edit addressing the added details:

Variance of a population is surely equal to or larger than its subset. You cannot accuse misconduct purely on this case. To reject their claim of a mean, median, or variance, you ought to do so with a formal test of hypothesis taking your own subset into consideration.

There are several standard materials available on test of hypothesis. For a quick reference, you may refer my concise expository paper here.

Ébe Isaac
  • 12,715
  • 2
  • 30
  • 73