28

I am currently reviewing a paper and feel suspicious about their numerical results. I feel their results have been slightly modified to exhibit their superiority compare to other comparative methods.

I cannot give an exact mathematical reason why I think their numerical results have been manipulated, but it is so odd that in all 15 scenarios they have been better. Actually, I am pretty sure that their method is better than his benchmarks but not always. Maybe they modified some scenarios.

As I don't have a mathematical reason why I think some of their numbers are altered, should I ask for their computer code and run them myself?

Jeromy Anglim
  • 20,310
  • 9
  • 72
  • 108
SAH
  • 966
  • 8
  • 20
  • 1
    I didn't ask the editor yet. Should I ask him first? @EnergyNumbers – SAH Jun 22 '15 at 06:05
  • 1
    If you have sufficiently many numbers and Benford’s law is applicable, you could use it to test the data. (But remember that you can always tweak the data such that it complies with Benford’s law.) – Wrzlprmft Jun 22 '15 at 18:33

6 Answers6

21

You say that you don't have a mathematical reason to be suspicious. But have some reason. You should formulate this as a reason in your report as well as you can.

Moreover, I think that it is totally reasonable to ask for code and data to reproduce the results. Probably the journal has some policy about reproducible research and you could refer to that.

Kimball
  • 32,248
  • 7
  • 64
  • 169
Dirk
  • 38,078
  • 9
  • 96
  • 154
  • I cannot formulate my feeling and speciousness. I have worked with these methods before, they don't always give the best results. In some case older method (his benchmarks) gives slightly better results. @Dirk – SAH Jun 22 '15 at 08:39
  • 8
    @Electricman I think you formulated them fairly well in your question, i.e. with reference to your surprise at the domination of one method over another. – fmark Jun 22 '15 at 09:56
  • So I should explain my reason fairly and then ask their computer code to reproduce the results? @fmark – SAH Jun 22 '15 at 11:45
  • 3
    "I have worked with these methods before, they don't always give the best results." This means that the editor picked the right person to review the paper. You know the field, and you have experience in practice. You should have confidence in your assessment that the results shown are too good to be true. – Wolfgang Bangerth Jun 22 '15 at 16:19
  • 1
    I think I will ask for the code to see if I can reproduce the same results. However, if their results were not too good I wouldn't be suspicious. However, regardless of this case, I think it s very difficult to catch manipulation in results generally. Why journals do not ask for the computer code as well as the manuscripts? Imagine there are methods A,B,C and the proposed method X. We know method X is brand new and A,B,C are older methods. So the author simply may put some bad numbers for A,B,C and then shows his X method is better. As X is brand new, no one get suspicious @WolfgangBangerth – SAH Jun 22 '15 at 17:26
12

Complementing the other answers, either in addition to, or apart from, asking for the codes and running them yourself, I don't think it is a bad idea to explicitly ask the authors, something like this:

The standard algorithms for executing (bla bla bla), e.g. as followed in the works of Refs. [1-3] (feed these in), are suggestive of a standard accuracy in typical algorithms. The current work appears to have bettered the same, which is a strong point in favor of publication. However, the authors should include a comment in the manuscript, mentioning what changes have they implemented to the typical algorithms so highly prevalent, which leads to this enhanced accuracy.

My jargon may have to be twisted a bit here, but I think I convey the point.

This being a query in the referee report, the authors shall be obligated to feed in the details, and if there is some manipulation involved, it may end up being exposed.

Hope that helps :)

299792458
  • 2,958
  • 3
  • 23
  • 42
5

If there are any "standard" or widely-used benchmark problems for this application, it would be reasonable to suggest they are used (and give a reference to the problem definitions, and other papers reporting the results, in your comments). Selective benchmarking is hardly new as a sales and marketing technique, though one might hope that an academic paper would present a balanced view.

Even if you can afford the time and effort to get the code and reproduce the results, you might end up with a "my test problems are harder than your test problems" debate, which may be irrelevant unless the paper claims its method is always better - and that would be a brave claim to make in any non-trivial situation. A method that is better "only" in some circumstances may well be worth publishing.

alephzero
  • 4,242
  • 15
  • 19
  • "Even if you can afford the time and effort to get the code and reproduce the results, you might end up with a "my test problems are harder than your test problems" > What if I ask his/her computer model of the given problem? So if I cannot regenerate their results by the given code, it means manipulation in results.... @alephzero – SAH Jun 22 '15 at 12:11
4

I think that it's mandatory for science to have open source code and opendata. It's not science otherwise. Using results of a computation without seein the code it's like trusting a theorem without asking for the proof. So absolutely ask the code.

asdf
  • 149
  • 3
  • 3
    I agree with you. Journals should modify their policies and ask for computer codes during submition of the manuscript. But it is not common at all. It has not yet happened to me at least. @asdf – SAH Jun 22 '15 at 18:18
  • 1
    I think that it's mandatory for science to have open source code and opendata. Not necessarily, but it should be. – Mast Jun 22 '15 at 20:40
  • Not necessarily before publication or during review, and it depends on how much code and data you're talking about; it would be unreasonable to insist on all source code including e.g. drawing a minor chart on p. 19. Generally reviewers should filter out the worst behavior, then after publication other researchers can try to get access to the data and replicate the results. If they can't (or find anomalies) they try to correspond with the authors. If that doesn't work, then they can publish the correspondence / report to oversight bodies. Irreproducibility is one thing, falsification another. – smci Jun 22 '15 at 20:46
  • 1
    Downvote explanation: What I do is still science, but no, I won't be putting people's medical records up on GitHub. – Fomite Jun 24 '15 at 03:44
4

You state that you don't have a mathematical reason to believe their numbers are altered, but in most cases, that shouldn't stop you from considering an application of Benford's Law to evaluate the digits of the results (bear in mind there are a few cases where it doesn't fit).

Edit: There is a quick free checker at http://benford.jplusplus.org/ just type in your numbers

robbat2
  • 141
  • 3
  • I don't think they manipulate the results manually. It seems they have run the algorithm with a high resolution and picked the best results while other methods have been run with lower resolution. This study is depondend on the selected resolution. But it worth to check. As there are only 21 numbers, so Is Benford applicable here? How can I apply them? is there any software for that? @robbat2 – SAH Jun 23 '15 at 06:05
  • 1
    i added a link to a site where you can check it easily.

    By resolution, if you mean screen resolution, beware that device distribution was one of the things found to NOT follow Benford's law, because few devices have such large resolutions, and very small resolutions caused unreadable text/layout.

     – robbat2 Jun 23 '15 at 07:49
  • I mean the searching resolution in optimization algorithm. @robbat2 – SAH Jun 28 '15 at 07:35
1

Whole point of publication is to provide information for reproducing the research work. Therefore the report needs to be written to make the work clear enough to justify the improvements. (I wish it was to exactly share it with public) You could refer to that and ask for more information to help your review process done faster, it can be a way to make the code or generate/collect data. Anything that is needed to reproducing it. obviously sharing it is much easier, but then it should be attached to the paper or hosted publicly for readers as well. reviewer and readers should be on a same ground.

Amir
  • 794
  • 4
  • 11