29

Especially in the the more experimental subfields of computer science like systems, how often are results faked? If there is no verification process for any code used, do researchers sometimes fake results to save time?

Lilienthal
  • 419
  • 3
  • 8

3 Answers3

29

Nobody knows. Unless someone tries to reproduce the results and cannot, there's not ever even going to be a challenge to the results. Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method. To my knowledge, we do not see a lot of retractions based on these kinds of studies, so I'd say the rate of faking is low.

Retraction Watch's CS section has about 15 articles in it, but most of them appear to be about retractions for plagiarism not for faking results.

Bill Barth
  • 48,733
  • 6
  • 112
  • 194
  • 10
    "Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method." It's unfortunate that it's very hard to get short papers of the form "yes, we were able to reproduce this result" published. – Joshua Taylor Apr 28 '15 at 17:28
  • 9
    Also, few papers publish executable code, and some not even pseudo code, along with their studies. Since we don't have a publication culture that appreciates effort of reproduction (cf @JoshuaTaylor) few results are checked. I suspect a high dark figure of published, peer-reviewed falsehoods. – Raphael Apr 28 '15 at 18:02
  • 16
    And the falsehoods aren't necessarily faked. Code has at least one bug in it. Suppose the bug in your unpublished code on which your paper relies happens to make it give incorrect but encouraging results, that you publish, and nobody ever checks. As a programmer, I'd say bugs that make your code appear to do what you wanted are the hardest to spot yourself. – Steve Jessop Apr 28 '15 at 21:11
  • 7
    @SteveJessop spot on. It must be very common for code to have bugs which give technically invalid results that nevertheless support the hypothesis. So the author is doing nothing wrong by publishing. However, one wonders how often a bug, even a significant one, might be discovered prior to publication and nevertheless conveniently ignored. – Keith Apr 29 '15 at 01:22
  • @SteveJessop Intentional or not, if the code is not there for people to check the authors don't act responsibly. – Raphael Apr 30 '15 at 17:28
  • @Raphael: CS academics keeping their code secret is another (related) debate, but I think this applies even if the code is published. The word "unpublished" can be struck from my comment above. Just because it's available to check doesn't mean anyone will check it, and if not then quite likely the only person with a chance of spotting the bug is the author. The rate of errors in papers (or sections of papers) that with hindsight are mostly insignificant, is probably elevated for this reason in all fields, because checking unimportant parts of a paper is a waste of time ;-) – Steve Jessop Apr 30 '15 at 17:41
  • @SteveJessop The difference is that once the code is published (in a useful way), the responsibility is shared between author and community. – Raphael May 01 '15 at 12:27
18

Let me split my answer into two sub-areas: reputable venues and crap venues.

In reputable venues, it is just as possible for somebody to commit fraud (or any of the other deadly sins of science) in experimental computer science as in any other experimental science. It also appears to be quite rare, because there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science (unlike, say, certain subfields of biology) and so fraud would often be relatively easy to detect. More to the point, however, the risk/reward tradeoff is terrible: one detected incident will likely destroy a career.

In crap venues, there might well be constant fraud---but who cares? If they'll accept machine-generated papers, they might accept anything, and I'm probably not going to cite it or even look at it in any case.

jakebeal
  • 187,714
  • 41
  • 655
  • 920
  • Thanks for the link. While I knew about isolated cases (eg, a high rep user here published one as a proof of concept), this really surprised me. Are proceedings of the IEEE considered "crap" these days? They used to have a decent reputation, didn't they? – gnometorule Apr 28 '15 at 15:08
  • 5
    @gnometorule The IEEE in general is quite good, but they've been less careful than they should in exercising oversight as they have continued to expand. Thus, while the IEEE brand is a very good sign, you still have to exercise judgement just as you would for any other venue. – jakebeal Apr 28 '15 at 15:32
  • 5
    Hey, machines have just as much right to publish as humans do! Their papers may not be up to our standards yet, but they are trying their best to learn. :) – Thomas Apr 28 '15 at 16:08
  • 3
    "there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science" Though there's an unfortunate tendency to publish results that are based on computation/simulation/etc, without always making the code available. That means that reproducibility is actually much harder, because it's difficult to know whether a difference is a result of a bug, data, or the actual intended method. – Joshua Taylor Apr 28 '15 at 17:30
  • 5
    @jakebeal the IEEE brand is a very good sign — [citation needed] The quality of IEEE publications in computer science varies so widely that the label itself indicates nothing about the quality of the venue. (In this respect, they are no different than any other large commercial publisher.) – JeffE Apr 29 '15 at 02:19
  • I would love to upvote this answer, but simply can't agree to "because there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science" - this leaves ample room for first coming up with a nice algorithmic idea together with a good theoretical reason for why this should work well, and to only then fabricate benchmarks to support this theory. This is difficult to detect, and a mere biased selection of benchmarks is already sufficient to imply the practical applicability of a new technique when there is none. – DCTLib Apr 29 '15 at 11:57
11

There are quite a few interesting articles on the topic of fraud rates. I didn't see anything specific to computer science, but the general sense is that it's very difficult to determine.

It's very tough to actually detect fraud. Most cases (that I'm familiar with, at least) involve someone falsifying data in a highly active field and publishing earth-shattering results that turn out to be false. As was said in other answers, detecting the fraud requires attempting to reproduce the results, failing, and then determining that the problem is on the other end.

With that said, there aren't many good proxy measures of fraud. Retractions are a start, but they're far and few between. We can almost be certain that not all fraud is retracted. Surveys can be used but they're also notoriously inaccurate. There really aren't many other ways to measure that would actually provide a useful metric.

eykanal
  • 48,364
  • 18
  • 119
  • 216
  • 3
    As most cases of exposed fraud happen due to researchers becoming to comfortable with falsifying data I think it's a very fair assessment to assume the large majority of fraud are not caught. – David Mulder Apr 28 '15 at 17:17