66

I have been working with my research group on a number of papers in a certain area of Computer Science. These papers have a certain portion which is theoretical (i.e. contains theorems, formalisms, algorithms) and a certain portion which is experimental (i.e. describes implementations and empirical results).

With regard to the experimental/practical sections, my supervisor generally suggests prioritizing the information that illuminates the general thrust of the paper (i.e. the section in the paper itself along with figures, etc) over the actual implementation details like links to code. Even if he considers releasing some tangible material, he seems to prefer to release executables rather than source code.

I can understand the rationale behind this because:

(1) In Academia, the final paper is the most important and recognized artifact of research;

and

(2) Releasing unpolished source code could be embarrassing, because it might contain errors which could damage the reputation of our research group.

While I agree with this, I am conflicted over what this means for the value of the paper and the ethics of research.

  • The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

  • If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

  • Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

  • The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Please help me understand if my argument has any merit, or if I am being irrational.

Trunk
  • 4,418
  • 9
  • 28
Grzegorczyk
  • 827
  • 1
  • 4
  • 8
  • 43
    "Releasing unpolished source code could be embarrassing" — a lot of science code is more script-like and unpolished. Would you judge some other team based on their code or be grateful that they released the code at all and understand that we all have circumstances? "embarrassing, because it might contain errors" — would you prefer to be wrong and non-embarrassed? Or to know the truth and have the chance to get your paper right? If someone finds an error in my code, I'm not embarrassed. I'm proud that someone found my code important enough to read it throroughly. And I'm grateful to them. – Džuris Jun 21 '22 at 03:35
  • 16
    What good is a physics paper if every paper isn't also shipped with the unique equipment necessary to reproduce the results? Or a biology paper without samples of what's being studied? It's not a perfect analogy, and one can certainly make a good case that one should always release the source code, but lack of released code doesn't make a paper useless. – NotThatGuy Jun 21 '22 at 11:06
  • 9
    @NotThatGuy that seems very focussed on the title - the text of the question is clear about relative value and the pros and cons of releasing the code. – mjaggard Jun 21 '22 at 13:18
  • 2
    Won't you be just as embarrassed if the output of the released executable contains errors? With source code available, others may be able to track down the issues to report and or correct the code. – doneal24 Jun 21 '22 at 15:51
  • 8
    Your code has errors. My code has errors. Everyone's code has errors. If I can look at the source, I will be more convinced that you don't have too many errors. And if you hide it instead, personally, I think you know you have more errors than you should. Also, source can be re-built in the future; a compiled binary may depend on some runtimes that become hard to find in the future. – Davidmh Jun 21 '22 at 22:08
  • I don't think I can write a full answer, but there is a case to be made that releasing the source code makes it too easy for others to reproduce the experiment too exactly - including any mistakes in the source code would also be present when they tried to reproduce the results. In contrast, if the results are reproduced by someone who writes their own implementation based on their reading and understanding the paper, then their implementation is unlikely to have exactly the same mistakes as yours, so it is more useful for the pursuit of knowledge if their experiment has the same result. – kaya3 Jun 22 '22 at 17:55

8 Answers8

88

I think you have this all correct. The world would be a better place if all papers released all of the software that is used to generate the results shown in it. And that's not just a personal thought of mine (and apparently of yours as well), it is empirically verifiable: Papers that release the software used get more citations than papers that do not -- in other words, others are also thinking that that is worthwhile.

People have all sorts of reasons not to release their software, including (i) they believe that they have a competitive advantage by keeping their software to themselves, (ii) they do not trust their own software, (iii) they do not comment their own software or otherwise use good software engineering practices, and don't want the world to see so, (iv) they do not want to provide support to others who would download the software and use it. I believe that (iv) is a legitimate reason. (i) is misguided in my opinion, because at least if the software is non-trivial, others trying to use it more likely than not will ask to collaborate with the authors of the software, rather than just use it themselves; in other words, the original authors would gain a competitive advantage rather than disadvantage from releasing the software. (On this point, I speak from many years of experience.) Finally, if someone does not want to release their software for reasons (ii) or (iii), I believe this is an ethically questionable approach: If you have no confidence in your work, you probably shouldn't publish it.

In any case, I believe that your arguments are correct. Whether they convince your adviser is, of course, a different question.

Wolfgang Bangerth
  • 94,697
  • 7
  • 201
  • 338
  • 13
    To your last comment point (iii); one can justifiably have confidence in their work being correct, even if it does not conform to good software engineering practices. Other than that, good answer. – Servaes Jun 20 '22 at 10:23
  • 2
    Thanks for this answer. IMHO, (i) is also misguided because research is not supposed to be like sports or business, or some other zero sum game. The goal of researchers should be to advance the understanding of the subject, not to prove themselves the best – Grzegorczyk Jun 20 '22 at 11:15
  • 7
    @Grzegorczyk I agree, but your output affects your livelihood in many systems of researchers' job evaluations, and so I'm tolerant of people wanting to have a competitive advantage. – Wolfgang Bangerth Jun 20 '22 at 12:06
  • 14
    @Thissitehasbecomeadump. Empirically, the software design and development community has found that code that is poorly written and poorly documented has a much higher density of bugs. If you know enough about software design to be embarrassed by your code, you should probably be also concerned about its correctness. – Wolfgang Bangerth Jun 20 '22 at 12:08
  • 11
    It's worth noting that well-regarded conferences (AAAI and ISWC come to mind) now have a "reproducibility" peer-review criterion, where reviewers are asked to assess the reproducibility of the work and the quality of provided supplementary materials. Just a few years ago, I didn't see this sort of thing. So the community is taking steps to push people to release code, data, etc. which supports the views expressed here (in both question and answer). – user3780968 Jun 20 '22 at 13:11
  • 1
    Regarding your point (iv) - do you mean "support" in the sense of technical support? I originally read it in the sense of "supporting them to carry out some aim" and got a very negative view of your answer, which I now think was incorrect. – preferred_anon Jun 20 '22 at 16:49
  • 1
    See also: the Community Research and Academic Programming License. – wizzwizz4 Jun 20 '22 at 22:17
  • 5
    @preferred_anon Maybe both. If you put code on the web for others to download, then others will ask you questions about it, will ask you for help to implement extensions, etc. They will, in other words, ask for your time. You may or may not be willing to offer that time. – Wolfgang Bangerth Jun 21 '22 at 00:38
  • 2
    Related to point (i), in some cases the software developed by the researchers is commercial software sold/licensed for profit. – Jordi Vermeulen Jun 21 '22 at 09:49
  • @WolfgangBangerth I agree with you that sacrifice of time is a legitimate reason (although you can just say no). My mistaken interpretation was nothing to do with that - it was more like "I, the software author, don't think that you, the user, will use my software for the Greater Good (TM), so I won't provide you with it". It seems clear that wasn't your meaning. – preferred_anon Jun 21 '22 at 11:11
  • 1
    Regarding supporting the code, I imagine one can fairly easily get away with adding a disclaimer saying e.g. the code is for illustrative purposes only and no support would be available to get the code running. If you post it on e.g. GitHub, it would be quite trivial for others to fork it and take over to fix any issues if they really feel so inclined. It doesn't seem like a particularly compelling reason to not release the code. – NotThatGuy Jun 21 '22 at 11:23
  • "They do not comment their own software". Poorly written code cannot be saved by comments, and clear, good code often doesn't need any comment at all. Did you mean "document" instead of "comment"? – Eric Duminil Jun 21 '22 at 14:13
  • 1
    I'd strongly add, also, things like releasing it with a docker file or similar can save a bunch of time - others can then figure out how to get it working outside a container, but making sure it runs when you install it will make the support process pretty trivial – lupe Jun 21 '22 at 16:51
  • I am searching for the reference to the claim "papers with code get cited more". Any graph, research, study I should be looking to? – Danfoa Oct 25 '22 at 06:38
14

Check out this famous paper (in AI) by Google. It described an algorithm by which an AI can learn to play a game - in this case chess and shogi - by playing against itself only, and how such an AI can beat the then-strongest engines at the games.

Google didn't release the source code. But! The papers contain enough information for someone else to duplicate their work. Using the same methodology, the chess engine community created Leela Chess Zero, and tuned it (Leela is likely stronger than AlphaZero at this point). The ripples of the new AI continue to be felt, since 1) it affected how Stockfish, the strongest traditional chess engine, was developed; Stockfish is likely stronger than Leela again, and 2) the gameplay ideas that Leela discovered (and continue to discover) have been incorporated into the repertoire the top human grandmasters.

Now we can ask the question: what good is this paper if the code is not open source? How do we know Google didn't fake their results? I'm sure you can see the answers in the second paragraph above.

Allure
  • 127,528
  • 50
  • 325
  • 493
  • 40
    I think this answer misses the point, though. You are considering a sample of one, written by a highly regarded collaboration, writing a paper for which it would have been clear to anyone who saw it as a draft that it will be a high-impact one. The question is whether you can generalize from this paper to recommendations for everyone -- and I don't think you can. Just because that one paper had enough information to be useful to others that they could reverse-engineer things doesn't mean that -- in general -- papers shouldn't bother with including their source code. – Wolfgang Bangerth Jun 20 '22 at 14:18
  • What good is your paper commercially if the code is open source? Even if it was the reverse and Google had released it, you can't take that as gospel either because you're not an internet monopoly poised to profit from its existence. Everybody's missing the point as usual : money. "The goal of researchers should be" to do what they're paid to do; to directly make us money, or make us look smart so other people pay us money to make them smart, which is stupid. – Mazura Jun 20 '22 at 18:12
  • 3
    @WolfgangBangerth Just because that one paper had enough information to be useful to others that they could reverse-engineer things doesn't mean that -- in general -- papers shouldn't bother with including their source code. The question isn't "should papers include their source code?" however. It asks about the value of the paper given the source code isn't available. – Allure Jun 20 '22 at 23:41
  • 1
    @Allure But then the same question applies. Just because there is one paper without source code that has value does not mean that this is true on average. It also doesn't answer the question whether on average papers with code have a greater value than those without. – Wolfgang Bangerth Jun 21 '22 at 00:41
  • 8
    @WolfgangBangerth I feel like we're not reading the question the same way. From my perspective, the question doesn't ask if papers with code have a greater value than those without, either. It only asks if the paper can have value if the code is not available. – Allure Jun 21 '22 at 01:47
  • 4
    @Allure Ah, fair enough, thanks for pointing out the disconnect. I guess the answer to that is "yes", as that's how we've been operating for a long time and sciences has progressed with that mantra. Though I continue to think that a sample of one is not the way to make that point :-) – Wolfgang Bangerth Jun 21 '22 at 01:58
  • note that in this case the situation is completely different. afaict AI papers generally describe new architectures - so the paper is easily reproducible even without source code, and even if it was released you'd still need to do the much more time intensive task of training the network anyway – somebody Jun 21 '22 at 07:04
  • 5
    @Mazura You might want to ask RedHat/SUSE, Oracle MySQL, Oracle Solaris, IBM Websphere, MongoDB, Docker, Chef, Puppet, etc. on how to succeed commercially with open source code. – doneal24 Jun 21 '22 at 15:59
14

While I won’t comment on your particular case, I think it’s useful to consider science as a whole. You should realize that computer scientists are uniquely able to distribute an entire exact experiment implementation broadly, instantly, to anywhere in the world by virtue of making the source code public. Anyone can reproduce such an experiment exactly, and this is (in principle) a benefit. However, for any physical or biological experiment, there is no way to transport the apparatus to everyone. The journal text, figures, published data, etc. are all anyone can hope to have to learn of an experiment and judge its conclusions and correctness. There is an implicit assumption (barring evidence otherwise) that the researchers competently performed the experiment and collected the data they describe. Trust is a fundamental component of scientific communication, and this is why malfeasance which abuses this trust (such as data fabrication) is taken so seriously.

All of this is to say that every other experimental science views journal articles, with all their flaws, as having value; they have been the primary vector for dissemination of cutting-edge results for well over a century. They work because there is generally a strong culture of scientific integrity among scientists, and people for the most part take the presented data at face value even though the experimenters could be incompetent or untrustworthy. So although you enumerate reasons why journal articles are to be mistrusted, they have served science through the period of the greatest expansion of knowledge in human history. And they shall serve you just as well.

Gilbert
  • 241
  • 1
  • 4
  • 2
    Your first sentence might be a little toooo optimistic: the precise platform, network situation, and competing loads on the machine(s) running something can have significant effects. – paul garrett Jun 21 '22 at 03:05
  • @paulgarrett good point. It depends on the experiment, I suppose. – Gilbert Jun 21 '22 at 03:07
  • 2
    I was going to say the same as @paulgarrett: Have you ever tried to provide a replicable environment for anything other than a 200-line Python script? Have you ever tried to actually reproduce a computational experiment? What do you do if that was run on a 10,000 core cluster? What if the software uses a dialect of some language that is no longer easily supported by modern compilers? – Wolfgang Bangerth Jun 21 '22 at 03:39
  • 5
    @WolfgangBangerth I’m no computer scientist, so perhaps I have a too romantic notion of it’s work. I imagine that at least measurements of relative performance comparisons would be possible to replicate nearly if not exactly. But the more wrong my offending statement is, the truer the rest of my answer: Journal articles have value, as evidenced by the last century+ of experimental science as a whole, wherein experiments are impossible to replicate exactly, and so the community must trust the written word. – Gilbert Jun 21 '22 at 05:02
  • @WolfgangBangerth if it's run on a 10,000 core cluster the answer is simple - the ones interested in verifying it probably have access to a similar cluster too. as for your second point - a) if it was published recently, whatever compiler that was used is probably still available; b) if it's old a working compiler is probably archived somewhere; c) the vast majority of compilers are backwards compatible anyway – somebody Jun 21 '22 at 07:07
  • 1
    @Gilbert Having worked in engineering/physics research, i can state that experimental details in papers are often quite flocculent and a lot of emphasis is given to insignificant things to no more avail than showing off the author's knowledge. What's more, vital experimental details are frequently omitted on the unspoken hauteur that these should be well-known to researchers of the field. Sometimes things observed by some are never observable to many others: it's just a folly no one wants to contradict. Computing may not be as variable but implementation details plus inputs are important. – Trunk Jun 21 '22 at 11:19
  • 4
    @Trunk yes, I agree with your assessment of experimental papers in engineering/physics research. My point is that despite all of the obvious flaws, journal articles are clearly regarded as valuable even for fields for which anything analogous to publishing source code is impossible. – Gilbert Jun 21 '22 at 11:56
  • Actually biological samples may not be available to everyone, but there's a big effort in many cases to make samples which are the same in specific, reproducible, standardized ways available to multiple teams, which has really enhanced research (although sometimes questionable in other ways). https://en.wikipedia.org/wiki/Henrietta_Lacks https://www.atcc.org/microbe-products/collections-and-projects/certified-reference-materials etc. – user3067860 Jun 21 '22 at 15:30
  • 1
    @Gilbert The field of computer and computational science has spawned a whole sub-field called "Reproducibility". Take my word for it: Reproducing computational experiments is not easy, even if you have the source code. – Wolfgang Bangerth Jun 22 '22 at 03:00
  • @WolfgangBangerth Thank you for introducing this to me! All the more reason to value a well-written manuscript whether or not source code is included. It sounds like computer science experiments are more akin to physical science experiments than I thought. – Gilbert Jun 22 '22 at 04:08
  • 1
    @Gilbert There is typically quite a bit of difficulty involved in building and deploying a non-trivial software project. I do commercial programming on projects expected to be useful in the long term, so we are generally a lot more concerned than academics are about long-term maintainability. Yet, even so, I've worked on plenty of projects where I've said only half-jokingly that the best thing to do to cripple a competitor would be to give them our source code and let them try to build it. – cjs Jun 22 '22 at 12:02
  • @WolfgangBangerth to be honest, the situation is particularly and un-representatively bad in numerics / HPC, both because of the large and nonstandardised computational platforms and the typically used languages (Python, C++, Fortran, Matlab) are all pretty horrible WRT versioning policies, backwards compatibility etc.. For a CS subdomain that can run its code on any CPU and using languages like Java, Haskell or Go (and making proper use of their dependency tools), the prospects are much better that you can relaunch a program from source 20 years later without major difficulties. – leftaroundabout Jun 22 '22 at 12:24
  • @leftaroundabout No, the programming language is not the problem. You can compile 20-year-old C++ with not very much trouble. The issue is that scientific software often builds on many other projects. Take for example my own project and the many other projects it can interface with: https://dealii.org/developer/readme.html#documentation If you wanted to build a 10-year-old version of deal.II, you'd also have to get 10-year-old versions of all of these dependencies. And you'd have to get a 10-year-old version of cmake. All of that is not likely going to be easy. – Wolfgang Bangerth Jun 23 '22 at 00:30
  • Just to be clear, it's the non-standardized parts of the process. The languages are well defined. The dependency libraries are not, and neither is cmake. They all change far faster and in non-backward compatible ways. – Wolfgang Bangerth Jun 23 '22 at 00:31
  • @WolfgangBangerth “The dependency libraries are not” yeah that's my point. If you have 10-year old project in Haskell, then stack build will fetch all those 10-year old dependencies no problem so you can just use the program exactly as-it-was; the .cabal files provide enough information to know how to go about updating the dependencies; and the strong type system ensures that any real conflicts will cause a compiler error immediately where the problem lies – instead of some cryptic TypeError exception somewhere else at runtime, or a 1000-line template instantiation failure mess·age. – leftaroundabout Jun 23 '22 at 06:56
  • @leftaroundabout Yea, well, that's good on the Haskell folks. Everyone else (that is, nearly everyone) is stuck without that benefit. That includes those who are building on Haskell stuff that isn't uploaded into whatever repository stack build draws from. – Wolfgang Bangerth Jun 23 '22 at 07:18
  • @WolfgangBangerth I used Haskell as the example because I'm most familiar with it, but I'm informed e.g. Java/Maven can do much the same thing. And apart from the language level, there's also NixOS and Docker, though these mostly just ensure you will be able to run the program as-it-was, not to move an outdated project forward. — My point was that solutions (at least partial ones) to the points you raised exist, and merely aren't used by the CompSci community as much as would be possible. – leftaroundabout Jun 23 '22 at 08:02
  • @leftaroundabout No no, they're all used alright. You make it sound like we're all uneducated hicks, but that's really not true. All of the big projects are offering docker images (and VMs, and debian packages, and ...). It's just that ten years ago, nobody used docker, so if you want to reproduce something from ten years ago, you have to do some actual work. And you can probably assume that ten years from now, people aren't going to be able to execute today's docker images, so you have to do some actual work. The same will be true for any other package manager you can probably name. – Wolfgang Bangerth Jun 23 '22 at 15:19
  • @WolfgangBangerth More could be said here but it has already veered of quite a bit from the question's topic. Just FTR, I did not write or imply anything about uneducated hicks, don't read ad-hominems where there are none. – leftaroundabout Jun 23 '22 at 21:59
9

Going point by point through your issues...

The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

This is undervaluing your contribution. Your paper, by definition, does have some useful theoretical basis otherwise you wouldn't be publishing it. It may seem obvious to you, but that's likely only because you've been living with this for a while. And I'll remind you that Thomas Henry Huxley's response on first reading Origin of Species was "How extremely stupid not to have thought of that!" What's apparently obvious after you know it may not be obvious before.

If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

And this is why it's important for other places to independently check results. It's not sexy, but it catches dodgy results produced through incompetence or malice.

Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

And if they've got the same dataset as you and the same code, they'd get the same results if they just ran your code. The important thing isn't the code, it's the algorithm. Which is NOT the same thing.

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

But what if the "interesting" feature of your results is actually a coding bug? Someone simply blindly copying your code will get the same results. Independently coding the algorithm though is unlikely to get the same bug twice, so they can report that they don't get the same results and your paper may be incorrect. Having the source code would let them analyse why your results were dodgy, sure, but that's less important than discovering the fact of it being dodgy. And more likely, they'll be in contact with you (or your supervisor) after they discover they can't reproduce it, and then you'll be checking your own source to find where it went wrong and publishing a correction.

Graham
  • 7,012
  • 18
  • 26
  • 1
    Having the source code would let them analyse why your results were dodgy, sure, but that's less important than discovering the fact of it being dodgy. But how may they discover it to be dodgy - assuming that the algorithm was verifiable - if they don't do an implementation themselves or use OP's implementation ? – Trunk Jun 21 '22 at 11:26
  • @Trunk I don't understand your question. It is only possible to reproduce results by carrying out the process yourself. That's the definition, the only definition ever, of "reproducing". At which point they've done an implementation themselves, right? – Graham Jun 21 '22 at 12:13
  • Or looked at OP's source and debugged/refined it. But I'm not too clear on your viewpoint here. At one point you are concerned about an algorithm error. Later you are concerned that paper readers using OP's source with or without his dataset will only generate the same results as OP. Are you concerned about the possibility of under-, over- or mis-specification in the algorithm ? Or are you suggesting that paper readers are better off doing their own implementations to reduce implementation errors and bugs ? – Trunk Jun 21 '22 at 12:43
  • 3
    @Trunk Re the algorithm error, it's possible (I've seen it myself) that an feature in the results gets misinterpreted as supporting the algorithm as written, when in fact it turns out to just be a feature of the implementation. If you were to implement the algorithm as written then it doesn't actually do what the paper says it does. :) Whether this is a bug in the implementation or just bad writing-up in the paper, either way it's something that can only be spotted with a clean implementation. – Graham Jun 21 '22 at 14:42
  • I see. So you are in the publish-the-source-code camp. But anyone with a mind to implement this algorithm will ipso facto have a bias towards or against a successful outcome. Maybe OP could ask a coding buddy to do an implementation schema for him. As regards the algo, maybe someone into formal verification in his department might lend a hand. – Trunk Jun 21 '22 at 15:16
8

The goal of a paper is to report a novel research finding. It can accomplish this perfectly well without providing source code.

It is not possible to describe empirical findings in full detail in a scientific paper (or arguably any format). It ultimately falls on the author to decide what detail is appropriate.

The scientific method is not about believing observations because they have been recorded in excellent detail, but about independently verifying them. So it doesn't particularly matter whether you provide the implementation code. What matters is whether it's clear to other researchers how they would implement their own version such that it confirms the results you obtained. Your code may or may not be an essential aid for that.

Some, including me, would argue that the virtue of a scientific finding is simplicity. In your paper, the more quirky things there are that have to be done "just right" - whether the algorithm works when implemented in a certain way with a certain language, or it only works on a certain kind of input - these all make the finding itself less interesting to begin with. So if your paper is really good enough to care about, the code is a moot point. I should be able to write my own code from scratch and verify your algorithms and theorems.

However, beyond the primary goal of justifying a claim, there are secondary goals that are well-served by making the implementation available:

  • If your implementation has a mistake, it will be found sooner
  • If your implementation turns out to be correct but hard to reproduce, you will not be accused of fabricating results
  • Researchers wishing to extend on your work will have a better starting point
  • Non-researchers (such as those in the industry) wishing to apply your findings will have a better starting point
  • People trying to learn about research in the field will have better resources
  • The code will serve as a public demonstration of your coding ability, if required by for example job applications

I personally would prefer if code was always published along with papers, but there have been many papers in CS and other fields that were published without code, and yet their claims are sufficiently credible and the findings they report are useful, so I would not consider it a hard requirement.

But to your concern about the code being unpolished: It doesn't really matter. No code is perfect. Either what you have works which means it's good enough, or it doesn't work which means your paper is wrong in the first place. You have little to lose from publishing code.

You also mention publishing binary executables, which is the really strange part to me. By publishing executables, you are asking people to trust that you did not introduce any malicious code (like viruses) into the binary, and also that your system and every other system in the chain (like the CDN actually hosting your binary) is secure from hackers and so forth. This is a very big claim, and completely unnecessary when you could just provide source code and completely obviate it. You are also training users to accept bad security practices. People who do publish binary code (eg. proprietary vendors) will at least take steps to mitigate the security hole, by employing security engineers and signing their binary builds. If your goal is to let people use your implementation without seeing the code, you should provide a web server so that the binary is executed on your system, or at least containerize it and provide something like a Docker image.

Jessica
  • 759
  • 1
  • 5
  • 1
    A comprehensive and well-balanced answer. The fourth paragraph is beautiful. However I can't quite agree with “the code being unpolished: It doesn't really matter”. In many cases, just spending one day to go over the code again, structuring it to be more reliable, putting it together with the right tests, and adding some comments, could save people trying to use it multiple days of work to get it to run, or even weeks of needing to re-do computations because they used the code wrong. – leftaroundabout Jun 22 '22 at 12:46
2

Unpublished papers, in and of themselves, are almost entirely worthless, like an unfilled lottery ticket, or a movie script that's never been filmed.

Once published, they get a little value, like a lottery ticket that has had the numbers filled in and registered at the checkout, or a movie that has been shot. They can all now be tested, and have a chance to win.

The community ascribes them value at this point, but this is equivalent to the face value of the scratchcard (a dollar, perhaps): there is a chance that the paper might turn out to be a "winner", based on correct data and correctly applied algorithms. Every panel on a scratchcard could be a winning number... or none of them may be. A movie could be a blockbuster... or a flop.

Reproducing the results is the proving ground: it's how they provide value to the consumer, like when lottery numbers are drawn, or a movie is released to the public.

The higher you make the bar of reproduction, the lower the chances of making any valuable contribution to the community.

The lower the bar, the more likely someone reading the journal you publish in is to say "hey, this is something I can give to my student to try" or "that's so easy to reproduce, I could try that out this lunchtime..."

There are those who argue "if you release the source, then people will run it and necessarily must reproduce your errors". This is very twisted logic: releasing the source saves them your entire development-cycle of time, meaning more people can work on what your paper gave. More eyes on your source can only mean more likelihood of finding errors: it is impossible for it to mean less.

There are cases (like the Google AI paper mentioned in another answer) where papers that involved software have been reproduced without source code, and managed to be beneficial anyway... but these are far less common. Even in the Google case, there was only one project which went through the trouble of rewriting that code. If they hadn't got lucky, if that project hadn't picked it up and run with it... what value would that paper have had to the community, then? How much MORE value would it have had if instead, they had released the source, data, and a docker file, so countless people could have just run one command and reproduced their results? How much bigger would the field of AI be now? Would we all be driving self-flying cars? We can't tell, but we can guess that the impact of the paper would not have been at all reduced by improving the reproducibility.

Assuming someone DOES reproduce your work, but gets different results... where is their error, if any? Even if you collaborate, and both use the same data to get different results... is it their error, or yours? How do they find out? What if you are not contactable? In software engineering it is considered a truism that you cannot easily identify or fix an error without reproducing it first, and if your output cannot be reproduced, it cannot be fixed.

TL;DR: All papers have some value, but without reproducibility, that value is minimal and aspirational, rather than real. Maximizing reproducibility maximizes real value and impact. Source code really helps these goals.

Dewi Morgan
  • 1,326
  • 10
  • 14
1

In Natural Sciences we should put forward boldly our hypotheses and results inviting debate, correction and, ultimately, falsification. Do publish your code. Why not clean it up, while are you at it? If for no one else then for yourself, who is the one most likely to re-use the code in the future.

Niels Holst
  • 581
  • 3
  • 5
1

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Let me answer from a software developer's point of view. Open sourcing the code has a huge benefits, even if it contains bugs or poorly written code. Point of being opensource is other people can contribute towards it. The code will be improved by others, bugs will be fixed and will be much more popular compared to a closed source one.

So, if you are confident on results, why hesitate?

Anish Sheela
  • 111
  • 4