36

There is a theoretical computer science paper that is published in a very good conference. The paper essentially comes up with an algorithm and an explicit function (mentioned in the paper) of some 20 parameters, f(r1, ..., r20), which ties the performance guarantee of their algorithm to the value generated by this function.

For example, if they find some values r1, r2, ..., r20 where their f function gives a result of 0.8, then their algorithm performance is at least 0.8.

The problem is that in the paper, the authors claim there exists a set of 20 r values where f(r1, ..., r20) >= 0.8. However, they do not write down explicitly in the paper the 20 r values that they used.

I tried using an optimizer to find 20 values where their stated function might indeed give 0.8 but I cannot find any. The closest I got was 0.74. This is not an improvement over prior algorithms. I emailed the authors 2 weeks ago for the 20 r values that they claim to have used, but they did not reply.

My question is, what options do I have? When should I send another email? If they do not respond, do I have any options? To be frank, I'm surprised that none of the reviewers asked for the 20 parameters that they claim to have used to be written explicitly in the paper.

user438383
  • 3,170
  • 2
  • 17
  • 25
user3508551
  • 3,888
  • 4
  • 22
  • 33
  • 17
    There are lots of situations in mathematics (of which theoretical computer science is arguably a branch) where you can prove that an object exists but cannot name it explicitly. There's nothing inherently wrong with that. It is quite possible that the set of values is outside the range you searched, possibly outside the range of values that your program can handle (e.g. they might be trillion-digit numbers for all we know). – Nate Eldredge Jun 05 '23 at 04:30
  • 4
    So in my opinion, the way to check their result is to read their proof. If you find a gap, then indeed their paper is incorrect. If you find a step that you can't verify, despite having done enough background research that you should be in a position to do so, then start by contacting the authors to ask for further explanation. – Nate Eldredge Jun 05 '23 at 04:46
  • 25
    @NateEldredge The numbers (ri) are between 0 and 1, so no magnitude issue. My issue is that they say that they found these numbers numerically. I am completely ok with the idea that they have a better optimiser/searching a larger search space/having a better optimizer than I do, but I still can't find values anywhere close to what they're claiming. It also seems strange the authors don't have 10 minutes to copy paste 20 values between 0 and 1 which significantly improve a known bound. – user3508551 Jun 05 '23 at 05:00
  • @user3508551 Please review my edits. Is this performance guarantee function itself derived and "proved", e.g. by some sort of limit analysis, in their paper ? Do any or all of the 20 parameters have any meanings in the context of the algorithm ? It seems to me that in the absence of a response from the authors, you are free to bring your concerns to the editor of the conference proceedings. Please note that in this type of situation 2 weeks is maybe too short a time for a response. Moreover if the authors are in consultation with the editors on this matter following requests from you et al. – Trunk Jun 05 '23 at 12:37
  • 4
    If the authors are not responding, then consider asking on pubpeer. – Dimitri Vulis Jun 06 '23 at 12:39
  • 1
    @user3508551, multidimensional optimizations are nasty, and get nastier as the number of parameters increases. Do you have reason to believe that the f function is very smooth? Otherwise you could be looking for a needle in a haystack where, maybe, r17 must be between 1.3334323e-9 and 1.3334325e-9 (and the other 19 parameters are similarly arbitrarily constrained) to get to a place where optimization will lead you to the proper optimum. Knowing the r's are between 0 and 1 is great, but knowing that the function is smooth for differences in r_n of 0.001 or less would be even more valuable. – The Photon Jun 08 '23 at 06:05
  • 3
    @ThePhoton The function is smooth, continuous, differentiable, and fairly well behaved. – user3508551 Jun 08 '23 at 06:32
  • How many extrema are there? Have you taken that into account? (BTW @ThePhoton, what do you mean by "very smooth"?) – Solomon Ucko Jun 08 '23 at 14:56
  • @SolomonUcko, sorry but the best mathematical definition of smoothness is beyond me (typically it might be defined by the number of continuous derivatives, but that might not be the most important thing here). And in any case this is Academia not Computational Science so it's probably not the right place to go into those details. – The Photon Jun 08 '23 at 17:15
  • 1
    20 variable function is not that high dimensional a space to search in some cases, can you prove (through computer-aided or non-computer-aided means) that the function is bounded above by some value < .8? In this case your statement would be much stronger, and would require them to accept your result or disprove it. Some thoughts off the top of my head: you may be able to reformulate the function as a MILP and use an off-the-shell solver to prove upper bounds. Even if its not an exact formulation the taylor expansion would still be a linear approx to the original and can be bounded. – ldog Jun 08 '23 at 20:52
  • Two weeks is not very long to wait for a reply. Maybe the corresponding author is busy with teaching, maybe they're on holiday, maybe they're on fieldwork (OK not so likely in theoretical computer science), maybe they're ill. – Daniel Hatton Jun 10 '23 at 13:57

4 Answers4

33

The conference format historically has not served theoretical computer science well. Some thirty years ago, I heard from well-known practitioners in the field that many published results were false, including sometimes their own. I hope things are better now. So, it is quite possible that there is an error that went unnoticed, because no referee had sufficient time to check the results.

Unfortunately also, making results replicable is still not always standard in CS. I certainly cannot find code for work I did some twenty years ago. This is definitely getting better.

The reward structure of computer science also does not help, as tender loving care of work stops after publication. After publication, the team often goes its own ways, and if the main work was done by a graduate student, that student might become hard to find after graduation. Also, results like those obtained by using an optimizer might not be reproducible because notes were not taken sufficiently accurately.

You end up now with a lot of scenarios that explain why you do not get an answer:

  1. Nobody feels particularly responsible to talk to you as the person that made the discovery is elsewhere.
  2. They genuinely believe in the result, but cannot reproduce it themselves because they forgot to write down the parameters used or because there is a newer version of the software used and the older one is gone.
  3. They are embarrassed at having been found out making a mistake.
  4. They are busy with the end of the quarter or are already in summer mode.
  5. Each author thinks that some other author should answer.

It could also be of course that the fault is entirely yours for using the wrong tool or using it in a naïve manner.

In an ideal world, you would spend more time trying to follow their procedure, and, if you cannot reproduce their result, submit a comment to the same conference. But we are not in an ideal world and conferences are bad at accepting rebuttals and do not have letters to the editor.

msh210
  • 103
  • 4
Thomas Schwarz
  • 22,862
  • 2
  • 54
  • 92
  • 17
    If you are a grad student, I would ask someone higher up for help. If they have connections and can forge an introduction your chances of an answer will increase. (It shouldn't be that way, but it's only human to give increased attention to people you know.) – Marianne013 Jun 05 '23 at 11:13
  • 1
    I am curious, in that field, are those known false publications officially retracted? – Chuu Jun 05 '23 at 16:16
  • I don't know, it was a reasonably well-known guy talking when I was in graduate school. – Thomas Schwarz Jun 05 '23 at 16:35
  • 2
    There's also the additional scenario in which this was not a "mistake", but where the authors (or at least one of them) outright invented the 0.8 claim. – Dan Romik Jun 05 '23 at 17:56
  • 2
    Another possibility is simply that OP's message ended up in the authors' spam folder. – Matt Jun 06 '23 at 15:14
  • 1
    @Chuu, known false results are not usually retracted in theoretical CS---at least within the conference circus. Polite authors attach comments to their arxiv versions. Others don't.. – Oxonon Jun 07 '23 at 02:56
7

To add to the otherwise excellent answer of @Thomas Schwarz, in general people tend to be extremely slow to react in such cases, if they even react. I have also sent many messages about supposed errors/missing details during my early years, sadly nobody ever answered. If you "need" your result for your research I would say be on the cautious side and assume the "real" result is the one you can reproduce yourself.

Another point, also important, is how old is the paper. The older it is the lower the chance of response for myriads of reasons. A real possibility is that the result in the meantime, correct or wrong, might have been improved by another paper which makes the whole situation "waste of time".

PsySp
  • 7,901
  • 6
  • 34
  • 58
0

Wait for two more weeks. Two weeks is nothing for busy people.

If you fail to get these parameters in the long run, you might want to publish this as a small technical report, citing the paper with the missing parameters. Before publishing it (or even writing it), talk to your supervisor, as this might be perceived as a hostile act. Also share a draft version with the authors first to give them the change to provide the right parameters. They might be more motivated facing a negative follow-up article.

usr1234567
  • 5,748
  • 14
  • 36
  • Writing a technical report might work in some cases. However, OP should make sure that the result still stands at this point and it is not obsolete by possibly some other paper(s), otherwise it would seem weird "look I improved this 1990 paper's parameter from 0.8 to 0.74, but since then people know how to go even further to 0.6 or 0.5" (I assume lower values are better) – PsySp Jun 07 '23 at 08:29
  • 4
    @PsySp It seems like larger numbers are better--the paper is claiming a very good result, better than other existing methods, but the OP is not able to reproduce this very good result. I am guessing something like "we are able to achieve 80% accuracy, which is an advance in the best possible results" vs. "actually you aren't achieving 80% accuracy, but at most 74%, which 1, we already had by other methods, and 2, I wish this did work because I really do need 80% for some important thing that I can't do otherwise". – user3067860 Jun 07 '23 at 13:31
  • 2
    How good are you at optimization problems with 20 floating variables. Optimizations are tricky. Have you done this with starting guesses that completely cover your solution space? In response to your question, it doesn't feel to me like much of a response is required in any case, and scale that to the confidence you hold in your own abilities in optimization. – Scott Seidman Jun 07 '23 at 16:17
  • @user3067860 Sure, but this doesn't change what I have elaborated in my comment and in my answer. – PsySp Jun 07 '23 at 16:25
  • @PsySp It makes a difference for your comment...if the result was "wrong" in that actually it should have been better than reported when it was reported but now it's obsolete is different than if a result is wrong in a worse way. The former type of issue is unlikely to be a problem for any other papers based on that result, whereas the latter type of issue easily could be a problem for work that hasn't even started yet. – user3067860 Jun 07 '23 at 16:41
  • @user3067860 We agree that I can also see some cases where what you propose could be a possible action. – PsySp Jun 07 '23 at 17:40
  • 1
    @PsySp Your example does not fit. OP cannot reproduce a result. If OP is not able to trivially get the parameter by himself and they are not given in the paper, what is he supposed to do? Reaching out to the authors gave no answer. Writing the paper informs the scientific community that OP struggled to use the published result. This might triggere a follow-up paper by the original authors or a third group explicitly stating the required parameters. This outcome would serve the scientific community as a whole. – usr1234567 Jun 08 '23 at 14:16
0

Probably the best thing to do is write a research article countering their claim, say they may have been mistaken given your new research, and try to just get the real result out there.

Its better as an opportunity for yourself to show that you can get the right idea. Not everything in science is accurate first time, we need people to notice incorrect claims and then to publish this information as a letter to the editor, or even journal article if there's a lot to say. This does happen even with the most famous scientists, many years later, and has an impact on the field.

apg
  • 651
  • 5
  • 15