How honest should I be in disclosing not-so-exciting results?

Question

I'm a sociology undegrad working on an essay for a methods class. I'm also planning on submitting it as a sample for my application to grad school. I don't want to be too specific, but I believe that this work is quite original and my hypothesis would confirm previous literature, and all in all I think it would would make a good impression on the admissions committee.

So basically I've run the tests and I'm getting conflicting results. Using one dataset (which has more observations) gives me very significant results, while using another one (which would arguably be more accurate) doesn't give me anything. So here I am at a crossroads, and I've come up with three possible options as to what to do:

Only show the significant results. After all, this is just a ten-page essay, it's not supposed to be publishable or anything, right?
Only use the better dataset and admit that there just isn't much there - maybe blaming it on the small sample size or on the not-so-good dependent variable. Hopefully the committee would appreciate the honesty and the relatively advanced methods that I used.
Show results from both datasets, suggesting that the differences might be due to the sample size or maybe to chance.

As I type this I'm leaning more towards option 3, but I'd like to hear from people with more experience in academia. What should I do?

Contradictory results are the first step towards a discovery. — henning, Dec 10 '18 at 17:21
@henning ...or a debunking of scientific credos. Embrace the contradiction. — Captain Emacs, Dec 10 '18 at 17:23
"this work is quite original and my hypothesis would confirm previous literature" It confirms existing previous results, but it's original? — Acccumulation, Dec 10 '18 at 18:23
+1 for asking. I strongly recommend you visit Andrew Gelman;s blog regularly for discussions of the proper way to do statistics, particularly in the social sciences, Here;s one example https://andrewgelman.com/?s=file+drawer — Ethan Bolker, Dec 10 '18 at 18:39
Turn the question around. Don't ask "how honest should I be?" Ask "how hard should I attempt to deceive my reviewers?" Is the answer to the question more straightforward when you ask it that way? — Eric Lippert, Dec 10 '18 at 22:28
Everybody seems to agree, but then bizarrely you have so many papers published with amazing results on hand-picked datasets that nobody can reproduce on any other dataset :-) — jcaron, Dec 11 '18 at 09:47
Once I saw the words "how honest" and "disclosing results", I knew the answer to your question would be "completely honest." — JaS, Dec 11 '18 at 18:33
I actually just listened to a podcast from the lovely folks at SYSK that hit on this exact topic. A huge problem with research is that only the "sexy" results (their descriptor, not mine) tend to get reported and it leads to misleading/outright false information (for those interested: https://www.stuffyoushouldknow.com/podcasts/research-tips-from-sysk.htm) — Broots Waymb, Dec 11 '18 at 18:56
@jcaron that's the powerlessness of the 'ought'. Unfortunately, publication bias is a reality. — henning, Dec 12 '18 at 09:06
There is a third option: bad methodology -- either in collecting the data or in deciding what data to collect. — Marxos, Dec 12 '18 at 18:07
You're forgetting the scientific adage that any theory can be considered proven if doing so involves throwing out fewer than half of your observations. (I'm being sarcastic.) — Mark Meuer, Dec 14 '18 at 16:37
I want to make sure I understand. Your essay covers two experiments. The first had a larger data set and turned out significant results, but its variables are lackluster. The second study has more interesting variables but the study was too small to turn out a significant result. Is that correct? Sounds like exactly what I would want to happen in order to justify repeating the second experiment with a larger study. Would that not be the obvious conclusion for the essay? — John Wu, Dec 16 '18 at 09:25

score 199 · Answer 1 · answered Dec 10 '18 at 17:36

199

In research, you don't set out to prove that something is true. You set out to discover whether or not it is true. This would be knowledge. The other is just propaganda.

Negative results are not a failure. They give you evidence just as do positive results. If you ignore, or obscure, results you are lying to yourself and others. If you design an "experiment" so that it is guaranteed a priori to produce positive results, it isn't research.

Hoping that something is true isn't evidence. Many researchers start out with that idea. I think this is true. I really want it to be true. But if it is false, it is just as valuable (possibly more so) to know that and to be able to investigate why.

Report all your results. Try to explain why different aspects lead you in different directions. Only then can your learning begin.

answered Dec 10 '18 at 17:36

Buffy

363,966
84
956
1,406

36

I really want this answer to be true, but is it... – user541686 Dec 11 '18 at 09:37
7

This, for the most part. I want to point out that there seems to be (and it's not good) this perception that somehow the only "good" results are ones that reach a novel conclusion. But this is far from true. Reaching an expected or conventional conclusion through an untested or novel path is just as much new knowledge, because it still moves that path from "predicted" to "knowledge". Moreover, even replication of an old path still has some use in that it increases confidence in those existing results, esp. if there was prior doubt about them. Replications are important. – The_Sympathizer Dec 11 '18 at 10:44
3

[And also, I might point out, verifying the predictions of an accepted theory through a novel test also serves to increase confidence. Those "confidently accepted" theories don't just get that way by magic or by fiat.] – The_Sympathizer Dec 11 '18 at 10:47
1

the main issue with not-so-exciting results imo is that it generally doesn't lead to more funding in this profit-driven world - finding funding for pure basic research without direct & easily commercializiable results is difficult, so as long as you are willing to change what exactly your researching rather than keep chasing a dead end, your likely to be fine - that said, multiple dead ends / lack of any exciting results will likely harm you business-wise, even though it shouldn't from a pure scientific point of view – user2813274 Dec 12 '18 at 17:27
3

"You set out to discover whether or not it is true." I agree the overall goal is to determine if a hypothesis is true/false, but the way you've stated this is critically incomplete, IMO. In order to determine if a hypothesis is true we try to disprove it. We design experiments, studies, etc. with the intent to try to falsify the predictions made by the hypothesis. Only after something has withstood attempts at disproving it do we consider it to likely be true, under the conditions tested. IMO, this is a critical area that is often misunderstood, usually due to people wanting to be "right". – Makyen Dec 12 '18 at 20:00
1

@Makyen, I would state it a bit differently. A positive result in a statistical study provides evidence that the Hypothesis might be true, not proof that it is. If the study is replicated often enough, the evidence builds. But only a complete population study can provide proof, and that may be impossible if the population changes with time, making exact replication impossible. We get evidence, not proof. – Buffy Dec 12 '18 at 20:05
"Try to explain why different aspects lead you in different directions." I feel like this could use a bit of elaboration. I presume you mean that the OP should start looking for new hypotheses that explain the difference (such as an unaccounted variable in one of the samples). – jpmc26 Dec 13 '18 at 03:58
1

This is a great answer and appropriate to OP's discipline, but the first paragraph would not apply in mathematics. – Randall Dec 13 '18 at 21:46
1

@Randall, from one mathematician to another, I disagree. Before you can prove something, you need to first study what it is that "might be true" and is interesting enough to study. Nobody hands theorem statements to you (after you graduate, anyway). A mathematical proof is only the final step in a process of discovery. The key phrase in my first paragraph is "set out...". – Buffy Dec 14 '18 at 00:25
@Buffy I guess I agree with you again, phrased that way. – Randall Dec 14 '18 at 01:36
@Mayken that's a good way to explain the (dominant) falsificationist paradigm. Bayesians probably place less emphasis on the value of a single falsification. – henning Dec 15 '18 at 21:25

henning · Answer 2 · 2018-12-10T17:36:05.990

83

Omitting negative findings and selectively reporting only the positive findings would be a breach of research ethics. As a researcher you are supposed to uncover knowledge,* not to obscure it. Findings are often contradictory and in need of interpretation. By explaining how you obtained these contradictory results (i.e. your methods), you help others to avoid dead ends in the future and to make sense of what looks confusing today.

_{*Interestingly, the knowledge that research creates often takes the form of higher-level confusion rather than ultimate certainty.}

edited Dec 10 '18 at 17:36

answered Dec 10 '18 at 17:30

henning

35,032
10
121
151

17

+1 because research ethics aren't something that applies only when something is "publishable" (as in "After all, this is just a ten-page essay, it's not supposed to be publishable or anything, right?") – De Novo Dec 10 '18 at 19:14

score 29 · Answer 3 · answered Dec 11 '18 at 00:49

29

How honest should I be in disclosing not-so-exciting results?

You should always be completely honest: Show the results of both datasets and let the conclusion follow from the data. Comment objectively on the quality of the two datasets, and their sample sizes, but don't exclude data merely because it gives undesirable or unexciting results. In terms of the differences between the datasets, if you know why they are different then explain this, and if you don't know why they differ, then say so - don't present your speculations as scientific conclusions.

answered Dec 11 '18 at 00:49

Ben

68,453
9
142
263

3

I very much like this answer. I have a lot of respect for papers which are honest — papers which "show off" results, and obscure the honest assessment of the author's results often cost other researchers a lot of time. If things are not "as true as the author claims" a lot of time can be wasted trying to learn a technique or reproduce a result, which turns out to be not useful at all... – Earthliŋ Dec 14 '18 at 19:24

score 13 · Answer 4 · answered Dec 11 '18 at 01:56

13

For option (3), add 'or there is something I do not yet understand going on".

This is much more interesting.

Your undergraduate course is there to teach you how to answer questions.

The important thing in research of any discipline is not getting the right answers but asking the right questions.

So, present both data sets, call out the discrepancy and try to explain why that is interesting and why it is worth following up.

Setting out a mini research problem like this could make you stand out much more than simply having a result.

answered Dec 11 '18 at 01:56

Keith

1,160
6
10

This does jump out as the best course. On one hand, it's crucial that proper methods are used, even if they lead you nowhere. On the other hand, if you seek publication, it has to be of interest to someone. Comparing jump heights of cat fleas and dog fleas does benefit from the conclusion that they are different. For you, the interesting bit is the difference between the datasets. Warning: you're sure to be asked about it, so you either have to delve into that a considerable bit, or explain why it falls outside the scope of your work. – kaay Dec 17 '18 at 11:33

score 5 · Answer 5 · answered Dec 11 '18 at 00:51

I'm only a student too (graduate level), but here are a couple more reasons to go with option 3 of showing both data sets:

As mentioned in henning's comment, perhaps you can use your unusual results as a stepping stone for further research, and include this in your application. Treating unsatisfactory results in such a way can show that you have motivation and resilience.
If you did good work and showed it, even without getting "good results", that can show that you at least have potential.
Furthermore, in the context of applications where people usually put only their best foot forward, your honesty may actually be appreciated and respected by the admission committee. It can show that you put science first.

score 4 · Answer 6 · answered Dec 10 '18 at 19:15

Are your significant results a large effect size, or just a tiny change that is significant because of the large sample size?

Are your non-significant results similar in direction and magnitude to the significant results from the other dataset?

Consider how much the size of the dataset is impacting what you are seeing - you may be able to frame one study as confirming the results of the other if they are in agreement apart from significance. Look at more than just the p-values, especially if they are coming from a very large dataset.

AdamO · Answer 7 · 2018-12-11T21:43:59.943

Consider for a moment that you may be comparing datasets (and results from them) incorrectly. "Significance" or rather the power is not independent of design. If Study A is done on 1,000 people but Study B is identical but includes only 100 volunteers, Study A is much more powerful, so (statistically) significant findings from A and (statistically) non-significant findings from B are non-surprising. There are better methods for comparing two studies, like a forest plot.

I only mention this because it all depends on the "you" you are trying to sell for this application. An undergraduate level sociologist doesn't need to have a graduate level statistics education, but if you are boasting it as a strength, you should be sure that you are correctly interpreting a set of findings.

The word "negative" (result or study) is an abuse of statistical terminology. There are issues of power, context, and precision; but adept researchers are readily throwing the baby out with the bathwater. Stop for a moment and think: "Do not reject H_0" means that the confidence limits include the null hypothesized value(s): 0 for differences or 1 for ratios. So what?

1) Was this study sufficiently powered or is it a complete shot-in-the-dark? Large, untenable confidence intervals can represent a crappy study or it can reflect substantial heterogeneity in the population. Were there issues with recruitment or compliance? Did you need to compensate people better? Did you administer an existing instrument and if so, did you assess yourself or the patients to be sure the wording is clear? If it's a trainwreck study you can focus on lessons learned. E.g.

we recruited 30 people based on an incorrect power calculation, our effect estimate had a much smaller magnitude than was noted in previous literature. This is a cause for some concern given our calculation was based on previous research which claimed that...

2) Is the CI narrowly on 0 or 1 excluding all other research? This is a significant finding because it is inconsistent with other literature. There's a whole field of research devoted to determining the effects of publication bias. Funnel Plots show the expected distribution of effects from meta-analyses. If the distribution is shifted with a gap at H0 it gives some pause as to whether the state of evidence is exaggerated by filtering out null findings? Important landmark research has been able to conclusively say, "No. A certain treatment does not / cannot cause a difference.

3) Is the CI wide but centered on a result which confirms previous research. For instance:

A 5,000 person study of salt reduction found that the HR for MI was 0.95 95%CI 0.92, 0.99 (p < 0.05). A confirmation study of 100 found a HR for MI of 0.95 95% CI 0.5, 1.45. (p > 0.05).

Importantly these studies agree 100%.

score 0 · Answer 8 · answered Dec 13 '18 at 20:06

0

That sounds so super interesting. There are statistical issues at play for sure, I don't want to dissuade you but you need to make sure that you did the math (including data collection and methodology are correct), but you can write a very powerful paper by comparing two methods. Something like:

Method A, which is cheap and easy to collect data on but we have concerns that it will contain bias gives a positive result.
On the other hand Method B that is difficult and expensive to collect data on but is far more thorough and not expected to contain bias gives a negative result.
Therefore, researchers should avoid using method A.

I'd be willing to bet that you could get a journal to publish a paper that is written like that, given that all of the analysis, data collection, etc. was above board, let alone get a good grade in the class.

answered Dec 13 '18 at 20:06

Ryan

267
1
2

Actually, your "Therefore..." isn't surprising at all, so likely not a candidate for publication. Using any methodology that is expected to contain bias is flawed, unless you have ways to measure it (adding cost...) – Buffy Dec 13 '18 at 20:21
Nah, I think you misunderstood me because I slapped this answer together on a lunch break. That's my bad. The way that I was thinking about it was that the OP had a bunch of data, apparently collected in two different ways. – Ryan Dec 14 '18 at 22:32
From OP "Using one dataset (which has more observations) gives me very significant results, while using another one (which would arguably be more accurate) doesn't give me anything." It wasn't known a priori that method A was biased. I believe the paper that I was envisioning is more of a demonstration that method A is biased when that wasn't known beforehand. – Ryan Dec 14 '18 at 22:40
Of course you have to do a thourough literature review and make sure that the bias you are reporting isn't known already, but that's what research is about. – Ryan Dec 14 '18 at 22:41

How honest should I be in disclosing not-so-exciting results?

8 Answers8