52

This may be a weird question, but I often wonder why the default way of writing about a given methodology is describing the method as if it "fell out of the blue". Admittedly, not all research work is written like this and perhaps this is not an issue for very experienced researchers/practitioners, but I believe it would be much more insightful if authors wrote the steps that led to the specific decisions when designing a new method. So, instead of describing a new method, the author would write the problem and motivation (this is usually done), and then elaborate on the thought process that led to the final design choices. I don't mean hand-holding on basic stuff, but to describe things like "we wondered if there was a way to solve X. Y is a popular method to achieve Z, a property required when solving X for the reason K. However, Y needs to be tweaked in the following manner so that it also asymptotically satisfies W" or "we tried this way of solving it, but it did not work due the following reasons [insert reasons], so we decided to try this instead". Was a certain property coincidentally satisfied by your decision choice, or did you reverse engineer a method that satisfies that property?

This may seem like a childish idea and most people probably don't have the time or interest in such a writing style (or may even feel insulted by overly detailed descriptions). However, I believe it would be a much more valuable contribution to science because:

  1. Researchers would learn from each other different ways of thinking about a research problem and the strategies to solve it
  2. Researchers interested in building on the presented work would know which ideas did not work (preventing them to pursue dead-ends), and if they know better ways of solving a particular sub-problem they could easily improve the method.

What are your thoughts on this?

user3653908
  • 673
  • 5
  • 8
  • 10
    This seems entirely opinion based. But researchers aren't cut off from that "process" stuff. It just appears elsewhere than in the papers themselves. Moreover much of it is irrelevant to the results and could even be misleading. – Buffy Jul 28 '21 at 12:57
  • 7
    With exceptions for reviews and the like, journals typically have page limits, e.g., eight pages. Even with online supplements available, conciseness helps readers keep their heads above the rising tide of publications. Maybe. – Ed V Jul 28 '21 at 13:04
  • 7
    I've heard of honors Calculus classes that do teach this way, but are very confusing for the student. Also, looks at books that do. For example, a Radical Approach to Real Analysis does describe the history. Last, the history of science field often covers the how science discoveries occurred and the thought process. – Richard Erickson Jul 28 '21 at 13:28
  • @Buffy Could elaborate on your answer? Where, exactly, does it appear? And how could describing the strategy used to solve a problem be misleading? If anything, I believe it could help clarify why things were solved in a particular way instead of other more obvious one. – user3653908 Jul 28 '21 at 13:49
  • I haven't given an "answer", actually. Scholars have discussions and answer questions about their work all the time. Classroom, conferences, invited talks, blogs, articles, letters, etc. I can for see a situation in which a "discussion" of "thoughts" leads a reviewer to accept something they should not, perhaps due to leaps of logic or whatever. – Buffy Jul 28 '21 at 13:54
  • 2
    @Buffy "I can for see a situation in which a "discussion" of "thoughts" leads a reviewer to accept something they should not, perhaps due to leaps of logic or whatever." Isn't it the case that more transparent reasoning about a new method would more easily expose said leaps of logic rather than solely relying on self-reported results in a medium where only positive results are encouraged to be published? – user3653908 Jul 28 '21 at 14:03
  • 5
    There's been a lot of prior discussion of this question in the scientific literature. See responses to Peter Medawar's Is the scientific paper a fraud?, for example. I also recall a similar sentiment expressed independently. – Ben Trettel Jul 28 '21 at 14:16
  • 2
    The science should stand on its own. If the methodology is valid and carried out faithfully then something is learned. The "why" is irrelevant to the result. And learning that something is not true is just as valid as learning that it is. Graduate students are too often misled to thinking that if they don't get "positive" results then they have failed. A negative result is a valid result. In science, we don't set out to "prove" our hypothesis, but to discover evidence about whether it is true or not. Math is a bit different but there is an equivalent formulation. – Buffy Jul 28 '21 at 14:17
  • @BenTrettel Thanks for the references!

    @ Buffy I am coming from the point of view of applied math research. Maybe in other fields this is not as big of an issue or it is actually the standard way the work is presented.

     – user3653908 Jul 28 '21 at 14:49
  • 2
    There were numerous discussions of this. As far as mathematics is concerned, you might be interested in looking at an old MO thread https://mathoverflow.net/questions/38639/thinking-and-explaining . The opinions generally seem to range from "It would be nice to communicate the thought process, but it is not clear how to do it without turning the paper into a messy labyrinth" to "Everybody thinks differently from others, so imposing on the readers your way of thinking only confuses them". I believe that some people try to do it now and then, though I cannot think of any success story off hand. – fedja Jul 28 '21 at 16:31
  • 1
    @EdV: Page limits exist in some fields, but they're far from universal. – Schmuddi Jul 29 '21 at 09:43
  • 3
    @Schmuddi Good to know! One nice thing about this particular stack exchange is learning just how varied academia is. – Ed V Jul 29 '21 at 10:10
  • 1
    I think it is likely that this would end up with the publication of a lot of misleading post-hoc rationalisations of people's actual thought processes, even with the best of intentions (c.f. Rashomon). – Dikran Marsupial Jul 29 '21 at 10:46
  • Essentially one wants to see the output of a thought process. It would give the "appearance of being unprofessional"... – JosephDoggie Jul 29 '21 at 12:29
  • This lecture is exactly about why "thinking process" is very different from "writing process" and two must not be mixed https://www.youtube.com/watch?v=vtIzMaLkCaM – aaaaa says reinstate Monica Jul 30 '21 at 03:18
  • Because we'd come to realize that so many a priori assumptions, data screening, and cherry picking went into the experimental design that we should ignore all the p-values ;-) – Scott Seidman Jul 30 '21 at 13:00
  • they could always do it in an appendix or a blog instead? i figured papers are mainly for the results and methodology – BCLC Jul 31 '21 at 06:20

11 Answers11

45

Thought processes are messy.

If you wrote out the thought process behind the typical papers I contribute to, you'd have to distill hours of weekly meetings, circular avenues where the same idea comes up 2 or 3 or 2 dozen different times before it gets incorporated definitively, endless iterations of experimental design, dead ends and failed experiments where everything starts over, side projects that unexpectedly inform a central one, shower thoughts, and pub inspiration.

In a good writer's hand maybe it could turn into a novel, but it wouldn't be the most direct way to explain the main findings of a work and contextualize those results within the literature. As Richard Erickson comments, history of science books often do get into the "process" behind discovery and they can be a great read (Ruse's The Darwinian Revolution is a personal favorite), but it's not reasonable to expect every scientist to go into this level of detail for every paper nor reasonable to expect their audience to invest the time to read it.

I do see "thought process" content in many papers, but at that point it is usually made into a much simpler, linear story to avoid all the meandering.

Bryan Krause
  • 114,149
  • 27
  • 331
  • 420
  • 19
    The "thought process" behind some discoveries went on for a hundred years. – Buffy Jul 28 '21 at 13:55
  • 1
    I agree. I am not suggesting adding every single step but rather the core, trimmed down ideas. – user3653908 Jul 28 '21 at 13:57
  • 1
    @user3653908 Maybe your field is different, then; I see the core/trimmed down thought process often, usually you'll find it in the introduction section. From a textbook perspective I would also say that biology textbooks I've experienced are usually written with history/thought process in mind, at least in part. They usually use a few key experiments to demonstrate "how we know what we know" and familiarize students with scientific methodology rather than just drilling facts. They don't go into everything like this, though, for the same reasons I mention in the answer. – Bryan Krause Jul 28 '21 at 14:43
  • @BryanKrause Right. Maybe I am biased to the ML field where the standard are 8-page conference papers where some new method is described and some properties are shown to hold for this method without any explanation as to how this method was derived. – user3653908 Jul 28 '21 at 14:48
  • 19
    @user3653908 My cynical view of a lot of papers in the ML field is that there often isn't much thought process, unfortunately. But I am definitely biased in thinking mostly of papers that overlap with my own field where the papers are less "new method in ML" and more "I know sklearn exists, what if I use some of the methods there on this brain data. Brains are cool right?" But less cynically, yes, length limits are surely going to cut down on prose when there is other meat to share. Hopefully they at least cite similar approaches; if not that's just sloppy rather than a style choice. – Bryan Krause Jul 28 '21 at 14:51
  • @BryanKrause Right. I am referring more specifically to new methodology papers and not applied papers. – user3653908 Jul 28 '21 at 14:54
  • 1
    Even though it's a bit messier, I find it more fair, relatable and educational to read "I did not know, so I asked on SE and Bryan Krause told me that [..]" instead of just stating the final result and including a link in the references. – Džuris Jul 29 '21 at 20:34
  • @Džuris Perhaps, but what I was getting at in this answer is that it is rarely that simple. How deep do you go? Why not start with what brought you to Stack Exchange in the first place? In the opposite direction, isn't "I did not know" redundant? At some point no one knows anything; a citation is a simple format for saying "this is where this knowledge came from". – Bryan Krause Jul 29 '21 at 20:43
  • 1
    ""I did not know, so I asked on SE and Bryan Krause told me that [..]" That is pretty much what a "personal communication" reference means. – Dikran Marsupial Jul 30 '21 at 06:30
  • 1
    @BryanKrause the depth would be determined by an optimal balance between clarity and conciseness. Tracing back an idea to when you were in the shower singing a song about yellow submarines is clearly too much. Describing how you empirically discovered during an experiment that your data lies on a curved manifold and thus you wondered whether using a particular differential geometry method would help your model, and how maybe adding this little tweak would confer it some desired property seems just right. – user3653908 Jul 30 '21 at 10:42
  • @user3653908 Sure, but... I'm not quite buying the premise that people don't already do that in the circumstance you describe. – Bryan Krause Jul 30 '21 at 14:06
  • @user3653908 Here's an example from a paper I coauthored (Hentschke et al 2017): "Here, we investigated the effects of isoflurane in mouse auditory TC brain slices on evoked and spontaneous neuronal responses, focusing on polysynaptic, propagating network activity (‘bursts’). Our interest in network bursts derives from the importance of the activity of ensembles, rather than of single neurones, in cortical processing.14–16" We tell you concisely what we're going to do and why we're doing it, with reference to 3 papers. – Bryan Krause Jul 30 '21 at 14:08
  • @user3653908 I'm looking at another paper of mine, https://doi.org/10.3389/fnsys.2014.00170 and the logical flow of the introduction starting in the 2nd paragraph is "Here is how people talk about things, here is how this is a problem because of contradictory evidence, here are more reasons this doesn't seem to apply in our special case, here's an alternative framework that we are going to use for the rest of the paper." – Bryan Krause Jul 30 '21 at 14:11
22

My answer is from a mathematics perspective.

I like the analogy of exploring an alien landscape for doing research in mathematics. In this analogy, the ideal mathematical paper reports on having found some astonishing landmark together with useful instructions of how to get there.

Good instructions for how to find a place will look very, very different than the journal of the first explorer to get there. In the latter, you'd have stuff like

I was trying to reach the summit of that mountain, but at some point I found myself separated from the summit by a deep gorge. But when I looked around, I spotted that serene lake between the tree tops. So I climbed back down and tried to make my way to the lake. Then I caught malaria and walked circles for a while hallucinating. After recovering, I made a lucky guess and stumbled upon the lake again, and it really was very beautiful.

But when we are actually dealing with math research, it probably makes much less sense. Reaching the point where you understand an idea well enough to even tell a fellow researcher about it can be hard work. There is "For a few months I thought about whether X could be a good way to attack my Y problem. I don't remember why I ever believed that this might have worked out". There is "one morning I woke up, and under the shower it suddenly all made sense". None of these are particularly helpful for a reader.

Arno
  • 43,962
  • 8
  • 123
  • 169
  • 3
    Waking up with an insight is pretty common, actually. The brain works even at rest. Sometimes even better at rest. Mathematical "dreams", on the other hand, often have very strange logic. – Buffy Jul 28 '21 at 15:03
  • 9
    @Buffy I think Arno's point is not that it's uncommon, but that it's irrelevant or at least unhelpful to a reader trying to understand the paper to know some step was a result of an insightful sleep. Well, maybe it could encourage the reader to develop better sleeping habits. – Bryan Krause Jul 28 '21 at 15:09
  • 2
    @BryanKrause, actually, I was "catapulting" the writers idea, not criticizing it. But yes, sleep and exercise are good research habits. – Buffy Jul 28 '21 at 15:11
  • @Arno while I agree with the sentiment expressed by your answer, I am not suggesting that people write how they came up with the solution while they were taking a walk in the park. Identifying the source of inspiration or how one has ideas is a separate discussion altogether. I am suggesting that you describe the motivation behind the different steps you used in your solution and why you thought each of these steps was well suited to solve the particular sub-problem at hand, even if it is just a "X is a method that is applied in vaguely related problems so we investigated further". – user3653908 Jul 28 '21 at 15:14
  • 5
    @user3653908 My point is that you seem to vastly underestimating how messy the proof process often is. I see explanations of the kind you wish for often enough that it is very plausible to me that most mathematicians include them whenever they had a clear rationale for trying something that they can express easily in words. – Arno Jul 28 '21 at 22:05
  • 1
    @user3653908 "X is a method that is applied in vaguely related problems so we investigated further." The good ML/NLP papers I read usually mentions this part. It's not really a thought process, but a justification as to why that method is suitable. Without the justification, usually it will be harder to be accepted, due to not situating their work in the context of existing work appropriately. Also usually the "failed methods" are also mentioned, but as an aside or in a footnote. But it's still there (usually to avoid questions like "why didn't you try ...?") – justhalf Jul 29 '21 at 06:13
  • @justhalf yes, the "failed methods" is now being included in some papers. Still, I think it should be a much more widespread practice. – user3653908 Jul 29 '21 at 06:56
  • 5
    @user3653908 I still think you're underestimating the scale of the question, here. I've written <10 page papers for which the those sections would stretch into hundreds of pages. – user3482749 Jul 29 '21 at 13:55
  • @user3482749 I agree that it could become too much. Common sense still applies here. So it would be a balance between maximum clarity and conciseness. Is this difficult? Most definitely. Could it make some papers way more accessible and let researchers learn with each other different strategies of attacking a problem (research meta-learning)? I believe so. – user3653908 Jul 30 '21 at 10:33
13

I can speak to why this is the case in mathematics; I suspect this bled over into the physical sciences as well but that's just a hunch.

When I was an undergrad I tried to read Rudin's Principles of Mathematical Analysis and at first I was totally perplexed -- seemingly out of the blue, he'd pull out the value of what some constant had to be in order to make the theorem go. I felt very stupid and asked my dad about this. He explained to me that this style of exposition is deeply ingrained in the culture of mathematics, going at least back to Carl Friedrich Gauss, who famously said, "A good building should not show its scaffolding when completed." He then gave me some strategies for reading, understanding, and writing proofs, and part of that involves reading and writing in an order other than the order in which it will be presented. None of my professors gave me or anyone this advice, we were just expected to figure it out or fail. Looking back, I think this was basically a form of gatekeeping.

There have been mathematical works that are presented in a different style, with much more background and exposition. I think one of the most notable examples is Melissa E O'Neil's work on random number generation. The PCG RNG paper did not appear in a peer-reviewed journal for some time because the ones she sent it to objected to the style of her writing, which presents much more of her thought process than would be customary for a math paper. You can read more about the history of the PCG paper from O'Neill herself here. The algorithm has since been included in several major software packages including numpy. Personally, I find the PCG paper to be a great read, and I think the fact that she had such a hard time publishing it is a damning indictment of the culture of mathematics.

So, to summarize, the reason more papers aren't written in this style is because (1) it's an ingrained habit that comes down to the present day through Gauss and other early 19th century mathematicians, and now (2) the consequences of trying to change the culture around this include doing a lot of unrewarding work and getting rejected repeatedly.

Daniel Shapero
  • 231
  • 1
  • 3
  • 1
    This answers my question. I was hoping to start a good debate on how things could be better, but it seems most are focusing too much on how involved the process is rather than how something would be vastly better than parachute definitions and the likes. Hopefully, this discussion will motivate more people to slowly abandon this terse style. – user3653908 Jul 30 '21 at 15:24
  • Robert Ghrist's book Elementary Applied Topology is an interesting example that very much breaks away from the traditional writing style, but that was self-published. A lot of people have also taken to blogging for the things that can't quite go in a refereed publication but which are still worth writing about. So it seems that the route is to seek other venues rather than to change the way traditional academic publishing works. – Daniel Shapero Jul 30 '21 at 15:45
  • @user3653908: " Hopefully, this discussion will motivate more people to slowly abandon this terse style." It certainly will. Look at the number of upvotes the question received. – Tho Re Aug 01 '21 at 21:25
12

In papers from my field (chemistry/materials science), it is actually quite common to do exactly what you suggest. However, you will not find anything of that in the typical methods/experimental sections because those are reserved for a description of what was done. The why is part of the discussion part of papers where authors explain what the underlying reasons are for their results, but also should justify their methodology if it is not obvious. Sometimes, this involves giving short mentions of attempts that failed to give the desired outcome.

Snijderfrey
  • 8,961
  • 2
  • 29
  • 48
8

I am not sure that we really have access to our actual thought processes behind our research, just the post-hoc internal explanation that we remember. It would require us to actively analyse and record our thought processes as we go along and who has the time for that. This may just be me, but if I am working on something my ideas and assumptions change and evolve as I perform experiments (partly because the next experiment are shaped by the results of the previous ones). I don't think I am able to roll back to the version of me at the start of the project and actually evaluate what I thought at the time. I think it is a mild case of the incommensurability of Thomas Kuhn's paradigms (but on a smaller scale).

Generally when I write a paper, I try to create a logical progression of ideas to explain what I found out during the course of the study. This is very rarely the same as the historical progression of ideas, with all of its blind alleys and "my brain hurts" moments (again that may be just me).

We are all prone to post-hoc rationalisation of events, it is human nature. I strongly recommend the film Rashomon to anybody that hasn't already seen it. I doubt the Rashomon effect named after it only applies to recollection of external events, or to situations involving personal gain/loss. I am not convinced that we are really objectively that aware of our thought processes and motivations.

Dikran Marsupial
  • 5,538
  • 20
  • 27
7

When you read a fiction book, you never have the author's thought process on how he designed the book. You may have the narrator's thought process in some books, but not the author. You only get the final output.

When you watch a movie, you don't have the thought process of the director included in the movie. You only get the final output. That would be dumb in many cases, as it would distract from the movie.

A research paper is this, the final output of a research process. A finely crafted piece of science whose goal is to convey information to fellow researchers in the most efficient and concise way.

If you want an example of reading about the thought process of some researcher, you can read Birth of a theorem by the mathematician and Field medalist Cedric Villani.

Taladris
  • 728
  • 6
  • 14
3

(I’m coming mainly from mathematics, with some crossover experience in theoretical CS.)

Most good academic writing does include a bit of “thought-process” explanation. The style you describe as “the default way” — writing as though everything “fell out of the blue” — does exist, but it’s not the default in fields I know (though it was more common a generation ago, in some areas of pure maths). Papers with no motivating explanation at all are seen as unpleasantly dry; it’s not uncommon for referees to ask for better motivating exposition.

But it’s usually just a little: Too much “thought-process” explanation is not as helpful to readers as you seem to expect. I’ve read papers that tried hard to explain a deep and subtle thought-process, but came across just as impenetrable waffling, and left me wishing the authors had just stuck to the facts. The main problem with such attempts is described well by Brent Yorgey: Abstraction, intuition, and the “monad tutorial fallacy”

So in sum, good authors are certainly conscious of this aspect, and usually choose to include some thought-process explanation, but not much. If you think more would be better, then be the change you wish to see in your field, and use a bit more in your own writing. But be aware of the reasons why most writers use only a little — remember the parable of Chesterton’s fence!

PLL
  • 10,210
  • 3
  • 33
  • 41
  • 1
    +1 Chesterton's fence is very apposite! Very relevant to debugging computer programs, but I suspect my students may appreciate the reference as much as I do! ;o) – Dikran Marsupial Jul 31 '21 at 18:27
3

For many research-level results I have obtained, the thought processes involved hundreds of wrong turns, including useless definitions and wrong proofs, and it would be really silly to include all those. Unfortunately, it is rare to be able to cogently deduce the results without significant trial and error, otherwise it would not be worthy of research, would it?

However, it is true that in many cases one can come up with a fictitious thought process obtained by splicing together the actually useful ideas that led to the final solution. And that ought to be included in any good exposition of the results. Sadly, there are still page limits for many peer-reviewed publication venues, putting a strain on the desire to give a verbose story of how to come up with the results.

user21820
  • 1,823
  • 14
  • 21
2

Different people follow extremely different thought processes. Thus, it is not, in general, extremely productive to force all people to arrive at a certain conclusion in the same manner. A closely related concept is described quite well by physicist Richard Feynman in this video on youtube(section beginning at 55:01) (from a BBC special "Fun to Imagine").

Consequently, presenting with a focus upon the important conclusion is a strong strategy. If it is necessary, one can subsequently build "walls" around the core idea being presented by describing other attempts to solve the problem that were eventually unsuccessful.

Name
  • 21
  • 1
1

Also from a mathematics perspective.

I was working on a problem for two years now, and, recently, in the process discovered another (in my opinion) nice connection/result. I wrote a paper about that (under review). I also felt it might be helpful for a potential reader, to better evaluate and understand the results, to give away some history and thought processes involved. However, from a logical point of view, it seemed more natural to me to present it the other way around: first present the discovered result and after that the application to the original problem that brought me there. Nevertheless, I gave a "personal story" in the introduction, I wrote something like this:

Now, maybe it is best if I give a little bit of a personal story about the results presented here, and that they are, in some sense, the result of two strands of thought.

[... following approximately one page of text, describing a collaboration, where the original problem came from and how it led to the present results...]

As is most often the case, and to have a clean separation between a more group-theoretical part and a more automata-theoretical part, the presentation does not follow the order of discovery in this sense.

After that, I continued to present the results. I hope the reviewers like my "historical account" ;)

StefanH
  • 361
  • 1
  • 10
1

Let me give a different perspective, coming from mathematics. Other answers have already explained the reasons why papers generally don't have so much thought process, so I won't rehash them here. Instead, let me contend that giving the full thought process has its own merits, and that more examples of this would be a positive thing.

The most prominent example I know of where the full thought process is given is in Grothendieck's Pursuing Stacks. It is hundreds and hundreds of pages long with endless digressions. Nevertheless, it has been influential, and even personally, it has taught me quite a lot per page, though I don't work in this field. In fact, it is by far the most enjoyable mathematical text I have ever tried to read.

Why do I think this is? Briefly, because I think it models discussing math with others much better than textbooks/papers. Generally speaking, discussing math with others is important for learning because it allows for the freedom of exploring ideas with another mind. Textbooks/papers are important because they give you shortest path to various goals and prove a lot of important results, which you can study on your own. So I see two needs which papers giving the whole thought process may fill:

  1. For people who don't have people they regularly discuss a certain area of math with.
  2. For people looking to just enjoy math, not concerned about publishing papers in a certain field.

Finally, let me add that even from a "practical" standpoint, texts like these may have the potential (apart from the actual ideas they may contain) to expose students to the unconstrained nature of doing research, which at least for certain students may be important to building their own tastes.

Cyclicduck
  • 268
  • 1
  • 4