What distinguishes top researchers from mediocre ones?

Question

I am a PhD student studying theoretical physics at a US institution. I am now beginning my fourth year in the program, and struggling to solve my first research problem. I see PhD students around me keep writing papers while I am still struggling to write the first one.

My advisor suggested a research problem that I am now working on. He is not helping me with the technical details of the problem, haven't suggested papers to read etc., but he suggested that, since I have been struggling with such a simple problem for a long time, then this is a sign that research is not for me.

As far as I know, I am not stupid. I passed all the qualifying exams and got the highest grades in all grad physics classes. I am also fascinated by the subject, so I do not know why I cannot solve the problem. I spent a great deal of time reading and trying to understand the subject (which is quantum field theory). However, I can not solve problems.

My questions are:

1. how does one approach a research problem (say, in theoretical physics) and solve it in a reasonable amount of time while absorbing the prerequisite background knowledge along the way?

2. What distinguishes PhD students who are productive in their PhDs and how do top researchers approach a research problem and solve it? is it talent? breadth of knowledge? or time management skills?

3. Is my advisor correct that struggling with a basic project for a long time is a sign that I lack the research talent? I came to do a PhD in physics hoping that someday I will be a top researcher in my field, so I am trying to see how to improve and become more productive.

Answer 1

To be successful in research, you need some combination of the following:

1. Background/breadth: You need to understand the problem(s) that are relevant to your field. Why are they worth solving? What makes them difficult to solve? What has been done before? The more problems you are familiar with, the more likely you are to find one that you can address.

2. Insight: Most problems are solved by applying existing ideas/techniques/tools/data/etc in new ways. Can you read papers from another field, understand the gist of what they do, and identify how that might translate to your area of interest?

3. Logical reasoning: Can you systematically evaluate a problem and potential solutions? Are your arguments watertight and convincing?

4. Technical skills: Maybe you're good at taking rats apart. Maybe you're fluent in Ancient Sumerian. Or COBOL. Anything that lets you do something that others couldn't.

5. Tenacity: Sometimes, if you keep bashing your head against the wall after everyone else has gone home, you'll find the wall moves.

6. Communication skills: No matter how amazing your work is, if you can't convince other people that you've done something useful, was there any point?

7. Resources: Whether that's dead rats, compute time, library books or pencils.

8. Luck.

Answer 2

To move from a student to a successful researcher, requires insight. Insight into the nature of problems that is deep enough to suggest solutions that might be worth pursuing. Those solutions aren't obvious and the insight is built on a strong base that you develop as a student, but insight is of a different nature.

One way I developed insight (math analysis) was to solve, by hand, a very large number of similar problems. Eventually I could look at a formula and understand it without the computations. That was the start (only the start).

Insight gives you the power to think about things that "might" be true and the skill gives you the power to test them.

It took Einstein ten years of deep thought and discussion with a few other people to come up with Special Relativity. That development was one of insight building and constant return to the problem at hand.

You might talk to your advisor about this sort of thing. What is it that you are missing in your level of insight that is holding you back.

---

Note also that some problems are harder than they appear. Research is looking into the unknown and there are no guarantees about success or especially about timelines. Math, theoretical physics, and some parts of CS are like that. Probably the same in other fields as well.

If you talk to your advisor about what approaches you have taken to solving your problem, you can get feedback on whether those are naive or not. If they are, then you probably lack insight in some way. Otherwise, you are dealing with a problem that may be much harder than it seems. Hopefully you have an advisor sophisticated enough to help with this.

Answer 3

I'm a fellow Ph.D. in theoretical physics that used to struggle with research, and occasionally still does, after two post-docs and one teaching position.

To succeed, I think you need to give yourself time and significantly change your mindset. As others have mentioned in the comments, part of the process is to become able to solve problems independently. And I mean textbook problems first. The only way to do so is to solve them on your own without looking at other people's solutions. Every little detail counts, every small derivation, every little disappointment. Often you need to sleep on problems and approach them again with a fresh mind, and, quite often, you need to ditch your approach for a new one worse comes to worst. But you need to fail a lot of times first.

What helped me realize these things was a teaching appointment that led me back to research after it was over. It helped me see myself through my students. Undergrad education is concerned with solvable, well-behaved problems for which even a closed-form solution often exists. Research is completely different. You often resort to unorthodox methods, new approaches, etc. After all, it's supposed to be new knowledge. And it won't come in a linear fashion (most of the time).

Another problem I faced was my ego. I couldn't accept that I was unable to solve even simple problems sometimes. Accepting it helped. Heck, I even accepted I'm not as smart as I thought.

Regarding your advisor, I must respectfully disagree with him. If you went all the way to having a physics degree, you can go further. But there needs to be that "phase transition" I'm describing above.

Much like the ABCs, physics research has its ABCs. You need to know the fundamentals of Classical Mechanics, EM, Thermo, Stat Mech, Quantum Mechanics, etc. Then you need to learn Math Methods, numerical methods, coding, etc. It's a hell lot of things so that needs time and guidance. Sometimes, it might be better to stop, take a step back, and assess what went wrong and try again. Try, fail, try again, fail again, etc. Sure, some lucky ones are extremely intelligent and get it on the first try. Most people don't though.

One last thing. Just jump into a problem and enjoy the process. Enjoy what you learned. Develop consistency. Spend some time on it every day. Get a small result. Discuss it with your advisor or some collaborator. Keep moving. Find friends and collaborators. Go to conferences, and keep discussing with people. Socializing that way helps. You need to find some passion for it. Be bold!

Answer 4

"He is not helping me with the technical details of the problem, haven't suggested papers to read."

Unable to evaluate you as a researcher, I can though safely say that your advisor has failed his job. Perhaps a prominent scientist, he is a plainly bad teacher.

My son wasted several years under the supervision of such an advisor, a man equally famous for his scientific accomplishments and his total disregard for his graduate students. While staying in his group, my son devised a clever strategy of creating collaborations with people, from both his university and other schools, who had experience in the area and were interested in the problems suggested to my son by his then advisor. In collaboration with such people, my son published several good works; and I would say that at that time those colleagues were effectively acting as his true advisors. Eventually, my son found a decent advisor at a different university, quickly wrote his thesis there, and got a postdoctoral fellowship at a stellar place. I would emphasise that the new advisor was not a world-known scholar -- but was a dedicated teacher caring for his grads.

In short, this is what I would do:

(a) Through publications or word of mouth, try to find people working on this class of problems. Get in touch with them. See if any of them could be interested in collaboration. (Don't be shy!)

(b) Meanwhile, look for a better advisor, at your school or, perhaps, elsewhere.

(c) Last, and by no means least, do NOT believe your advisor's words that "research is not for you". Do NOT let him discourage you. Quite possibly, you are not genius -- but one does not to be genius to write a PhD thesis. And in the situation you have described the principal problem is not you but your worthless advisor (who may at the same time be a brilliant scholar, why not?).

Answer 5

I am really sorry about your situation. Based on what you have indicated, in my opinion, your supervisor might not be doing his(her) work very well, which is to support you and help you to succeed. Autonomy and independence in research come with time and training, and PhD supervisors play a key role for this to happen.

Would you mind elaborating a bit more on the relationship with your supervisor? If you are not really getting support to your growth from it, it may be better to quit and start somewhere else. I am curious that you had not got input from your supervisor even for simple things as suggesting a paper to read.

On the other hand, be cautious when it comes to comparing yourself to others, as every person undergoes a different learning process. Furthermore, not all research problems are the same in terms of difficulty and the work required to solve them.

Soft skills, such as time management, are also important to succeed. Time management is more than just filling up a calendar with things and there is a "science" behind that. You may benefit from training in this respect. You find valuable tips about this on the internet and there are some courses available on platforms like Linkedin Learning as well.

You may also want to read this article: Nature article on failed PhD

Best wishes, AME

Answer 6

Is my advisor correct that struggling with a basic project for a long time is a sign that I lack the research talent? I came to do a PhD in physics hoping that someday I will be a top researcher in my field, so I am trying to see how to improve and become more productive.

I think it is important not to see this as an evaluation of your personal worth. As you have pointed out, by becoming a PhD student you have proven that you possess the necessary skills and qualifications. Your lack of success can be due to a multitude of factors, some of them may be the fault of your supervisor (e.g., he might be misjudging the simplicity of the problem or not providing enough guidance) or due to bad luck (e.g., you have chosen a field or a problem that are not suitable to your abilities, or your best skills are not really for this type of the problem.)

It might be also that you are less suitable for academic research - which does not mean that you are a bad researcher. After all academic research is a pageant, where the winner is not always the most intelligent one - success in academic research is also about choosing fashionable problems, publishing your results quickly, popularizing your achievements, making connections, etc. There are plenty of examples of strong students who were given difficult problems, struggled and eventually abandoned academia, whereas more mediocre ones did good career, because they had a caring and/or influential supervisor or accidentally struck the problem that attracted much interests.

If the situation persists by the end of your PhD, you will have to consider changing career, rather than following a standard path PhD-postdoc-professor. On the other hand, if this is just your first year and your professor makes such remarks, it is more reasonable to consider changing the supervisor.

Answer 7

Given that it is Y4, then you must be mindful of the danger of PhD failure above all.

I feel that you need to refer to the Head of Department and get some positive input from beyond your superisor.

It's likely that you took on an easy-looking problem that had a share of hidden complexities. A brontosaurus of complexity situation, as they say.

I am surprised that this matter hasn't come to a head earlier than year 4 of the program.

Supervision is certainly in question here.

But so is your own lack of alacrity in either seeking another problem or getting more help with your approach to it.

So go see your HoD and seek informal views from other staff (if possible).

I've ignored your question of what distinguishes top researchers from mediocre ones. The answer I believe is humility and perseverance. Luck will help a small few researchers make a splash. But without the latter two virtues such fortunate success will be lost in either hubris or lack of will to make further advances. But right now you need to focus on being a competent researcher not a top one.

Answer 8

I am also fascinated by the subject, so I do not know why I cannot solve the problem.

Not every problem is for everybody. Specialist fields such as your are as much art as they are science. Very few artists, in any field, are universalists. They like certain approaches and takes on the subject, and dislike or even disdain others. Perhaps that problem is just not your problem. And that's not to say that there is anything "wrong" with you. I'm an electrical engineer. I am supposed to solve a wide variety of problems, but there are only some that I really enjoy working on and got good results in. Even though my training was broader than that.

Look back at what you enjoyed doing in classes. There probably is some pattern there. Look at classes that were most specific to the sub-field you're facing in the research problem, and see if you can figure a way to build a bridge from what you know to what you don't know. Try to digest your knowledge gap into manageable chunks. Read what others wrote about the topic.

I spent a great deal of time reading and trying to understand the subject (which is quantum field theory). However, I can not solve problems.

I'd go on a limb and suggest: try explaining the field from basic principles to a non-specialist with some grounding in undergrad physics/engineering/mathematics. That's one way to evaluate your understanding. Explaining things so that a non-specialist could solve a very simple textbook problem requires exquisite understanding first.

Whatever you find, no one should be suggesting that you're "stupid" - don't think that way. You clearly are not - just asking the question here is a sign that you see what's going on and try to do something to improve the situation.

Answer 9

My advisor suggested a research problem that I am now working on. He is not helping me with the technical details of the problem, haven't suggested papers to read etc., but he suggested that, since I have been struggling with such a simple problem for a long time, then this is a sign that research is not for me.

This is absolutely an issue with your advisor not providing sufficient supervision, mentorship, and training. A friend of mine was told by an advisor that they'd never met anyone who got accepted into graduate school who wasn't capable of finishing grad school. Grad school is about training to become an independent researcher. Being good at that isn't some innate quality. It's a collection of skills that you learn, develop, and refine over the years through guided practice.

For a few years, I struggled to make much progress with my proposed dissertation project (analytical work in theoretical astrophysics). I was stubborn and kept telling my advisor and committee that I could do it, but eventually had a proper conversation with my advisor. I needed to finish my dissertation and the only way for that to happen was putting the original project on the shelf and shifting to projects I could complete in a more reasonable timeframe. I met with my committee members and other colleagues I'd met over the years to chat about other research projects. No one ever told me my lack of progress meant I wasn't cut out to do research. Your supervisor shouldn't be saying that to a student.

Does your programme require students to have a committee? If so, how often do you meet with your committee? Do you know their thoughts on this situation?

I'll emphasise again that I don't think you're the issue . I think it's an issue with your advisor. Are there other faculty or postdocs in your department or whom you've worked/interacted with that you could reach out to? A new (related) project and some extra guidance could make a world of difference.

To answer your questions:

1. Get help. Trying to complete a project in theoretical physics as a graduate student without having regular help, guidance, and feedback is extraordinarily challenging. Are you struggling with developing the requisite skills? Go through relevant sections/chapters in a grad-level textbook and foundational papers. Chat with other researchers in your group and department (students, postdocs, faculty, etc.) when you're stuck. Unsuccessfully trying to figure it out on your own for several days (or even worse, weeks) isn't an effective use of time.

2. Good supervision, mentorship, and guidance.

3. Nope.

Answer 10

Is my advisor correct that struggling with a basic project for a long time is a sign that I lack the research talent?

No. What might be lacking is the skills of project management, people management and expectations management. Notice how I repeat three times the word management. Tons of persons in research world are not “popular” by the bizarr rating metrics of todays research community. Stay bold and clear the issues in your path without looking what others do.