This is an excellent topic and question. You have partially given us an answer by acknowledging, in your first sentence, that you do not have a source for this being a problem.
Within Orthodox communities, and communities in general, there is pressure to adhere to a status norm. This can be healthy (or unhealthy) depending on the delivery, tone, circumstances, and the individual.
It is a big problem for a Jewish man not to wear tzitzit in the same sense that it is a big problem for a Jewish man not to keep the Torah and all of its written and oral laws.
The reason for this is a huge series of rewards and punishments - but to go into those details somewhat circumvents the main point. Which is this: G-d gave us the Torah as an instruction manual on how to live life. Thank G-d that we have this guide and the opportunity to live an extremely moral and meaningful existence based on the Torah's teachings.
If someone were to give you an incredible gift, and you were to not use that gift, or to let it sit in the closet, that person would be somewhat sad that you did not take advantage of what they have given you. This is an interesting analogy, but consider how much more significant the Torah is than any human gift. It is Divine. It is literally our source of life. When we decide between following the Torah and not, it it literally deciding between life and death, as stated in Deuteronomy 30:19
I have put before you life and death, blessing and curse. Choose life - and you and your offspring would live - by loving the Lord your God, heeding His commands, and holding fast to Him.
This is not to pressure you into following every set law. Everyone is constantly growing on their own path, at their own pace, over a very long period of time. We spend our entire lives studying the Torah, and more so.
We can function by understanding that G-d loves us and has put us into a specific point in time with unique skills. We are not here to perfect the world, but neither are we here to neglect it. May your journey be joyful and filled with blessings.