I think you are asking several different but related questions. It also sounds like there is a specific situation you want to ask about... But in general, I would say academia is like most other professions - I don't think it's any better or worse than other career paths in the senses you mention.
Like with all professions, interpersonal dynamics are crucial, though arguably it matters more for some fields and at some places more than others. For example, you really need reasonable interpersonal skills if you want to run a lab or research group properly - but you would need it even if you were more of traditional solo humanities scholar who always publishes alone, since you need to work with your colleagues to agree on teaching and governance as well as with non-academic staff for administration of the department, managing grants, organising conferences, etc. Like with every profession, some people are better at this than others: so many young scholars leave academia because of awful senior people who are just bad managers, bullies, or only interested in growing their own power, but there have recently been high-profile cases (in the UK) where the bullies have faced consequences so there may be hope yet. The general rule of 'be a good person' should be followed.
You can't escape bureaucracy - but that's true of all professions really.
It seems like a lot of energy and time is expended on balancing ones work with having to navigate other variables, such as making the right connections, finding supervisors, and not stepping on the wrong toes.
I would consider finding the right connections and supervisors/collaborators a part of work, not something to be balanced against it. Like in most professions, networking is important. Stepping on the wrong toes is bad, but you can't control other people's reactions, so if you follow the general rule of 'be a good person', you should be fine.
From the perspective of academics who have worked the system, is this seen as 'paying your dues'? Or is it be considered to be a deterrent to certain students who might be gifted/ creative, with the potential for significant contributions in research?
None of this should be 'paying your dues' - you will have to do admin, work with people, and manage relationships throughout your career (as in most careers). It can be annoying, and I personally find networking very hard, but it shouldn't be a deterrent as you'd have to deal with this whatever you do. Amazing research - even of the most world-changing kind - will rarely get you out of these duties/responsibilities.
Is there room for visionary/ idealistic type students to be able to learn the necessary academic rigors and discipline without hampering their unique ways of seeing? Can such students thrive if they end up pandering to egos or limited vision (if that is the case)?
Noone wants to make you a worse scholar - the only way your vision gets limited is if you let it by, as you say, pandering to others who don't have your best interests at heart. But why would you pander? At your stage, the most important thing is to find a good supervisor and a good department: ask the current graduate students what the atmosphere is like in their department (they are often very frank). Like when you search for a job in any profession, you need to do your research about the company/department you join. And if you hear bad things, avoid that department. Sometimes, bad things happen, but overly egotistical colleagues are not the norm and interpersonal disasters are not the default situation in academia.
I get the sense you are talking about internal politics. This, I avoid like the plague, because it has the potential to take over your life. But sometimes internal battles have to be fought, if it's over the direction of your field or over moral issues like creating a safe and inclusive learning environment. The key is not to let disagreement over a single issue spill over into other areas of interactions with your colleagues. Some colleagues are better at this than others, but again this is not unique to academia and this should be discoverable from asking around. (Also, drama tends to find people who like drama.) Finally, you will usually escape most of this as a graduate student, especially a masters student - they are rarely involved in internal politics. As a PhD student, you will be exposed to it a bit more, but you will usually be on the sidelines of internal politics unless you make yourself (or are made) a pawn - but most departments are sane enough to leave students out of any mess. Choose a sane supervisor and sane department by asking around before applying to ensure your chances of encountering trouble like this is minimised.
Academia can be a lovely place. In a well-functioning department, your life could be amazing. In a dysfunctional department, your life could be hell. I have been in both. In my corner of academia, to judge from discussions with colleagues, the two are just as common as each other. I wouldn't be put off by this (I would put more weight on job prospects in general and levels of pay), but just go in without rose-tinted glasses (which you clearly don't have).
