A PhD is usually narrowed down to a very peculiar research topic and as such missing some education in other general areas might not be particularly dangerous for the sake of getting the PhD itself. Moreover, what matters the most when undergoing a research path is the attitude to learn and to question the scientific method more than the titles you have previously obtained, which, however, do matter when applying for admission to a PhD programme. Committees might prefer someone with already some background just because the selection processes usually see many applicants and Universities need a way to filter them out. A good recommendation letter goes however far beyond any selection criteria and if you can get one it should not be too difficult to enter a PhD programme. In particular, especially in Europe, GRE, GPA and all these quiz results are irrelevant for anything.
Concerning the topic itself, as you have formal background in mathematics,
do consider that many PhD in theoretical physics are essentially pure mathematics, therefore you might even find it along your lines and not too difficult to enjoy.
This said, if you want to have a good research profile and a thorough understanding of what you are doing (rather than just getting the PhD), a good knowledge of physics is fundamental. I would suggest to first achieve a master degree and only then go for a PhD (which should always be the rule, even if you manage to get admitted before).
