People are accepted into a physics program because they prove that they can do well in physics graduate school through their past performance in courses, their personal statement, their letters of recommendation, test scores, undergraduate research experience, etc. If you have a strong mathematics background, you might be able to convince an admissions committee to accept you if you demonstrate an understanding of undergraduate physics outside of course work (like performing well on the physics GRE). If you have not, at least self studied undergraduate physics material (at the very least at a Sophomore/Junior level), then it is not likely that you would accepted into a physics graduate program (even a masters program). Now if you were exceptional at math (e.g. have published research in a peer reviewed math journal), perhaps its possible that a physics program would accept you, with a very good personal statement explaining why you want to study physics at the graduate level.
But why get a degree in physics? At least part of the answer should be that you enjoy doing physics and are (at least somewhat) good at doing physics. If you aren't 100% certain that you love physics, you shouldn't go to graduate school in physics. Instead, self study physics by reading a book, doing problems and watching online lectures. Once you can determine whether you enjoy doing physics, only then should you go to graduate school in the subject.
In addition you haven't explained why "knowing about physics could improve a philosopher's thinking about a number of questions". At least at the vaguest level, this could be said about nearly any discipline; it doesn't mean you should get a degree in those fields. You can take physics courses while studying philosophy at many universities. If you think you will derive deep satisfaction from this, you should seek out philosophy programs that allow flexibility in your course work so you can pursue an interest in physics.
