You've rather painted yourself into a hypothetical corner here. There is an important distinction between "I just don't happen to like X" and "X is a real problem for me because of my medical issues" - but you don't want to tell your boss about your medical issues because you think they're irrelevant to your work.
I think the solution is to reveal the part of your medical issue that is relevant, and nothing further. For example:
Thankyou for that generous offer, but I have some auditory processing issues that make it impossible for me to be in a large noisy crowd.
If your boss apologizes or appears embarrassed:
You couldn't have known; this quiet working environment is great for me but I can't go to a stadium at all.
If you boss pooh-poohs your demurral and tells you it will be fine:
I regret that I really must decline. I would love to be part of the team activity but a live sporting event is simply not possible for me because of my auditory issues. [Look regretful, like you wish you could go.]
If you feel that your auditory processing issues might (or might not) be relevant for some work-related events that might occur in the future, such as attending a conference, flying somewhere, or even attending a very large meeting, then you can mention these after you have successfully communicated I can't go to the sporting event:
I can handle a conference keynote if it's less than 90 minutes and I get a break right afterwards, but I know a [football game] will be a lot louder and longer than a conference keynote.
or
I'm actually unable to even attend a conference keynote, though breakout sessions are fine for me. When I attended [conference] I watched the live stream of the keynote from elsewhere in the convention centre [or used the overflow room which was quieter, or whatever] and that worked fine.
or
I know you haven't offered to send me to any conferences yet, but I'll just mention for completeness that my auditory issues also prevent me from attending sessions with more than 100 people or so in them. That rules out X and Y, but not A and B as conferences I'd love to attend some time.
Don't invent an issue for this purpose; I just chose "auditory processing issue" based on some cues in the question. Be clear and precise (but succinct) when stating your reason for being unable to attend the sporting event, and optionally connecting it to similar working events like conferences. Three sentences should be enough.
Stay firm and polite and there should be no consequence. Your boss may suggest you attend part of the activity, such as a get together before and after. If you can, you should, even if it will be difficult for you. You got out of the part that would be impossible, so make an effort and do the difficult. People will appreciate it.