Rashi - Berachos 35b s.v. אזוקי מזיק
דגבי ברכה ואכלת כתיב
Rashi explains that the reason you do not make a blessing on drinking oil is that it is not considered an act of eating, and the paradigm of blessings on which Chazal based their creation is associated with eating.
This Rashi would explain the lack of extension to other pleasures, but does not explain why smells have blessings. I would conjecture that maybe Chazal created blessings in things like eating, where there is ingestion - smelling is the olfactory absorption of follicles of the object being smelled. This would exclude all other pleasures.
Tosefos - Pesachim 53b s.v. אין מברכין
הקשה הר"ר יוסף למה אין מברכין בכל שעה שנהנה ממנו, דאסור ליהנות מן העוה"ז בלא ברכה, ותירץ דה"מ כשגופו נהנה אבל שאר הנאה לא
Tos' explains that you do not make a blessing on enjoying the light of a flame because the prohibition to enjoy is only on pleasures that the body gets a tangible pleasure from. This would rule out playing or watching sports and the like, but would still not explain something like a hot shower.
The בית יוסף סימן ריז סעיף ג echoes this idea of Tosefos, but his parameters allow a more flexible application. In explaining the opinion of the Rambam that you do not make a blessing on a ריח שאין לו עיקר:
אינו צורך הגוף כל כך
"It is not such a need of the body" - only bodily needs that meet a certain critical point of enjoyment are included, such as strong smells and foods.
The גליוני הש"ס to Berachos 35a discusses that there should be a blessing on marital relations, and concludes that Birkas Ha'erusin accomplishes that. It would seem he held this concept can extend to other pleasures, although this could still be in the realm of "a need of the body."
The מגיד תעלומות in his commentary to the Rif in Berachos 35a, partially quoting his father in law, says that really there should have been blessings on a number of pleasures, but for technical reasons they were not made. He gives the example of marital relations, which he says are not meant to be for pleasure but rather for the mitzvah, and therefore even when it is for pleasure it would be inappropriate to establish a blessing for it as that would imply condoning the involvement for the sake of pleasure. However, he does conclude that it would be appropriate to make a Shehakol on something else before engaging in relations, and that would "cover" the other pleasures as well. This could be the source for the idea mentioned in a comment above that one should make a Shehakol on something before taking a shower. However, it is difficult to extend this to non-bodily pleasures, as above. On the other hand, there doesn't seem to be any reason why not to do it.