I can't see that the rabbi's sermon holds any special halachic status that should require someone to stay in shul to hear it, just on its own. The requirement to be there may come from its alternatives.
If you're out in the hallway, you're probably shmoozing and mevatel Torah (i.e. "wasting time") which is a problem, when the rav is talking Divrei Torah (assuming that's the focus of his sermon. The sermon is not always Divrei Torah, you know.) However, if you leave the shul to learn Torah by yourself or with a chavruta, then, you're on an equal plane, most likely - possibly better!
I have seen, unfortunately, too many people use the rabbi's sermon time to join the schnapps "club". This has been such a huge problem in my neighborhood, BTW, because men and sometimes teenagers were getting drunk. So much so, that the OU and Young Israel intervened and made their affiliated shuls completely ban schnapps from the shul. (Some don't even have wine in the shul for kiddush.) This alternative causes Chillul Hashem, among other problems. So, it's obvious, in this case, that you should be listening to the sermon.
Most rabbis probably don't mind if you're looking at the Chumash or learning during the sermon. But, to be safe, you may want to ask him, unless you know him well enough that he wouldn't mind or is "oblivious" to who is listening. (I think many rabbis have no clue how many people are and how many aren't listening.) If you really want to learn, than, of course, your best option is to leave before the sermon. Just make sure you return in time for Musaph, of course :-)
Leaving in the middle of a sermon is rude, unless you can't help it. If your kid is next to you and starts to shriek or get fidgety, seriously, you need to take him / her out. (I can't understand why some parents insist on keeping a screaming kid in shul during the sermon because they think it's rude to leave.) Leaving a child, there competes with the rabbi and the other congregants who are trying to hear him. That's probably some sort of halachic problem (a form of stealing, perhaps?) if you cause a disturbance or, in this case, do nothing to prevent one.
I should add that there is an "indirect" halachic requirement to listen to the sermon in these areas:
Kavod Harav (A derivation from "Mipnei Seivah Takum", perhaps. One should show respect to someone who has wisdom.
Chinuch - educating your kids. It is extremely important that young kids see their father listening to words of Torah from the rav. It teaches them that when they are in yeshiva and their rebbe, or rosh yeshiva speaks that they have to be respectful and listen. Sadly, these days, there is far less respect for rebbes and teachers than there was 30 - 40 years ago. I am quite certain that having fathers leave the shul during the rav's drasha has been a contributory factor.
As mentioned above, the "kiddush / schnapp's club" has been the biggest damage to chinuch. I don't understand why so many people - esp. fathers who bring young kids to shul - do this.