Aside from Harry who wouldn't have gone if Dumbledore hadn't told him to, no one is recorded as rejecting an invitation (even if they didn't want to go, like Hermione).
"Anyway," said Hermione, continuing their interrupted conversation as though a lump of wood had not just attacked them, "Slughorn’s going to have a Christmas party, Harry, and there’s no way you’ll be able to wriggle out of this one because he actually asked me to check your free evenings, so he could be sure to have it on a night you can come."
Half Blood Prince — Chapter 14: Felix Felicis
"Harry, that’s three of my little suppers you’ve missed now!" said Slughorn, poking him genially in the chest. "It won’t do, m’boy, I’m determined to have you! Miss Granger loves them, don’t you?"
"Yes," said Hermione helplessly, "they’re really –"
"So why don’t you come along, Harry?" demanded Slughorn.
"Well, I’ve had Quidditch practice, Professor," said Harry, who had indeed been scheduling practices every time Slughorn had sent him a little violet-ribbon-adorned invitation. This strategy meant that Ron was not left out and they usually had a laugh with Ginny imagining Hermione shut up with McLaggen and Zabini.
"Well, I certainly expect you to win your first match after all this hard work!" said Slughorn. "But a little recreation never hurt anybody. Now, how about Monday night, you can’t possibly want to practise in this weather …"
"I can’t, Professor, I’ve got – er – an appointment with Professor Dumbledore that evening."
"Unlucky again!" cried Slughorn dramatically. "Ah, well … you can’t evade me for ever, Harry!"
Half Blood Prince — Chapter 12: Silver and Opals
"Listen, it’s not my fault Slughorn invited Hermione and me to his stupid party, neither of us wanted to go, you know!" said Harry, firing up.
Half Blood Prince — Chapter 11: Hermione’s Helping Hand
(emphasis mine)