To put it in factual terms it's not called karma, saints and pandits have been passing along the wrong word or I say some lack of correct words, to put in terms of Veda's and it is actually called prarabdha karma (प्रारब्ध).
Well, there are many definitions and meanings to it along with prarabdha-gati, and prarabhdha-yoga. It is the collection of all the good and bad deeds done with the current incarnated body and spirit goes with you. It's not just good karma. And it is to be believed that you have to pay off the balance with sorrow, pain and happiness, joy etcetera. in the next incarnated body.
I know there is some confusion among the people that only good karma goes but what I know of that is the Prarabdha does its work in the afterlife including both good and bad deeds.