Since we don't yet know if any of the events/plots
in System Down's book spoiler exist in the TV show world, here's a show-only answer.
Short answer: actively joining them is incredibly risky. Not just for the individual(s) making an epic journey to negotiate with an unknown, unpredictable queen - it also risks marking your entire family as traitors.
At no point in the show has it looked like there is a credible, active Targaryen counter rebellion with a strong chance of taking the throne worth overtly risking everything like that for - the closest are, a terrifyingly unpredictable dothraki horde, and an impressive army perplexingly marching the wrong way.
So, the loyalists quietly keep their heads down for now - the only exceptions are exiles and fugitives with nothing to lose.
Long answer: There were Targaryen loyalists. King Robert was worried that they might join a credible counter-rebellion:
If the Targaryen boy crosses with a Dothraki horde at his back, the scum will join him!
Robert was well aware that many of those who surrendered to him were reluctant and were potential traitors, and he took this risk seriously. Any pro-Targaryen action would be a huge risk - you'd mark yourself a traitor and a danger to the most powerful houses, and we see what happens to families who mark themselves to be enemies of the powerful...
To take that sort of risk, you'd need some kind of inspiring leader who fills you with hope that they have a real chance of winning and that even if they fail, it's such a noble glorious cause that it's worth risking everything for.
For the decade and a half or so between the start of the show and the rebellion, that "inspiring leader", traveling the free cities trying to rally support, was this guy:
Feel inspired yet? No, nor did they. Viserys ends up desperate enough to resort to trying to piggyback on the back of the Dothraki as the khalessi's brother.
When he dies, the Targaryen loyalists have another problem. The person they would now be risking everything to support might now be less uninspiring, but much more terrifying and unpredictable. He's probably not what they had in mind when they said they wanted a new king:
Robert might have been right that the loyalists might join a Dothraki/Targaryen army if it landed on their side of the sea. But before then? Far too risky and unpredictable. The only man who does is Jorah, an exile with nothing to lose (and a spy hoping to earn a ticket home).
Then Drogo dies, the khalasar scatters, and it looks like it's all over.
By the time anyone in Westeros knows Dany has, against all the odds, become a credible power in her own right, she's the wrong side of the world, landed her ships, baffling any would-be supporters by going the wrong way. Does she even want to rule Westeros? They have no reason to be sure that she does.
She's too far to easily get in contact with - you'd have to send a valuable trusted envoy on a dangerous journey, risking losing them or giving away your intentions. There's a storyline(s) in the books that appears to have been dropped from the show that makes explicit just how difficult, dangerous and risky this is for both the envoys (risking their lives on seas and roads where slavers are not the only danger), and the family who sent them (risking being discovered and knowing there's a risk this unknown, unpredictable queen might reject them).
Yet, people do spontaneously join her, uninvited. Who are they, and what do they have in common?
- Jorah was an exile with nothing to lose
- Barristan is effectively an exile with nothing to lose, having fallen out of favour with, then overtly opposed, the most powerful family
- Tyrion is a fugitive with nothing to lose, with a bounty on his head
All those other loyalists and their families have too much to lose to stick their necks out.
But are they secretly plotting, or preparing themselves for a credible counter-rebellion? We'll have to wait and see...
