What makes the journal "prestigious" in the first place? I can't myself find any sensible answer.
One candidate reason is that some journals have better reviewers. But reviewers are usually not paid, so if it were true that the good reviewers flock to these select few journals, they must be doing so because those journals were prestigious or attractive in some sense in the first place, and so we are back at our original question.
Another possible reason is that some journals only publish papers of a high quality. This seems wrong as well. For example, if I decided to create a journal which only published extreme-super-duper-mega-high quality papers, nobody would care about it. Rather it seems to be the other way around: these journals publish the best papers because they are prestigious and therefore receive the attention of the best academics and their best papers. So again, we are back at the original question.
A third reason could be readership, but, again, the readership is a consequence of the prestige the journal already possesses. We still don't know where it came from.