This is described in the C.S.Lewis novel "Prince Caspian". In short, the relatives of the Telmar were Pacific pirates who fell through a portal housed in a cave:
You, Sir Caspian,” said Aslan, “might have known that you could be no
true King of Narnia unless, like the Kings of old, you were a son of
Adam and came from the world of Adam’s sons. And so you are. Many
years ago in that world, in a deep sea of that world which is called
the South Sea, a shipload of pirates was driven by storm on an island.
And there they did as pirates would: killed the natives and took the
native women for wives, and made palm wine, and drank and were drunk,
and lay in the shade of the palm trees, and woke up and quarreled, and
sometimes killed one another. And in one of these frays six were put
to flight by the rest and fled with their women into the center of the
island and up a mountain, and went, as they thought, into a cave to
hide. But it was one of the magical places of that world, one of the
chinks or chasms between that world and this. There were many chinks
or chasms between worlds in old times, but they have grown rarer. This
was one of the last: I do not say the last. And so they fell, or rose,
or blundered, or dropped right through, and found themselves in this
world, in the Land of Telmar which was then unpeopled. But why it was
unpeopled is a long story: I will not tell it now. And in Telmar their
descendants lived and became a fierce and proud people; and after many
generations there was a famine in Telmar and they invaded Narnia,
which was then in some disorder (but that also would be a long story),
and conquered it and ruled it. Do you mark all this well, King
Caspian?”.
There's no special reason to assume the Telmarines in the film came from somewhere different.