Trying to locate a sci-fi story I read in the '90s, and the story is probably written in '50s-'70s. Paraphrasing, might have fudged details, but the essence follows.
The narrator A is a cop from the future, arriving in our time to check on B, a (political?) enemy-of-the-state / criminal, also from the future, who has been exiled to our time as punishment. In the years that B has been here, he has settled down, made a life, has a wife, kids, etc, and despite being an anachronism in his mind, has found happiness. A and B talk about things for a bit.
The story ends with A revealing to B, to B's dismay and horror, that A is here to impose the real punishment now, a second exile for B, uprooting him from the life he has painstakingly built for himself.
The story ends with A saying something to the reader like "I sent him back to a small village in Eurasia (?) in the __th century, 3 years before Genghis Khan (/ Attila the Hun?) ransacked the region."
[Update 01:] Mark Olson has identified it as "My Object All Sublime" by Poul Anderson. He found the story in this Google Books link. (I was clearly muddled on story specifics.)