A few verses that I know regarding sex with slaves are:
- Al Qur'an, Al Ahzab, 50
- Al Qur'an, Al Mu'minun, 5-7
Those verses clearly states that Allah permits Muslim to have sex with their slaves, especially in terms of "guarding their private parts". Having sex with slaves in the above verses are literals, not metaphors. However, this applies to a world where slavery is the norm, i.e. the past. Even then, having sex to slaves might not be the norm in the Muslim world because Islam encourages Muslim to treat slaves well or even release slaves as ways to repent or achieve high reward from Allah. Thus, allowing Muslim to have sex with slaves should not be considered as allowing them to have sex with as many person as possible. It's the interpretation that is incorrect, not the verses, right?
Fast forward to today. One who reads those verses should ask themselves, "what constitutes a slave?" A few examples of past slaves are captives of war, bought as slaves, or born from slaves. Do we have them today? No, we don't. Current laws prohibit slavery. Even captives of war can't be taken as slaves. Housekeepers or maids must not be considered as slaves. In a way, even if Allah allows us to have sex with slaves, we can't really do that because there are no more slaves in this world.
Then why do the verses exist? They are to be used whenever conditions apply. The rulings of Islam is eternal so when, Allah forbids, humans reverts back to it's old way of life which includes slavery, the above rulings apply. I'll refrain talking about this possibility since it is out of context.