The place I start with this question is by thinking about what kind of event the end of the Fourth Age would have to be. The geography of Middle Earth at the time of Lord of the Rings is clearly very different to the world now. The timeframe doesn't allow anywhere near enough time for this to have been a gradual change (either by continental drift, or something more "fantasy"ish) so at some point between the start of the Fourth Age and the present day there must have been some kind of sudden and dramatic event, in which entire continents got reshaped. That sounds distinctly like the sort of thing that happens at the ends of ages. In fact, even if nothing else happened other than the whole world being reshaped, I'd say that alone would be enough to qualify for the start of a new age. Obviously this event must have occurred before recorded history. Since Tolkien thought we're now at the end of the Sixth Age, and doubtless considered Christ's death and resurrection to be the start of a new age, logically this world-shaping disaster must be the beginning of the Fifth Age (with Christ's coming the beginning of the Sixth).
I therefore believe that whenever the Fourth Age ended, it must have been in some terrible cataclysm which reshaped the whole world, and which happened before, or at the beginning of, recorded history.
This could, of course, simply be too long ago to find any records, but let's suppose it isn't, and it's an event that happened in our own stories. Are there any suitable events? The one which seems to fit the best in my opinion is the Flood. It sounds like a Middle-Earthish sort of event; similar to the downfall of Numenor but on a much vaster scale. The idea that Eru/God might change the world's geography while it was flooded to make it less blatantly supernatural seems fairly plausible, from the perspectives of both Tolkien's myths (it's more or less happened before at the end of the Second Age) and the Biblical account (where God is talking about remaking the world from scratch). It's also a story which appears (in various forms) in a lot of different cultures, which is what we'd expect if the whole world (or at least, what would become the Eurasian/African supercontinent) was flooded and there were only a few survivors. (I see no reason to assume that no one but Noah was preserved: if we're accepting Tolkien's myths then it's clear the earliest bits of Genesis can't all be literal historical descriptions. And Noah believing his family might be the only survivors seems quite sufficient to explain the Biblical account if you're willing to drop that assumption.)
I like this theory, partly because it has an interesting consequence. The most important family line to survive, the ones who are most "good" and given a special task by God, are apparently notably long-lived. And some of them have Numenorian sounding names - "Arphaxad" is the most obvious example. The language they would go on to develop has some apparent connections with the Elvish languages - "adam" being equivalent to "atan"/"edain", and "el" being used in connection with heaven and divinity. Could Noah be a distant heir of Aragorn and Arwen, in the same way that Aragorn himself was the heir of Elros, and hence of Beren and Luthien, and Tuor and Idril?
Of course, that would also make Abraham, the Jews, and Christ himself, a (very) distant part of that whole family tree. It would also (but I'm very much into speculation here) link back to that prophecy about the Dagor Dagorath, and how Turin would be the one to strike the killing blow to Morgoth. There's at least one Biblical prophecy which refers to David, but is clearly actually talking about the Messiah, the descendent and heir of David. Turin had no children, so Tuor's family line would presumably be his heirs. One could perhaps argue (and if the prophecy were real, people certainly would) that the Dagorath prophecy is talking about an heir of Turin, who like Turin would suffer unjustly due to Morgoth's particular attention being focussed on him.