Clarke was always fairly clear that the four novels did not share a strict continuity. Writing in the afterword to 3001 he says:
Obviously there is no way in which a series of four science-fiction
novels, written over a period of more than thirty years of the most
breathtaking developments in technology (especially in space
exploration) and politics could be mutually consistent. As I wrote in
the introduction to 2061, “Just as 2010: Odyssey Two was not a
direct sequel to 2001: A Space Odyssey, so this book is not a linear
sequel to 2010. They must all be considered as variations on the same
theme, involving many of the same characters and situations, but not
necessarily happening in the same universe.”
Hence in the version of the Odyssey universe described in 2010, Lucifer continued to burn until at least 20001, whereas in 2061 it only lasted until 3001.
One of the themes of the series is mankind being watched and judged by the civilisation and machines that preceded us; Lucifer starting to dim and triggering TMA01 for the second time appears to be a narrative device showing that, as far as humanity had progressed in that millennium, there were still greater forces in the universe.