Harry and Hagrid leave the Dursley family on the shack on the rock and take the only boat.
This is a good days drive from their home.
Harry and Hagrid have a day out in London and Hagrid drops Harry at Paddington* with the assumption that he is going back home.
How were the Durslies meant to get back in time? Has this ever been discussed?
* Most trains to Surrey leave from Victoria or Waterloo, JKR often seems to get London stations mixed up
