I re-read GOF and noticed that on p 570, something Voldemort said did give a clue about the protection Harry receives from living at the Dursley's, officially revealed in OOTP.
"Dumbledore invoked an ancient magic, to ensure the boy's protection as long as he is in his relations care. Not even I can touch him there..."
It's interesting how there are clues like that, I'd forgotten all about that until I read it again!