Real reason Harry didn't stay in Charles's home on flying