The one 'unanswered question' that's stopping Harry spending time with Charles