Harry 'left out in the cold' as fulfilling royal duties 'impossible' from the US