King Charles and Queen Camilla 'had enough' of the Sussexes