'Invisible' Poulter torn apart by top Tory over 'betrayal' after Labour switch