Barnaby Joyce 'too close' to PM: minister