The former Attorney General will replace Karen Andrews, who is taking on the Home Affairs portfolio.