The ex-health secretary says politicians should "go where the people are" after facing a backlash.