Cabinet minister to face no action over leaflet asking if people want traveller site next to them.