Something that holds you back in your progress is sometimes called an albatross.