What are 'Gentlemen's agreements'? How binding are they? What happens if they are broken? Why do the hold sometimes, but not others?

Image: MachoCarionca via Wikipedia