The boundaries that you have, whether you realize you have them or not, are there for a reason. Someone choosing to ignore your boundaries should affect you and it's okay to get upset over it.