It's ok to shift priorities, it means you're progressing in your life.  Don't force it though, if you feel yourself being tugged, examine your priorities (and listen to this podcast).