Duffy and Tommy came to the realization that maybe they're in the minority when it comes to how they feel about the holidays