Well, most of the time that I'm actually talking to one of those people, I'm probably not in a situation that I can easily get out of without "offending" them and making things worse for myself. I've learned that the more I react, whether it's leaving, or saying something insulting, generally just makes things worse.
These people are the "cool" people who like starting conversations with me more or less to subtly make fun of me and such, in a way that I can't respond to because it'll just "offend" them. It's kinda hard to describe exactly what I'm talking about. This also might be a somewhat slanted view because after all the bullying I've been through, I tend to automatically hate the "cool" people unless they actually do something to show that they're trying to be friends. I can usually tell just by the way someone talks, whether they really care about talking to me, and whether they're baiting me or not.
Of course, that could be somewhat slanted by me as well. I've developed some social anxiety thanks to all that bullying, and unless I know who I'm talking to, or at least have some idea of their personality and social standing and all that, I tend to be very anxious, and don't know what to say and how to act. It's really problematic, but I've got 8 straight years of bullying to thank for that.