Ever have an Ex you can't get out of your mind (for whatever reason)?
that depends, as a prefix ex can refer to a variety of relationships. 
True enough, let's say an EX, well not lover, but someone who you causally dated
I think most people understand that. As it is often used as a derogatory tinge. 
What do you mean 'derogatory tinge'?
Well, its usually not positive. sometimes its assumed to be a former romantic partner or spouse. 
Well, yes, in this case, it was a negative relationship as things turned out.
And it you say exes, you are now talking plural, so that is a lot of them.