Ever have an Ex you can't get out of your mind (for whatever reason)?
that depends, as a prefix ex can refer to a variety of relationships. 
True enough, let's say an EX, well not lover, but someone who you causally dated
I think most people understand that. As it is often used as a derogatory tinge.