Disclaimer*: I do know with bosses etc it is different, I'm talking about your spouse, parents, siblings, friends etc.

I recently discovered this, while being emotionally abused by my spouse.

Just because someone demands to talk to me, doesn't mean I HAVE to talk to them back.

I owe my attention to no one. 

I need to keep me healthy.

And if talking with someone makes me emotionally unhealthy, then backing away is PERFECTLY REASONABLE. 

No matter the anxiety it causes. And it probably *will* cause anxiety. But the more you practice self-care by distancing, the easier and less-anxious you will be.

Please, do not give in to that pressure. You are not someone's servant who has to come when called. You are an individual who is autonomous and can make their own decisions.