TLDR I’m venting and would like words of encouragement or advice. How do manage relationship anxiety?

I don’t know what to do and I’ve tried to explain my feelings to my partner but he says things like I need to figure out what makes me happy, or that my actions or what I’m feeling is crazy, his peace is important blah blah blah. What about my peace? My anxiety is ruining my peace. 

We’ve been together for 10 years but only lived together for less than 2 and not consecutively. We live apart now, have a child together and he is usually here daily. Lately he’s been taking our child to school and popping in for coffee and breakfast then off to do his thing. We’ll chat on the phone between meetings we both work from our homes). After work ~5 pm he either has a friend over his place or goes out to a hookah bar and is there for a few hours, then he pops in around 8 or 9, sometimes 10 then he leaves an hour or two later. Weekends I get some of his mornings and sometimes most of Sunday. 

He is with friends everyday. Today he played board games with them for 8 plus hours at his apartment. I was cool with that, but then my anxiety kicked in after we spoke around 5:40 and he said they were playing a last game and he’d call back. I called around 7:25 and texted, no reply. I called again 8:30 and he picked up and got mad when I told him I thought he was going to call me back. I told him I was fine all day until then and he could’ve given me a thumbs up reply or something to my text to acknowledge me. He can read and answer his friends texts when he’s around me but not mine when he’s with them? Granted he did pick up my other calls during the day but I told him I felt ignored and after waiting an hour I got worked up. He told me I need to learn how to deal with it when he’s busy and can’t come to the phone. 

So while I agree I was acting a little unreasonable and shouldn’t have called so much or made a big deal about it, it was his reply and way of handling it that sets me off. I told him I felt ignored because he’s with his friends more than me and he said no he’s with me more… I told him I feel like he prioritizes his friends more because he’s with them everyday and he says that I always have a problem with something. I told him it makes me feel unimportant when he spends so much time with them. 

 Any way the whole conversation was not constructive and he said maybe he gives me too much time and should give me less, that I sounded crazy, that I should be more pleasant and I should make him want to be with me more than his friends…. He says I sound crazy but I think he sounds insensitive and only focused on his “peace”. He says I shouldn’t rely on him making me happy but I said I don’t need him to be happy, I want him around to add to my peace and that waiting around for him doesn’t do it for me. 

Am I being toxic?