TLDR: Sort of SO and I hit a really rough patch due to both of our mental health being bad (mine especially).  Some time later, we're both in much better places and living together still, and neither of us are sure about the situation.  I get too anxious whenever I try to talk about it 

The strength of panic I'm feeling right now is utterly exhausting and I can never tell whether my worries are valid or not, and either way I can't even calm down enough to think about the situation with the slightest bit of rationality, much less talk about it with him rationally.

I've been spending the last several months of forced unemployment thanks to COVID focusing on myself, getting to the root of my mental problems, after becoming briefly suicidal early in the pandemic.  These are issues I've had for a long time and it's been a long process to dismantle all the layers of bad mental habits, assumptions, and behaviors.  As a result of all of these, my relationship with my sort-of-partner, we'll call him S (we started dating about 2.5 years ago, and have been living together for some time. Last year we moved across the country together, which was shortly before this rough patch), became difficult and very strained.  I often experienced overwhelming emotions during the worst of my mental state and unfortunately handled them in such a way that my outbursts would affect him dramatically (Indirect - essentially I would tell him that I hate myself and that he didn't care about me and run away, and just ruminate unhealthily on those topics making myself more and more anxious and upset - but extremely painful for him, I'm sure), and I completely acknowledge the damage I have done to him.  

After a friend called the police on me because I had a plan (scared the shit out of S), I took about six weeks to stay with my family and focus on myself before returning feeling a little better, with much better control over myself, though I was far from completely cured.  When I returned S was very affectionate and it was lovely, but it dropped off almost immediately.  Since then it's only become less common.  I've continued to do little but work on myself, and it's been an extremely long and frustrating road.  A few steps forward, almost as many back, as I'm sure many here have experienced.

I want to add that S is dealing with many problems of his own, including untreated severe ADHD and low libido.  The latter is as a result of a medication he took, and it makes him feel incredibly bad about himself, as well as feel less able to form connections and feel affection toward anyone, and he's incredibly worried that it's permanent. (I have expressed worry that it's my fault or because I'm around him, but he doesn't think so since he experiences the same situation when thinking about others or trying to watch porn or anything)  During my rough patch and all the drama and stress of the move, we were both depressed, also, which compounded all existing problems.

Anyway, today, I started being a little affectionate toward S, and he straight up said "Why are you being so affectionate today?"  I immediately backed off, and then asked if he wanted me to stop.  He basically said "if you want to".  He said it wasn't bad.

We sleep in the same bed and often cuddle and hug.  I (this is me being reactive, it's a problem and I still need to apologize for letting it happen again) said "Maybe it would be better if I stop sleeping in your bed."  He said "Why? You like to, and I like cuddles."

I'm honestly just panicking now, and I know that's completely ridiculous, because theoretically we're going to be friends anyway and that's what's important, so I guess that answers the question of whether this is indicative of my anxiety or of a problem that is worth being concerned about.  It's not that I need him to want to be with me forever - we've talked about both being unsure about a relationship, (though he still asks me to come with him all the time and we hang out a looot) and that we care about each other/are friends regardless - it's just that I'm so confused about where we currently stand.  And every time I try to talk about it, I get so caught up in my own worry that I end up being too reactive and the conversation isn't nearly as fruitful as it needs to be.  It doesn't help that he really struggles with conversations like that and tends to just clam up and be unable to contribute, which only makes me feel worse, and end up talking nervously and going around in circles in hopes of getting any kind of information from him.

I'm sorry, I have no idea what the point of this post is, but I'm at a loss and I don't know what else to do.