Hi everyone

I started a new job about 4 weeks ago as an activities assistant in a care home. I have social anxiety but I wanted this job because it's not many hours a week and I thought spending time with people would help me overcome my anxiety. For the most part, it has been okay. 

But today something happened. There's a resident who I've spent quite a bit of time talking to. I liked spending time with him because he talks a lot and doesn't expect me to say much back in return. Today I gave him a jigsaw puzzle of dogs to do because he mentioned in the past that he likes jigsaw puzzles. He said he won't have time to do it today but asked me to sit down so we can talk about dogs. I mentioned that I was scared of dogs and he said that makes sense because I don't look strong enough to hold them back if they try to attack.  
Later on, he randomly said I have nice skin and it looks really smooth. He asked if he can touch my arm. I felt uncomfortable but said okay. He asked me if I think I have nice skin to which I replied "no because I get lots of pimples, especially on my face." He then asked if he can touch my face and again, I let him. He again said I have really nice skin. Then he asked me to close the door (he can't walk) because he's cold, so I did. And then he asked to touch my hair because it looks really soft. I let him and then he asked me to touch his hair so I did quickly. But then I got up to leave because I was super uncomfortable. As I got up, he patted his lap and I was like "what?" And he said "sit on my lap" and I said "no, I have to go now, I have a lot of things to do" and he said "I just want to whisper something in your ear." I said no again and then left.

Typing this all out makes me realise how stupid I was. I've always had a problem setting boundaries but this was really bad. Part of me wanted to tell someone at work what had happened but I was so afraid whoever I told would think I'm stupid or blame me or something and it would make me feel worse. I couldn't really concentrate the rest of the day because that incident is all I could think about. What I want to know is, is he wrong for touching my arm and hair? Or is it not wrong because I gave him permission? Also, should I tell my manager or someone about what happened?