To my ears it sounds a little unfinished without an 'it' or 'them' or something else rounding off the sentence but I'd say its still acceptable. How would the sentence change if I wanted to say 'I am not allowed' ? would it be one of the times when Poles literally say 'I have not allowed' i.e. nie mam pozwalać [though I'm pretty sure pozwalać needs declining here somehow.
It clearly refers to what has just been said in the real context. It does sound natural in Polish.
"I am not allowed" = "I have no permission" (Nie mam pozwolenia/zgody), also what would sound good here is "Nie wolno mi" ('wolno' is a defective verb, it doesn't refer to any grammatical gender).
What else... "Nie pozwalają mi" ('they' don't allow me), "Nie pozwolili/Nie pozwoliły mi" ('they' did not allow me), "Nie pozwolono mi" (I was not allowed, subjectless)... "Nie wolno mi" sounds the best to me.