Nits is another word for lice. Head lice is a type of insect infestation that children can experience. 
Lousy means very poor or bad, and it can also mean infested with lice. The person in the joke has a "lousy memory", with lousy being cleverly used since the memory relates to lice. The joke is playing on the word "lousy" and its double meaning. 
I can ' t remember ever getting nits as a kid , although I do have a lousy memory .