To "pay attention" is to give attention to someone by watching and/or listening to them. To "pay someone" is to give that someone money for work done, or goods received. The joke is talking about someone that is poor. A poor person is someone that often lacks sufficient money to purchase goods or services. "We were so poor... we couldn't even afford to pay attention." The joke is playing on the word "pay" and its different meanings and uses.
We were so poor when I was growing up we couldn ' t even afford to pay attention .