This is a play on words. Having a rough time means having a difficult time doing something. However, "rough" also means phsyically of coarse text, like sandpaper would be.
The inventor of sandpaper had a rough time of it .