"For granite" sounds like "for granted". To take something for granted means to fail to appreciate something. A rock is a hard, naturally occurring mass made up of mineral material. Granite is a type of igneous rock. This joke is playing on the term "for granite" because it sounds like "for granted". 
You should never take rocks for granite .