This is a pun on the phrase "marks the end of his sentence". Here it is taking the word 'sentence' literally in that a sentence does end with a period. But 'sentence' can also mean the punishment assigned to a defendant found guilty by a court, or fixed by law for a particular offense.
A prisoner ' s favorite punctuation mark is the period . It marks the end of his sentence .