code: def _tokenize(self, text):
          
          sentence_tokenizer = TokenizeSentence('latin')
          sentences = sentence_tokenizer.tokenize_sentences(text.lower())
  
          sent_words = []
          punkt = PunktLanguageVars()
          for sentence in sentences:
              words = punkt.word_tokenize(sentence)
  
              assert isinstance(words, list)
              words_new = []
              for word in words:
                  if word not in self.punctuation or self.abbreviations or self.numbers or self.abbreviations:  # pylint: disable=line-too-long
                      words_new.append(word)
  
              # rm all numbers here with: re.compose(r'[09]')
              sent_words.append(words_new)
  
          return sent_words
docs: Use NLTK's standard tokenizer, rm punctuation.
          :param text: pre-processed text
          :return: tokenized text
          :rtype : list