code: def tokenize(text, format=None):
      
      text = Text(text)
      text = text.replace("\t", " ")
      tokens = re.findall(patterns, text)
      tokens = [token[0] for token in tokens]
      if format == "text":
          return " ".join(tokens)
      else:
          return tokens
docs: tokenize text for word segmentation
  
      :param text: raw text input
      :return: tokenize text