code: def jieba_tokenize(text, external_wordlist=False):
      
      global jieba_tokenizer, jieba_orig_tokenizer
      if external_wordlist:
          if jieba_orig_tokenizer is None:
              jieba_orig_tokenizer = jieba.Tokenizer(dictionary=ORIG_DICT_FILENAME)
          return jieba_orig_tokenizer.lcut(text)
      else:
          if jieba_tokenizer is None:
              jieba_tokenizer = jieba.Tokenizer(dictionary=DICT_FILENAME)
  
          # Tokenize the Simplified Chinese version of the text, but return
          # those spans from the original text, even if it's in Traditional
          # Chinese
          tokens = []
          for _token, start, end in jieba_tokenizer.tokenize(simplify_chinese(text), HMM=False):
              tokens.append(text[start:end])
          return tokens
docs: Tokenize the given text into tokens whose word frequencies can probably
      be looked up. This uses Jieba, a word-frequency-based tokenizer.
  
      If `external_wordlist` is False, we tell Jieba to default to using
      wordfreq's own Chinese wordlist, and not to infer unknown words using a
      hidden Markov model. This ensures that the multi-character tokens that it
      outputs will be ones whose word frequencies we can look up.
  
      If `external_wordlist` is True, this will use the largest version of
      Jieba's original dictionary, with HMM enabled, so its results will be
      independent of the data in wordfreq. These results will be better optimized
      for purposes that aren't looking up word frequencies, such as general-
      purpose tokenization, or collecting word frequencies in the first place.