code: def _split_token_to_subtokens(token, subtoken_dict, max_subtoken_length):
    
    ret = []
    start = 0
    token_len = len(token)
    while start < token_len:
      # Find the longest subtoken, so iterate backwards.
      for end in xrange(min(token_len, start + max_subtoken_length), start, -1):
        subtoken = token[start:end]
        if subtoken in subtoken_dict:
          ret.append(subtoken)
          start = end
          break
      else:  # Did not break
        # If there is no possible encoding of the escaped token then one of the
        # characters in the token is not in the alphabet. This should be
        # impossible and would be indicative of a bug.
        raise ValueError("Was unable to split token \"%s\" into subtokens." %
                         token)
    return ret
docs: Splits a token into subtokens defined in the subtoken dict.