code: def _split_string_to_tokens(text):
    
    if not text:
      return []
    ret = []
    token_start = 0
    # Classify each character in the input string
    is_alnum = [c in _ALPHANUMERIC_CHAR_SET for c in text]
    for pos in xrange(1, len(text)):
      if is_alnum[pos] != is_alnum[pos - 1]:
        token = text[token_start:pos]
        if token != u" " or token_start == 0:
          ret.append(token)
        token_start = pos
    final_token = text[token_start:]
    ret.append(final_token)
    return ret
docs: Splits text to a list of string tokens.