code: def tokenize(self, line: str, expand: bool = True) -> List[str]:
          
  
          # expand shortcuts and aliases
          if expand:
              line = self._expand(line)
  
          # check if this line is a comment
          if line.lstrip().startswith(constants.COMMENT_CHAR):
              return []
  
          # split on whitespace
          tokens = shlex_split(line)
  
          # custom lexing
          tokens = self._split_on_punctuation(tokens)
          return tokens
docs: Lex a string into a list of tokens. Shortcuts and aliases are expanded and comments are removed
  
          :param line: the command line being lexed
          :param expand: If True, then aliases and shortcuts will be expanded.
                         Set this to False if no expansion should occur because the command name is already known.
                         Otherwise the command could be expanded if it matched an alias name. This is for cases where
                         a do_* method was called manually (e.g do_help('alias').
          :return: A list of tokens
          :raises ValueError if there are unclosed quotation marks.