code: def lex(self, string):
          
          self.lexer.input(string)
          for token in self.lexer:
              yield token
docs: Lex a string.
          
          :param string: The input to lex. 
          :return: A generator which will continue yielding tokens until the end
                   of the input is reached.