code: def generate_tokens(self, text):
          
          if text != self.last_text:
              string_io = io.StringIO(text)
              self.last_tokens = list(
                  tokenize.generate_tokens(string_io.readline)
              )
              self.last_text = text
          return self.last_tokens
docs: A stand-in for tokenize.generate_tokens().