code: def tokenize(readline):
      
      # This import is here to avoid problems when the itertools module is not
      # built yet and tokenize is imported.
      from itertools import chain, repeat
      encoding, consumed = detect_encoding(readline)
      rl_gen = iter(readline, b"")
      empty = repeat(b"")
      return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
docs: The tokenize() generator requires one argument, readline, which
      must be a callable object which provides the same interface as the
      readline() method of built-in file objects.  Each call to the function
      should return one line of input as bytes.  Alternatively, readline
      can be a callable function terminating with StopIteration:
          readline = open(myfile, 'rb').__next__  # Example of alternate readline
  
      The generator produces 5-tuples with these members: the token type; the
      token string; a 2-tuple (srow, scol) of ints specifying the row and
      column where the token begins in the source; a 2-tuple (erow, ecol) of
      ints specifying the row and column where the token ends in the source;
      and the line on which the token was found.  The line passed is the
      logical line; continuation lines are included.
  
      The first token sequence will always be an ENCODING token
      which tells you which encoding was used to decode the bytes stream.