code: def token_offsets(tokens):
      
      end_offset = 0
      previous_end_row = 0
      previous_end_column = 0
      for t in tokens:
          token_type = t[0]
          token_string = t[1]
          (start_row, start_column) = t[2]
          (end_row, end_column) = t[3]
  
          # Account for the whitespace between tokens.
          end_offset += start_column
          if previous_end_row == start_row:
              end_offset -= previous_end_column
  
          # Record the start offset of the token.
          start_offset = end_offset
  
          # Account for the length of the token itself.
          end_offset += len(token_string)
  
          yield (token_type,
                 token_string,
                 start_offset,
                 end_offset)
  
          previous_end_row = end_row
          previous_end_column = end_column
docs: Yield tokens and offsets.