code: def _generate_tokens(pat: GenericAny, text: str) -> Iterator[Token]:
      
      rules = _TYPE_RULES
      keys = _TYPE_KEYS
      groupindex = pat.groupindex
      scanner = pat.scanner(text)
      for m in iter(scanner.match, None):
          lastgroup = m.lastgroup
          func = rules[keys[groupindex[lastgroup] - 1]]
          if func is not None:
              yield func(m.group(lastgroup))
docs: Generate a sequence of tokens from `text` that match `pat`
  
      Parameters
      ----------
      pat : compiled regex
          The pattern to use for tokenization
      text : str
          The text to tokenize