code: def NextToken(self):
      
      current_state = self.state
      for token in self.tokens:
        # Does the rule apply to us?
        if not token.state_regex.match(current_state):
          continue
  
        # Try to match the rule
        m = token.regex.match(self.buffer)
        if not m:
          continue
  
        # The match consumes the data off the buffer (the handler can put it back
        # if it likes)
        # TODO: using joins might be more efficient here.
        self.processed_buffer += self.buffer[:m.end()]
        self.buffer = self.buffer[m.end():]
        self.processed += m.end()
  
        next_state = token.next_state
        for action in token.actions:
  
          # Is there a callback to handle this action?
          callback = getattr(self, action, self.Default)
  
          # Allow a callback to skip other callbacks.
          try:
            possible_next_state = callback(string=m.group(0), match=m)
            if possible_next_state == self._CONTINUE_STATE:
              continue
            # Override the state from the Token
            elif possible_next_state:
              next_state = possible_next_state
          except errors.ParseError as exception:
            self.Error(exception)
  
        # Update the next state
        if next_state:
          self.state = next_state
  
        return token
  
      # Check that we are making progress - if we are too full, we assume we are
      # stuck.
      self.Error('Expected {0:s}'.format(self.state))
      self.processed_buffer += self.buffer[:1]
      self.buffer = self.buffer[1:]
      return self._ERROR_TOKEN
docs: Fetch the next token by trying to match any of the regexes in order.