(self, text)
Return an iterable of (tokentype, value) pairs.
        In subclasses, implement this method as a generator to
        maximize effectiveness.
Return an iterable of (tokentype, value) pairs.
        In subclasses, implement this method as a generator to
        maximize effectiveness.
Return an iterable of ( tokentype value ) pairs . In subclasses implement this method as a generator to maximize effectiveness .
def get_tokens_unprocessed(self, text):
        """
        Return an iterable of (tokentype, value) pairs.
        In subclasses, implement this method as a generator to
        maximize effectiveness.
        """
        raise NotImplementedError
def get_tokens_unprocessed ( self , text ) : raise NotImplementedError