Stopping criteria for Llama2 70b
I have used the following code for defining the stopping criteria for Llama2from transformers import StoppingCriteria, StoppingCriteriaList

# define custom stopping criteria object
class StopOnTokens(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        for stop_ids in stop_token_ids:
            if torch.eq(input_ids[0][-len(stop_ids):], stop_ids).all():
                return True
        return False

stopping_criteria = StoppingCriteriaList([StopOnTokens()])This is my stop_list = ['\nHuman: ', ‘\n```\n’, '\nAI: ', ‘\nQuestion’]However to my question “Who is the CEO of Meta?”, llama2 doesn’t stop on any of these stop tokens.Also attaching the code for conversion of tokens to longtensorstop_token_ids = [tokenizer(x, return_tensors='pt')['input_ids'].squeeze() for x in stop_list]
stop_token_ids
stop_token_ids = [torch.LongTensor(x).to(device) for x in stop_token_ids]
stop_token_idsAny help is appreciated! Thanks!
2023-10-19T17:07:30Z
I think you have to add “add_special_token=False” to restrict the tokenization to the phrases only.
2024-01-03T23:11:11Z
Adding a link toadd_special_tokens docsfor reference. This fixed my problem, thanks!