GPT2: many bad_words_ids leading to slow text generation?
For many models, like GPT2, thegeneratefunction acceptsbad_words_ids. We’re currently passing about 2500 tokenized phrases into this, and finding that it works well, but also finding that it slows down inference considerably… with 2500 phrases, we find that ageneratethat would take250mswithout thebad_words_ids, takes about500ms.Maybe there is just no solution for this, and we need to simply curtail our usage ofbad_words_ids?We were also looking at this code:github.comhuggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/generation_logits_process.py#L345)for hypo_idx in range(num_hypos)]for i, banned_tokens in enumerate(banned_batch_tokens):scores[i, banned_tokens] = -float("inf")return scoresclass NoBadWordsLogitsProcessor(LogitsProcessor):""":class:`transformers.LogitsProcessor` that enforces that specified sequences will never be sampled.Args:bad_words_ids (:obj:`List[List[int]]`):List of list of token ids that are not allowed to be generated. In order to get the tokens of the wordsthat should not appear in the generated text, use :obj:`tokenizer(bad_word,add_prefix_space=True).input_ids`.eos_token_id (:obj:`int`):The id of the `end-of-sequence` token.A thebad_words_idsare required to be passed in as alist. If somehow we could use atensorinstead, and put that tensor on the GPU, would that speed it up?Any other suggestions?thanks