What model-pairs are supported by the assistant decoding generation in Huggingface AutoModelForCausalLM?
The assistant decoding model as described inAssisted Generation: a new direction toward low-latency text generationis implemented inGenerate: Add assisted generation by gante · Pull Request #22211 · huggingface/transformers · GitHubQ Part 1.What model-pairings are known to be supported by themodel.generate(..., assistant_model='')feature?Q Part 2.Does it work for decoder-only model too? Anyone tried any pairs of decoder-only models available on the huggingface hub?The assumption for the assistant decoding model are:the tokenizer must be the same for assistant and main modelthe model is supported byAutoModelForCausalLMfrom transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = 'EleutherAI/pythia-1.4b-deduped'
assistant = 'EleutherAI/pythia-160m-deduped'

tokenizer = AutoTokenizer.from_pretrained(checkpoint) #, bos_token_id=101, eos_token_id=102)
model = AutoModelForCausalLM.from_pretrained(checkpoint) #, bos_token_id=101, eos_token_id=102)

assistant_model = AutoModelForCausalLM.from_pretrained(assistant)

tokenized_inputs = tokenizer("Alice and Bob", return_tensors="pt")

outputs = model.generate(**tokenized_inputs, assistant_model=assistant_model)

tokenizer.batch_decode(outputs, skip_special_tokens=True)I’ve tried the following and this works:EleutherAI/pythia-1.4b-deduped+EleutherAI/pythia-160m-dedupedBut these didn’t:google-bert/bert-large-uncased+google-bert/bert-base-uncased(also had to add, bos_token_id=101, eos_token_id=102)to the model and/or tokenizer initialization to avoid None type when assistant model is scoping down the vocabulary)FacebookAI/xlm-roberta-large+FacebookAI/xlm-roberta-base(ended up withTypeError: object of type 'NoneType' has no len()error when looking for candidate generation)
2024-03-13T19:54:01Z
Also asked onpython - What model-pairs are supported by the assistant decoding generation in Huggingface AutoModelForCausalLM? - Stack OverflowandGenerate: Add assisted generation by gante · Pull Request #22211 · huggingface/transformers · GitHub