How to generate a sequence using inputs_embeds instead of input_ids?
Hello, I am struggling with generating a sequence of tokens using model.generate() with inputs_embeds.For my research, I have to use inputs_embeds (word embedding vectors) instead of input_ids (token indices) as an input to the GPT2 model.I want to employ model.generate() which is a convenient tool for generating a sequence of tokens, but there is no argument for inputs_embeds. I tried to edit " transformers.generation_utils", but it was not easy to figure out which lines I should change.Is there any idea that I can easily generate tokens with default settings for hyper-paremeters as in model.generate()? If there is any idea, help me please.
2021-10-18T14:01:22Z
Were you able to figure something out for this ? My needs are similar. I’m using BART model
2021-10-18T22:07:22Z
@StackSmasher,consider you have the tensorinputs_embedswhich I believe will be in the shape of(batch_size, seq_length, dim), or If you have ahidden_statein the shape of(batch_size, dim)justunsqueeze(dim=1)it to become(batch_size,1,dim)Then smoothly u can achieve the desired output by:create anattention mask(batch_size, seq_len)createdecoder_input_ids(batch_size, 1)tracing these two parameters, leads to two functions below:attention mask you are already familiar with it.github.comhuggingface/transformers/blob/d5ff69fce92bb1aab9273d674e762a8eddcb2e3f/src/transformers/generation_utils.py#L394-L403def _prepare_attention_mask_for_generation(self, input_ids: torch.Tensor, pad_token_id: int, eos_token_id: int) -> torch.LongTensor:is_pad_token_in_inputs_ids = (pad_token_id is not None) and (pad_token_id in input_ids)is_pad_token_not_equal_to_eos_token_id = (eos_token_id is None) or ((eos_token_id is not None) and (pad_token_id != eos_token_id))if is_pad_token_in_inputs_ids and is_pad_token_not_equal_to_eos_token_id:return input_ids.ne(pad_token_id).long()return input_ids.new_ones(input_ids.shape, dtype=torch.long)decoder_input_idsit’sonestoo (multiplied by config.decoder_start_token_ids).github.comhuggingface/transformers/blob/d5ff69fce92bb1aab9273d674e762a8eddcb2e3f/src/transformers/generation_utils.py#L419-L426def _prepare_decoder_input_ids_for_generation(self, input_ids: torch.LongTensor, decoder_start_token_id: int = None, bos_token_id: int = None) -> torch.LongTensor:decoder_start_token_id = self._get_decoder_start_token_id(decoder_start_token_id, bos_token_id)decoder_input_ids = (torch.ones((input_ids.shape[0], 1), dtype=torch.long, device=input_ids.device) * decoder_start_token_id)return decoder_input_idswrapping it all:generator = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
inputs_embeds = # a 3D tensor, [batch, seq_length, dim]
attention_mask = torch.ones(inputs_embeds.shape[:2], dtype=torch.long)
decoder_input_ids = torch.ones((inputs_embeds.shape[0], 1), dtype=torch.long)*generator.config.decoder_start_token_id
output_ids = generator.generate(attention_mask=attention_mask, decoder_input_ids=decoder_input_ids, inputs_embeds=inputs_embeds, max_length=100,num_beams=4)
2021-11-25T10:47:14Z
Hey thanks a lot for the reply and it works nice !Slight clarification, would I need to setdecoder_input_idswhen I’m training the said model usinginputs_embedsas well ?
2022-04-17T16:59:06Z
For BART, you can use encoder_outputs, which you should get from the encoder part of BART model.