Llama position_ids
I’m trying to understand the Llamaposition_ids.If I call the LlamaForCausalLM forward directly, providinginput_idsandattention_mask,position_idswill be generated as an arange from 0 to the sequence length, with batch size 1 regardless of the input batch:past_seen_tokens = 0
...
cache_position = torch.arange(
    past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
)
...
if position_ids is None:
    position_ids = cache_position.unsqueeze(0)On the other hand, if I call thegeneratefunction, providinginput_idsandattention_mask, it will callprepare_inputs_for_generation. This creates a batched position_ids with 1 values whereattention_mask == 0position_ids = attention_mask.long().cumsum(-1) - 1
position_ids.masked_fill_(attention_mask == 0, 1)
if past_key_values:
    position_ids = position_ids[:, -input_ids.shape[1] :]Code Links:github.comhuggingface/transformers/blob/831bc25d8fdb85768402f772cf65cc3d7872b211/src/transformers/models/llama/modeling_llama.py#L999past_seen_tokens = past_key_values.get_seq_length()if cache_position is None:if isinstance(past_key_values, StaticCache):raise ValueError("cache_position is a required argument when using StaticCache.")cache_position = torch.arange(past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device)if position_ids is None:position_ids = cache_position.unsqueeze(0)causal_mask = self._update_causal_mask(attention_mask, inputs_embeds)# embed positionshidden_states = inputs_embeds# decoder layersall_hidden_states = () if output_hidden_states else Noneall_self_attns = () if output_attentions else Nonenext_decoder_cache = Nonegithub.comhuggingface/transformers/blob/831bc25d8fdb85768402f772cf65cc3d7872b211/src/transformers/models/llama/modeling_llama.py#L1275if (max_cache_length is not Noneand attention_mask is not Noneand cache_length + input_ids.shape[1] > max_cache_length):attention_mask = attention_mask[:, -max_cache_length:]position_ids = kwargs.get("position_ids", None)if attention_mask is not None and position_ids is None:# create position_ids on the fly for batch generationposition_ids = attention_mask.long().cumsum(-1) - 1position_ids.masked_fill_(attention_mask == 0, 1)if past_key_values:position_ids = position_ids[:, -input_ids.shape[1] :]if self.generation_config.cache_implementation == "static":# generation with static cachecache_position = kwargs.get("cache_position", None)if cache_position is None:past_length = 0else:
2024-03-05T03:01:41Z
The reason why ingeneratetheposition_idstensor is computed fromattention_maskis that batch generation require (or at least is muuuuch simpler) when doing left padding.So let’s say you are preparing inputs for callinggeneratethrough your tokenizer, passingpadding="left"will ensure proper paddingat the beginningof the prompts (as opposed to the end).attention_maskwill look like something like[[0, 0, 1, 1, 1], [1, 1, 1, 1, 1]]which in turns will give[[-1, -1, 0, 1, 2], [0, 1, 2, 3, 4]]for theposition_ids
2024-03-05T16:18:42Z
Thanks@VictorSanh.My question was more why the two code paths differ in how they generateposition_ids. Both code paths have access to batch input andattention_mask, but produce differentposition_ids.I agree that the version inprepare_inputs_for_generationmakes more sense.Minor nit, themasked_fill_inserts 1, not -1. Not sure it matters, but your example would be[[1, 1, 0, 1, 2], [0, 1, 2, 3, 4]]