RuntimeError: The size of tensor a (4096) must match the size of tensor b (4097) at non-singleton dimension 3
In the process of inference of thevicuna-13b-16kmodel I get the following error when the context length isgreater than 4096tokens:RuntimeError: The size of tensor a (4096) must match the size of tensor b (4097) at non-singleton dimension 3As I understand it, there are two parameters in the config that allow you to achieve the length of the context (prompt and model response) of16384tokens:"max_position_embeddings": 4096,
"rope_scaling": {
    "factor": 4.0,The error occurs in theself._update_causal_maskfunction in the following line:padding_mask = causal_mask[..., :mask_length].eq(0.0) * attention_mask[:, None, None, :].eq(0.0)causal_maskhas dimensions[1, 1, 4096, 4096].The cinditionif seq_length > self.causal_mask.shape[-1]:is not met because the generation uses the kv-cache and the dimensioninputs_embeds(input_tensor) is[1, 1, 5120].Are there any ideas on how to fix this error?
2024-08-24T14:07:19Z
The problem was in the transformers version4.38. In version4.44the problem was fixed and the model works fine.