ValueError: model.embed_tokens.weight doesn't have any device set
Hi, I am having this error and I don’t know what it means. Can someone explain?This code triggers error:model = LlamaForCausalLM.from_pretrained(base_model,#load_in_8bit=True,torch_dtype=torch.float16,device_map=device_map,quantization_config=quantization_config,)device_map = {“transformer.word_embeddings”: 0,“transformer.word_embeddings_layernorm”: 0,“lm_head”: “cpu”,“transformer.h”: 0,“transformer.ln_f”: 0,}quantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)
2023-06-21T00:26:52Z
I am also facing the same error.  Please help with the fix
2023-06-23T22:24:30Z
Same here. Any new?
2023-07-19T11:47:12Z
while using device map add “model.embed_tokens”. In my case I have set everything to “cpu” since I don’t have any GPUs, you can set your’s as required.you may need to add “model.layers” and “model.norm” alsodevice_map = {“transformer.word_embeddings”: “cpu”,“transformer.word_embeddings_layernorm”: “cpu”,“lm_head”: “cpu”,“transformer.h”: “cpu”,“transformer.ln_f”: “cpu”,“model.embed_tokens”: “cpu”,“model.layers”:“cpu”,“model.norm”:“cpu”}
2023-10-11T08:50:16Z
In llama2-70b, we have encountered the same issue.Does anyone know why this error occurs in the device_map, or perhaps have a clear solution?If anyone has detailed documentation regarding llama2’s tokenizer, sharing it would be greatly appreciated.
2023-12-29T13:34:05Z
Hi,Adevice_mapspecifies where to place each of the individual parameters of the model. If a model supports it, it’s advised to usedevice_map="auto", which will automatically determine where to place each of the layers (using the priority of GPUs > CPUs > disk). Read more about it here:Handling big models for inference.In your case, as you specify thedevice_mapof the individual parameters, it will complain if you don’t include ALL of the model’s parameter names. In this case, the token embedding matrix of theLlamaForCausalLMmodel is calledmodel.embed_tokensas can be seenhere. Hence, the error will be raised because this was not specified in yourdevice_map. I see you’ve specifiedtransformer.word_embeddings, but that’s not the name of the token embedding matrix for this model.