Getting the same embedding from llama 2 class token for any input
Hello,I’m trying to extract an embedding for several sentences from llama2-7b, and I see that I am getting the same embedding vector for the class token from the last hidden layer, no matter what the input is.from transformers import LlamaTokenizer, LlamaModel
llama2_tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
llama2_model = LlamaModel.from_pretrained("meta-llama/Llama-2-7b-hf")

inputs = llama2_tokenizer("put any sentence here", return_tensors="pt")
outputs = llama2_model(**inputs, return_dict=True)
print(outputs.last_hidden_state[0,0])The class token is the first token, hence I index from [0,0] for a batch of size 1.Would appreciate your thoughts on this. Thanks!
2023-12-04T20:32:33Z
What happened to me was that the LLAMA tokenizer added a<SOS>token to the front of each sentence, theoutputs.last_hidden_state[0,0]you are querying is indeed the hidden state for the<SOS>token and it’s the same no matter what sentence you are using. Probably you should tryoutputs.last_hidden_state[:,1]to get the hidden state of the first token?