Embeddings from fine-tuned ModelForSequenceClassification
Hi,I am fine-tuning a Model for Sequence Classification i.e., I want to modify both the weights from the pre-trained language model and the weights from the classification head. Once I do that, I want to compare the embeddings from the freshly fine-tuned model and the ones from the original pre-trained model.How can I retrieve the embeddings from the Model For Sequence Classification as it does not have anylast_hidden_statesfeatures?Cheers,GiuseppeI found a solution: it was simple but IMHO it was nowhere on the web clearly stated. You can save the model, then reload it and use it with hidden states.trainer.save_model(working_data_path)
model = AutoModel.from_pretrained(working_data_path).to(device)
preds = model(**inputs, output_hidden_states=True)The usuallast_hidden_statecan be retrieved frompreds["hidden_states"][-1].