How to load weights in the pix2struct delpot model
processor = Pix2StructProcessor.from_pretrained('deplot_model')
model = Pix2StructForConditionalGeneration.from_pretrained('deplot_model')Hi,I am trying to fine-tune the model by extending the vocabulary. To do this, I have added tokens and resized the embedding shape, but I would like to know how to directly access the weights since I will be initializing the newly added weights using different initialization options (random, Xavier, He, etc…). For the other transformer models,model.embeddings.word_embeddings.weightseemed to work but I realized that model.embeddings does not exist in Pix2StructForConditionalGeneration. So is there a way to directly access the weights for the Pix2struct model?
2023-07-24T12:28:12Z
Hi,You can use themodel.resize_token_embeddingsmethod for this purpose:from transformers import Pix2StructProcessor , Pix2StructForConditionalGeneration

processor = Pix2StructProcessor.from_pretrained("google/deplot")
model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")

print("Number of initial token embeddings:", model.get_input_embeddings().num_embeddings)

processor.tokenizer.add_tokens(['newWord', 'newWord2'])
model.resize_token_embeddings(len(processor.tokenizer))

print("Number of final token embeddings:", model.get_input_embeddings().num_embeddings)
2023-07-25T02:00:35Z
Thank you@nielsr!