Fine tune vocab size of pre-trained Causal Language Model
Hello everyone,I’d like to know if there’s a way to adjust the vocab size of a pre-trained Causal Language Model so that, for example, instead of being able to predict one of 50k words, it will be able to predict only n words (with n being predefined). Is this possible?Thanks for your time,
2022-10-11T12:53:34Z
In general you would have to replace the embedding matrix. In this matrix the embeddings of all the vocabulary items are stored. The most easy way is to define a new matrix (Embedding — PyTorch 1.12 documentation) of the correct size (n x hidden) and initialize the vectores randomly. The problem is that this ruins (more or less) the entire model.If there is large overlap between old and new matrix (vocabularies are similar) you can initialize the new embeddings by the corresponding old ones.I used this example once:Medium – 21 Aug 21Faster than training from scratch — Fine-tuning the English GPT-2 in any...Tutorial on how to use fastai v2 over Hugging Face’s libraries to fine-tune English pre-trained GPT-2 to any language other than English.Reading time: 40 min read
2022-10-17T13:20:52Z
thanks thies!