Gradual Unfreezing support for Fine tuning models
Does huggingface library has support for gradual unfreezing? As thats one of the key strategy to fine tune model effectively for down stream task.
2020-08-25T11:04:32Z
You will have to do that manually. TheTrainerwon’t do it automatically for you. To freeze the pytorch model, you can do something like this:model = AutoModel.from_pretrained(model_id)
for param in model.parameters():
    param.requires_grad = FalseIn this way, you’d have more granular control over which layers you want to freeze. For your use case, you’d want to train just the linear layer initially. You can do this by freezing just the encoderfor param in model.encoder.parameters():
    param.requires_grad = FalseYou’d have to check ifencoderis an attribute of all models to be fully sure. At later stages, you can turn the bool to True, to train the encoder.I hope this gives you an idea.
2020-08-25T15:51:42Z
I would be very happy to help support this if anyone has results that demonstrate its effectiveness!There is also afreeze_paramshelper function here:https://github.com/huggingface/transformers/blob/e92efcf7286c955e6901f894be39cf6154af48b7/examples/seq2seq/utils.py#L277
2020-08-26T23:28:04Z
hi@sshleifer,what counts as a “layer” for freezing?I’ve created a model based on BertModel (with a custom head), and I’d like to freeze the embedding layer and some of the attention layers.   I’ve appliedct = 0for child in model.children():ct += 1if ct < 11:          # this freezes layers 1-10for param in child.parameters():param.requires_grad = Falsebut I’m not sure what that will have frozen.How can I tell ?