Gradual Layer Freezing with huggingface model
Hi I would like to freeze only the first few encoder layers in a bert model.I can do it by looping through the 202 layers and freeze them by ordermodel = BertForMaskedLM.from_pretrained('bert-base-uncased')

for param in model.parameters()[6:60]:
   param.requires_grad = FalseBut isn’t there a better way to freeze some layers not having to rely on the order?Thanks
2021-02-10T14:52:40Z
Hi,I don’t think you can freeze the encoder layers without effectively freezing the embedding layer too.  (The changes won’t be able to propagate back through the frozen layers to reach the earlier layers.)