Hey, I am trying to figure out how to freeze layers of a model and read that I had to use for param in model.base_model.parameters(): param.requires_grad = False if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer? I tried the following: …

How to freeze layers using trainer?

sgugger March 19, 2021, 4:37pm 7

You should inspect the weights to see where they difer then. Trainer will not change the requires_grad value of your parameters.

3 Likes

Freeze Lower Layers with Auto Classification Model

Topic		Replies	Views
How to freeze some layers of BertModel Beginners	8	17162	August 25, 2022
The point of using pretrained model if I don't freeze layers Beginners	1	7537	May 31, 2023
Gradual Layer Freezing with huggingface model 🤗Transformers	1	838	February 10, 2021
Freeze Lower Layers with Auto Classification Model 🤗Transformers	6	17011	May 25, 2023
Freezing layers when using gradient checkpointing 🤗Transformers	0	677	March 20, 2022