Trainer and TrainingArguments - gradual unfreezing
Hello!Since further pre-training BERT might suffer from catastrophic forgetting, I am wondering, is it possible to use gradual unfreezing somehow with the Trainer module? I could do it with PyTorch and a classical training loop, but I was looking if you already have it implemented in the Trainer class.
2022-05-11T08:58:42Z
petarulev:unfreezing somehow with the Trainer module? I could do it with PyTorch and a classical training loop, but I was looking if you already have it implemented in the Trainer class.or TrainingArguments class.
2022-08-25T10:52:06Z
I think you can add SkipConnection to avoid Catastrophing forgetting in for ex BERT. It should be possible by subclassing and modifying main class, but I don’t know how to do thatclass BertForMultilabelSequenceClassification(BertForSequenceClassification):
....