How can you delete BERT Layers after Finetuning

Intermediate
1

Hello,

I was wondering if it is possible to finetune a BERT model on SQuAD, then delete the last x layers and then perform a SQuAD evaluation on the remaining layers? How would one go about doing this?
Would this help?

transformers.modeling_utils.prune_layer

I am not exactly clear on what it does from reading the documentation but from my understanding, it only removes linear layers, not full BERT layers.

Any help would be greatly appreciated,
Thank you

1 Like