How to add RNN layer on top of Huggingface BERT model
I am working on a binary classification task and would like to try adding RNN layer on top of the last hidden layer of huggingface BERT PyTorch model. How can I extract the layer-1 and contact it with LSTM layer?tokenizer = BertTokenizer.from_pretrained(model_path)
# Load BertForSequenceClassification, the pretrained BERT model with a single linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(model_path, num_labels=len(lab2ind))
2021-01-19T03:10:52Z
We can useBertModelinstead ofBertForSequenceClassificationhttps://huggingface.co/transformers/model_doc/bert.html#bertmodelAnd feeds hidden states output to LSTM
2021-01-19T09:53:53Z
Thank you for your reply. I am trying this now. But does add LSTM on top of Bert needs to train BERT from scratch?
2021-01-19T12:49:56Z
You could useBertModel.from_pretrainedand then LSTM