I meet the zero gradient descent
I want use transformers to do text classification, I want code myself rather than useTFBertForSequenceClassification,so I write the model withTFBertModelandtf.keras.layers.Dense,but this is no gradient descent in my code, I try to find what wrong with my code but I can’t. So I submit this issues to ask for some help.my code is here:Model:and I know train data is test data,just for quick debug.and when I train this model ,
2020-11-12T13:08:51Z
maybe@jpluhas some idea here.If you could share a google colab exhibiting the behavior it would be a lot better than screen caps.
2020-11-12T14:47:38Z
thanks for your reply. I upload my code on the colab,this is the link :https://colab.research.google.com/drive/1S4Y01Pr64R8uQi6cdNx53BzVx9iROZEH?usp=sharingand I have gave the edit permissionsIt might be a little chaos cause I wrote it just for a test.Thank you very much for answering my question.
2020-11-12T15:05:36Z
At a first glance I would say it comes from two reasons:You should not flatten the output of the BERT modelYou take the sequence output while you have to take only the pooled output.Look at how it is done inTFBertForSequenceClassification.
2020-11-12T15:14:30Z
I have seen the document of them for a hold afternoon.You mean the souce code ofTFBertForSequenceClassification? But I’m too weak to find the souce code because there are a lot of call…How can I find the souce code.
2020-11-12T15:31:01Z
Here:transformers/modeling_tf_bert.py
2020-11-12T15:37:52Z
Thank you very much!!! I will read the  source code  tommorrow because is too late.Thanks again for your patient !！But I still can’t understand why the gradient descent was zero.I see the documentlast_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) – Sequence of hidden-states at the output of the last layer of the model.It is the parameters of bert after all,why Dense(FC) can’t match  them and do the gradient descent?
2020-11-13T04:40:37Z
I solve this problem,thanks a lot for patient  teaching.