[Solved] Issue on translating DPR to TFDPR on loading pytorch weights to TF model
Hi Huggingface team,I would love to contribute translating DPR to TFDPR model .This is my first time trying to contribute so please bear with me for my simple question.I have followed@sshleifer'sgreat PR on TFBart modelon 4 files :__init__.py,convert_pytorch_checkpoint_to_tf2.py,utils/dummy_tf_objects.pyand (newly created)modeling_tf_dpr.pyNow the TF code can run properly with examples in Pytorch’s DPR (correct input/output tensors) . However, it seems the method.from_pretrained(..., from_pt=True)does not work properly as there are always warning messages that weights could not be loaded :Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFDPRContextEncoder: ['ctx_encoder.bert_model.encoder.layer.7.intermediate.dense.bias', 'ctx_encoder.bert_model.encoder.layer.7.attention.output.dense.weight', ... (long list of every variable)]I have a colab code for the modified 4 mentioned files here :https://colab.research.google.com/drive/1lU4fx7zkr-Y3CXa3wmHIY8yJhKdiN3DI?usp=sharing(It would be great if Sam or Patrick@patrickvonplatencan take a quick look, as this may be an easy fix)(to easily navigate the change, please “find on page” for e.g.TFDPRContextEncoder)
2020-10-27T14:38:19Z
try passing for example, name=‘ctx_encoder’, as a kwarg to more components (like you do for layers) and callingsuper().__init__(config, **kwargs)inside of them.Like this:That’s how I debugged some tensorflow layer name issues (like the one you have),but I’m no expert.
2020-10-29T23:40:57Z
@sshleiferThanks for the insights!!I am now able to clear up most of the naming mismatches.The last issue is thatDPREncoderhas a variable fromBertModelclass.(which I already solved but seeking confirmation, see below)And there isone important different implementationbetweenPytorch’sBertModelvs.TFBertModelWhile inTFBertModelwe have one more nested classTFBertMainLayer(which initialize withname='bert'), in Pytorch’sBertModelwe have none of thisMainLayerclass, so the naming is inherently different.In Pytorch name is likectx_encoder.bert_model.encoder.layer.3.attention.self.key.weightand TF name has‘extra’bertinside due to theTFBertMainLayerclass:ctx_encoder.bert_model.bert.encoder.layer.3.attention.self.key.weight===========SolutionSo the solution I made is to simply change fromTFBertModeltoTFBertMainLayer, and now all pytorch weights are correctly loading forctx_encoder,question_encoderandreader. To progress in thisTFDPR, I will open issue in Github seeking for PR suggestion .Thank you again!