Unable to load TF2-checkpoints into Huggingface
I am unable to import a pretrained BERT checkpoints trained with Tensorflow 2.I get the error “AttributeError: ‘BertForPreTraining’ object has no attribute ‘shape’”. I have tried both the “transformers-cli convert” and the “.from_pretrained”, basically stopping at the same issue.It also affects the official BERT models. I have no issues loading the official BERT-model trained for TF1.x, however trying to load the official TF2.0/Keras-model is giving me the same error.This Colab Gist downloads the three official models hosted at gs://cloud-tpu-checkpoints and tries to load them into Huggingface:https://colab.research.google.com/gist/peregilk/6b97384d3443ca9354a1898efd9819af/loading-tf-2-0-checkpoints-in-huggingface.ipynb
2022-11-09T16:31:45Z
Hey Pere,Were you able to solve this issue? I have the a similar error but with bias. That is the error says “AttributeError: ‘BertForPreTraining’ object has no attribute ‘bias’”.
2022-11-09T17:00:21Z
Long ago. From what I remembered I gave up on this. One of the reasons I switched to using PyTorch and Flax.