 Overview  BERT (Bidirectional Encoder Representations from Transformers) provides dense vector representations for natural language by using a deep, pre-trained neural network with the Transformer architecture. It was originally published by  Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: ["BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"](https://arxiv.org/abs/1810.04805), 2018.  This model uses the implementation of BERT from the TensorFlow Models repository on GitHub at [tensorflow/models/official/legacy/bert](https://github.com/tensorflow/models/tree/master/official/legacy/bert). It uses L=12 hidden layers (i.e., Transformer blocks), a hidden size of H=768, and A=12 attention heads. For other model sizes, see the [BERT](https://tfhub.dev/google/collections/bert/1) collection.  The weights of this model are those released by the original BERT authors. This model has been pre-trained for English on the Wikipedia and BooksCorpus. Text inputs have been normalized the "cased" way, meaning that the distinction between lower and upper case as well as accent markers have been preserved. For training, random input masking has been applied independently to word pieces (as in the original BERT paper).  All parameters in the module are trainable, and fine-tuning all parameters is the recommended practice.  