Fine-tuning BERT with sequences longer than 512 tokens
The BERT models I have found in theModel’s Hub handle a maximum input length of 512. Using sequences longer than 512 seems to require training the models from scratch, which is time consuming and computationally expensive. However, the only limitation to input sequences longer than 512 in a pretrained BERT model is the length of the position embeddings. Therefore,Would it be okay if I expand the matrix of pretrained position embeddings in order to handle longer sequences and avoid the training from scratch?This can be achieved by re-using the first 512 embeddings to expand the matrix of position embeddings.I know BERT’s attention mechanism is quadratic and inputting longer sequences requires more computations; however, this seems manageable for sequences below 2k tokens by reducing the batch size. In fact, I tried this expansion of position embeddings usingmax_position_embeddings=1024and managed to fine-tune abert-mediummodel using less than 12GB of GPU memory with abatch_size=8at a rate of 4 minutes per epoch (4k records in the training set). The improvement in accuracy compared to using a max length of 512 seems negligible (~1%); however, having the opportunity to use longer sequences might be useful in applications where truncation is too inconvenient and using larger models such as Longformer and BigBird is difficult in terms of computational resources. I suspect this “hacky” expansion of position embeddings works because the expanded matrix of embeddings still carries the positional information BERT needs.In summary, my quick test suggests the simple expansion of the position embeddings works; however, I am asking to see if anybody has an opinion about any potential issues that may arise with this approach.Below is the code I used to expand the position embeddings to length 1024 by simply stacking the first 512 positions twice. Note that theposition_idsandtoken_type_idsalso need to be expanded.max_length = 1024
model_checkpoint = "prajjwal1/bert-medium"
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)

tokenizer.model_max_length = max_length
model.config.max_position_embeddings = max_length
model.base_model.embeddings.position_ids = torch.arange(max_length).expand((1, -1))
model.base_model.embeddings.token_type_ids = torch.zeros(max_length).expand((1, -1))
orig_pos_emb = model.base_model.embeddings.position_embeddings.weight
model.base_model.embeddings.position_embeddings.weight = torch.nn.Parameter(torch.cat((orig_pos_emb, orig_pos_emb)))
2022-02-27T14:04:52Z
why not use a longformer?
2022-02-28T19:57:17Z
In my experience, LongFormer and BigBird require a lot of GPU memory. I tried using these on a 14GB GPU, but I was limited tobatch_size=1, which took for ever to train and yielded rather poor results.
2022-02-28T20:13:21Z
Interesting, was just thinking that if a bert model is passed in longer sequences of tokens. But the bert model was initially pretrained with a limited (512) tokens. Wouldn’t the weights be “confused” by the longer sequences. hence perform badly?
2022-03-01T06:10:50Z
Perhaps, but I assume the “confusion” will not be significant, given that the only element in BERT’s architecture that depends on the sequence length is the position embeddings. My empirical test on an IMDB text classification task suggests that using longer sequences does not degrade the performance, but it does not significantly improve it either (see the original post where I mention that performance only increased by 1% when using longer sequences). However, for other datasets and tasks, the performance may vary when using longer sequences.
2022-04-03T10:44:03Z
do you know if the max seq length is 512 character token or word token?
2022-04-03T20:33:20Z
HI@moma1820, I saw you posted a similar question in another thread, so I replied there. See the link below:Is 512 token in bert, token or character level?BeginnersBERT uses a subword tokenizer (WordPiece), so the maximum length corresponds to 512 subword tokens. See the example below, in which the input sentence has eight words, but the tokenizer generates a sequence with length equal to nine. Note that the extra token in this example is due to the word outstandingly, which the subword tokenizer represents using two tokens:  outstanding and ##ly. 
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased"…
2022-04-04T06:17:02Z
thanks man!