Attention type 'block_sparse' is not possible if sequence_length: 458 <= num global tokens:
I am using the pre-trainedgoogle/bigbird-pegasus-large-arxivmodel.But I receive the following update during the forward pass.Attention type 'block_sparse' is not possible if sequence_length: 458 <= num global tokens: 2 * config.block_size + min. num sliding tokens: 3 * config.block_size + config.num_random_blocks * config.block_size + additional buffer: config.num_random_blocks * config.block_size = 704 with config.block_size = 64, config.num_random_blocks = 3.Changing attention type to 'original_full'...I understand the update and I am aware of benefit of time and memory it saves while usingblock_sparsethanoriginal_full.So, how should I go about selecting the suitableblock_sizeandnum_random_blockswhen I know that there is a lot of variation in the sequence length of my inputs?
2021-08-14T00:20:08Z
I think one workaround is to pad your sequence length to a fixed number. i.e., >= 512/1024/2048 etc
2021-08-14T00:36:00Z
That’s very inefficient since the seq length varies from 400-10k.truncation=Truein tokenizer works.
2021-09-06T04:49:10Z
I have encountered the same problem. Do you have a solution now? If you ignore this, will it have an impact
2021-09-06T14:02:17Z
truncation=Trueworked for me, did you try it at all?