Is there any way to control the input of a `Longformer` layer?
Hello,Is there any way that I can directly control the input to each layer ofLongformerForMultipleChoice, similar toGPT2.transformer.h[]?I triedbest_model_longformer.longformer.encoder.layer[layer_index](input_hidden_state_for_layer)but it’s giving this error:Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 426, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 388, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 205, in fill_MC_loss_accuracy_tensor
    best_model_longformer.longformer.encoder.layer[j](input_hidden_state)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 852, in forward
    output_attentions=output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 796, in forward
    output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 241, in forward
    attention_mask = attention_mask.squeeze(dim=2).squeeze(dim=1)
AttributeError: 'NoneType' object has no attribute 'squeeze':S thank you,
2020-10-14T09:28:55Z
If you want full control you can just copy paste the code and tweak it the way you want