Replace Causal Mask of T5 to custom mask
I’m a beginner of nlp,and i’m using t5 for my experiment.My goal it’s tomodify the mask of t5 decoder,as the official doc said,t5 use default causal mask for decoder.transformers/src/transformers/modeling_utils.py at main · huggingface/transformers · GitHubI wonder how i can modify this default mask and define it myself.
2024-02-21T14:05:06Z
Hey@yoohell, you can create your owndecoder_attention_maskand pass that to T5 instead of using the default
2024-02-21T15:16:06Z
do you meandecoder_attention_maskofforward()function?I found it in t5 doc,it’s said that :decoder_attention_mask (torch.BoolTensorof shape(batch_size, target_sequence_length), optional) — Default behavior: generate a tensor that ignores pad tokens indecoder_input_ids. Causal mask will also be used by default.and the casual mask matrix is built by buil-in function,is there anyway i can replace it with my custom matrix?thanks for replying me!