LongT5 masking tokens
While Longt5 has the same tokenizer as T5, in its paper (and github code) it uses Pegasus for pretraining. In Pegasus there are two types of tokens (MASK_1 and MASK_2). These are assigned token_id 2 and 3 respectively:github.comgoogle-research/pegasus/blob/99104470bb79fdf8b344bebec1267e425cfe5960/pegasus/ops/pretrain_parsing_ops.cc#L60-L69constexpr int64 kPadTokenId = 0;// End of Sequence token ID.constexpr int64 kEosTokenId = 1;// Masked Sentence token ID.constexpr int64 kMaskSentenceTokenId = 2;// Masked Word token ID.constexpr int64 kMaskWordTokenId = 3;Since LongT5 uses the same pretraining masking strategy:github.comgoogle-research/longt5/blob/eacd0c06fe40508f4a85c22af63bf8bf4391f665/longt5/preprocessors.py#L142-L202pegasus_parser, _ = parsers.string_features_for_pretraining_parser(vocab_filename='gs://t5-data/vocabs/cc_all.32000.100extra/sentencepiece.model',encoder_type='sentencepiece_noshift',  # Matches tokenizer used by T5.max_input_len=4096,max_target_len=910,max_total_words=0,parser_strategy='dynamic_rouge',parser_masked_sentence_ratio=0.2,parser_masked_words_ratio=0,parser_mask_word_option_prob=[0.8, 0.1, 0.1],parser_mask_sentence_option_prob=[.9, 0, .1, 0],parser_rouge_ngrams_size=1,parser_rouge_metric_type='F',parser_rouge_compute_option='standard',# The stopwords file used is here: https://gist.github.com/sebleier/554280parser_rouge_stopwords_filename='',shift_special_token_id=t5.data.DEFAULT_EXTRA_IDS - 2,  # 2's for eos and padmode='',parser_rouge_noise_ratio=.2,parser_dynamic_mask_min_ratio=.33,This file has been truncated.show originalOne would assume that token ids 2 and 3 are those used for masking. However these are assigned as ‘’, ‘▁’ in longT5 tokenizer. Were those ids changed when porting the model to HF? If so, which are the masking tokens for LongT5 models?