text to text pretraining T5
233296501
mT6: Multilingual Pretrained Text to Text Transformer with Translation Pairs
Multilingual T5 (MT5; Xue et al. 2020) pretrains a sequence to sequence model on massive monolingual texts, which has shown promising results on many cross lingual tasks. In this paper, we improve multilingual textto text transfer Transformer with translation pairs (MT6) Specifically, we explore three cross lingual text to text pre training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially nonautoregressive objective for text to text pretraining. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed MT6 improves cross lingual transferability over MT5.
ArXiv
Zewen Chi Shuming Ma Shaohan Huang Xian-Ling Mao Heyan Huang Furu Wei
220546439
Investigating Pretrained Language Models for Graph to Text Generation
Graph to text generation, a subtask of data to text generation, aims to generate fluent texts from graph based data. Many graph to text models have shown strong performance in this task employing specialized graph encoders. However, recent approaches employ large pretrained language models (PLMs) achieving state of the art results in data to text generation. In this paper, we aim to investigate the impact of large PLMs in graph to text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. Our analysis shows that PLMs such as BART and T5 achieve state of the art results in graph to text benchmarks without explicitly encoding the graph structure. We also demonstrate that task adaptive pretraining strategies are beneficial to the target task, improving even further the state of the art in two benchmarks for graph to text generation. In a final analysis, we investigate possible reasons for the PLMs' success on graph to text tasks. We find evidence that their knowledge about the world gives them a big advantage, especially when generating texts from KGs.
ArXiv
Leonardo F R Ribeiro Martin Schmitt Hinrich Schutze Iryna Gurevych
235352948
nmT5 Is parallel data still relevant for pre training massively multilingual language models?
Recently, mT5 a massively multilingual version of T5 leveraged a unified text to text format to attain state of the art results on a wide variety of multilingual NLP tasks. In this paper, we investigate the impact of incorporating parallel data into mT5 pre training. We find that multi tasking language modeling with objectives such as machine translation during pretraining is a straightforward way to improve performance on downstream multilingual and cross lingual tasks. However, the gains start to diminish as the model capacity increases, suggesting that parallel data might not be as essential for larger models. At the same time, even at larger model sizes, we find that pre training with parallel data still provides benefits in the limited labelled data regime.
ACL/IJCNLP
Mihir Kale Aditya Siddhant Noah Constant Melvin Johnson Rami Al-Rfou Linting Xue
230437660
Improving Sequence to Sequence Pre training via Sequence Span Rewriting
In this paper, we propose Sequence Span Rewriting (SSR) a self supervised task for sequence to sequence (Seq2Seq) pre training. SSR learns to refine the machine generated imperfect text spans into ground truth text. SSR provides more fine grained and informative supervision in addition to the original textinfilling objective. Compared to the prevalent text infilling objectives for Seq2Seq pretraining, SSR is naturally more consistent with many downstream generation tasks that require sentence rewriting (e.g. text summarization, question generation, grammatical error correction, and paraphrase generation) We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre trained model T5 in a continual pre training setting and show substantial improvements over T5 on various natural language generation tasks.1
ArXiv
Wangchunshu Zhou Tao Ge Ke Xu Furu Wei
236486212
Attribute Value Generation from Product Title using Language Models
Identifying the value of product attribute is essential for many e commerce functions such as product search and product recommendations. Therefore, identifying attribute values from unstructured product descriptions is a critical undertaking for any e commerce retailer. What makes this problem challenging is the diversity of product types and their attributes and values. Existing methods have typically employed multiple types of machine learning models, each of which handles specific product types or attribute classes. This has limited their scalability and generalization for large scale real world e commerce applications. Previous approaches for this task have formulated the attribute value extraction as a Named Entity Recognition (NER) task or a Question Answering (QA) task. In this paper we have presented a generative approach to the attribute value extraction problem using language models. We leverage the large scale pretraining of the GPT 2 and the T5 text to text transformer to create fine tuned models that can effectively perform this task. We show that a single general model is very effective for this task over a broad set of product attribute values with the open world assumption. Our approach achieves state of the art performance for different attribute classes, which has previously required a diverse set of models.
ECNLP
Kalyani Roy Pawan Goyal Manish Pandey
236562740
Towards Improving Open Student Answer Assessment using Pretrained Transformers
The transfer learning pretraining finetuning paradigm has revolutionized the natural language processing field yielding state of the art results in several subfields such as text classification and question answering. However, little work has been done investigating pretrained language models for the open student answer assessment task. In this paper, we fine tune pretrained T5, BERT, RoBERTa, DistilBERT, ALBERT and XLNet models on the DT Grade dataset which contains freely generated (or open) student answers together with judgment of their correctness. The experimental results demonstrated the effectiveness of these models based on the transfer learning pretraining finetuning paradigm for open student answer assessment. An improvement of 8% 15% in accuracy was obtained over previous methods. Particularly, a T5 based method led to state of the art results with an accuracy and F1 score of 0.88.
FLAIRS Conference
Nisrine Ait Khayi Vasile Rus Lasang Jimba Tamang
231821641
\cal P^2} A Plan and Pretrain Approach for Knowledge Graph to Text Generation
Text verbalization of knowledge graphs is an important problem with wide application to natural language generation (NLG) systems. It is challenging because the generated text not only needs to be grammatically correct (fluency) but also has to contain the given structured knowledge input (relevance) and meet some other criteria. We develop a planand pretrain approach, P 2 which consists of a relational graph convolutional network (RGCN) planner and the pretrained sequence tosequence (Seq2Seq) model T5. Specifically, the R GCN planner first generates an order of the knowledge graph triplets, corresponding to the order that they will be mentioned in text, and then T5 produces the surface realization of the given plan. In the WebNLG+ 2020 Challenge, our submission ranked in 1st place on all automatic and human evaluation criteria of the English RDF to text generation task.
WEBNLG
Qipeng Guo Zhijing Jin Ning Dai Xipeng Qiu X Xue David P Wipf Zheng Zhang
222208721
TextSETTR: Label Free Text Style Extraction and Tunable Targeted Restyling
We present a novel approach to the challenging problem of label free text style transfer. Unlike previous approaches that use parallel or non parallel labeled data, our technique removes the need for labels entirely, relying instead on the implicit connection in style between adjacent sentences in unlabeled text. We show that T5 (Raffel et al. 2020) a strong pretrained text to text model, can be adapted to extract a style vector from arbitrary text and use this vector to condition the decoder to perform style transfer. As the resulting learned style vector space encodes many facets of textual style, we recast transfers as "targeted restyling" vector operations that adjust specific attributes of the input text while preserving others. When trained over unlabeled Amazon reviews data, our resulting TextSETTR model is competitive on sentiment transfer, even when given only four exemplars of each class. Furthermore, we demonstrate that a single model trained on unlabeled Common Crawl data is capable of transferring along multiple dimensions including dialect, emotiveness, formality, politeness, and sentiment.
ArXiv
Parker Riley Noah Constant Mandy Guo Girish Kumar David C Uthus Zarana Parekh
235621503
TextSETTR: Few Shot Text Style Extraction and Tunable Targeted Restyling
We present a novel approach to the problem of text style transfer. Unlike previous approaches requiring style labeled training data, our method makes use of readily available unlabeled text by relying on the implicit connection in style between adjacent sentences, and uses labeled data only at inference time. We adapt T5 (Raffel et al. 2020) a strong pretrained text to text model, to extract a style vector from text and use it to condition the decoder to perform style transfer. As our labelfree training results in a style vector space encoding many facets of style, we recast transfers as "targeted restyling" vector operations that adjust specific attributes of the input while preserving others. We demonstrate that training on unlabeled Amazon reviews data results in a model that is competitive on sentiment transfer, even compared to models trained fully on labeled data. Furthermore, applying our novel method to a diverse corpus of unlabeled web text results in a single model capable of transferring along multiple dimensions of style (dialect, emotiveness, formality, politeness, sentiment) despite no additional training and using only a handful of exemplars at inference time.
ACL/IJCNLP
Parker Riley Noah Constant Mandy Guo Girish Kumar David C Uthus Zarana Parekh
230104595
P: A Plan and Pretrain Approach for Knowledge Graph to Text Generation
Text verbalization of knowledge graphs is an important problem with wide application to natural language generation (NLG) systems. It is challenging because the generated text not only needs to be grammatically correct (fluency) but also has to contain the given structured knowledge input (relevance) and meet some other criteria. We develop a planand pretrain approach, P, which consists of a relational graph convolutional network (RGCN) planner and the pretrained sequence tosequence (Seq2Seq) model T5. Specifically, the R GCN planner first generates an order of the knowledge graph triplets, corresponding to the order that they will be mentioned in text, and then T5 produces the surface realization of the given plan. In the WebNLG+ 2020 Challenge, our submission ranked in 1st place on all automatic and human evaluation criteria of the English RDF to text generation task.1
Qipeng Guo Zhijing Jin Ning Dai Xipeng Qiu X Xue David P Wipf Zheng Zhang