point out the typora.

This work explores alternatives to the standard pretraining-finetuning paradigm, aiming at more drastic efficiency improvement in lower cost. We propose a simple, efficient, finetune-free framework which understand, generate and represent (GUR) text in zeroshot after unsupervised pretraining. Based on the examples of similar text pairs chosen by their LCS (longest common string) from raw unlabeled documents, our pretraining scheme consists of masked language modeling combined with an unsupervised contrastive learning task. The experiments show that GUR achieves results similar to than PLMs in NLU and NLG tasks as a language model and better than BM25 in recall tasks as a dense retriever in zeroshot. The model architecture is optimized for inference in different sceneries.
