What are the pros and cons of different approaches aiming to find verbalizers for prompting language models?
Verbalizer Design The verbalizer plays a crucial role in prompting as it connects model outputs and labels, significantly influencing performance. (Schick and Schütze, 2021a) design human written verbalizers for prompting, however, they are highly biased towards personal vocabulary with inadequate coverage. Apart from manually designed verbalizers, some recent studies explore automatic verbalizer construction. Auto-L (Gao et al., 2021) uses re-ranking to find the label words set by fine-tuning the model on the candidates searched by RoBERTa; AutoPrompt (Shin et al., 2020) applies gradient-based search to create both prompts and label words automatically with a few trigger examples. But these approaches need to update parameters with gradient descent, which turns out to be infeasible without access to the model weights (e.g., GPT-3). KPT (Han et al., 2021) incorporates external knowledge into the verbalizer in which the unlabeled dataset is needed to refine the label words and thus is not applicable to scenarios where only label names are known. In contrast, our approach NPPrompt directly finds, without any gradient update, relevant words to label names with PLM’s initial word embedding only.
Making Pre-trained Language Models Better Few-shot Learners
The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF—better few-shot fine-tuning of language models—a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.
[{"first":"Tianyu","last":"Gao","middle":[]},{"first":"Adam","last":"Fisch","middle":[]},{"first":"Danqi","last":"Chen","middle":[]}]
PTR: Prompt Tuning with Rules for Text Classification
Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is still challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classification and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical and complicated many-class classification task, and the results show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.
[{"first":"Xu","last":"Han","middle":[]},{"first":"Weilin","last":"Zhao","middle":[]},{"first":"Ning","last":"Ding","middle":[]},{"first":"Zhiyuan","last":"Liu","middle":[]},{"first":"Maosong","last":"Sun","middle":[]}]
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with “task descriptions” in natural language (e.g., Radford et al., 2019). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.
[{"first":"Timo","last":"Schick","middle":[]},{"first":"Hinrich","last":"Schütze","middle":[]}]
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.
[{"first":"Taylor","last":"Shin","middle":[]},{"first":"Yasaman","last":"Razeghi","middle":[]},{"first":"IV","last":"RobertL.Logan","middle":[]},{"first":"Eric","last":"Wallace","middle":[]},{"first":"Sameer","last":"Singh","middle":[]}]
Pre-trained Language Models Can be Fully Zero-Shot Learners
How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, paraphrasing, and multiple-choice question answering. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Our source code is available at https://anonymous.4open. science/r/NPPrompt.
[{"first":"Xuandong","last":"Zhao","middle":[]},{"first":"Siqi","last":"Ouyang","middle":[]},{"first":"Zhiguo","last":"Yu","middle":[]},{"first":"Ming","last":"Wu","middle":[]},{"first":"Lei","last":"Li","middle":[]}]
Making Pre-trained Language Models Better Few-shot Learners
The GPT-3 model (Brown et al., 2020) has made waves in the NLP community by demonstrating astounding few-shot capabilities on myriad language understanding tasks. Given only a natural language prompt and a few demonstrations of the task, GPT-3 is able to make accurate predictions without updating any of the weights of its underlying lan-guage model. However, while remarkable, GPT-3 consists of 175B parameters, which makes it challenging to use in most real-wold applications.
Making Pre-trained Language Models Better Few-shot Learners
In this work, we study a more practical scenario in which we only assume access to a moderatelysized language model such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019), and a small number of examples (i.e., a few-shot setting), which we can use to fine-tune the weights of the language model. This setting is appealing as (1) such models can be trained on typical research hardware;
Making Pre-trained Language Models Better Few-shot Learners
(2) few-shot settings are realistic, as it is generally both easy to acquire a few annotations (e.g., 32 examples) and efficient to train on them; and (3) updating parameters typically leads to better performance. Inspired by GPT-3's findings, we propose several novel strategies for expanding its few-shot learning abilities to our setting, considering both classification and-for the first time-regression.
Making Pre-trained Language Models Better Few-shot Learners
First, we follow the route of prompt-based prediction, first developed by the GPT series (Radford et al., 2018(Radford et al., , 2019Brown et al., 2020) for zero-shot prediction and recently studied by PET (Schick and Schütze, 2021a,b) for fine-tuning. Prompt-based prediction treats the downstream task as a (masked) language modeling problem, where the model directly generates a textual response (referred to as a label word) to a given prompt defined by a taskspecific template (see Figure 1(c)). Finding the right prompts, however, is an art-requiring both domain expertise and an understanding of the language model's inner workings. Even if significant effort is invested, manual prompts are likely to be suboptimal. We address this issue by introducing automatic prompt generation, including a pruned brute-force search to identify the best working label words, and a novel decoding objective to automatically generate templates using the generative T5 model (Raffel et al., 2020)-all of which only require the few-shot training data. This allows us  Figure 1: An illustration of (a) masked language model (MLM) pre-training, (b) standard fine-tuning, and (c) our proposed LM-BFF using prompt-based fine-tuning with demonstrations. The underlined text is the task-specific template, and colored words are label words.
Making Pre-trained Language Models Better Few-shot Learners
to cheaply obtain effective prompts that match or outperform our manually chosen ones. Second, we adopt the idea of incorporating demonstrations as additional context. GPT-3's naive "in-context learning" paradigm picks up to 32 randomly sampled examples, and concatenates them with the input. This method is not guaranteed to prioritize the most informative demonstrations, and mixing random examples from different classes together creates long contexts which can be hard to learn from. Additionally, the number of usable demonstrations is bounded by the model's maximum input length. We develop a more refined strategy, where, for each input, we randomly sample a single example at a time from each class to create multiple, minimal demonstration sets. We also devise a novel sampling strategy that pairs inputs with similar examples, thereby providing the model with more discriminative comparisons.
Making Pre-trained Language Models Better Few-shot Learners
We present a systematic evaluation for analyzing few-shot performance on 8 single-sentence and 7 sentence-pair NLP tasks. We observe that given a small number of training examples, (1) promptbased fine-tuning largely outperforms standard finetuning; (2) our automatic prompt search method matches or outperforms manual prompts; and (3) incorporating demonstrations is effective for finetuning, and boosts few-shot performance. Together, these simple-yet-effective methods contribute towards a dramatic improvement across the tasks we evaluate on, and we obtain gains up to 30% absolute improvement (11% on average) compared to standard fine-tuning. For instance, we find that a RoBERTa-large model achieves around 90% accuracy on most binary sentence classification tasks, while only relying on 32 training examples. We refer to our approach as LM-BFF, better few-shot fine-tuning of language models: a strong, taskagnostic method for few-shot learning.
Making Pre-trained Language Models Better Few-shot Learners
Task formulation. In this work, we assume access to a pre-trained language model L that we wish to fine-tune on a task D with a label space Y. For the task, we only assume K training examples per class 3 for the task's training set D train , such that the total number of examples is K tot = K × |Y|,
Making Pre-trained Language Models Better Few-shot Learners
Our goal is then to develop task-agnostic learning strategies that generalize well to an unseen test set (x test in , y test ) ∼ D test . For model selection and hyper-parameter tuning, we assume a development set D dev , of the same size as the few-shot training set, i.e., |D dev | = |D train |. This distinction is important: using a larger development set confers a significant advantage (see our experiments in Appendix A), and subverts our initial goal of learning from limited data. 4 For all of the following experiments (unless specified otherwise), we take L = RoBERTa-large and K = 16.
Making Pre-trained Language Models Better Few-shot Learners
Evaluation datasets. We conduct a systematic study across 8 single-sentence and 7 sentence-pair English tasks, including 8 tasks from the GLUE benchmark (Wang et al., 2019), SNLI (Bowman et al., 2015), and 6 other popular sentence classification tasks (SST-5, MR, CR, MPQA, Subj, TREC). All of the dataset details are provided in Appendix B. For single-sentence tasks, the goal is to make a prediction based on an input sentence x in = x 1 , such as whether a movie review is positive or not. For sentence-pair tasks, the goal is to take a pair of input sentences x in = (x 1 , x 2 ) and predict the relationship between them. We also interchangeably refer to the inputs as <S 1 > or (<S 1 >, <S 2 >). Note that we mainly use SST-2 and SNLI for pilot experiments and model development, making it close to a true few-shot setting, at least for all the other datasets we evaluate on.
Making Pre-trained Language Models Better Few-shot Learners
Evaluation protocol. Systematically evaluating few-shot performance can be tricky. It is wellknown that fine-tuning on small datasets can suffer from instability (Dodge et al., 2020;Zhang et al., 2021), and results may change dramatically given a new split of data. To account for this, we measure average performance across 5 different randomly sampled D train and D dev splits. This issue has also been discussed in Schick and Schütze (2021b)they suggest using a fixed set of training examples. We argue that sampling multiple splits gives a more robust measure of performance, and a better estimate of the variance. We also observe that hyperparameters can make a significant difference, thus we sweep multiple hyper-parameters for each data sample, and take the best setting as measured on the D dev of that sample (see Appendix C.1).
Making Pre-trained Language Models Better Few-shot Learners
Given a masked language model L, we first convert input x in to a token sequencex, and the language model L then mapsx to a sequence of hidden vectors {h k ∈ R d }.  , and W o ∈ R |Y|×d is a set of randomly initialized parameters introduced at the start of fine-tuning. Similarly, for a regression task, we can introduce w o ∈ R d and optimize the mean squared error between w o ·h [CLS] and the gold label. In either case, the number of new parameters can be substantialfor example, a simple binary classification task will introduce 2,048 new parameters for a RoBERTalarge model-making it challenging to learn from a small amount of annotated data (e.g., 32 examples). An alternative approach to solving this problem is prompt-based fine-tuning, in which L is directly tasked with "auto-completing" natural language prompts. For instance, we can formulate a binary sentiment classification task using a prompt with input x 1 (e.g., "No reason to watch it .") as:
Making Pre-trained Language Models Better Few-shot Learners
and let L decide whether it is more appropriate to fill in "great" (positive) or "terrible" (negative) for [MASK]. We now formalize this approach for classification and regression ( §4.1 and §4.2), and discuss the importance of prompt selection ( §4.3).
Making Pre-trained Language Models Better Few-shot Learners
Let M : Y → V be a mapping from the task label space to individual words 5 in the vocabulary 5 More generally, we can consider a one-to-many mapping M : Y → 2 |Y| in which we map labels to sets of words. However, we did not find significant gains in our experiments.
Making Pre-trained Language Models Better Few-shot Learners
V of L. Then for each x in , let the manipulation x prompt = T (x in ) be a masked language modeling (MLM) input which contains one [MASK] token. In this way, we can treat our task as an MLM, and model the probability of predicting class y ∈ Y as:
Making Pre-trained Language Models Better Few-shot Learners
where h [MASK] is the hidden vector of [MASK] and w v denotes the pre-softmax vector corresponding to v ∈ V. When supervised examples {(x in , y)} are available, L can be fine-tuned to minimize the cross-entropy loss. It is important to note that this approach re-uses the pre-trained weights w v and does not introduce any new parameters. It also reduces the gap between pre-training and fine-tuning, making it more effective in few-shot scenarios.
Making Pre-trained Language Models Better Few-shot Learners
We assume the same basic setup as in classification, but treat the label space Y as a bounded interval [v l , v u ]. Inspired by Mettes et al. (2019), we model the problem as an interpolation between two opposing poles, {y l , y u }, with values v l and v u respectively. For instance, we can formulate our previous sentiment analysis task as a regression problem in the range [0, 1], where we slide between "terrible" (v l = 0) and "great" (v u = 1). In this way, we can express y as a mixture model:
Making Pre-trained Language Models Better Few-shot Learners
where p(y u | x in ) is the probability of y u , and p(y l | x in ) = 1 − p(y u | x in ). Then we define  M : {y l , y u } → V, and model p(y u | x in ) the same as Eq. (1). We fine-tune L to minimize the KL-divergence between the inferred p(y u | x in ) and the observed mixture weight,
Making Pre-trained Language Models Better Few-shot Learners
The key challenge is to construct the template T and label words M(Y)-we refer to these two together as a prompt P. Previous works (Schick and Schütze, 2021a,b) hand-craft both the templates and label words, which usually requires domain expertise and trial-and-error. Table 1 summarizes manual templates and label words chosen for each dataset in our experiments. These templates and label words were designed by intuition, and by considering formats used in previous literature.
Making Pre-trained Language Models Better Few-shot Learners
To better understand what constitutes a good template or label word, we conduct a pilot study on SST-2 and SNLI. Table 2 shows that different prompts can lead to substantial differences in final accuracy. Specifically, when a template is fixed, the better the label words match the "semantic classes", the better the final accuracy is (great/terrible > good/bad > cat/dog). In extreme cases where we swap plausible label words (e.g., terrible/great), we achieve the worst overall performance. 6 Furthermore, with the same set of label words, even a small change in the template can make a difference. For example, for SNLI, if we put [MASK] at the end, or swap sentence order, we observe a >10% drop. The above evidence clearly underlines the importance of selecting good templates and label words. Searching for prompts, however, is hard, as the search space can be very large-especially for the template. Even worse, we only have a few examples to use to guide our search, which can easily overfit. We will address these issues next.
Making Pre-trained Language Models Better Few-shot Learners
We now explore principled ways of automating the search process for label words ( §5.1) and templates ( §5.2). Our goals are to reduce the human involvement required to design prompts, and to find more optimal settings than those that we manually choose. Here, we assume a classification task, but the process for regression is analogous.
Making Pre-trained Language Models Better Few-shot Learners
We first study how to construct a label word mapping M that maximizes accuracy on D dev after fine-tuning, given a fixed template T . Naively searching all possible assignments, however, is (1) generally intractable, as the search space is exponential in the number of classes; and (2) prone to overfitting, as we will tend to uncover spurious correlations given only a few annotations. As a simple solution, for each class c ∈ Y, we construct a pruned set V c ⊂ V of the top k vocabulary words based on their conditional likelihood using the initial L. That is, let D c train ⊂ D train be the subset of all examples of class c. We take V c as
Making Pre-trained Language Models Better Few-shot Learners
where P L denotes the output probability distribution of L. To further narrow down the search space, we find the top n assignments over the pruned space that maximize zero-shot accuracy on D train (both n and k are hyper-parameters, see Appendix C.2). Then we fine-tune all top n assignments, and rerank to find the best one using D dev . This approach is similar to the automatic verbalizer search methods in Schick and Schütze (2021a); Schick et al. (2020), except that we use a much simpler search process (brute-force) and also apply re-rankingwhich we find to be quite helpful.
Making Pre-trained Language Models Better Few-shot Learners
Next, we study how to generate a diverse set of templates {T } automatically from a fixed set of label words M(Y). To address this challenging problem, we propose to use T5 (Raffel et al., 2020),  a large pre-trained text-to-text Transformer. T5 is pre-trained to fill in missing spans (replaced by T5 mask tokens, e.g., <X> or <Y>) in its input. For example, given the input "Thank you <X> me to your party <Y> week", T5 is trained to generate "<X> for inviting <Y> last <Z>", meaning that "for inviting" is the replacement for <X> and "last" is the replacement for <Y>. This is well suited for prompt generation: we can simply take input sentences from D train and let the T5 model construct the template T , without having to specify a predefined number of tokens for it.
Making Pre-trained Language Models Better Few-shot Learners
Given an input example (x in , y) ∈ D train , we consider the following simple conversions, denoted as T g (x in , y), for formulating the T5 model inputs: 7
Making Pre-trained Language Models Better Few-shot Learners
As shown in Figure 2, we rely on the T5 model to fill in the placeholders. When decoding, our goal here is to find an output that can work well for all examples in D train , i.e., the output template T that maximizes (x in ,y)∈D train log P T5 (T | T g (x in , y)), where P T5 denotes the output probability distribution of T5. It can be decomposed according to:
Making Pre-trained Language Models Better Few-shot Learners
where (t 1 , . . . , t |T | ) are the template tokens.
Making Pre-trained Language Models Better Few-shot Learners
We use beam search to decode multiple template candidates. Concretely, we use a wide beam width (e.g., 100) to cheaply obtain a large set of diverse templates. We then fine-tune each generated template on D train and use D dev to either pick the single template with the best performance (Table 3), or the top k templates to use as an ensemble (Table 4). Though it might appear to be expensive to fine-tune the model on each individual template, this is fast in practice due to the small size of D train , and is also fully automated: making it easy to use, compared to manually tuning prompts for each dataset.
Making Pre-trained Language Models Better Few-shot Learners
In this section, we study whether we can leverage demonstrations when fine-tuning medium-sized LMs, and find better ways to exploit them.
Making Pre-trained Language Models Better Few-shot Learners
GPT-3's naive approach to in-context learning simply involves concatenating the input with up to 32 examples randomly drawn from the training set. This approach is suboptimal as (1) the number of available demonstrations is bounded by the model's maximum input length; 8 and (2) mixing numerous random examples from different classes together creates extremely long contexts which can be hard to leverage, especially for a smaller model. To address these issues, we propose a simpler solution: at each training step, we randomly sample one 9 example x  (Figure 1(c)):
Making Pre-trained Language Models Better Few-shot Learners
Here ⊕ denotes concatenation of input sequences. During both training and inference we sample multiple demonstration sets for each x in . Note that both x in and demonstration examples are sampled from the same set D train during training. At testing time, we still sample demonstration sets from D train and ensemble predictions across all sets.
Making Pre-trained Language Models Better Few-shot Learners
We observe that controlling the construction of in are all dramatically different-from each other, or from the query x in -then it becomes challenging for the language model to decipher meaningful patterns. As a result, the model may simply ignore   (Reimers and Gurevych, 2019) model to obtain embeddings for all input sentences (for sentence-pair tasks, we use the concatenation of the two sentences). Here we just feed the raw sentences without the templates into SBERT. For each query x in and each label c ∈ Y, we sort all training instances with the label x ∈ D c train by their similarity score to the query cos(e(x in ), e(x)), and only sample from the top r = 50% instances for each class to use as demonstrations.
Making Pre-trained Language Models Better Few-shot Learners
We present our main results, and address several research questions pertaining to our LM-BFF approach. Implementation details are in Appendix C.
Making Pre-trained Language Models Better Few-shot Learners
We use a RoBERTa-large model and set K = 16 in our experiments. A comparison of using RoBERTa vs BERT can be found in Appendix D. For automatic prompt search, in our main table we report automatic template search only (which consistently performs the best, see Table 5). To put our results in perspective, we compare to a number of baselines, namely (1) standard fine-tuning in our few-shot setting; (2) standard fine-tuning using the full training set; (3) simply taking the most frequent class (measured on the full training set); (4) prompt-based zero-shot prediction where we take our manual prompts and use L "out-of-thebox" without using any training examples; and (5) "GPT-3" in-context learning, where we use the same prompt-based zero-shot setting, but augment the context with randomly sampled 32 demonstrations (and still use RoBERTa-large, not GPT-3).
Making Pre-trained Language Models Better Few-shot Learners
Single-prompt results. Table 3 shows our main results using a single prompt, either from our manually designed ones (Table 1) , or the best generated ones. First, prompt-based zero-shot prediction achieves much better performance than the majority class, showing the pre-encoded knowledge in RoBERTa. Also, "GPT-3" in-context learning does not always improve over zero-shot prediction, likely because smaller language models are not expressive enough to use off-the-shelf like GPT-3.   Table 5: Comparison between manual prompts and different automatic prompt generation methods: autogenerated templates (Auto T), auto-generated label words (Auto L), and their combination (Auto T + L).
Making Pre-trained Language Models Better Few-shot Learners
Second, prompt-based fine-tuning can greatly outperform standard fine-tuning, both when using a manual prompt or a generated one. CoLA is one interesting exception, as the input may be a nongrammatical sentence which is out of the distribution of L. Generally, our automatically searched templates can achieve comparable or even higher results than manual ones, especially for tasks in which constructing strong manual templates is less intuitive (e.g., TREC, QNLI and MRPC).
Making Pre-trained Language Models Better Few-shot Learners
Finally, using demonstrations in context leads to consistent gains in a majority of tasks. In summary, our combined solution-fine-tuning with automatically searched templates and sampled demonstration sets-achieves a 30% gain on SNLI compared to standard fine-tuning, and 11% gain on average.
Making Pre-trained Language Models Better Few-shot Learners
Ensemble results. An advantage of automatic prompt search is that we can generate as many prompts as we want, train individual models, and create large ensembles. PET (Schick and Schütze, 2021a,b) also ensembles multiple models trained with manual prompts. 10 In Table 4, we make a direct comparison of our searched prompts and PET's manual prompts on MNLI and RTE (two 10 They then use unlabeled data and distillation to get a single model, which is outside of our scope.
Making Pre-trained Language Models Better Few-shot Learners
SST-2 (positive/negative)
Making Pre-trained Language Models Better Few-shot Learners
Auto L T (x in ) = <S 1 > It was [MASK]. #1. irresistible/pathetic #2. wonderful/bad #3. delicious/bad
Making Pre-trained Language Models Better Few-shot Learners
, <S 2 > #1. Alright/Watch/Except #2. Hi/Watch/Worse #3. Regardless/Fortunately/Unless Table 6: Examples of our automatically generated templates (Auto T) and label words (Auto L).
Making Pre-trained Language Models Better Few-shot Learners
datasets that we evaluate in common). 11 As the results show, an ensemble with multiple templates always improves performance. An ensemble of the same number of automatic templates achieves comparable or better performance than the ensemble of PET's manual prompts. Increasing the number of automatic templates brings further gains. Table 5 gives the results of using manual vs automatic prompts. For automatic prompts, we compare template search (Auto T), label word search (Auto L), and a joint variant (Auto T + L) in which we start from manual label words, apply Auto T, and then Auto L. In most cases, Auto T achieves comparable or higher performance than manual ones, and is consistently the best variant. Auto L outperforms manual prompts on TREC and MRPC-but is considerably worse on SNLI. Auto T + L is often better than Auto L, but only sometimes better than Auto T. Table 6 shows examples from Auto T and Auto L (A full list in Appendix E). Auto T templates generally fit the context and label words well, but can contain biased peculiarities (e.g., "{Yes/No}, no" in SNLI). For Auto L words, things are mixed: while most look intuitively reasonable, there are also some mysterious abnormalities (e.g., "Hi" for the "entailment" class in SNLI).   Table 7 compares the performance of demonstrations using uniform sampling to selective sampling by SBERT. We acknowledge that SBERT is trained on SNLI and MNLI datasets, thus we also tried a simple sentence encoder using mean pooling of hidden representations from RoBERTa-large. We find that in either case, using selective sampling outperforms uniform sampling, highlighting the importance of sampling similar examples for incorporating demonstrations in context. Figure 3 illustrates how standard fine-tuning and our LM-BFF compare as K increases. For a simple task such as SST-2 (also see MR, CR and MPQA in Table 3), despite using only 32 total examples, LM-BFF has already nearly saturated its performance and is comparable to standard fine-tuning over the entire dataset. On the harder task of SNLI, LM-BFF continues to improve as K increases while still maintaining a performance gap over standard finetuning, until the two converge around K = 256.
Making Pre-trained Language Models Better Few-shot Learners
Reformulating NLP tasks as MLM has exciting implications for few-shot learning, but also has limitations. First, while LM-BFF greatly outperforms standard fine-tuning, Table 3 shows that, overall, the performance still substantially lags behind finetuning with thousands of examples, especially for harder tasks. Additionally, just like standard finetuning, our results also suffer from high variance. As described in §2, several recent studies have tried to counter instability in few-shot fine-tuning and we expect these methods to also help here.
Making Pre-trained Language Models Better Few-shot Learners
With respect to automatic prompt generation, despite its effectiveness, we still find it practically challenging to expand the search space, or generalize well based on only approximately 32 examples. This is partly due to our lingering reliance on some manual design-either manual templates (for label word search) or manual label words (for template search), which allows us to get our search off the ground, but does also bias it towards areas of the search space that we might have already imagined.
Making Pre-trained Language Models Better Few-shot Learners
Finally, it is important to clarify that LM-BFF favors certain tasks which (1) can be naturally posed as a "fill-in-the-blank" problem; (2) have relatively short input sequences; and (3) do not contain many output classes. Issues (2) and (3) might be ameliorated with longer-context language models (e.g., Beltagy et al., 2020). For tasks that are not straightforward to formulate in prompting, such as structured prediction, issue (1) is more fundamental. We leave it as an open question for future work.
Making Pre-trained Language Models Better Few-shot Learners
In this paper we presented LM-BFF, a set of simple but effective techniques for fine-tuning language models using only a few examples. Our approach proposes to (1) use prompt-based finetuning with automatically searched prompts; and (2) include selected task demonstrations (training examples) as part of the input context. We show that our method outperforms vanilla fine-tuning by up to 30% (and 11% on average). We concluded by discussing the limitations of our approach, and posed open questions for future study.
Making Pre-trained Language Models Better Few-shot Learners
A Impact of Development Sets Table A.1 shows how the size of the development sets can affect the final performance of the model. For "No D dev ", we take the same hyper-parameters from Schick and Schütze (2021a,b): batch size = 16, learning rate = 1e-5 and training steps = 250. We also experiment with a variant that we sample a development set of 10 times larger than the training set. We can see that using larger development sets leads to better performance, and this is why we stick to |D train | = |D dev | in our few-shot setting. 
Making Pre-trained Language Models Better Few-shot Learners
For SNLI (Bowman et al., 2015) and datasets from GLUE (Wang et al., 2019), including SST-2 (Socher et al., 2013), CoLA (Warstadt et al., 2019), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005;Bar Haim et al., 2006;Giampiccolo et al., 2007;Bentivogli et al., 2009), MRPC (Dolan andBrockett, 2005), QQP 12 and STS-B (Cer et al., 2017), we follow Zhang et al. (2021) and use their original development sets for testing. For datasets which require a cross-validation evaluation-MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), MPQA (Wiebe et al., 2005), Subj (Pang and Lee, 2004)-we simply randomly sample 2,000 examples as the testing set and leave them out from training. For SST-5 (Socher et al., 2013) and TREC (Voorhees and Tice, 2000), we use their official test sets. We show dataset statistics in Table B.1.
Making Pre-trained Language Models Better Few-shot Learners
For grid search, we take learning rates from {1e-5, 2e-5, 5e-5} and batch sizes from {2, 4, 8}. These 12 https://www.quora.com/q/quoradata/ numbers are picked by pilot experiments on the SST-2 and SNLI datasets. We also use early stopping to avoid overfitting. For each trial, we train the model for 1,000 steps, validate the performance every 100 steps, and take the best checkpoint. Table 1 shows all the manual templates and label words we use in experiment. For automatically template generation, we take the T5-3B 13 model, which is the largest publicly available one that can fit on a single GPU. For automatically searching label words, we set k to 100 for all tasks except SST-5 and TREC. For SST-5 we set a smaller k = 30, as it is a 5-way classification task. For TREC, we observe that filtering V c using conditional likelihood alone is still noisy, thus we set k = 1000, and then re-rank V c by the nearest neighbors of the original manual label words and take the top 30 per class. We set n to 100 in all experiments. Due to the large number of trials in automatic search, we take a fixed set of hyper-parameters in this part: batch size of 8 and learning rate of 1e-5.
Making Pre-trained Language Models Better Few-shot Learners
Since the idea of prompt-based fine-tuning is to make the input and output distribution close to the pre-training, the implementation details are crucial. For templates, we put extra space before sentences if it is not at the beginning of the input. Also, we lowercase the first letter of the sentence if it is concatenated with a prefix (e.g., <S 2 > in Table 1). Also if one sentence is appended any punctuation (e.g., <S 1 > in Table 1), then the last character of the original sentence is discarded. Finally, we prepend a space for label words in M(Y). For example, we use " great" instead of "great" in the RoBERTa vocabulary, where " " stands for space.
Making Pre-trained Language Models Better Few-shot Learners
When using demonstrations, we sample 16 different sets of demonstrations for each input and average the predicted log probability for each class during inference. We find that further increasing the number of samples does not bring substantial improvement. Additional, we have tried different aggregation methods like taking the result with the maximum confidence and we did not find a meaningful improvement. For selective demonstrations, we take roberta-large-nli-stsb   Table D.1 compares the results of BERT-large (uncased) and RoBERTa-large in our settings. Pretrained BERT provides two segment embeddings (A/B) for different parts of input. The common practice, when fine-tuning BERT, is that using only segment A for single-sentence tasks, and using segment A/B for the two sentences in sentence-pair tasks. In our case of incorporating demonstrations, however, we have more than two sentences. Thus we explore the following different strategies for segments: (1) using the A segment for all sentences 14 https://github.com/UKPLab/ sentence-transformers (1-seg); (2) using the A segment for the original input and the B segment for the demonstrations (2-seg); (3) using different segment embeddings for each sentence (n-seg), e.g., for SNLI, we use different segments for each premise and hypothesis in both the original input and the demonstrations, which leads to a total number of 8 segment embeddings. This introduces new segment embeddings (randomly initialized and learned during fine-tuning) as the pre-trained BERT only has two. Table D.1 shows that prompt-based fine-tuning with demonstrations also works for BERT, and 2seg works the best when incorporating demonstrations. Still, we take RoBERTa-large as our main model, for RoBERTa performs much better than BERT and RoBERTa saves the trouble to tune the usage of segment embeddings.
Making Pre-trained Language Models Better Few-shot Learners
We demonstrate the top 3 automatically generated templates and label words for all tasks in Table E.1. In general, most automatic templates are reasonable and grammatically correct. For the label words, the generated results look intuitive for most single sentence tasks. For other tasks, the automatic ones can be counterintuitive in some cases. It is still unclear why the language model picks these words and sometimes they actually work well. We leave this for future study. 
PTR: Prompt Tuning with Rules for Text Classification
Recently, pre-trained language models (PLMs) such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019), RoBERTa , and T5 (Raffel et al., 2020) have emerged as a powerful instrument for language understanding and generation. By applying self-supervised learning, PLMs can capture a wealth of linguistic (Jawahar et al., 2019), semantic (Yenicelik et al., 2020), syntactic (Hewitt and Manning, 2019), and world * Corresponding author: Z. Liu(liuzy@tsinghua.edu.cn) knowledge (Petroni et al., 2019) from large-scale corpora, which is the key to the success of PLMs. By fine-tuning these PLMs with additional taskspecific data, rich knowledge distributed in PLMs can be adapted to various downstream tasks. In the past few years, PLM fine-tuning has shown awesome performance on almost all important NLP tasks. It is now a consensus of the NLP community to fine-tune PLMs for specific tasks instead of learning models from scratch (Qiu et al., 2020).
PTR: Prompt Tuning with Rules for Text Classification
Despite the success of fine-tuning PLMs, some recent studies find one of its critical challenges is the significant gap of objective forms in pretraining and fine-tuning, which restricts taking full advantage of knowledge in PLMs. Take Figure 1 for example, pre-training is usually formalized as a cloze-style task to predict target words (e.g. sequential language models and masked language models), yet downstream tasks in fine-tuning may exhibit different objective forms such as classification, generation, and sequence labeling. This gap will hinder the transfer and adaptation of knowledge in PLMs to downstream tasks.
PTR: Prompt Tuning with Rules for Text Classification
To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020;Schick et al., 2020). As shown in Figure 1, a typical prompt consists of a template (e.g. "<S 1 > It was [MASK].") and a set of label words (e.g. "great" and "terrible"). The set of label words serves as the candidate set for predicting [MASK]. By fusing the original input with the prompt template for predicting [MASK] and then mapping predicted words to corresponding labels, prompt tuning can convert a binary sentiment classification task into a cloze-style task. In other words, by checking that PLMs predict "great" or "terrible" at the masked position, we can determine whether <S 1 > is positive or negative. Besides sentiment classification, existing prompt tuning methods have also achieved promising results on some other few-class arXiv:2105.11259v3 [cs.CL] 15 Sep 2021
PTR: Prompt Tuning with Rules for Text Classification
Input Template
PTR: Prompt Tuning with Rules for Text Classification
[CLS] I would highly recommend this .
PTR: Prompt Tuning with Rules for Text Classification
[CLS] I like this . classification tasks such as natural language inference (Schick and Schütze, 2021;Liu et al., 2021). For instance, by using "<S 1 > .
PTR: Prompt Tuning with Rules for Text Classification
[MASK] , <S 2 >" as the prompt template, and {"yes", "maybe", "no"} as the set of label words, we can also easily use PLMs to predict the discourse relation (entailment, neutral, or contradiction) between <S 1 > and <S 2 >.
PTR: Prompt Tuning with Rules for Text Classification
However, for those tasks with many classes, it becomes challenging to manually find appropriate templates and suitable label words to distinguish different classes. For example, relation classification, a typical many-class classification task, requires models to predict semantic relations between two marked entities in the text. Given the relation "person:parent" and the relation "organization:parent", it is hard to pick label words to distinguish them. A straightforward solution is automatically generating prompts: Schick et al. (2020); Schick and Schütze (2021) automatically identify label words from the vocabulary for human-picked templates; Shin et al. (2020) utilize gradient-guided search to automatically generate both templates and label words;  propose to generate all prompt candidates and apply development sets to find the most effective ones. Although autogenerated prompts can avoid intensive labor, it is not surprising that most auto-generated prompts cannot achieve comparable performance to humanpicked ones. Meanwhile, auto-generated prompts require extra computation costs for generation and verification. The expensive extra computation costs make auto-generated prompts more suitable for few-shot learning, rather than those normal learning settings with many instances and classes.
PTR: Prompt Tuning with Rules for Text Classification
In this paper, we propose prompt tuning with rules (PTR) for many-class classification tasks. For a many-class classification task, we manually design essential sub-prompts and apply logic rules to compose sub-prompts into final task-specific prompts. Compared with existing prompt tuning methods, PTR has two advantages:
PTR: Prompt Tuning with Rules for Text Classification
(1) Prior Knowledge Encoding. PTR can apply logic rules to encode prior knowledge about tasks and classes into prompt tuning. We also take relation classification as an example, the predicted results are usually correlated to both relational semantics of sentences and types of entities. We can build the prompt for the relation "person:parent" and the relation"organization:parent" with two subprompts, one sub-prompt is to determine whether marked entities are human beings or organizations, and the other sub-prompt is to determine whether sentences express the semantics of the parent-child relationship.
PTR: Prompt Tuning with Rules for Text Classification
(2) Efficient Prompt Design. Compared with manually designing templates and individual label words for all classes, it is easier to design several simple sub-prompts and then combine these subprompts to form task-specific prompts according to logic rules. Moreover, compared with automati-cally generating prompts based on machine learning models, using logic rules to generate prompts is more efficient and interpretable.
PTR: Prompt Tuning with Rules for Text Classification
To verify the effectiveness of PTR, we conduct experiments on relation classification, a typical and complicated many-class classification task. Four popular benchmarks of relation classification are used for our experiments, including TA-CRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021), and Se-mEval 2010 Task 8 (Hendrickx et al., 2010). Sufficient experimental results show that PTR can significantly and consistently outperform existing stateof-the-art baselines.
PTR: Prompt Tuning with Rules for Text Classification
Before introducing PTR, we first give some essential preliminaries, especially some details about prompt tuning for classification tasks. Formally, a classification task can be denoted as T = {X , Y}, where X is the instance set and Y is the class set. For each instance x ∈ X , it consists of several tokens x = {w 1 , w 2 , . . . , w |x| }, and is annotated with a label y x ∈ Y.
PTR: Prompt Tuning with Rules for Text Classification
Given a PLM M, vanilla fine-tuning first converts the instance x = {w 1 , w 2 , . . . , w |x| } to the input sequence {[CLS], w 1 , w 2 , . . . , w |x| , [SEP]}, and then uses M to encode all tokens of the input sequence into corresponding vectors
PTR: Prompt Tuning with Rules for Text Classification
For a downstream classification task, a task-specific head is used to compute the probability distribution over the class set Y with the softmax function p(·|x) = Softmax(Wh [CLS] 
PTR: Prompt Tuning with Rules for Text Classification
is the hidden vector of [CLS], b is a learnable bias vector, and W is a learnable matrix randomly initialized before fine-tuning. The parameters of M, b, and W are tuned to maximize 1 |X | x∈X log p(y x |x).
PTR: Prompt Tuning with Rules for Text Classification
To apply cloze-style tasks to tune PLMs, prompt tuning has been proposed. Formally, a prompt consists of a template T (·) and a set of label words V. For each instance x, the template is first used to map x to the prompt input x prompt = T (x). The template defines where each token of x is placed and whether to add any additional tokens. Besides retaining the original tokens in x, at least one [MASK] is placed into x prompt for M to fill label words. For instance, for a binary sentiment classification task, we set a template T (·) = " · It was [MASK].", and map x to x prompt = "x It was [MASK].".
PTR: Prompt Tuning with Rules for Text Classification
By input x prompt into M, we compute the hidden vector h [MASK] of [MASK]. Given v ∈ V, we produce the probability that the token v can fill the masked position p([MASK] = v|x prompt ) = exp(v·h [MASK] ) ṽ∈V exp(ṽ·h [MASK] ) , where v is the embedding of the token v in the PLM M.
PTR: Prompt Tuning with Rules for Text Classification
There also exists an injective mapping function φ : Y → V that bridges the set of classes and the set of label words. In some papers, the function φ is named "verbalizer". With the verbalizer φ, we can formalize the probability distribution over Y with the probability distribution over V at the masked position, i.e. p(y|x) = p([MASK] = φ(y)|x prompt ). Here, we also take a binary sentiment classification task as an example. We map the positive sentiment to "great" and the negative sentiment to "terrible". According to M fills the masked position of x prompt with "great" or "terrible", we can know the instance x is positive or negative.
PTR: Prompt Tuning with Rules for Text Classification
For prompt tuning, with the template T (·), the set of label words V, and the verbalizer φ, the learning objective is to maximize Figure 1, we show pre-training, fine-tuning, and prompt tuning, which can indicate the connections and differences among them clearly.
PTR: Prompt Tuning with Rules for Text Classification
As we mentioned before, prompt tuning is well developed for few-class classification tasks such as sentiment classification and natural language inference. Considering the difficulty of designing prompts for many-class classification tasks, we propose PTR to incorporate logic rules to compose task-specific prompts with several simple subprompts. In this section, we will introduce the details of PTR, and select relation classification as an example to illustrate how to adapt PTR for many-class classification tasks.
PTR: Prompt Tuning with Rules for Text Classification
PTR is driven by basic human logic inferences. For example, in relation classification, if we wonder whether two marked entities in a sentence have the relation "person:parent", with prior knowledge, we would check whether the sentence and two 
PTR: Prompt Tuning with Rules for Text Classification
Logic Rules Logic Rules Figure 2: Illustration of PTR. By composing the sub-prompts of the conditional functions f es (·, ·), f eo (·, ·) and f es,eo (·, ·, ·), we could easily get an effective prompt to distinguish "person:parent" and "organization:parent". marked entities meet certain conditions: (1) The two marked entities are human beings; (2) the sentence indicates the parental semantics between the two marked entities.
PTR: Prompt Tuning with Rules for Text Classification
Inspired by this, for any text classification task T = {X , Y}, we design a conditional function set F. Each conditional function f ∈ F determines whether the function input meets certain conditions. For instance, the conditional function f (x, person) can determine whether the input x is a person, the conditional function f (x, 's parent was, y) can determine whether y is the parent of x. Intuitively, these conditional functions are essentially the predicates of first-order logic.
PTR: Prompt Tuning with Rules for Text Classification
For each conditional function f ∈ F, PTR set a template T f (·) and a set of label words V f to build its sub-prompt. According to the semantics of Y, we can use logic rules to transform the classification task into the calculation of a series of conditional functions. As shown in Figure 2, for relation classification, determining whether the entities x and y have the relation "person:parent" can be formalized as
PTR: Prompt Tuning with Rules for Text Classification
where f es (·, ·) is the conditional function to determine the subject entity types, f eo (·, ·) is the conditional function to determine the object entity types, and f es,eo (·, ·, ·) is the conditional function to determine the semantic connection between entities. Similarly, determining the relation "organization:parent" can be formalized as
PTR: Prompt Tuning with Rules for Text Classification
Based on the above-mentioned logic rules and conditional functions, we can compose the subprompts of rule-related conditional functions to handle the task T . Next, we will introduce how to design sub-prompts for conditional functions, as well as how to aggregate sub-prompts for tasks.
PTR: Prompt Tuning with Rules for Text Classification
For each conditional function f ∈ F, we manually design a sub-prompt for it. Similar to the conventional prompt setting, a sub-prompt also consists of a template and a set of label words.
PTR: Prompt Tuning with Rules for Text Classification
The basic conditional function is a unary function. For binary sentiment classification, unary functions can determine whether a sentence is positive or negative; For entity typing, unary functions can indicate entity types; For topic labeling, unary functions can be used to predict article topics.
PTR: Prompt Tuning with Rules for Text Classification
Here we still take relation classification as an example.
PTR: Prompt Tuning with Rules for Text Classification
Given a sentence x = {. . . e s . . . e o . . .}, where e s and e o are the subject entity and the object entity respectively, we set f es (·, {person|organization| . . .}) to determine the type of e s . The sub-prompt template and label word set of f es can be formalized as
PTR: Prompt Tuning with Rules for Text Classification
(1)
PTR: Prompt Tuning with Rules for Text Classification
For the object entity e o , the sub-prompt template and label word set of f eo are similar to Eq. (1). Another important type of conditional functions is a binary function. For natural language inference, binary functions can determine the relation between two sentences is entailment, neutral, or contradiction; For relation classification, such functions can be used to determine the complex connections between two entities. If we set f es,eo (·, {'s parent was|was born in| . . .}, ·), the sub-prompt template and label word set of f es,eo can be formalized as
PTR: Prompt Tuning with Rules for Text Classification
V fe s,eo = {"'s parent was", "was born in", . . .}.
PTR: Prompt Tuning with Rules for Text Classification
( 2) Normally, unary and binary functions are sufficient to compose logic rules for classification tasks. For classification tasks with complex semantics, these design methods of unary and binary functions can be extended to multi-variable functions, which can provide more powerful sub-prompts.
PTR: Prompt Tuning with Rules for Text Classification
Since we transform a classification task into the calculation of a series of conditional functions, we need to compose sub-prompts of various conditional functions into a complete task-specific prompt. In this paper, we apply a simple strategy: we use logic rules with conjunctive normal form and then directly concatenate the sub-prompts of all rule-related functions.
PTR: Prompt Tuning with Rules for Text Classification
For instance, in Figure 2, by aggregating the sub-prompts in Eq. (1) and Eq. (2), the complete prompt template is as follows,
PTR: Prompt Tuning with Rules for Text Classification
where [·; ·; ·] is the aggregation function of subprompts. And the aggregated sets of label words are given as 
PTR: Prompt Tuning with Rules for Text Classification
As shown in Figure 2, we can also add some learnable tokens, whose parameters are randomly initialized, to make the template more effective.  As the aggregated template may contain multiple [MASK] tokens, we must consider all masked positions to make predictions, i.e.,
PTR: Prompt Tuning with Rules for Text Classification
where n is the number of masked positions in T (x), and φ j (y) is to map the class y to the set of label words V [MASK] j for the j-th masked position [MASK] j . Eq. (5) can be used to tune PLMs and classify classes. The final learning objective of PTR is to maximize
PTR: Prompt Tuning with Rules for Text Classification
We conduct experiments on relation classification to show the effectiveness of PTR on many-class classification tasks.
PTR: Prompt Tuning with Rules for Text Classification
We carry out our experiments on four relaction classification datasets: TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021), and SemEval 2010 Task 8 (Se-mEval) (Hendrickx et al., 2010). TACRED: one of the largest and most widely used datasets for relation classification. It is obtained via crowd-sourcing and contains 42 relation types (including "no_relation").
PTR: Prompt Tuning with Rules for Text Classification
TACREV: one dataset built based on the original TACRED dataset. They find out and correct the errors in the original development set and test set of TACRED, while the training set is left intact.
PTR: Prompt Tuning with Rules for Text Classification
ReTACRED: another version of TACRED dataset. They address some shortcomings of the original TACRED dataset, refactor its training set, development set, and test set. ReTACRED also modifies a few relation types.
PTR: Prompt Tuning with Rules for Text Classification
SemEval: a traditional dataset in relation classification covering 9 relations with two directions and one special relation "Other".   More details of these datasets are shown in Table 1. For all the above datasets, we use F 1 scores as the main metric for evaluation.
PTR: Prompt Tuning with Rules for Text Classification
Our model is implemented based on the toolkit Transformers (Wolf et al., 2020) and Open-NRE (Han et al., 2019). Most hyper-parameters are set following previous works: our model is optimized with Adam using the learning rate of 3e − 5 on ROBERTA_LARGE, with a linear warmup for the first 10% steps. The weight decay is set to 1e − 2. For all datasets, we fine-tune our model for 5 epochs with the batch size 64. The best model checkpoint is selected based on the performance on the development set. In Table 2 and Table 3, we show some details of relation-specific prompts in TACRED, TACREV, ReTACRED, and SemEval.
PTR: Prompt Tuning with Rules for Text Classification
In TACRED, reversing some relations, such as reversing (e s ,"organization: member_of", e o ) into (e o ,"organization: member", e s ), can lead to better prompts. Hence, in our experiments, we apply two settings for PTR, one is to build prompts without changing any relations, and the other is to build prompts after reversing a part of relations. For models tuning with some reversed relations, we name them ". . . (Reversed)".
PTR: Prompt Tuning with Rules for Text Classification
In this part, we compare PTR with several typical models for relation classification, including:
PTR: Prompt Tuning with Rules for Text Classification
(1) Learning models from scratch: for relation classification, the typical approach is learning neural models from scratch. Here we choose PA-LSTM (Zhang et al., 2017) and C-GCN (Zhang et al., 2018) as our baselines, as these two models are the most effective ones based on recurrent neural networks and graph neural networks.
PTR: Prompt Tuning with Rules for Text Classification
(2) Fine-tuning pre-trained models: as PLMs have achieved promising results on various NLP tasks, some efforts are also devoted to finetuning PLMs for relation classification. For fine-tuning vanilla PLMs, we directly select ROBERTA_LARGE as our baseline.
PTR: Prompt Tuning with Rules for Text Classification
(3) Fine-tuning knowledge-enhanced pretrained models: Considering the entity information is quite important for models to under-Model Extra Data TACRED TACREV ReTACRED SEMEVAL Learning models from scratch PA-LSTM (Zhang et al., 2017) w  Table 4: F 1 scores (%) on the TACRED, TACREV, ReTACRED and SEMEVAL. For ROBERTA_LARGE, we report the results of the vanilla version ROBERTA_LARGE without adding any entity markers. More adaptions of ROBERTA_LARGE for relation classification will be reported in the next tables. In the "Extra Data" column, "w/o" means that no additional data is used for pre-training and fine-tuning, yet "w/" means that extra data or knowledge bases are used for data augmentation. The best results are bold, and the best results under normal settings (without reversing relations) are underlined.
PTR: Prompt Tuning with Rules for Text Classification
stand relational semantics, a series of knowledgeenhanced PLMs have been further explored, which use knowledge bases as additional information to enhance PLMs. For knowledge-enhanced PLMs, we select SPANBERT (Joshi et al., 2020), KNOW-BERT (Peters et al., 2019), LUKE (Yamada et al., 2020), and MTB (Baldini Soares et al., 2019) as our baselines. And these four baselines are typical models that use knowledge to enhance learning objectives, input features, model architectures, and pre-training strategies respectively.
PTR: Prompt Tuning with Rules for Text Classification
For the above baselines, we report the results of their papers as well as some results reproduced by Zhou and Chen (2021). Table 4 shows the overall performance on four datasets. From the table, we can see that:
PTR: Prompt Tuning with Rules for Text Classification
(1) Conventional methods design complicated models, including various recurrent neural layers, attention neural layers, and graph neural layers. Although learning these complicated models from scratch have achieved effective results, the vanilla PLM ROBERTA_LARGE can easily outperform them, indicating that the rich knowledge of PLMs captured from large-scale unlabeled data is important for downstream tasks.
PTR: Prompt Tuning with Rules for Text Classification
(2) Within PLMs, those knowledge-enhanced PLMs defeat the vanilla PLM ROBERTA_LARGE. On the one hand, this shows that introducing extra task-specific knowledge to enhance models is effective. On the other hand, this indicates that simply fine-tuning PLMs cannot completely cover the knowledge required for downstream tasks.
PTR: Prompt Tuning with Rules for Text Classification
(3) Within knowledge-enhanced PLMs, LUKE and KNOWBERT are better than other models. This is because these two models respectively use knowledge as data augmentation and architecture enhancement in the fine-tuning phase. In contrast, SPANBERT and MTB inject task-specific knowledge into parameters in the pre-training phase and then fine-tune their parameters without extra knowledge. All the above results indicate that even if the task-specific knowledge is already contained in PLMs, it is difficult for fine-tuning to stimulate the knowledge for downstream tasks.
PTR: Prompt Tuning with Rules for Text Classification
(4) Either for the normal setting or the setting of reversing some relations, PTR achieves significant improvements than all baselines. Even compared with those knowledge-enhanced models, PTR still outperforms these models on TACRED.
PTR: Prompt Tuning with Rules for Text Classification
In general, all these experimental results show that the pre-training phase enables PLMs to capture sufficient task-specific knowledge, how to stimulate the knowledge for downstream tasks is crucial. Compared with typical finetuning methods, PTR can better stimulate taskspecific knowledge.
PTR: Prompt Tuning with Rules for Text Classification
In this part, we compare PTR with several recent methods using prompts for relation classification, including:  Table 5: F 1 scores (%) of various models using prompts on the datasets TACRED, TACREV, and ReTACRED. All these models are based on ROBERTA_LARGE or BERT_LARGE. For each model, we also provide the processed input of an example sentence "Mark Twain was born in Florida". We also provide the information that whether these models add extra neural layers to generate features or use extra human annotations to provide more information. "w/o" indicates that the model does not use additional human annotations or neural layers, yet "w/" indicates the model requires additional human annotations or neural layers. The best results are bold, and the best results under normal settings (without reversing relations) are underlined.
PTR: Prompt Tuning with Rules for Text Classification
(1) Fine-tuning PLMs with entity markers: for relation classification, some methods add additional entity markers to the input as additional information, which is similar to prompt tuning. Among these marker-based models, ENT MARKER methods inject special symbols to index the positions of entities and TYP MARKER methods additionally introduce the type information of entities. Intrinsically, the ENT MARKER methods are similar to prompting by introducing extra serial information to indicate the position of special tokens, i.e., named entities. Analogously, the TYP MARKER methods could be regarded as a type of template for prompts but require additional annotation of type information.
PTR: Prompt Tuning with Rules for Text Classification
". . . (PUNCT)" indicates using the vocabulary of PLMs to build markers. ". . . (CLS)" indicates using the hidden vector of [CLS] for classification. Other models use additional neural layers to aggregate all entity features for classification. For more details of marker-based models, we refer to the paper of Zhou and Chen (2021).
PTR: Prompt Tuning with Rules for Text Classification
(2) Prompt tuning for relation classification:  introduce an adaptive label words selection mechanism for prompt tuning, which scatters the relation label into a variable number of label words to handle the complex multiple label space. The model proposed by  is named ADAPROMPT. As ADAPROMPT has not released its code, we only show its reported results on TACREV.
PTR: Prompt Tuning with Rules for Text Classification
In Table 5, we can find that using the vocabulary of PLMs for prompts can lead to better results. By designing particular sub-prompts, our PTR method has the same effect as the TYP MARKER methods. Furthermore, PTR establishes the effectiveness in experiments without additional human annotations and neural layers. Note that in the training procedure, TYP MARKER methods optimize the parameters of the encoder and additional output layer, PTR just optimizes the encoder and several learnable tokens in prompts.
PTR: Prompt Tuning with Rules for Text Classification
To further compare the models using the vocabulary of PLMs for prompts, we introduce the few-shot learning scenario. Following the few-shot setting in , we sample K training instances and K validation instances per class from the original training set and development set, and  Table 6: F 1 scores (%) of various models with different sizes of training instances on the datasets TACRED, TACREV, and ReTACRED.  evaluate models on the original test set. We set K from {8, 16, 32} respectively and use a fixed set of 5 random seeds to sample instances.
PTR: Prompt Tuning with Rules for Text Classification
From Table 6, we find that although PTR performs poorer than TYP MARKER (PUNCT) using the full training dataset, PTR achieves comparable or even better results in the few-shot scenario, especially in ReTACRED where the data has fewer annotation errors. It is worth mentioning that PTR does not use extra human annotations to bring entity types, but to stimulate the type knowledge in PLMs. Compared with TYP MARKER (PUNCT) and ADAPROMPT, we attribute our success to the use of both human knowledge and model knowledge. As shown in Figure 3, besides effectiveness, PTR can lead to faster convergence. TThis indicates the potential positive effect of prompt tuning and logic rules on the convergence.
PTR: Prompt Tuning with Rules for Text Classification
We introduce the reversed setting of TACRED, TACREV, ReTACRED in Section 4.2 and report the corresponding results in Table 4. Surprisingly, we find that after manipulating the labels into the reversed version, PTR gains significant improvements on all the three TACRED datasets. The results demonstrate the tremendous potential for prompt-based tuning. It suggests that small modification of prompts may result in a huge impact on whether the prior knowledge distributed in PLMs can be stimulated. Specifically, as we only change the direction of the relations and yield such improvements, prompts are position-aware. We further apply the reversed relation setting into another two baselines, ENTITY MARKER and TYPED MARKER. As shown in Table 7 , these two methods yield 0.2% and 1.4% absolute improvements on TA-CRED, which are less than the improvements PTR gains (3.5%) but still considerable. The results indicate that PLMs may not encode the bidirectional relation into "symmetric" representations. Based on PTR, we believe more efforts about the construction of effective prompts should be made. And theoretical analysis to better understand the mechanisms underlying prompting strategies is also urgently needed.
PTR: Prompt Tuning with Rules for Text Classification
In this paper, we propose prompt tuning with rules (PTR) for many-class text classification. By composing sub-prompts into task-specific prompts according to logic rules, prior human knowledge can be encoded into prompt tuning. Meanwhile, the introduction of sub-prompts can further alleviate the difficulty of designing templates and sets of label words. The experimental results on relation classification, a typical many-class classification task, show that PTR can significantly outperform existing state-of-the-art baselines without introducing any additional model layers, manual annotations, and augmented data. In the future, we will further combine the existing exploration of auto-generated prompts and our PTR, which will lead to a more practical approach to prompt tuning.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Learning from examples is the predominant approach for many NLP tasks: A model is trained on a set of labeled examples from which it then generalizes to unseen data. Due to the vast number of languages, domains and tasks and the cost of annotating data, it is common in real-world uses of NLP to have only a small number of labeled examples, making few-shot learning a highly important research area. Unfortunately, applying standard supervised learning to small training sets often performs poorly; many problems are difficult to grasp from just looking at a few examples. For instance, assume we are given the following pieces of text:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
• T 1 : This was the best pizza I've ever had.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
• T 2 : You can get better sushi for half the price.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
• T 3 : Pizza was average. Not worth the price. 1 Our implementation is publicly available at https:// github.com/timoschick/pet.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Best pizza ever! +1 ) ∈ T ( Best pizza ever! It was . Furthermore, imagine we are told that the labels of T 1 and T 2 are l and l , respectively, and we are asked to infer the correct label for T 3 . Based only on these examples, this is impossible because plausible justifications can be found for both l and l . However, if we know that the underlying task is to identify whether the text says anything about prices, we can easily assign l to T 3 . This illustrates that solving a task from only a few examples becomes much easier when we also have a task description, i.e., a textual explanation that helps us understand what the task is about.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
With the rise of pretrained language models (PLMs) such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and let the PLM predict continuations that solve the task (Radford et al., 2019;Puri and Catanzaro, 2019). So far, this idea has mostly been considered in zero-shot scenarios where no training data is available at all.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
In this work, we show that providing task descriptions can successfully be combined with standard supervised learning in few-shot settings: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that uses natural language patterns to reformulate input examples into cloze-style phrases. As illustrated in Figure 1, PET works in three steps: First, for each pattern a separate PLM is finetuned on a small training set T . The ensemble of all models is then used to annotate a large unlabeled dataset D with soft labels. Finally, a standard classifier is trained on the soft-labeled dataset. We also devise iPET, an iterative variant of PET in which this process is repeated with increasing training set sizes.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
On a diverse set of tasks in multiple languages, we show that given a small to medium number of labeled examples, PET and iPET substantially outperform unsupervised approaches, supervised training and strong semi-supervised baselines. Radford et al. (2019) provide hints in the form of natural language patterns for zero-shot learning of challenging tasks such as reading comprehension and question answering (QA). This idea has been applied to unsupervised text classification (Puri and Catanzaro, 2019), commonsense knowledge mining (Davison et al., 2019) and argumentative relation classification (Opitz, 2019). Srivastava et al. (2018) use task descriptions for zero-shot classification but require a semantic parser. For relation extraction, Bouraoui et al. (2020) automatically identify patterns that express given relations. Mc-Cann et al. (2018) rephrase several tasks as QA problems. Raffel et al. (2020) frame various problems as language modeling tasks, but their patterns only loosely resemble natural language and are unsuitable for few-shot learning. 2 Another recent line of work uses cloze-style phrases to probe the knowledge that PLMs acquire during pretraining; this includes probing for factual and commonsense knowledge (Trinh and Le, 2018;Petroni et al., 2019;Wang et al., 2019;Sakaguchi et al., 2020), linguistic capabilities (Ettinger, 2020;Kassner and Schütze, 2020), understanding of rare words (Schick and Schütze, 2020), and ability to perform symbolic reasoning (Talmor et al., 2019). Jiang et al. (2020) consider the problem of finding the best pattern to express a given task.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Let M be a masked language model with vocabulary V and mask token ∈ V , and let L be a set of labels for our target classification task A. We write an input for task A as a sequence of phrases x = (s 1 , . . . , s k ) with s i ∈ V * ; for example, k = 2 if A is textual inference (two input sentences). We define a pattern to be a function P that takes x as input and outputs a phrase or sentence P (x) ∈ V * that contains exactly one mask token, i.e., its output can be viewed as a cloze question. Furthermore, we define a verbalizer as an injective function v : L → V that maps each label to a word from M 's vocabulary. We refer to (P, v) as a pattern-verbalizer pair (PVP).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Using a PVP (P, v) enables us to solve task A as follows: Given an input x, we apply P to obtain an input representation P (x), which is then processed by M to determine the label y ∈ L for which v(y) is the most likely substitute for the mask. For example, consider the task of identifying whether two sentences a and b contradict each other (label y 0 ) or agree with each other (y 1 ). For this task, we may choose the pattern P (a, b) = a?
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
, b. combined with a verbalizer v that maps y 0 to "Yes" and y 1 to "No". Given an example input pair x = (Mia likes pie, Mia hates pie), the task now changes from having to assign a label without inherent meaning to answering whether the most likely choice for the masked position in P (x) = Mia likes pie?
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
, Mia hates pie.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
is "Yes" or "No".
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Let p = (P, v) be a PVP. We assume access to a small training set T and a (typically much larger) set of unlabeled examples D. For each sequence z ∈ V * that contains exactly one mask token and w ∈ V , we denote with M (w | z) the unnormalized score that the language model assigns to w at the masked position. Given some input x, we define the score for label l ∈ L as
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
and obtain a probability distribution over labels using softmax:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
l ∈L e sp(l |x)
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We use the cross-entropy between q p (l | x) and the true (one-hot) distribution of training example (x, l) -summed over all (x, l) ∈ T -as loss for finetuning M for p.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
In our application scenario, only a few training examples are available and catastrophic forgetting can occur. As a PLM finetuned for some PVP is still a language model at its core, we address this by using language modeling as auxiliary task. With L CE denoting cross-entropy loss and L MLM language modeling loss, we compute the final loss as
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
This idea was recently applied by Chronopoulou et al. (2019) in a data-rich scenario. As L MLM is typically much larger than L CE , in preliminary experiments, we found a small value of α = 10 −4 to consistently give good results, so we use it in all our experiments. To obtain sentences for language modeling, we use the unlabeled set D. However, we do not train directly on each x ∈ D, but rather on P (x), where we never ask the language model to predict anything for the masked slot.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
A key challenge for our approach is that in the absence of a large development set, it is hard to identify which PVPs perform well. To address this, we use a strategy similar to knowledge distillation (Hinton et al., 2015). First, we define a set P of PVPs that intuitively make sense for a given task A. We then use these PVPs as follows:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
(1) We finetune a separate language model M p for each p ∈ P as described in Section 3.1.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
As T is small, this finetuning is cheap even for a large number of PVPs.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
(2) We use the ensemble M = {M p | p ∈ P} of finetuned models to annotate examples from D. We first combine the unnormalized class scores for each example x ∈ D as
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
where Z = p∈P w(p) and the w(p) are weighting terms for the PVPs. We experiment with two different realizations of this weighing term: either we simply set w(p) = 1 for all p or we set w(p) to be the accuracy obtained using p on the training set before training. We refer to these two variants as uniform and weighted. Jiang et al. (2020) use a similar idea in a zero-shot setting.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We transform the above scores into a probability distribution q using softmax. Following Hinton et al. (2015), we use a temperature of T = 2 to obtain a suitably soft distribution. All pairs (x, q) are collected in a (soft-labeled) training set T C .
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
(3) We finetune a PLM C with a standard sequence classification head on T C .
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
The finetuned model C then serves as our classifier for A. All steps described above are depicted in Figure 2; an example is shown in Figure 1. (2) The final set of models is used to create a soft-labeled dataset T C .
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
(3) A classifier C is trained on this dataset.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Distilling the knowledge of all individual models into a single classifier C means they cannot learn from each other. As some patterns perform (possibly much) worse than others, the training set T C for our final model may therefore contain many mislabeled examples.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
To compensate for this shortcoming, we devise iPET, an iterative variant of PET. The core idea of iPET is to train several generations of models on datasets of increasing size. To this end, we first enlarge the original dataset T by labeling selected examples from D using a random subset of trained PET models ( Figure 2a). We then train a new generation of PET models on the enlarged dataset (b); this process is repeated several times (c).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
More formally, let M 0 = {M 0 1 , . . . , M 0 n } be the initial set of PET models finetuned on T , where each M 0 i is trained for some PVP p i . We train k generations of models M 1 , . . . , M k where M j = {M j 1 , . . . , M j n } and each M j i is trained for p i on its own training set T j i . In each iteration, we multiply the training set size by a fixed constant d ∈ N while maintaining the label ratio of the original dataset. That is, with c 0 (l) denoting the number of examples with label l in T , each T j i contains c j (l) = d · c j−1 (l) examples with label l. This is achieved by generating each T j i as follows:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
1. We obtain N ⊂ M j−1 \ {M j−1 i } by randomly choosing λ · (n − 1) models from the previous generation with λ ∈ (0, 1] being a hyperparameter.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
2. Using this subset, we create a labeled dataset
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
To avoid training future generations on mislabeled data, we prefer examples for which the ensemble of models is confident in its prediction. The underlying intuition is that even without calibration, examples for which labels are predicted with high confidence are typically more likely to be classified correctly (Guo et al., 2017). Therefore, when drawing from T N , we set the probability of each (x, y) proportional to s N (l | x).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
3. We define T j i = T ∪ l∈L T N (l). As can easily be verified, this dataset contains c j (l) examples for each l ∈ L.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
After training k generations of PET models, we use M k to create T C and train C as in basic PET.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
With minor adjustments, iPET can even be used in a zero-shot setting. To this end, we define M 0 to be the set of untrained models and c 1 (l) = 10/|L| for all l ∈ L so that M 1 is trained on 10 examples evenly distributed across all labels. As T N may not contain enough examples for some label l, we create all T N (l) by sampling from the 100 examples x ∈ D for which s N (l | x) is the highest, even if l = arg max l∈L s N (l | x). For each subsequent generation, we proceed exactly as in basic iPET.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We evaluate PET on four English datasets: Yelp Reviews, AG's News, Yahoo Questions (Zhang et al., 2015) and MNLI (Williams et al., 2018). Additionally, we use x-stance (Vamvas and Sennrich, 2020) to investigate how well PET works for other languages. For all experiments on English, we use RoBERTa large (Liu et al., 2019) as language model; for x-stance, we use XLM-R (Conneau et al., 2020). We investigate the performance of PET and all baselines for different training set sizes; each model is trained three times using different seeds and average results are reported.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
As we consider a few-shot setting, we assume no access to a large development set on which hyperparameters could be optimized. Our choice of hyperparameters is thus based on choices made in previous work and practical considerations. We use a learning rate of 1 · 10 −5 , a batch size of 16 and a maximum sequence length of 256. Unless otherwise specified, we always use the weighted variant of PET with auxiliary language modeling. For iPET, we set λ = 0.25 and d = 5; that is, we select 25% of all models to label examples for the next generation and quintuple the number of training examples in each iteration. We train new generations until each model was trained on at least 1000 examples, i.e., we set k = log d (1000/|T |) . As we always repeat training three times, the ensemble M (or M 0 ) for n PVPs contains 3n models. Further hyperparameters and detailed explanations for all our choices are given in Appendix B.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We now describe the patterns and verbalizers used for all tasks. We use two vertical bars ( ) to mark boundaries between text segments. 3
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Yelp For the Yelp Reviews Full Star dataset (Zhang et al., 2015), the task is to estimate the rating that a customer gave to a restaurant on a 1to 5-star scale based on their review's text. We define the following patterns for an input text a: P 1 (a) = It was . a P 2 (a) = Just ! a P 3 (a) = a. All in all, it was .
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
P 4 (a) = a In summary, the restaurant is . 3 The way different segments are handled depends on the model being used; they may e.g. be assigned different embeddings (Devlin et al., 2019) or separated by special tokens (Liu et al., 2019;Yang et al., 2019). For example, "a b" is given to BERT as the input "
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We define a single verbalizer v for all patterns as
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
AG's News AG's News is a news classification dataset, where given a headline a and text body b, news have to be classified as belonging to one of the categories World (1), Sports (2), Business (3) or Science/Tech (4). For x = (a, b), we define the following patterns:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We use a verbalizer that maps 1-4 to "World", "Sports", "Business" and "Tech", respectively.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Yahoo Yahoo Questions (Zhang et al., 2015) is a text classification dataset. Given a question a and an answer b, one of ten possible categories has to be assigned. We use the same patterns as for AG's News, but we replace the word "News" in P 5 with the word "Question". We define a verbalizer that maps categories 1-10 to "Society", "Science", "Health", "Education", "Computer", "Sports", "Business", "Entertainment", "Relationship" and "Politics".
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
MNLI The MNLI dataset (Williams et al., 2018) consists of text pairs x = (a, b). The task is to find out whether a implies b (0), a and b contradict each other (1) or neither (2). We define
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
and consider two different verbalizers v 1 and v 2 :
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Combining the two patterns with the two verbalizers results in a total of 4 PVPs.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
X-Stance The x-stance dataset (Vamvas and Sennrich, 2020) is a multilingual stance detection dataset with German, French and Italian examples.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Each example x = (a, b) consists of a question a concerning some political issue and a comment b; the task is to identify whether the writer of b  supports the subject of the question (0) or not (1). We use two simple patterns
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
and define an English verbalizer v En mapping 0 to "Yes" and 1 to "No" as well as a French (German) verbalizer v Fr (v De ), replacing "Yes" and "No" with "Oui" and "Non" ("Ja" and "Nein"). We do not define an Italian verbalizer because x-stance does not contain any Italian training examples.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
English Datasets Table 1 shows results for English text classification and language understanding tasks; we report mean accuracy and standard deviation for three training runs. Lines    Table 2 shows that PET and iPET substantially outperform both methods across all tasks, clearly demonstrating the benefit of incorporating human knowledge in the form of PVPs.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We evaluate PET on x-stance to investigate (i) whether it works for languages other than English and (ii) whether it also brings improvements when training sets have medium size. In contrast to Vamvas and Sennrich (2020), we do not perform any hyperparameter optimization on dev and use a shorter maximum sequence length (256 vs 512) to speed up training and evaluation. To investigate whether PET brings benefits even when numerous examples are available, we consider training set sizes of 1000, 2000, and 4000; for each of these configurations, we separately finetune French and German models to allow for a more straightforward downsampling of the training data. Additionally, we train models on the entire French (|T Fr | = 11 790) and German (|T De | = 33 850) training sets. In this case we do not have any additional unlabeled data, so we simply set D = T . For the French models, we use v En and v Fr as verbalizers and for German v En and v De (Section 4.1). Finally, we also investigate the performance of a model trained jointly on French and German data (|T Fr + T De | = 45 640) using v En , v Fr and v De .
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Results are shown in Table 3; following Vamvas  Table 4: Minimum (min) and maximum (max) accuracy of models based on individual PVPs as well as PET with and without knowledge distillation (|T | = 10). and Sennrich (2020), we report the macro-average of the F1 scores for labels 0 and 1, averaged over three runs. For Italian (column "It"), we report the average zero-shot cross-lingual performance of German and French models as there are no Italian training examples. Our results show that PET brings huge improvements across all languages even when training on much more than a thousand examples; it also considerably improves zero-shot cross-lingual performance.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Combining PVPs We first investigate whether PET is able to cope with situations were some PVPs perform much worse than others. For |T | = 10, Table 4 compares the performance of PET to that of the best and worst performing patterns after finetuning; we also include results obtained using the ensemble of PET models corresponding to individual PVPs without knowledge distillation. Even after finetuning, the gap between the best and worst pattern is large, especially for Yelp. However, PET is not only able to compensate for this, but even improves accuracies over using only the bestperforming pattern across all tasks. Distillation brings consistent improvements over the ensemble; additionally, it significantly reduces the size of the final classifier. We find no clear difference between the uniform and weighted variants of PET.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We analyze the influence of the auxiliary language modeling task on PET's performance. Figure 3 shows performance improvements from adding the language modeling task for four training set sizes. We see that the auxiliary task is extremely valuable when training on just 10 examples. With more data, it becomes less important, sometimes even leading to worse performance. Only for MNLI, we find language modeling to consistently help.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Iterative PET To check whether iPET is able to improve models over multiple generations, Figure 4 shows the average performance of all generations of models in a zero-shot setting. Each additional iteration does indeed further improve the ensemble's performance. We did not investigate whether continuing this process for even more iterations gives further improvements. Another natural question is whether similar results can be obtained with fewer iterations by increasing the training set size more aggressively. To answer this question, we skip generations 2 and 3 for AG's News and Yahoo and for both tasks directly let ensemble M 1 annotate 10 · 5 4 examples for M 4 . As indicated in Figure 4 through dashed lines, this clearly leads to worse performance, highlighting the importance of only gradually increasing the training set size. We surmise that this is the case because annotating too many examples too early leads to a large percentage of mislabeled training examples. In-Domain Pretraining Unlike our supervised baseline, PET makes use of the additional unlabeled dataset D. Thus, at least some of PET's performance gains over the supervised baseline may arise from this additional in-domain data.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
To test this hypothesis, we simply further pretrain RoBERTa on in-domain data, a common technique for improving text classification accuracy (e.g., Howard and Ruder, 2018;Sun et al., 2019). As language model pretraining is expensive in terms of GPU usage, we do so only for the Yelp dataset. Figure 5 shows results of supervised learning and PET both with and without this indomain pretraining. While pretraining does indeed improve accuracy for supervised training, the supervised model still clearly performs worse than PET, showing that the success of our method is not simply due to the usage of additional unlabeled data. Interestingly, in-domain pretraining is also helpful for PET, indicating that PET leverages unlabeled data in a way that is clearly different from standard masked language model pretraining.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We have shown that providing task descriptions to pretrained language models can be combined with standard supervised training. Our proposed method, PET, consists of defining pairs of cloze question patterns and verbalizers that help leverage the knowledge contained within pretrained language models for downstream tasks. We finetune models for all pattern-verbalizer pairs and use them to create large annotated datasets on which standard classifiers can be trained. When the initial amount of training data is limited, PET gives large improvements over standard supervised training and strong semi-supervised approaches. 
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Our implementation of PET and iPET is based on the Transformers library (Wolf et al., 2020) and PyTorch (Paszke et al., 2017).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Except for the in-domain pretraining experiment described in Section 5, all of our experiments were conducted using a single GPU with 11GB RAM (NVIDIA GeForce GTX 1080 Ti).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Relevant training hyperparameters for both individual PET models and the final classifier C as well as our supervised baseline are listed in Table 5. All hyperparameters were selected based on the following considerations and experiments:
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Batch size / maximum length Both batch size and maximum sequence length (or block size) are chosen so that one batch fits into 11GB of GPU memory. As Devlin et al. (2019) and Liu et al.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
(2019) use larger batch sizes of 16-32, we accumulate gradients for 4 steps to obtain an effective batch size of 16.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Learning rate We found a learning rate of 5e−5 (as used by Devlin et al. (2019)) to often result in unstable training for regular supervised learning with no accuracy improvements on the training set. We therefore use a lower learning rate of 1e−5, similar to Liu et al. (2019). Experiments with various learning rates can be found in Appendix D.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Training steps As the number of training epochs recommended by Liu et al. (2019) in a data-rich scenario is in the range 2-10, we perform supervised training for 250 training steps, corresponding to 4 epochs when training on 1000 examples. For individual PET models, we subdivide each batch into one labeled example from T to compute L CE and three unlabeled examples from D to compute L MLM . Accordingly, we multiply the number of total training steps by 4 (i.e., 1000), so that the number of times each labeled example is seen remains constant (16 · 250 = 4 · 1000). For the final PET classifier, we train for 5000 steps due to the increased training set size (depending on the task, the unlabeled set D contains at least 20 000 examples). Deviating from the above, we always perform training for 3 epochs on x-stance to match the setup of Vamvas and Sennrich (2020) more closely. The effect of varying the number of training steps is further investigated in Appendix D.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Temperature We choose a temperature of 2 when training the final classifier following Hinton et al. (2015).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Auxiliary language modeling To find a suitable value of α for combining language modeling loss and cross-entropy loss, we first observed that in the early stages of training, the former is a few orders of magnitude higher than the latter for all tasks considered. We thus selected a range {1e−3, 1e−4, 1e−5} of reasonable choices for α and performed preliminary experiments on Yelp with 100 training examples to find the best value among these candidates. To this end, we split the training examples into a training set and a dev set using both a 90/10 split and a 50/50 split and took the value of α that maximizes average dev set accuracy. We adopt this value for all other tasks and training set sizes without further optimization.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Models per ensemble As we always train three models per pattern, for both iPET and training the final classifier C, the ensemble M (or M 0 ) for n PVPs contains 3n models. This ensures consistency as randomly choosing any of the three models for each PVP would result in high variance. In preliminary experiments, we found this to have only little impact on the final model's performance.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
iPET dataset size For iPET, we quintuple the number of training examples after each iteration (d = 5) so that only a small number of generations is required to reach a sufficient amount of labeled data. We did not choose a higher value because we presume that this may cause training sets for early generations to contain a prohibitively large amount of mislabeled data.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
iPET dataset creation We create training sets for the next generation in iPET using 25% of the models in the current generation (λ = 0.25) because we want the training sets for all models to be diverse while at the same time, a single model should not have too much influence.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Others For all other hyperparameters listed in Table 5, we took the default settings of the Transformers library (Wolf et al., 2020).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
As PET does not require any additional learnable parameters, the number of parameters for both PET and iPET is identical to the number of parameters in the underlying language model: 355M for RoBERTa (large) and 270M for XLM-R (base). For MixText, we use the original implementation 5 and the default set of hyperparameters. Specifically, each batch consists of 4 labeled and 8 unlabeled examples, we use layers 7, 9 and 12 for mixing, we set T = 5, α = 16, and use a learning rate of 5 · 10 −6 for RoBERTa and 5 · 10 −4 for the final classification layer. We optimize the number of training steps for each task and dataset size in the range {1000, 2000, 3000, 4000, 5000}.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
For UDA, we use a PyTorch-based reimplementation 6 . We use the same batch size as for MixText and the hyperparameter values recommended by Xie et al. (2020); we use an exponential schedule for training signal annealing and a learning rate of 2 · 10 −5 . We optimize the number of training steps for each task and dataset size in the range {500, 1000, 1500, . . . , 10000}.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
For in-domain pretraining experiments described in Section 5, we use the language model finetuning script of the Transformers library (Wolf et al., 2020); all hyperparameters are listed in the last column of Table 5. Pretraining was performed on a total of 3 NVIDIA GeForce GTX 1080 Ti GPUs. 5 https://github.com/GT-SALT/MixText 6 https://github.com/SanghunYun/UDA_ pytorch
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
For each task and number of examples t, we create the training set T by collecting the first t/|L| examples per label from the original training set, where |L| is the number of labels for the task. Similarly, we construct the set D of unlabeled examples by selecting 10 000 examples per label and removing all labels. For evaluation, we use the official test set for all tasks except MNLI, for which we report results on the dev set; this is due to the limit of 2 submissions per 14 hours for the official MNLI test set. An overview of the number of test examples and links to downloadable versions of all used datasets can be found in Table 6.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Preprocessing In some of the datasets used, newlines are indicated through the character sequence "\n". As the vocabularies of RoBERTa and XLM-R do not feature a newline, we replace this sequence with a single space. We do not perform any other preprocessing, except shortening all examples to the maximum sequence length of 256 tokens. This is done using the longest first strategy implemented in the Transformers library. For PET, all input sequences are truncated before applying patterns.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Evaluation metrics For Yelp, AG's News, Yahoo and MNLI, we use accuracy. For x-stance, we report macro-average of F1 scores using the evaluation script of Vamvas and Sennrich (2020).
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
To analyze the importance of hyperparameter choices for PET's performance gains over supervised learning, we look at the influence of both the learning rate (LR) and the number of training steps on their test set accuracies.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
We try values of {1e−5, 2e−5, 5e−5} for the learning rate and {50, 100, 250, 500, 1000} for the number of training steps. As this results in 30 different configurations for just one task and training set size, we only perform this analysis on Yelp with 100 examples, for which results can be seen in Figure 6. For supervised learning, the configuration used throughout the paper (LR = 1e−5, 250 steps) turns out to perform best whereas for PET, training for fewer steps consistently performs even better. Importantly, PET clearly outperforms regular supervised training regardless of the chosen learning rate and number of training steps. ---* temperature --2.0 -weight decay 0.01 0.01 0.01 0.01 0.0   
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Pretrained language models (LMs) have had exceptional success when adapted to downstream tasks via finetuning (Peters et al., 2018;Devlin et al., 2019). Although it is clear that pretraining improves accuracy, it is difficult to determine whether the knowledge that finetuned LMs contain is learned during the pretraining or the finetuning * First three authors contributed equally.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
process. How can we directly evaluate the knowledge present in pretrained LMs, be it linguistic, factual, commonsense, or task-specific?
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Numerous techniques have been proposed to elicit such knowledge by analyzing pretrained LMs' internal representations. A common strategy is to use probing classifiers-shallow classifiers that predict certain attributes using an LMs' representations as features (Conneau et al., 2018;Liu et al., 2019). However, probing classifiers require additional learned parameters and are thus susceptible to false positives; high probing accuracy is not a sufficient condition to conclude that an LM contains a certain piece of knowledge (Hewitt and Liang, 2019;Voita and Titov, 2020). Attention visualization, another common technique, has a similar failure mode: attention scores may be correlated with, but not caused by the underlying target knowledge, leading to criticism against their use as explanations (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019). Both probing and attention visualizations also struggle to evaluate knowledge that cannot be represented as simple token-or sequencelevel classification tasks.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
A more direct approach for eliciting knowledge from these models, since they are language models after all, is prompting, i.e. converting tasks into a language model format. For example, Radford et al. (2019) frame summarization as a language modeling task by appending "TL;DR:" to the end of an article and then generating from an LM. Similarly, Petroni et al. (2019) manually reformulate a knowledge base completion task as a cloze test (i.e., a fill-in-the-blank problem). Compared to existing model analysis methods, prompting is noninvasive: it does not introduce large amounts of additional parameters or require direct inspection of a model's representations. Thus prompting pro- AUTOPROMPT a real joy. atmosphere alot dialogue Clone totally [MASK].
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Trigger Tokens atmosphere, alot, dialogue, Clone... Figure 1: Illustration of AUTOPROMPT applied to probe a masked language model's (MLM's) ability to perform sentiment analysis. Each input, x inp , is placed into a natural language prompt, x prompt , which contains a single [MASK] token. The prompt is created using a template, λ, which combines the original input with a set of trigger tokens, x trig . The trigger tokens are shared across all inputs and determined using a gradient-based search (Section 2.2). Probabilities for each class label, y, are then obtained by marginalizing the MLM predictions, p([MASK]|x prompt ), over sets of automatically detected label tokens (Section 2.3).
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
vides a lower bound on what the model "knows", and is therefore a more useful analysis tool. However, prompting unfortunately requires manually crafting the context to feed into the model. Not only is this time consuming and non-intuitive for many tasks (e.g., textual entailment), more importantly, models are highly sensitive to this context: improperly-constructed contexts cause artificially low performance (Jiang et al., 2020). Overcoming the need to manually specify prompts would make prompting a more widely useful analysis tool.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
In this paper, we introduce AUTOPROMPT-an automated method for generating prompts for any task, illustrated in Figure 1. Given a task, e.g., sentiment analysis, AUTOPROMPT creates a prompt by combining the original task inputs (e.g. reviews) with a collection of trigger tokens according to a template. The same set of trigger tokens is used for all inputs, and is learned using a variant of the gradient-based search strategy proposed in . The LM predictions for the prompt are converted to class probabilities by marginalizing over a set of associated label tokens, which can either be learned or specified ahead of time, enabling the LM to be evaluated the same as one would any other classifier.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
We validate the effectiveness of AUTOPROMPT in numerous experiments. First, we use AUTO-PROMPT to construct prompts that test pretrained masked language models (MLMs) on sentiment analysis and natural language inference (NLI). Our tests reveal that, without any finetuning, MLMs perform well on both of these tasks-a properly-prompted RoBERTa achieves 91% accuracy on SST-2 (better than a finetuned ELMo model (Peters et al., 2018)), and 69% accuracy on a balanced variant of the SICK-E dataset (Marelli et al., 2014). Next, we apply AUTOPROMPT to the fact retrieval tasks of LAMA (Petroni et al., 2019), where we are able to construct prompts that more effectively elicit MLM's factual knowledge than existing prompts generated using manual and corpusmining methods. Concretely, we achieve 43.3% precision-at-1, compared to the current best singleprompt result of 34.1% (Jiang et al., 2020). We also introduce a variant of this task, similar to relation extraction (RE), that tests whether MLMs can extract knowledge from a given piece of text. We show that MLMs can actually outperform existing RE models when context sentences with real facts are provided, however, they struggle when context sentences are artificially falsified.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Finally, although the goal of AUTOPROMPT is to analyze models, we find that it provides certain practical advantages over finetuning. First, AU-TOPROMPT achieves higher average-and worstcase accuracy than finetuning in low-data regimes. Moreover, unlike finetuning, prompting LMs does not require large amounts of disk space to store model checkpoints; once a prompt is found, it can be used on off-the-shelf pretrained LMs. This is beneficial when serving models for multiple tasks.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
A natural way to elicit knowledge from pretrained LMs is to pose tasks as fill-in-the-blank problems.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
However, writing prompts is not only time consuming, but it is not clear that the same phrasing will be effective for every model, nor is it clear what criteria determine whether a particular phrasing the best to elicit the desired information. In light of this, we introduce AUTOPROMPT, a method that constructs customized prompts for a specific task and MLM of interest, to cause the MLMs to produce the desired knowledge. 1 An illustration of AUTOPROMPT is provided in Figure 1. The prompt is constructed by taking the original task inputs-a collection of one or more sequences of tokens (e.g., the review in Figure 1)-and mapping them to a sequence of tokens using a template. In the following sections, we describe how AUTOPROMPT uses labeled training data to construct prompts, and how it uses the output of the MLM as a prediction for the task.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
For the purpose of prompt construction, we distinguish the original task inputs x inp (e.g., the review in Figure 1, "a real joy.") from the prompt x prompt (e.g., "a real joy. atmosphere alot dialogue Clone totally [MASK].") that is fed into the MLM. The mapping from x inp to x prompt is performed using a template, λ. This template defines where each input sequence will be placed in the prompt, as well as the placement of any additional tokens. In particular, it must also define the placement of a special [MASK] token for the MLM to fill in (denoted by [P] in the template to distinguish it from other [MASK] tokens that might appear). Feeding the prompt into the MLM produces a probability distribution p([MASK]|x prompt ) describing which tokens most likely fill in the blank.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
If class labels naturally correspond to tokens in the vocabulary (e.g., entity names in knowledge base completion tasks), this distribution may be readily interpreted as a distribution over class labels. However, for tasks such as sentiment analysis, there may be a set of label tokens V y that correspond to a particular label y. For example, in Figure 1, "Cris", "marvelous", and "philanthrop" all indicate positive sentiment. In this case, the class probability is obtained by marginalizing over the 1 Although we focus only on MLMs in this work, our method is trivially extendable to autoregressive LMs. The only adjustment is that the predict token must occur at the end of the prompt. set of label tokens:
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
(1)
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
So far, we have shown how to reformulate a classification task as a language modeling task using prompts. Here, we propose a method for automatic prompt construction based on . The idea is to add a number of "trigger" tokens that are shared across all prompts (denoted by [T] in the example template in Figure 1). These tokens are initialized to [MASK] tokens, and then iteratively updated to maximize the label likelihood (Equation (1)) over batches of examples.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Formally, at each step, we compute a first-order approximation of the change in the log-likelihood that would be produced by swapping the jth trigger token x (j) trig with another token w ∈ V. Then we identify a candidate set V cand of the top-k tokens estimated to cause the greatest increase:
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
where w in is the input embedding of w, and the gradient is taken with respect to the input embedding of x (j) trig . Note that computing this candidate set is roughly as expensive as a single forward pass and backward pass of the model (the dot-products require the same amount of multiplications as computing the LM output projection). For each candidate in this set, we then re-evaluate Equation (1) on the updated prompt, and retain the prompt with the highest probability in the next step-this requires k forward passes of the model. An example prompt produced by this method for the task of sentiment analysis is shown in Figure 1.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
While in some settings the choice of label tokens is obvious (e.g., when class labels directly correspond to words in the vocabulary), it is less clear what label tokens are appropriate for problems involving more abstract class labels (e.g., NLI). In this section, we develop a general two-step approach to automate the selection of the sets of label tokens V y . In the first step, we train a logistic classifier to predict the class label using the contextualized embedding of the [MASK] token as input:
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
We write the output of this classifier as:
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
where y and β y are the learned weight and bias terms for the label y, and i represents the index of the [MASK] token.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
In the second step, we substitute h (i) with the MLM's output word embeddings w out to obtain a score s(y, w) = p(y|w out ). Intuitively, because w out · h and y · h are large for words and labels that are relevant to a particular context, s w ∝ exp(w out · y + β y ) should be large for words that are typically associated with a given label. The sets of label tokens are then constructed from the k-highest scoring words:
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Our work fits into a body of work that probes language model's knowledge via prompts. Previous works have used manually defined prompts to study an LM's ability to perform: commonsense reasoning ( (2020) use manually constructed prompts in conjunction with semi-supervised learning for fewshot learning. We instead automatically create prompts for any task, which leads to higher accuracy and opens up new phenomena to analyze.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
In the following sections, we apply AUTOPROMPT to probe BERT BASE 2 (110M parameters) and RoBERTa LARGE 's (355M parameters) knowledge of the following tasks: sentiment analysis, natural language inference (NLI), fact retrieval, and relation extraction. We use the PyTorch implementations and pretrained weights provided by the transformers Python library (Wolf et al., 2019). For sentiment analysis and NLI, we find label tokens using the logistic-regression-based heuristic described in Section 2.3. For fact retrieval and relation extraction, we skip this step as the labels (entities) directly correspond to tokens in the vocabulary. For all tasks, we perform the prompt 2 For brevity, we will omit subscripts in the model names.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
search described in Section 2.2 for multiple iterations. In each iteration, we use a batch of training data to identify the candidate set V cand of replacement trigger tokens. We then evaluate the label likelihoods of the updated prompts on a separate batch of data, and we retain the best trigger token in the next iteration of the search. At the end of every iteration, we measure the label likelihood on withheld development data, and return the best prompt found during the entire search as the final output. Performance is evaluated using the appropriate task-specific metrics-e.g., accuracy for sentiment analysis and NLI, and precision@k for fact retrieval-on a separate withheld test set.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Our AUTOPROMPT implementation is publicly available at http://ucinlp.github.io/autoprompt, and supports prompt generation for pretrained models in the HuggingFace transformers library (Wolf et al., 2019) on arbitrary datasets.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Sentiment analysis is a fundamental task in NLP, both for natural language understanding research and real-world applications. It is also difficult to probe the extent to which MLMs understand sentiment without finetuning.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Setup We apply our method to convert instances from the binary Stanford Sentiment Treebank (Socher et al., 2013, SST-2) into prompts, using the standard train/test splits. We find label tokens using a prompt based on the template in Table 3. For our gradient-based prompt search, we perform a grid search over the following hyperparameters: |V cand | ∈ {10, 100}, |V y | ∈ {1, 3, 5}, |x trig | ∈ [3, 6]. 3 All prompts are initialized with the same template used to find the label set.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
We also construct a prompt manually (before automated prompts are generated, to avoid bias) based on the intuition that SST-2 is comprised of movie reviews. We use "{sentence} this movie was [P]." as the template, and use "terrible" and "fantastic" for the negative and positive label tokens, respectively. and RoBERTa have a strong knowledge of sentiment analysis: without any finetuning, BERT performs comparably to a supervised BiLSTM, and RoBERTa achieves an accuracy on-par with finetuned BERT and ELMo models. In addition, we observe that our automatically constructed prompts are more effective than manual prompts, and that they are difficult to construct using human intuition: the best template for RoBERTa is "{sentence} atmosphere alot dialogue Clone totally [P]." We include results on the effect of the AUTOPROMPT hyperparameters in Appendix A.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Accuracy in Low-Data Settings Although the goal of AUTOPROMPT is to probe a model's knowledge, we also find that it may be a viable alternative to finetuning in the low-data regime. To show this, we measure the development set accuracy of AU-TOPROMPT prompts when using random subsets of 10, 100, and 1000 instances from the training data. We run our prompt search with |x trig | = 10, |V y | = 3, and |V cand | = 10. We compare to the performance of BERT and RoBERTa finetuned on the same data. For fair comparison between AU-TOPROMPT and finetuning, we use Mosbach et al.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
(2020)'s recommended parameters for finetuning on small datasets: trained for 20 epochs, using AdamW (Loshchilov and Hutter, 2018) with bias correction and a learning rate that linearly increases to 2 × 10 −5 in the first 10% of iterations, and linearly decreases to 0 afterwards. Experiments are repeated 10 times on random subsets of data (and seeds for the finetuned models). Best-case, worstcase, and average performance are shown in Figure 2. Note that results in the EMNLP version had a bug that has since been fixed. We observe that while finetuning outperforms AUTOPROMPT on sentiment analysis, AUTO-PROMPT can perform better than finetuning on NLI. Notably, AUTOPROMPT elicits better average performance from both BERT and RoBERTa given only 10 training examples. Furthermore, results for RoBERTa are more stable across all sample sizes whereas finetuning can result in "failed runs" (consistent with Dodge et al. 2020). This behavior in the low-data regime is an interesting phenomenon, and suggests that there are barriers that MLMs must surmount when they are converted to finetuned classifiers that are not encountered when the task is presented as masked language modeling.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
To evaluate the semantic understanding of MLMs, we experiment on Natural Language Inference  (NLI). NLI is crucial in many tasks such as reading comprehension and commonsense reasoning (Bowman et al., 2015), and it is used as a common benchmark for language understanding.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Setup We use the entailment task from the SICK dataset (Marelli et al., 2014, SICK-E) which consists of around 10,000 pairs of human-annotated sentences labeled as entailment, contradiction, and neutral. The standard dataset is biased toward the neutral class which represent 56.7% of instances. We also experiment on an unbiased variant with 2-way classification of contradiction vs. entailment (2-way), as well as an unbiased 3-way classification variant (3-way). The template used for AUTOPROMPT is provided in Table 3. We search over the following parameters: |V cand | ∈ {10, 50}, |V y | ∈ {1, 3, 5, 10}, |x trig | ∈ [1, 5], and choose the best prompt according to development set accuracy.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Results Table 2 shows that AUTOPROMPT considerably outperforms the majority baseline in all experiments. For example, on the 2-way SICK-E dataset, AUTOPROMPT is comparable to a supervised finetuned BERT. We also test linear probeslinear classifiers trained on top of frozen MLM representations with average pooling -and find AUTOPROMPT has comparable or higher accuracy, despite linear probes being susceptible to false positives. Overall, these results demonstrate that both BERT and RoBERTa have some inherent knowledge of natural language inference. We also examine the efficacy of AUTOPROMPT in the low-data regime (using the same procedure as SST-2) on the unbiased 3-way SICK-E data. The results in Figure 2 show that AUTOPROMPT performs on par with finetuned BERT and significantly better than finetuned RoBERTa in low data settings.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
We find that the label tokens are more interpretable for con-tradiction compared to entailment or neutral (examples in Table 3). We investigate if this hurts the model performance on entailment and neutral classes. We measure the precision for each label in the 3-way balanced SICK-E dataset. BERT achieves 74.9%, 54.4%, and 36.8% precision for contradiction, entailment, and neutral cases, respectively, while RoBERTa obtains 84.9%, 65.1%, and 57.3%. These results suggest that AUTOPROMPT may be more accurate for concepts that can be easily expressed using natural label tokens.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
An important question is whether pretrained MLMs know facts about real-world entities. The LAMA dataset (Petroni et al., 2019) evaluates this using cloze tests that consist of (sub, rel, obj) triples, e.g. (Obama, bornIn, Hawaii), and manually created prompts with missing objects, e.g. "Obama was born in [MASK].". LPAQA (Jiang et al., 2020) extends this idea by systematically creating prompts that are generated by mining Wikipedia, paraphrasing, and crowdsourcing. In this section, we use the same cloze-style setup but automatically generate prompts in order to better evaluate the factual knowledge of MLMs. We compare our approach against LAMA and LPAQA, which are explicitly designed for the task of fact retrieval.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Setup We reformulate fact retrieval by mapping (sub,rel,obj) triples to a prompt using the template "{sub}[T]. . . [T] [P].", where the trigger tokens are specific to the relation rel and the correct object obj is the label token. We use the original test set from LAMA (Petroni et al., 2019), henceforth Original. To collect training data for AUTOPROMPT, we gather at most 1000 facts for each of the 41 relations in LAMA from the T-REx dataset (ElSahar et al., 2018). For the relations that still have less than 1000 samples, we gather extra facts straight from Wikidata. We ensure that none of the T-REx triples are present in the test set, and we split the data 80-20 into train and development sets. Moreover, because the collected T-REx data is from a slightly different distribution than the LAMA test set, we also consider a separate evaluation where we split the T-REx triples into a 60-20-20 train/dev/test split and evaluate on the test set. This T-REx dataset is used to measure the performance of our prompts when the train and test data is from the same distribution.  We use AUTOPROMPT with 5 or 7 tokens, and select the search parameters using the T-REx development set. We prevent proper nouns and tokens that appear as gold objects in the training data from being selected as trigger tokens. This is done to prevent AUTOPROMPT from "cheating" by embedding common answers inside the prompt. To evaluate, we observe the rank of the true object in label token distribution of the MLM, and use standard ranking metrics: mean reciprocal rank (MRR), precision-at-1 (P@1), and precision-at-10 (P@10).
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Results Table 4 shows the performance of MLMs with different prompting methods, and we show qualitative examples in Table 3 and in Appendix C. Prompts generated using AUTOPROMPT can extract factual knowledge from BERT more effectively than their manual and mined counterparts: we improve P@1 by up to 12 points. Moreover, despite AUTOPROMPT using only one prompt per relation, it still outperforms LPAQA's ensemble method (which averages predictions for up to 30 prompts) by approximately 4 points. Using 7 trigger tokens achieves slightly higher scores than 5 trigger tokens, although the difference is not substantial. This indicates that our approach is stable to the choice of trigger length, which is consistent with our sentiment analysis results. Overall, these results show that AUTOPROMPT can retrieve facts more effectively than past prompting methods, thus demonstrating that BERT contains more factual knowledge than previously estimated.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
We also provide a detailed breakdown of the prompts found by Petroni et al. (2019) and AUTOPROMPT, and their associated accuracies in Appendix C, Table 7. Manual prompts are competitive when the prompt is easy to specify, e.g., the prompt "was born in" for the PLACE OF BIRTH relation. On the other hand, AUTOPROMPT performs especially well for relations that are difficult to specify in a natural language prompt. For example, Petroni et al. (2019)'s prompt for the PO-SITION PLAYED ON TEAM relation is "{sub} plays in [MASK] position", which is not as specific as the relation requires. Although the prompt from AU-TOPROMPT is not grammatical ("{sub} ediatric striker ice baseman defensive {obj}"), it does contain tokens that are directly related to sports.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
BERT outperforms RoBERTa We finally directly compare BERT and RoBERTa. To do so, we subsample the LAMA test set to consist of examples where the object is a single token for both BERT and RoBERTa (Original-RoBERTa). 4 BERT actually slightly outperforms RoBERTa, and we find that the prompts generated for RoBERTa tend to contain more irrelevant words (see Appendix C, Table 7). For example, the prompt generated by RoBERTa for the PLAYS INSTRUMENT relation contains words such as "Trump" and symbols such as "," ()," for the POSITION Table 4: Factual Retrieval: On the left, we evaluate BERT on fact retrieval using the Original LAMA dataset from Petroni et al. (2019). For all three metrics (mean reciprocal rank, mean precision-at-10 (P@10), and mean precision-at-1(P@1)), AUTOPROMPT significantly outperforms past prompting methods. We also report results on a T-REx version of the data (see text for details). On the right, we compare BERT versus RoBERTa on a subset of the LAMA data using AUTOPROMPT with 5 tokens. perform better than BERT, and it is worthy of investigating this further in future work. Additionally, recall that prompting is a lower bound on a model's knowledge: the lower relative performance does not mean that the model actually knows less.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Apart from evaluating whether MLMs know facts, it is also important to evaluate whether they can extract knowledge from text. In this section, we use the task of relation extraction (RE)-to identify how entities are related in a given sentence-an important task in information extraction. We create RE prompts in a similar fashion as fact retrieval: for a given triple (subj,rel,obj) and sentence that expresses this relation, we construct a prompt as "{sent}{sub} [T]. . . [T] [P].", where the trigger tokens are specific to the relation, and label token is the correct object obj (see Table 3 for an example).
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Setup We use the T-Rex dataset for RE because each T-REx fact comes with context sentences that mention the subject and object surface forms. We compare AUTOPROMPT to LAMA and LPAQA (their prompts are still useful here), as well as a recent supervised relation extraction model (Sorokin and Gurevych, 2017) that was also used by Petroni et al. (2019). To make the evaluation fair for the supervised RE model, we modify the standard RE evaluation. We give the model credit as long as it does not predict a different relation for the subject and object, i.e. we ignore the "no relation" prediction and all other relations. We also drop all sentences from evaluation for which the model's named entity extractor failed to identify the subject and the object as entities. See Appendix B for further details. For the evaluation of all systems, we treat a prediction as correct if it is either the canonical version of the object (e.g., "USA") or the rendered surface form (e.g., "American") for any of the context sentences in a given triple.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Results Table 5 shows the results for BERT and RoBERTa. MLMs can extract relational information more effectively than the supervised RE model, providing up to a 33% increase on the task when using AUTOPROMPT. RoBERTa also outperforms the supervised RE model, although it is worse than BERT (likely for similar reasons as we outline in Section 5). For both BERT and RoBERTa, we notice that the trigger tokens consist of words related to their corresponding relations (see Appendix D, Table 8 for full list), e.g. RoBERTa selects "defy trademarks of namesake manufacturer" for relation MANUFACTURER/PRODUCER OF PRODUCT.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Perturbed Sentence Evaluation A possible explanation for the strong results of MLMs in the RE setting is that they may already know many of the relations. Thus, they may directly predict the objects instead of extracting them. To separate this effect, we synthetically perturb the relation extraction dataset by replacing each object in the test data with a random other object and making the same change to the prompt. For example, "Ryo Kase (born November 9, 1974 in Yokohama→Yorkshire) is a Japanese actor" where Ryo Kase is the subject, Yokohama is the original object, and Yorkshire is the new object. We regenerate the prompts using the perturbed version of the data.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
The accuracy of the RE model does not change significantly on the perturbed data (Table 5), however, the accuracy of the MLMs decreases significantly. This indicates that a significant portion of MLM accuracy comes from background information rather than relation extraction. Nevertheless, our prompts for BERT outperform their LAMA and LPAQA counterparts, which provides further evidence that AUTOPROMPT produces better probes.  
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Prompting as an Alternative to Finetuning The goal of prompting a language model is to probe the knowledge that the model acquired from pretraining. Nevertheless, prompting has some practical advantages over finetuning for solving realworld tasks. First, as shown in Section 3, prompts generated using AUTOPROMPT can achieve higher accuracy than finetuning in the low-data regime.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Moreover, prompting has advantages over finetuning when trying to solve many different tasks (e.g., the many users of the OpenAI GPT-3 API (Brown et al., 2020)). In particular, finetuning requires storing large language model checkpoints for each individual task, and drastically increases system cost and complexity because it requires deploying many different models at the same time. Prompting alleviates both of these issues. Only prompts are stored for each individual task, while the same pretrained model is used across all of the tasks.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Limitations of Prompting There are certain phenomena that are difficult to elicit from pretrained language models via prompts. In our preliminary evaluation on datasets such as QQP (Iyer et al., 2017) and RTE (Dagan et al., 2005), prompts generated manually and with AUTOPROMPT did not perform considerably better than chance. However, we cannot conclude that BERT does not know paraphrasing or entailment from these results. In general, different probing methods have different tasks and phenomena they are suitable for: AUTO-PROMPT makes prompt-based probes more generally applicable, but, it still remains just one tool in the toolbox of the interpretability researcher.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Limitations of AUTOPROMPT One downside of AUTOPROMPT is that it requires labeled training data. Although this is also required for other probing techniques (e.g., linear probing classifiers), manual prompts rely on domain/language insights instead of labeled data. Compared to human-designed prompts, AUTOPROMPT generated prompts lack interpretability, which is similar to other probing techniques, such as linear probing classifiers. Another limitation of AUTOPROMPT is that it can sometimes struggle when the training data is highly imbalanced. For example, in Sections 4 and 5 we show that the prompts often just increase the likelihood of the majority label. Rebalancing the training data can help to mitigate this problem. Finally, due to the greedy search over the large discrete space of phrases, AUTOPROMPT is sometimes brittle; we leave more effective crafting techniques for future directions.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
In this paper, we introduce AUTOPROMPT, an approach to develop automatically-constructed prompts that elicit knowledge from pretrained MLMs for a variety of tasks. We show that these prompts outperform manual prompts while requiring less human effort. Furthermore, the results for sentiment analysis and textual entailment suggest that, in some data-scarce settings, it may be more effective to prompt language models than to finetune them for the task. Although we focus only on masked language models in this paper, our method can be trivially extended to standard language models, and thus maybe useful for constructing inputs for models like GPT-3 (Brown et al., 2020). Source code and datasets to reproduce the results in this paper is available at http://ucinlp.github.io/autoprompt. To measure the effects of the AUTOPROMPT search hyperparameters, we plot the validation accuracy as a function of label set size |V y | and the number of trigger tokens |x trig | in Figure 3. We fix the number of candidates at |V cand | = 100. We observe similar trends when |V cand | = 10.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Varying the number of trigger tokens generally has little effect. On the other hand, there is a substantial increase in accuracy when increasing the label set size from 1 to 3 (approximately +5% for BERT, and +10% for RoBERTa). After analyzing the label sets, we find that our method generally produces intuitive results-"marvelous" and "philanthrop" are associated with positive sentiment, whereas "worse" and "incompetence" are associated with negative sentiment for RoBERTa.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Following Petroni et al. (2019), we use the pretrained RE model from Sorokin and Gurevych (2017) as our baseline. To encode the sentence, this model uses a combination of an LSTM-based relation encoder and an attention mechanism. To make predictions, the model constructs a knowledge graph whose edges are the extracted relation triples. The standard RE evaluation measures how well the model predicts the relation types of entity pairs on the sentence level.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Since our goal is to extract the object of relation triplets, rather than the relation itself, we tweak the standard RE evaluation. We feed the RE model sentences from test facts and we query the resulting graph for all edges that contain the given subject and relation. Then we select the triple with the highest confidence and compare it's object to the gold object. We do this for every fact and take the average across all relations to get the overall precision. The RE model is not trained to predict two of the original T-REx relations. For fair comparison, we exclude these two relations for our evaluation.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Natural language understanding (NLU) has been important in many applications such as intelligent dialog assistants, online search, and social media analysis. Recent advancement of NLU has been driven by emergent pre-trained language models (PLMs) including BERT (Devlin et al., 2019;Liu et al., 2019b), GPT (Radford et al., 2018Brown et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020). Prior studies show that PLMs obtain substantial knowledge during pretraining on raw text corpus (Petroni et al., 2019;Feldman et al., 2019). By fine-tuning on taskspecific labeled data, PLMs exploit such knowl-edge and gain impressive accuracy on a wide range of NLP tasks, such as text classification (Kowsari et al., 2019), question answering (Rajpurkar et al., 2016), machine reading comprehension (Campos et al., 2016), etc.
Pre-trained Language Models Can be Fully Zero-Shot Learners
However, fine-tuning approaches are expensive. It requires labeled datasets, which are rarely available for many tasks. Significant computational efforts are needed to update PLMs' parameters for multiple tasks. In addition, fine-tuning results in one distinct model for each task to maintain.
Pre-trained Language Models Can be Fully Zero-Shot Learners
How can we generalize a pre-trained model to many NLP tasks, without labeled or additional unlabeled data? Existing few-shot and zero-shot approaches propose to construct prompts to elicit desired predictions from PLMs (Brown et al., 2020). The main idea of prompting PLMs is to convert an input utterance to one with masked templates. For example, in text classification an input can be "The Warriors won the NBA championship 2022" and it is instead converted to "The Warriors won the NBA championship 2022. This topic is about [MASK]". A PLM (e.g. BERT) takes the converted text and produces predictions for the masked token, along with the probability. Ideally, a PLM will generate a higher probability for the word "sports" than "politics" on the [MASK] token.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Although these prompting-based methods are effective, they require unlabeled data for training or huge human efforts to construct prompts and to choose designated tokens to represent class labels (Schick and Schütze, 2021a,b;Gao et al., 2021). In addition, these manually constructed verbalizers, i.e. mapping from words (e.g. "basketball") to class labels (e.g. SPORTS), do not extend to new emerging categories after PLMs are deployed.
Pre-trained Language Models Can be Fully Zero-Shot Learners
In this paper, we investigate the fully zero-shot learning problem for NLU where only the target label names are available but not the extra raw text. We propose nonparametric prompting PLM (NPPrompt), a novel method to generate predic-tions for semantic labels without any fine-tuning. NPPrompt uses PLM's own embeddings to automatically find relevant words to labels (e.g. "basketball" and "NBA" for SPORTS), therefore it does not need humans to construct verbalizers. Our key idea is to search for the top k nearest neighbors to a label name in the embedding manifold and then generate and aggregate PLM's predicted logits from masked prompts. In the above case, both predicted values for "basketball" and "NBA" contribute to the final prediction for the SPORTS category. In this way, NPPrompt can be easily generalized to any new categories as long as the category names are semantically meaningful.
Pre-trained Language Models Can be Fully Zero-Shot Learners
The contributions of this paper are as follows. a) We develop NPPrompt, a novel method for fully zero-shot learning with PLMs. b) We conduct extensive experiments on diverse language understanding tasks including text classification, text entailment, similar text retrieval, paraphrasing, and multiple-choice question answering. Experimental results show that NPPrompt outperforms the previous zero-shot methods by absolute 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Surprisingly, NPPrompt is on a par with the best prior method that trained with manual verbalizers, an additional knowledge base, and extra unlabeled data.
Pre-trained Language Models Can be Fully Zero-Shot Learners
We first provide standard paradigms, prompt-based tuning, that perform well in few-shot scenarios, before introducing our approach for the zero-shot case. Take N way text classification as an example. We aim to predict the label y ∈ Y for each sentence, where Y is the label set with N distinct classes. Prompt-based tuning tunes PLM using customized prompts (Brown et al., 2020). The regular prompt-based tuning converts a specific task to a cloze-style mask language modeling problem. For each input example x (single sentence or sentence pair), we first apply a task template T on it, converting original input x to x prompt . For instance, we concatenate the template "T (·) = This topic is about [MASK]" with the original input "The Warriors won the NBA championship 2022" and wrap it into:
Pre-trained Language Models Can be Fully Zero-Shot Learners
The verbalizer f in vanilla prompt engineering maps a set of selected words V from the vocabulary to the original label space Y, i.e., f : V → Y. Inversely, we use M(y j ) to denote the label words in V that are mapped into a specific label y j , ∪ y j ∈Y M(y j ) = V. Then we calculate the probability of label y j :
Pre-trained Language Models Can be Fully Zero-Shot Learners
where g(·) is for aggregating the probability of label words into the probability of the label. Then PLMs can be fine-tuned by minimizing the crossentropy loss with supervised examples.
Pre-trained Language Models Can be Fully Zero-Shot Learners
We inherit PLM with verbalizers framework but keep PLM's parameters frozen (Gao et al., 2021).
Pre-trained Language Models Can be Fully Zero-Shot Learners
The key idea of NPPrompt is using PLM's word embeddings to automatically construct verbalizers -mapping from words to labels -in a fully zeroshot way. It does not need any additional raw text corpus for fine-tuning. NPPrompt consists of two steps to compute predictions for any labels in a nonparametric form (Figure 1). 1) We search for all label words closely related to each class y j in PLM's token embedding manifold. 2) Then we use the PLM to predict values for [MASK], filter them using each class's set of label words, and aggregate the properly weighed outputs to produce the final prediction. In the following, we describe NPPrompt for text classification but it generalizes to other language understanding tasks.
Pre-trained Language Models Can be Fully Zero-Shot Learners
For each class label (e.g. "SPORTS"), we search over the whole vocabulary V for the top-k words nearest to the label name in the PLM's embedding space. Here, the distance between words and label names is measured using the cosine similarity score. Other distance metrics work as well and are examined in Section 5. We denote k as the neighborhood number. Assuming the embeddings of word v i and label name y j are emb(v i ) and emb(y j ) respectively, the label words of the verbalizer for y j are selected by top-k ranking: (1) where S(·) is the cosine similarity function:
Pre-trained Language Models Can be Fully Zero-Shot Learners
∥emb(y j )∥ . Since the PLM is already pre-trained on raw text corpus, it acquires sensible semantic knowledge and relatedness of words in the vocabulary. We use PLM's embedding to search for label words semantically relevant to given label names. For illustration, we show the found label words of two categories in the AG News dataset (Zhang et al., 2015) and the corresponding similarity scores in Table 1. We also extend our verbalizer to support label names with longer expressions in Appendix A.2.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Sim Word Sim " sports" 1.00 " business" 1.00 " Sports" 0.77 " Business" 0.78 " sport" 0.75 " businesses" 0.74 " sporting" 0.68 "business" 0.72 " athletics" 0.65 "Business" 0.67 "sports" 0.65 " businessman" 0.59 "Sports" 0.65 " corporate" 0.58 " Sport" 0.62 " company" 0.56 " athletic" 0.61 " enterprise" 0.55 " athletes" 0.61 " businessmen" 0.55 
Pre-trained Language Models Can be Fully Zero-Shot Learners
This topic Figure 1: The illustration of NPPrompt. We generate the label words by searching the related words from the initial word embedding of the pre-trained language model. By aggregating logits from the label words, we predict the category with the largest score (SPORTS).
Pre-trained Language Models Can be Fully Zero-Shot Learners
for [MASK]. In contrast to previous prompting methods which directly calculate the probability over the surface labels, we use the nearest label words from above to compute the probability for each output label. Only the words in a label's top-k neighborhood will contribute to the class prediction. The contribution from each label word is non-equal.
Pre-trained Language Models Can be Fully Zero-Shot Learners
To be specific, with T (x), a PLM produces the logit vector Θ [MASK] for all possible words at the [MASK] token. Notice that if the whole vocabulary is V, Θ [MASK] ∈ R |V| . Then we compute the class probability for a label y j by aggregating the logits filtered by the verbalizer's label words. We use kernel smoothing to aggregate as follows:
Pre-trained Language Models Can be Fully Zero-Shot Learners
Where the weight between label word v i and class name y j is defined as:
Pre-trained Language Models Can be Fully Zero-Shot Learners
Finally, the best class prediction is selected from the maximum of all labels:
Pre-trained Language Models Can be Fully Zero-Shot Learners
Notice since we use kernel smoothing on logits instead of probability, Q is also unnormalized probability.
Pre-trained Language Models Can be Fully Zero-Shot Learners
For example, AG News has two classes y 1 = {SCIENCE}, y 2 = {SPORTS}. From Table 1, the verbalizer for SPORTS M(y 1 ) includes label words "sports", "athletics", etc, and the verbalizer for BUSINESS M(y 2 ) includes label words "business", "corporate", etc. Given an input text x "The Warriors won the NBA championship 2022", the prompt-augmented sequence x prompt will be "The Warriors won the NBA championship 2022. This topic is about [MASK]". The PLM computes logits for every word Θ([MASK] = v|x prompt ). NPPrompt computes the unnormalized probabilities for SPORTS and BUSINESS:
Pre-trained Language Models Can be Fully Zero-Shot Learners
If the aggregated prediction Q for SPORTS is larger than BUSINESS, NPPrompt outputs SPORTS.
Pre-trained Language Models Can be Fully Zero-Shot Learners
There are certain conditions where one class has label names containing little semantic meaning or where several keywords are needed to define a label. For instance, in the DBPedia dataset (Lehmann et al., 2015), one class is related to NATURALPLACE, then we can use the keywords {"river", "lake", "mountain"} to represent this class. In this setting, we pick out the keyword with the maximum score calculated by Equation 2 to represent each label first. Then we choose the label with the largest score. We use Φ(y j ) to denote all keywords in class y j , and the final prediction is :
Pre-trained Language Models Can be Fully Zero-Shot Learners
(4)
Pre-trained Language Models Can be Fully Zero-Shot Learners
We conduct extensive zero-shot learning experiments to demonstrate the effectiveness of our method. We provide detailed information on our implementation and address several research questions related to NPPrompt.  We adopt sentiment classification tasks on two datasets, IMDB (Maas et al., 2011) and Amazon (McAuley and Leskovec, 2013), and topic classification tasks on another two datasets, AG News (Zhang et al., 2015) and DBPedia (Lehmann et al., 2015). All datasets are in the English language. For each task, we directly use the test set to assess model performances, without incorporating validation or training sets for post-tuning or cherrypicking hand-crafted prompts. The statistics of each dataset are shown in Table 2.
Pre-trained Language Models Can be Fully Zero-Shot Learners
To concentrate on the verbalizer and reduce the influence of templates, we adopt multiple fixed manual templates following (Hu et al., 2022). We report the best template used for the RoBERTalarge model in Table 3.  We implement our experiments based on an open-source toolkit OpenPrompt (Ding et al., 2021), which aims to conduct prompt learning easily. We choose RoBERTa-large (Liu et al., 2019b) as our pre-trained language model. We report the best accuracy of classification results for all experiments using different neighborhood numbers. Since we directly use the pre-trained models for testing, there is no randomness (random seed) in this process. All experiments are conducted on Nvidia A6000 GPUs and more details can be found in Appendix A.1.
Pre-trained Language Models Can be Fully Zero-Shot Learners
We evaluate the following baseline methods.
Pre-trained Language Models Can be Fully Zero-Shot Learners
We utilize sentence embedding models (Reimers and Gurevych, 2019) to obtain the embedding for each sentence and descriptions for each class. Then we calculate the cosine similarity between sentences and label descriptions. We assign the most similar class labels to the sentence. Particularly, we use all-mpnet-base-v2 from Hugging Face as the sentence embedding model, and the descriptions for each class can be found in Appendix A.1. (Sun et al., 2021) propose text entailment tasks to replace text classification tasks and then use the Next Sentence Prediction (NSP) head to predict the results. We show the template we use in Appendix A.1.
Pre-trained Language Models Can be Fully Zero-Shot Learners
ManualVerb Manual verbalizers are defined by human experts with domain knowledge and we simply use the label words provided by OpenPrompt (Ding et al., 2021).
Pre-trained Language Models Can be Fully Zero-Shot Learners
LOTClass (Meng et al., 2020) employ pretrained neural language models with unlabeled data for category understanding, i.e., finding words similar to label names. They then introduce a selftraining approach to the entire unlabeled corpus to generalize the model.
Pre-trained Language Models Can be Fully Zero-Shot Learners
GPT-3 with descriptions Following (Brown et al., 2020), we manually write the descriptions for each class and query GPT-3 where the predicted token serves as the prediction. We show the descriptions in Appendix A.1.
Pre-trained Language Models Can be Fully Zero-Shot Learners
ChatGPT with descriptions In the case of Chat-GPT (OpenAI, 2022), we employ the same descriptions as those used for GPT-3. We query the Chat-GPT model using these descriptions, and the predicted token is considered as the corresponding prediction. Our experimentation is based on the March 2023 version of ChatGPT.   Multi-Null prompting  find that simply introducing a few prompt [MASK]s can improve the performance and robustness of the Null Prompt in the zero-shot settings.
Pre-trained Language Models Can be Fully Zero-Shot Learners
We demonstrate our experimental results in Table  4. Overall NPPrompt outperforms Null Prompt and Multi-Null Prompt remarkably by over 10 percent in a fully zero-shot setting. NPPrompt achieves an accuracy of over 85% on AG News and DBPedia and over 90% on IMDB and Amazon. We conjecture that topic classifications in AG News and DBPedia are more complicated than binary sentiment classifications in IMDB and Amazon, hence the higher accuracy on the latter. NPPrompt is only slightly worse than KPT and SimPTC but outperforms most baseline methods in which human efforts/external knowledge or unlabeled data are strictly required. It's worth noting that NPPrompt performs much better than ManualVerb, suggesting that the label words generated by our method are more comprehensive and unbiased than human-designed ones. Besides, NPPrompt can beat GPT-3 by 4% in terms of average accuracy, a strong sign of the great potential for RoBERTa-large with 355M parameters compared to 175B parameters giant GPT-3.
Pre-trained Language Models Can be Fully Zero-Shot Learners
To explore how our method NPPrompt performs on different kinds of tasks, we also conduct experiments on the GLUE benchmark (Wang et al., 2018).  (Socher et al., 2013) for Sentiment Classification; The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) for Linguistic Acceptability.
Pre-trained Language Models Can be Fully Zero-Shot Learners
As shown in Table 5, NPPrompt outperforms all other methods in fully zero-shot setting. Auto-L (Gao et al., 2021) and AMuLaP  are both automatic label words searching methods utilizing few-shot examples. Our method NPPrompt can even outperform them without any unlabeled data or few-shot training examples.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Both weight and similarity functions play a critical role in the design of NPPrompt and we test how NPPrompt performs on AG News with different configurations. The "Default" setting is as stated in Equation 1 and 3. We fix the similarity function
Pre-trained Language Models Can be Fully Zero-Shot Learners
∥emb(y j )∥ , set w(v i , y j ) = 1 for the "Same weight" setting and w(v i , y j ) = S(emb(v i ),emb(y j )) v k ∈M(y j ) S(emb(v k ),emb(y j )) for the  "Average weight" setting. Besides cosine similarity, the Euclidean distance and the dot product are also common similarity measures for embeddings. Consequently, we fix the weight w(v i , y j ) = 1, choose S (emb(v i ), emb(y j )) = −∥emb(v i ) − emb(y j )∥ for the "Euclidean distance" setting and S (emb(v i ), emb(y j )) = emb(v i )·emb(y j ) for the "Dot product" setting. It can be informed from Figure 2 that with a fixed similarity function, different weight calculations yield comparable results, but with a fixed weight, cosine similarity is the optimal similarity measure. 
Pre-trained Language Models Can be Fully Zero-Shot Learners
NPPrompt sums up all logits for a label word set as shown in Equation 2. Another possible approach is to sum up the probabilities from PLM's prediction for the label words and choose the argmax for all different labels as the prediction:
Pre-trained Language Models Can be Fully Zero-Shot Learners
We conduct experiments on AG News to compare the above two approaches, one that sums up logits ("sum logit") and one that sums up probabilities ("sum prob"). Figure 3 presents the results and we find that "sum logit" performs better at small k but "sum prob" delivers better results when k exceeds 30. "sum logit" achieves the best result at k = 12 among all experiments.
Pre-trained Language Models Can be Fully Zero-Shot Learners
The number of label words impacts the performance of our method NPPrompt as well. In Figure 4, we display the performances of different models with varied neighborhood numbers. In general, NPPrompt attains similar test accuracy across different neighborhood numbers. Regardless of the choice for neighborhood number, NPPrompt-RoBERTa-large achieves over 80% accuracy in topic classification tasks on AG News and DBPedia, and it gains over 90% accuracy in sentiment classification tasks on IMDB and Amazon. In realworld applications, we can simply choose a fixed neighborhood number (e.g. 8-10) to achieve decent performance.  The performance of NPPrompt heavily relies on the choice of the pre-trained language model. This is due to the variations in label words for different categories, which stem from the distinct initial word embeddings and vocabularies employed by each PLM. Additionally, NPPrompt can be adapted for text generation models such as T5 (Raffel et al., 2020) and GPT-2 (Radford et al., 2019)) with minor modifications. In our approach, we utilize T5base/GPT2-base to generate the missing spans at the end of the prompt text. The first predicted token serves as the input to the verbalizer, and we follow the nonparametric aggregation steps outlined in Appendix A.1 to determine the category.
Pre-trained Language Models Can be Fully Zero-Shot Learners
To investigate the impact of employing different PLMs, we conduct additional experiments using BERT-base-cased, BERT-large-cased, RoBERTabase, T5-base, and GPT2-base models. The results are presented in Table 6. Notably, NPPrompt with RoBERTa-large achieves the highest performance, which can be attributed to the model's extensive parameter count and the fact that it is pretrained on a large corpus. As anticipated, larger models such as RoBERTa-large and BERT-large outperform their base counterparts (RoBERTa-base and BERT-base) on average, with RoBERTa consistently exhibiting superior accuracy compared to BERT models. While NPPrompt-T5-base and NPPrompt-GPT2-base demonstrate commendable performance, they do not surpass the performance of NPPrompt-RoBERTa-large.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Our research extends beyond text classification and encompasses experiments on multiple-choice question answering (QA) tasks as well. Specifically,  we assess the performance of NPPrompt using the widely-utilized CommonsenseQA (CQA) dataset (Talmor et al., 2019). In this new setting, we use the prompt template "x The answer is [MASK].", e.g. "What do animals do when an enemy is approaching? The answer is [MASK].". Subsequently, we search for the k-nearest neighbors for each target answer, setting k as 15. The prediction is obtained by applying the same process employed for text classification tasks. The results of our experiments are presented in Table 7 (few-shot results obtained from (Zelikman et al., 2022)). Notably, NPPrompt not only achieves satisfactory performance on the CommonsenseQA dataset but even outperforms few-shot GPT-J (Wang, 2021) as well. This demonstrates the versatility and flexibility of NPPrompt across various NLP scenarios.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Our proposed method, NPPrompt, demonstrates exceptional performance in zero-shot text classification tasks. We attribute this success to two key factors. Firstly, by utilizing the initial word embedding from pre-trained language models (PLMs), we are able to identify cognates of the label words. For instance, in Table 1, we observe variations of the word "business" such as "Business" and "businesses" for the BUSINESS category. Secondly, we effectively leverage the capabilities of pre-trained language models by reformulating the zero-shot classification problem as a masked token prediction task, which aligns with the pre-training process. Furthermore, NPPrompt offers a promising solution for dynamic and open zero-shot classification problems, where new classes may arise or old classes may be removed. With the use of efficient PLMs and category names, as well as the key word design in Equation 4, NPPrompt can also be applied in scenarios where label names do not possess semantic meaning (e.g. categories with label names "A", "B", "C"). This technique has the potential for wide deployment in real-world applications.
Pre-trained Language Models Can be Fully Zero-Shot Learners
In this paper, we propose NPPrompt, a novel and effective method for fully zero-shot learning with pre-trained language models. We use initial word embedding of PLM to automatically find related words for category names, which enables us to construct the verbalizers without manual design or unlabeled corpus. Experimental results show that NPPrompt outperforms the previous zero-shot methods by large margins.
Pre-trained Language Models Can be Fully Zero-Shot Learners
For those label names without semantic meanings, several keywords are still required for NPPrompt to work well. Furthermore, this study focuses exclusively on the zero-shot setting. However, there are potential avenues for exploration in the few-shot scenario, which is prevalent in practical applications. The applicability of NPPrompt to other tasks, such as ranking and relation extraction, remains uncertain and warrants further investigation. Designing a refinement method to jointly search for label words and templates can be a promising direction for future research.
Pre-trained Language Models Can be Fully Zero-Shot Learners
A.1 Experimental Details Table 8 shows all the manual templates of NSP-BERT. We show the prompt templates for NPPrompt-T5 in Table 9. Table 11 summarizes manual designed descriptions of each dataset for Semantic Retrieval. As for GPT-3, we query the OpenAI API 1 and test with Davinci-001 model. The prompts for GPT-3 are shown in Table 12. We list all templates and label names for NPPrompt of all experiments in Table 13. We also list the related words result in sentiment classification (GOOD/BAD) and NLI (YES/NO)) tasks in Table  14.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Template AG News News: label name.
Pre-trained Language Models Can be Fully Zero-Shot Learners
News: label name. IMDB This text shows label name sentiment. Amazon The attitude of this text is label name.  1.00 " school" 1.00 "school" 1.00 "school" 1.00 " School" 0.80 " School" 0.75 "School" 0.69 "School" 0.70 " schools" 0.77 " schools" 0.71 "schools" 0.63 "schools" 0.63 "school" 0.74 "school" 0.70 "college" 0.55 "college" 0.54 " SCHOOL" 0.69 "School" 0.70 "university" 0.50 "university" 0.51 "School" 0.68 " SCHOOL" 0.56 "student" 0.42 "College" 0.40 " university" 0.66 " college" 0.50 "church" 0.41 "church" 0.40 " college" 0.65 " university" 0.50 "house" 0.38 "student" 0.37 " Schools" 0.65 " Schools" 0.49 "education" 0.38 "students" 0.37 " schooling" 0.64 " schooling" 0.45 "students" 0.37 "Schools" 0.37 " preschool" 0.63 " preschool" 0.44 "class" 0.37 "academy" 0.37 " kindergarten" 0.63 " kindergarten" 0.41 "town" 0.37 "class" 0.36 " classroom" 0.60 " student" 0.41 "College" 0.36 "education" 0.36 " student" 0.58 " students" 0.39 "Schools" 0.36 "University" 0.35 " education" 0.58 " classroom" 0.38 "work" 0.35 "house" 0.35 We summarize the label words of different PLMs for SCHOOL category in DBPedia in Table 10. RoBERTa-large and RoBERTa-base share similar sets of label words yet with a minor discrepancy 1 https://openai.com/api/ between their similarity scores. RoBERTa-large usually produces larger similarities than RoBERTabase. In contrast, the label words in RoBERTa are quite different from those in BERT.
Pre-trained Language Models Can be Fully Zero-Shot Learners
Here we extend our method to support multi-word label names like NATURALPLACE, MEANOFTRANSPORTATION and etc. The major part is to obtain related words to a multi-word label name. Once we obtain the related words, the rest non-parametric aggregation step remains identical. We consider two scenarios:
Pre-trained Language Models Can be Fully Zero-Shot Learners
The label name is multi-word (i.e., phrase) and related words are still single-words To model the phrase, we use average contextualized embedding instead of word embedding for both label names and related single-words to compute cosine similarity. As suggested in (Su et al., 2021), we whiten the contextualized output of RoBERTa by a linear transformation obtained from the contextualized embedding of all words in vocabulary. To obtain the best result, we select the output of layer 6 of RoBERTa. This extension achieves 61% accuracy on the DBPedia dataset using the original multi-word label names (original label names can be found at https://rdrr.io/cran/textdata/ man/dataset_dbpedia.html).
Pre-trained Language Models Can be Fully Zero-Shot Learners
Both the label name and related words are phrases Since the search space of a related phrase is exponentially large in its length, we use another prompt to filter candidate words. The template we use is "[LABEL_NAME] can also be called [MASK] * n.", where n is the length of the candidate. For example, if the label name is MEANOFTRANS-PORTATION and n = 2, the template will look like "Mean of transportation can also be called [MASK] [MASK].". We feed it to RoBERTa and filter top-k candidate phrases of masked prediction. Since masked prediction is conditionally independent of each mask, we further re-rank the top-k candidate phrases by either the contextualized embedding method mentioned above or another autoregressive LM. For the latter one, we evaluate the perplexity of the template with [MASK] filled by candidate phrases. This generates 71% accuracy on DBPedia if the length of the phrase is two and the re-ranking is performed by GPT-2 (Radford et al., 2019).
Pre-trained Language Models Can be Fully Zero-Shot Learners
Sim Word Sim Word Sim Word Sim " good" 1.00 " bad" 1.00 " Yes" 1.00 " No" 1.00 " Good" 0.73 " Bad" 0.71 " yes" 0.79 " no" 0.80 " GOOD" 0.72 " terrible" 0.69 " YES" 0.73 "No" 0.74 "good" 0.69 " BAD" 0.69 "Yes" 0.72 " NO" 0.70 " great" 0.66 " horrible" 0.68 " Yeah" 0.72 " Nope" 0.62 " excellent" 0.66 "bad" 0.65 " Yep" 0.65 " Yes" 0.62 " decent" 0.66 " awful" 0.64 " Sure" 0.62 "no" 0.61 "Good" 0.65 "Bad" 0.64 " No" 0.62 " Nobody" 0.59 " nice" 0.64 " good" 0.63 " Indeed" 0.61 " Nos" 0.57 " bad" 0.63 " badly" 0.62 " yeah" 0.60 " The" 0.57 " better" 0.62 " crappy" 0.60 "yes" 0.59 " Yeah" 0.57 " wonderful" 0.58 " lousy" 0.60 " Wow" 0.59 " Nothing" 0.56 " best" 0.58 " worst" 0.60 " Absolutely" 0.58 " Not" 0.56 " terrific" 0.57 " horrendous" 0.60 " Nope" 0.58 " Never" 0.56 " fantastic" 0.57 " worse" 0.59 " Okay" 0.57 " None" 0.55 " mediocre" 0.57 " nasty" 0.59 " Oh" 0.57 " Number" 0.55 " lousy" 0.57 " shitty" 0.59 " Hello" 0.57 " So" 0.54 " satisfactory" 0.56 " dreadful" 0.59 " Hey" 0.57 " Any" 0.54 " marvelous" 0.56 " rotten" 0.58 " Nevertheless" 0.57 " And" 0.54 " GREAT" 0.56 " harmful" 0.58 " However" 0.56 "NO" 0.53 Table 14: The top 20 similar words of label names in sentiment classification (GOOD/BAD) and NLI (YES/NO) tasks.