How have contrastive learning techniques been applied to learn dense sentence representations?
Recently, contrastive learning has been adopted
to learn sentence representations by designing dif-ferent augmentation methods (Zhang et al., 2020;
Carlsson et al., 2021; Giorgi et al., 2021; Yan et al.,
2021; Kim et al., 2021; Gao et al., 2021). A typical
example SimCSE uses dropout as a data augmen-
tation strategy and is also the foundation of many
following works. ArcCSE (Zhang et al., 2022) en-
hances the pairwise discriminative power and mod-
els the entailment relation among triplet sentences.
DCLR (Zhou et al., 2022) alleviates the influence of
improper negatives. DiffCSE (Chuang et al., 2022)
introduces equivariant contrastive learning to Sim-
CSE. PCL (Wu et al., 2022a) proposes contrastive
representation learning with diverse augmentation
strategies for an inherent anti-bias ability. InfoCSE
(Wu et al., 2022b) learns sentence representations
with the ability to reconstruct the original sentence
fragments. Generative learning techniques (Wang
et al., 2021; Wu and Zhao, 2022) have also been
proposed to enhance the linguistic interpretability
of sentence representations. Although achieving
promising results, these methods fail to capture
the fine-grained ranking knowledge among the sen-
tences.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using “entailment” pairs as positives and “contradiction” pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman’s correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show—both theoretically and empirically—that contrastive learning objective regularizes pre-trained embeddings’ anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.
[{"first":"Tianyu","last":"Gao","middle":[]},{"first":"Xingcheng","last":"Yao","middle":[]},{"first":"Danqi","last":"Chen","middle":[]}]
Self-Guided Contrastive Learning for BERT Sentence Representations
Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.
[{"first":"Taeuk","last":"Kim","middle":[]},{"first":"Kang Min","last":"Yoo","middle":[]},{"first":"Sang-goo","last":"Lee","middle":[]}]
Debiased Contrastive Learning of Unsupervised Sentence Representations
Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space.However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives (false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space.To address it, we present a new framework DCLR (Debiased Contrastive Learning of unsupervised sentence Representations) to alleviate the influence of these improper negatives.In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space.Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: bluehttps://github.com/RUCAIBox/DCLR.
[{"first":"Kun","last":"Zhou","middle":[]},{"first":"Beichen","last":"Zhang","middle":[]},{"first":"Xin","last":"Zhao","middle":[]},{"first":"Ji-Rong","last":"Wen","middle":[]}]
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning, which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other “harmful” types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.
[{"first":"Yung-Sung","last":"Chuang","middle":[]},{"first":"Rumen","last":"Dangovski","middle":[]},{"first":"Hongyin","last":"Luo","middle":[]},{"first":"Yang","last":"Zhang","middle":[]},{"first":"Shiyu","last":"Chang","middle":[]},{"first":"Marin","last":"Soljacic","middle":[]},{"first":"Shang-Wen","last":"Li","middle":[]},{"first":"Scott","last":"Yih","middle":[]},{"first":"Yoon","last":"Kim","middle":[]},{"first":"James","last":"Glass","middle":[]}]
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Learning high-quality sentence representations is a fundamental problem of natural language processing which could benefit a wide range of downstream tasks. Though the BERT-like pre-trained language models have achieved great success, using their sentence representations directly often results in poor performance on the semantic textual similarity task. Recently, several contrastive learning methods have been proposed for learning sentence representations and have shown promising results. However, most of them focus on the constitution of positive and negative representation pairs and pay little attention to the training objective like NT-Xent, which is not sufficient enough to acquire the discriminating power and is unable to model the partial order of semantics between sentences. So in this paper, we propose a new method ArcCSE, with training objectives designed to enhance the pairwise discriminative power and model the entailment relation of triplet sentences. We conduct extensive experiments which demonstrate that our approach outperforms the previous state-of-the-art on diverse sentence related tasks, including STS and SentEval.
[{"first":"Yuhao","last":"Zhang","middle":[]},{"first":"Hongji","last":"Zhu","middle":[]},{"first":"Yongliang","last":"Wang","middle":[]},{"first":"Nan","last":"Xu","middle":[]},{"first":"Xiaobo","last":"Li","middle":[]},{"first":"Binqiang","last":"Zhao","middle":[]}]
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Sentence embeddings are an important component of many natural language processing (NLP) systems. Like word embeddings, sentence embeddings are typically learned on large text corpora and then transferred to various downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sentence embeddings require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. In this paper, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Inspired by recent advances in deep metric learning (DML), we carefully design a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. When used to extend the pretraining of transformer-based language models, our approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. Importantly, our experiments suggest that the quality of the learned embeddings scale with both the number of trainable parameters and the amount of unlabelled training data. Our code and pretrained models are publicly available and can be easily adapted to new domains or used to embed unseen text.
[{"first":"John","last":"Giorgi","middle":[]},{"first":"Osvald","last":"Nitski","middle":[]},{"first":"Bo","last":"Wang","middle":[]},{"first":"Gary","last":"Bader","middle":[]}]
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.
[{"first":"Yuanmeng","last":"Yan","middle":[]},{"first":"Rumei","last":"Li","middle":[]},{"first":"Sirui","last":"Wang","middle":[]},{"first":"Fuzheng","last":"Zhang","middle":[]},{"first":"Wei","last":"Wu","middle":[]},{"first":"Weiran","last":"Xu","middle":[]}]
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multi-augmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings.
[{"first":"Qiyu","last":"Wu","middle":[]},{"first":"Chongyang","last":"Tao","middle":[]},{"first":"Tao","last":"Shen","middle":[]},{"first":"Can","last":"Xu","middle":[]},{"first":"Xiubo","last":"Geng","middle":[]},{"first":"Daxin","last":"Jiang","middle":[]}]
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods. Our code are available at https://github.com/caskcsg/sentemb/tree/main/InfoCSE.
[{"first":"Xing","last":"Wu","middle":[]},{"first":"Chaochen","last":"Gao","middle":[]},{"first":"Zijia","last":"Lin","middle":[]},{"first":"Jizhong","last":"Han","middle":[]},{"first":"Zhongyuan","last":"Wang","middle":[]},{"first":"Songlin","last":"Hu","middle":[]}]
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Learning universal sentence embeddings is a fundamental problem in natural language processing and has been studied extensively in the literature (Kiros et al., 2015;Hill et al., 2016;Conneau et al., 2017;Logeswaran and Lee, 2018;Cer et al., 2018;Reimers and Gurevych, 2019, inter alia).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In this work, we advance state-of-the-art sentence embedding methods and demonstrate that a contrastive objective can be extremely effective when coupled with pre-trained language models such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019). We present SimCSE, a simple contrastive sentence embedding framework, which can produce superior sentence embeddings, from either unlabeled or labeled data.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Our unsupervised SimCSE simply predicts the input sentence itself with only dropout (Srivastava et al., 2014) used as noise (Figure 1(a)). In other words, we pass the same sentence to the pre-trained encoder twice: by applying the standard dropout twice, we can obtain two different embeddings as "positive pairs". Then we take other sentences in the same mini-batch as "negatives", and the model predicts the positive one among negatives. Although it may appear strikingly simple, this approach outperforms training objectives such as predicting next sentences (Logeswaran and Lee, 2018) and discrete data augmentation (e.g., word deletion and replacement) by a large margin, and even matches previous supervised methods. Through careful analysis, we find that dropout acts as minimal "data augmentation" of hidden representations, while removing it leads to a representation collapse.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Our supervised SimCSE builds upon the recent success of using natural language inference (NLI) datasets for sentence embeddings (Conneau et al., 2017;Reimers and Gurevych, 2019) and incorporates annotated sentence pairs in contrastive learning (Figure 1(b)). Unlike previous work that casts it as a 3-way classification task (entailment, neutral and contradiction), we leverage the fact that entailment pairs can be naturally used as positive instances. We also find that adding corresponding contradiction pairs as hard negatives further improves performance. This simple use of NLI datasets achieves a substantial improvement compared to prior methods using the same datasets. We also compare to other labeled sentence-pair The pets are sitting on a couch. Different hidden dropout masks in two forward passes There are animals outdoors.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
There is a man.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
The man wears a business suit.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
A kid is skateboarding.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
A kit is inside the house.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Two dogs are running.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
A man surfing on the sea.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
A kid is on a skateboard.  datasets and find that NLI datasets are especially effective for learning sentence embeddings.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
To better understand the strong performance of SimCSE, we borrow the analysis tool from Wang and Isola (2020), which takes alignment between semantically-related positive pairs and uniformity of the whole representation space to measure the quality of learned embeddings. Through empirical analysis, we find that our unsupervised Sim-CSE essentially improves uniformity while avoiding degenerated alignment via dropout noise, thus improving the expressiveness of the representations. The same analysis shows that the NLI training signal can further improve alignment between positive pairs and produce better sentence embeddings. We also draw a connection to the recent findings that pre-trained word embeddings suffer from anisotropy (Ethayarajh, 2019;Li et al., 2020) and prove that-through a spectrum perspective-the contrastive learning objective "flattens" the singular value distribution of the sentence embedding space, hence improving uniformity.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We conduct a comprehensive evaluation of Sim-CSE on seven standard semantic textual similarity (STS) tasks (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016Cer et al., 2017;Marelli et al., 2014) and seven transfer tasks (Conneau and Kiela, 2018). On the STS tasks, our unsupervised and supervised models achieve a 76.3% and 81.6% averaged Spearman's correlation respectively using BERT base , a 4.2% and 2.2% improvement compared to previous best results. We also achieve competitive performance on the transfer tasks. Finally, we identify an incoherent evaluation issue in the literature and consolidate results of different settings for future work in evaluation of sentence embeddings.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Contrastive learning aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors (Hadsell et al., 2006). It assumes a set of paired examples
SimCSE: Simple Contrastive Learning of Sentence Embeddings
, where x i and x + i are semantically related. We follow the contrastive framework in Chen et al. (2020) and take a cross-entropy objective with in-batch negatives (Chen et al., 2017;Henderson et al., 2017): let h i and h + i denote the representations of x i and x + i , the training objective for (x i , x + i ) with a mini-batch of N pairs is:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
where τ is a temperature hyperparameter and sim(h 1 , h 2 ) is the cosine similarity
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In this work, we encode input sentences using a pre-trained language model such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019): h = f θ (x), and then fine-tune all the parameters using the contrastive learning objective (Eq. 1).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Positive instances. One critical question in contrastive learning is how to construct (x i , x + i ) pairs. In visual representations, an effective solution is to take two random transformations of the same image (e.g., cropping, flipping, distortion and rotation) as Dosovitskiy et al., 2014). A similar approach has been recently adopted in language representations (Wu et al., 2020;Meng et al., 2021) by applying augmentation techniques such as word deletion, reordering, and substitution. However, data augmentation in NLP is inherently difficult because of its discrete nature. As we will see in §3, simply using standard dropout on intermediate representations outperforms these discrete operators.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In NLP, a similar contrastive learning objective has been explored in different contexts (Henderson et al., 2017;Gillick et al., 2019;Karpukhin et al., 2020). In these cases, (x i , x + i ) are collected from supervised datasets such as question-passage pairs. Because of the distinct nature of x i and x + i , these approaches always use a dual-encoder framework, i.e., using two independent encoders f θ 1 and f θ 2 for x i and x + i . For sentence embeddings, Logeswaran and Lee (2018) also use contrastive learning with a dual-encoder approach, by forming current sentence and next sentence as (x i , x + i ). Alignment and uniformity. Recently, Wang and Isola (2020) identify two key properties related to contrastive learning-alignment and uniformityand propose to use them to measure the quality of representations. Given a distribution of positive pairs p pos , alignment calculates expected distance between embeddings of the paired instances (assuming representations are already normalized):
SimCSE: Simple Contrastive Learning of Sentence Embeddings
(2)
SimCSE: Simple Contrastive Learning of Sentence Embeddings
On the other hand, uniformity measures how well the embeddings are uniformly distributed:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
where p data denotes the data distribution. These two metrics are well aligned with the objective of contrastive learning: positive instances should stay close and embeddings for random instances should scatter on the hypersphere. In the following sections, we will also use the two metrics to justify the inner workings of our approaches.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
The idea of unsupervised SimCSE is extremely simple: we take a collection of sentences
SimCSE: Simple Contrastive Learning of Sentence Embeddings
and use x + i = x i . The key ingredient to get this to work with identical positive pairs is through the use of independently sampled dropout masks for x i and x + i . In standard training of Transformers (Vaswani et al., 2017), there are dropout masks placed on fully-connected layers as well as attention probabilities (default p = 0.1). We denote h z i = f θ (x i , z) where z is a random mask for dropout. We simply feed the same input to the encoder twice and get  The two columns denote whether we use one encoder or two independent encoders. Next 3 sentences: randomly sample one from the next 3 sentences. Delete one word: delete one word randomly (see Table 1).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
two embeddings with different dropout masks z, z , and the training objective of SimCSE becomes:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
for a mini-batch of N sentences. Note that z is just the standard dropout mask in Transformers and we do not add any additional dropout.  
SimCSE: Simple Contrastive Learning of Sentence Embeddings
, z) and g is a (random) discrete operator on x. We note that even deleting one word would hurt performance and none of the discrete augmentations outperforms dropout noise. We also compare this self-prediction training objective to the next-sentence objective used in Logeswaran and Lee (2018), taking either one encoder or two independent encoders. As shown in Table 2, we find that SimCSE performs much better than the next-sentence objectives (82.5 vs 67.4 on STS-B) and using one encoder instead of two makes a significant difference in our approach.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Why does it work? To further understand the role of dropout noise in unsupervised SimCSE, we try out different dropout rates in Table 3 and observe that all the variants underperform the default dropout probability p = 0.1 from Transformers. We find two extreme cases particularly interesting: "no dropout" (p = 0) and "fixed 0.1" (using default dropout p = 0.1 but the same dropout masks for the pair). In both cases, the resulting embeddings for the pair are exactly the same, and it leads to a dramatic performance degradation. We take the checkpoints of these models every 10 steps during training and visualize the alignment and uniformity metrics 3 in Figure 2, along with a simple data augmentation model "delete one word". As clearly shown, starting from pre-trained checkpoints, all models greatly improve uniformity. However, the alignment of the two special variants also degrades drastically, while our unsupervised SimCSE keeps a steady alignment, thanks to the use of dropout noise. It also demonstrates that starting from a pretrained checkpoint is crucial, for it provides good initial alignment. At last, "delete one word" improves the alignment yet achieves a smaller gain on the uniformity metric, and eventually underperforms unsupervised SimCSE.  Figure 2: alignuniform plot for unsupervised SimCSE, "no dropout", "fixed 0.1", and "delete one word". We visualize checkpoints every 10 training steps and the arrows indicate the training direction. For both align and uniform , lower numbers are better.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We have demonstrated that adding dropout noise is able to keep a good alignment for positive pairs (x, x + ) ∼ p pos . In this section, we study whether we can leverage supervised datasets to provide better training signals for improving alignment of our approach. Prior work (Conneau et al., 2017;Reimers and Gurevych, 2019) has demonstrated that supervised natural language inference (NLI) datasets (Bowman et al., 2015;Williams et al., 2018) are effective for learning sentence embeddings, by predicting whether the relationship between two sentences is entailment, neutral or contradiction. In our contrastive learning framework, we instead directly take (x i , x + i ) pairs from supervised datasets and use them to optimize Eq. 1.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Choices of labeled data. We first explore which supervised datasets are especially suitable for constructing positive pairs (x i , x + i ). We experiment with a number of datasets with sentence-pair examples, including 1) QQP 4 : Quora question pairs; 2) Flickr30k (Young et al., 2014): each image is annotated with 5 human-written captions and we consider any two captions of the same image as a positive pair; 3) ParaNMT (Wieting and Gimpel, 2018): a large-scale back-translation paraphrase dataset 5 ; and finally 4) NLI datasets: SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We train the contrastive learning model (Eq. 1) with different datasets and compare the results in Table 4. For a fair comparison, we also run experiments with the same # of training pairs. Among all the options, using entailment pairs from the NLI (SNLI + MNLI) datasets performs the best. We think this is reasonable, as the NLI datasets consist of high-quality and crowd-sourced pairs. Also, human annotators are expected to write the hypotheses manually based on the premises and two sentences tend to have less lexical overlap. For instance, we find that the lexical overlap (F1 measured between two bags of words) for the entailment pairs (SNLI + MNLI) is 39%, while they are 60% and 55% for QQP and ParaNMT.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Contradiction as hard negatives. Finally, we further take the advantage of the NLI datasets by using its contradiction pairs as hard negatives 6 . In NLI datasets, given one premise, annotators are required to manually write one sentence that is absolutely true (entailment), one that might be true (neutral), and one that is definitely false (contradiction). Therefore, for each premise and its entailment hypothesis, there is an accompanying contradiction hypothesis 7 (see Figure 1 for an example).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Formally, we extend (
SimCSE: Simple Contrastive Learning of Sentence Embeddings
i and x − i are entailment and contradiction hypotheses. The training objective i is then defined by (N is mini-batch size):
SimCSE: Simple Contrastive Learning of Sentence Embeddings
(5) As shown in Table 4, adding hard negatives can further improve performance (84.9 → 86.2) and this is our final supervised SimCSE. We also tried to add the ANLI dataset (Nie et al., 2020) or combine it with our unsupervised SimCSE approach, but didn't find a meaningful improvement. We also considered a dual encoder framework in supervised SimCSE and it hurt performance (86.2 → 84.2).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Recent work identifies an anisotropy problem in language representations (Ethayarajh, 2019; Li et al., 2020), i.e., the learned embeddings occupy a narrow cone in the vector space, which severely limits their expressiveness. Gao et al. (2019) 6 We also experimented with adding neutral hypotheses as hard negatives. See Section 6.3 for more discussion. 7 In fact, one premise can have multiple contradiction hypotheses. In our implementation, we only sample one as the hard negative and we did not find a difference by using more.  demonstrate that language models trained with tied input/output embeddings lead to anisotropic word embeddings, and this is further observed by Ethayarajh (2019) in pre-trained contextual representations.  show that singular values of the word embedding matrix in a language model decay drastically: except for a few dominating singular values, all others are close to zero. A simple way to alleviate the problem is postprocessing, either to eliminate the dominant principal components (Arora et al., 2017;Mu and Viswanath, 2018), or to map embeddings to an isotropic distribution (Li et al., 2020;Su et al., 2021). Another common solution is to add regularization during training (Gao et al., 2019;. In this work, we show that-both theoretically and empirically-the contrastive objective can also alleviate the anisotropy problem.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
The anisotropy problem is naturally connected to uniformity (Wang and Isola, 2020), both highlighting that embeddings should be evenly distributed in the space. Intuitively, optimizing the contrastive learning objective can improve uniformity (or ease the anisotropy problem), as the objective pushes negative instances apart. Here, we take a singular spectrum perspective-which is a common practice in analyzing word embeddings (Mu and Viswanath, 2018;Gao et al., 2019;, and show that the contrastive objective can "flatten" the singular value distribution of sentence embeddings and make the representations more isotropic.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Following Wang and Isola (2020), the asymptotics of the contrastive learning objective (Eq. 1) can be expressed by the following equation when the number of negative instances approaches infinity (assuming f (x) is normalized):
SimCSE: Simple Contrastive Learning of Sentence Embeddings
where the first term keeps positive instances similar and the second pushes negative pairs apart. When p data is uniform over finite samples
SimCSE: Simple Contrastive Learning of Sentence Embeddings
we can derive the following formula from the second term with Jensen's inequality:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Let W be the sentence embedding matrix corresponding to {x i } m i=1 , i.e., the i-th row of W is h i . Optimizing the second term in Eq. 6 essentially minimizes an upper bound of the summation of all elements in WW , i.e.,
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Since we normalize h i , all elements on the diagonal of WW are 1 and then tr(WW ) (the sum of all eigenvalues) is a constant. According to Merikoski (1984), if all elements in WW are positive, which is the case in most times according to Figure G.1, then Sum(WW ) is an upper bound for the largest eigenvalue of WW . When minimizing the second term in Eq. 6, we reduce the top eigenvalue of WW and inherently "flatten" the singular spectrum of the embedding space. Therefore, contrastive learning is expected to alleviate the representation degeneration problem and improve uniformity of sentence embeddings.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Compared to post-processing methods in Li et al. (2020); Su et al. (2021), which only aim to encourage isotropic representations, contrastive learning also optimizes for aligning positive pairs by the first term in Eq. 6, which is the key to the success of SimCSE. A quantitative analysis is given in §7.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We conduct our experiments on 7 semantic textual similarity (STS) tasks. Note that all our STS experiments are fully unsupervised and no STS training sets are used. Even for supervised SimCSE, we simply mean that we take extra labeled datasets for training, following previous work (Conneau et al., 2017). We also evaluate 7 transfer learning tasks and provide detailed results in Appendix E. We share a similar sentiment with Reimers and Gurevych (2019) that the main goal of sentence embeddings is to cluster semantically similar sentences and hence take STS as the main result.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Semantic textual similarity tasks. We evaluate on 7 STS tasks: STS 2012-2016(Agirre et al., 2012, 2013, 2015, 2016, STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). When comparing to previous work, we identify invalid comparison patterns in published papers in the evaluation settings, including (a) whether to use an additional regressor, (b) Spearman's vs Pearson's correlation, and (c) how the results are aggregated (Table B.1). We discuss the detailed differences in Appendix B and choose to follow the setting of Reimers and Gurevych (2019) in our evaluation (no additional regressor, Spearman's correlation, and "all" aggregation). We also report our replicated study of previous work as well as our results evaluated in a different setting in Table B.2 and Table B (Su et al., 2021), we only report the "NLI" setting (see Table C.1).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We compare unsupervised and supervised Sim-CSE to previous state-of-the-art sentence embedding methods on STS tasks. Unsupervised baselines include average GloVe embeddings (Pennington et al., 2014), average BERT or RoBERTa embeddings 10 , and post-processing methods such as BERT-flow (Li et al., 2020) and BERTwhitening (Su et al., 2021). We also compare to several recent methods using a contrastive objective, including 1) IS-BERT (Zhang et al., 2020), which maximizes the agreement between global and local features; 2) DeCLUTR (Giorgi et al., 2021), which takes different spans from the same document as positive pairs; 3) CT (Carlsson et al., 2021), which aligns embeddings of the same sentence from two different encoders. 11 Other supervised 10 Following Su et al. (2021), we take the average of the first and the last layers, which is better than only taking the last. 11 We do not compare to CLEAR (Wu et al., 2020), because they use their own version of pre-trained models, and the numbers appear to be much lower. Also note that CT is a concurrent work to ours. methods include InferSent (Conneau et al., 2017), Universal Sentence Encoder (Cer et al., 2018), and SBERT/SRoBERTa (Reimers and Gurevych, 2019) with post-processing methods (BERT-flow, whitening, and CT). We provide more details of these baselines in Appendix C. Table 5 shows the evaluation results on 7 STS tasks. SimCSE can substantially improve results on all the datasets with or without extra NLI supervision, greatly outperforming the previous stateof-the-art models. Specifically, our unsupervised SimCSE-BERT base improves the previous best averaged Spearman's correlation from 72.05% to 76.25%, even comparable to supervised baselines. When using NLI datasets, SimCSE-BERT base further pushes the state-of-the-art results to 81.57%. The gains are more pronounced on RoBERTa encoders, and our supervised SimCSE achieves 83.76% with RoBERTa large .
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In Appendix E, we show that SimCSE also achieves on par or better transfer task performance compared to existing work, and an auxiliary MLM objective can further boost performance.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Unsup. Sup.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
[   
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We investigate the impact of different pooling methods and hard negatives. All reported results in this section are based on the STS-B development set.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We provide more ablation studies (normalization, temperature, and MLM objectives) in Appendix D.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Pooling methods. Reimers and Gurevych (2019); Li et al. (2020) show that taking the average embeddings of pre-trained models (especially from both the first and last layers) leads to better performance than [CLS]. Hard negatives. Intuitively, it may be beneficial to differentiate hard negatives (contradiction examples) from other in-batch negatives. Therefore, we extend our training objective defined in Eq. 5 to incorporate weighting of different negatives:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
where 1 j i ∈ {0, 1} is an indicator that equals 1 if and only if i = j. We train SimCSE with different values of α and evaluate the trained models on the development set of STS-B. We also consider taking neutral hypotheses as hard negatives. As shown in Table 7, α = 1 performs the best, and neutral hypotheses do not bring further gains.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In this section, we conduct further analyses to understand the inner workings of SimCSE.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Uniformity and alignment. Figure 3 shows uniformity and alignment of different sentence embedding models along with their averaged STS results. In general, models which have both better alignment and uniformity achieve better performance, confirming the findings in Wang and Isola (2020). We also observe that (1) though pre-trained embeddings have good alignment, their uniformity is poor (i.e., the embeddings are highly anisotropic);
SimCSE: Simple Contrastive Learning of Sentence Embeddings
(2) post-processing methods like BERT-flow and BERT-whitening greatly improve uniformity but also suffer a degeneration in alignment; (3) unsupervised SimCSE effectively improves uniformity of pre-trained embeddings whereas keeping a good alignment; (4) incorporating supervised data in SimCSE further amends alignment. In Appendix F, we further show that SimCSE can effectively flatten singular value distribution of pre-trained embeddings. In Appendix G, we demonstrate that SimCSE provides more distinguishable cosine similarities between different sentence pairs. Qualitative comparison. We conduct a smallscale retrieval experiment using SBERT base and SimCSE-BERT base . We use 150k captions from Flickr30k dataset and take any random sentence as query to retrieve similar sentences (based on cosine similarity). As several examples shown in Table 8, the retrieved sentences by SimCSE have a higher quality compared to those retrieved by SBERT.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Query: A dog runs on the green grass near a wooden fence. #1 A dog runs on the green grass near a grove of trees.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
The dog by the fence is running on the grass. #2 A brown and white dog runs through the green grass.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Dog running through grass in fenced area. #3 The dogs run in the green field.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
A dog runs on the green grass near a grove of trees.  with n-gram embeddings leads to strong results. Several recent (and concurrent) approaches adopt contrastive objectives (Zhang et al., 2020;Giorgi et al., 2021;Wu et al., 2020;Meng et al., 2021;Carlsson et al., 2021;Kim et al., 2021;Yan et al., 2021) by taking different views-from data augmentation or different copies of models-of the same sentence or document. Compared to these work, SimCSE uses the simplest idea by taking different outputs of the same sentence from standard dropout, and performs the best on STS tasks.  Su et al., 2021;Huang et al., 2021) to alleviate the representation degeneration problem (as discussed in §5), and yields substantial improvement over pre-trained language models.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
In this work, we propose SimCSE, a simple contrastive learning framework, which greatly improves state-of-the-art sentence embeddings on semantic textual similarity tasks. We present an unsupervised approach which predicts input sentence itself with dropout noise and a supervised approach utilizing NLI datasets. We further justify the inner workings of our approach by analyzing alignment and uniformity of SimCSE along with other baseline models. We believe that our contrastive objective, especially the unsupervised one, may have a broader application in NLP. It provides a new perspective on data augmentation with text input, and can be extended to other continuous representations and integrated in language model pre-training. 
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We implement SimCSE with transformers package (Wolf et al., 2020). For supervised Sim-CSE, we train our models for 3 epochs, evaluate the model every 250 training steps on the development set of STS-B and keep the best checkpoint for the final evaluation on test sets. We do the same for the unsupervised SimCSE, except that we train the model for one epoch. We carry out grid-search of batch size ∈ {64, 128, 256, 512} and learning rate ∈ {1e-5, 3e-5, 5e-5} on STS-B development set and adopt the hyperparameter settings in Table A.1. We find that SimCSE is not sensitive to batch sizes as long as tuning the learning rates accordingly, which contradicts the finding that contrastive learning requires large batch sizes (Chen et al., 2020). It is probably due to that all SimCSE models start from pre-trained checkpoints, which already provide us a good set of initial parameters.  Finally, we introduce one more optional variant which adds a masked language modeling (MLM) objective (Devlin et al., 2019) as an auxiliary loss to Eq. 1: + λ · mlm (λ is a hyperparameter). This helps SimCSE avoid catastrophic forgetting of token-level knowledge. As we will show in Table D.2, we find that adding this term can help improve performance on transfer tasks (not on sentence-level STS tasks).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We elaborate the differences in STS evaluation settings in previous work in terms of (a) whether to use additional regressors; (b) reported metrics; (c) different ways to aggregate results.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Additional regressors. The default SentEval implementation applies a linear regressor on top of  (2018) Pearson mean Reimers and Gurevych (2019) Spearman all Zhang et al. (2020) Spearman all Li et al. (2020) Spearman wmean Su et al. (2021) Spearman wmean Wieting et al. (2020) Pearson mean Giorgi et al. (2021) Spearman mean Ours Spearman all Table B.1: STS evaluation protocols used in different papers. "Reg.": whether an additional regressor is used; "aggr.": methods to aggregate different subset results.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
frozen sentence embeddings for STS-B and SICK-R, and train the regressor on the training sets of the two tasks, while most sentence representation papers take the raw embeddings and evaluate in an unsupervised way. In our experiments, we do not apply any additional regressors and directly take cosine similarities for all STS tasks.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Metrics. Both Pearson's and Spearman's correlation coefficients are used in the literature. Reimers et al. (2016) argue that Spearman correlation, which measures the rankings instead of the actual scores, better suits the need of evaluating sentence embeddings. For all of our experiments, we report Spearman's rank correlation.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Aggregation methods. Given that each year's STS challenge contains several subsets, there are different choices to gather results from them: one way is to concatenate all the topics and report the overall Spearman's correlation (denoted as "all"), and the other is to calculate results for different subsets separately and average them (denoted as "mean" if it is simple average or "wmean" if weighted by the subset sizes). However, most papers do not claim the method they take, making it challenging for a fair comparison. We take some of the most recent work: SBERT (Reimers and Gurevych, 2019), BERT-flow (Li et al., 2020) and BERT-whitening (Su et al., 2021) 12 as an example: In Table B.2, we compare our reproduced results to reported results of SBERT and BERT-whitening, and find that Reimers and Gurevych (2019) take the "all" setting but Li et al. (2020); Su et al. (2021) take the "wmean" setting, even though Li et al. (2020) claim that they take the same setting as Reimers Table B.2: Comparisons of our reproduced results using different evaluation protocols and the original numbers. ♣: results from Reimers and Gurevych (2019); ♠: results from Su et al. (2021); Other results are reproduced by us. From the table we see that SBERT takes the "all" evaluation and BERT-whitening takes the "wmean" evaluation. and Gurevych (2019). Since the "all" setting fuses data from different topics together, it makes the evaluation closer to real-world scenarios, and unless specified, we take the "all" setting. We list evaluation settings for a number of previous work in Table B.1. Some of the settings are reported by the paper and some of them are inferred by comparing the results and checking their code. As we can see, the evaluation protocols are very incoherent across different papers. We call for unifying the setting in evaluating sentence embeddings for future research. We will also release our evaluation code for better reproducibility. Since previous work uses different evaluation protocols from ours, we further evaluate our models in these settings to make a direct comparison to the published numbers. We evaluate SimCSE with "wmean" and Spearman's correlation to directly compare to Li et al. (2020) and Su et al. (2021) in Table B.3.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
We elaborate on how we obtain different baselines for comparison in our experiments:
SimCSE: Simple Contrastive Learning of Sentence Embeddings
• For average GloVe embedding (Pennington et al., 2014) • For BERT-flow (Li et al., 2020), since their original numbers take a different setting, we retrain their models using their code 15 , and evaluate the models using our own script.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
• For BERT-whitening (Su et al., 2021), we implemented our own version of whitening script following the same pooling method in Su et al. (2021), i.e. first-last average pooling. Our implementation can reproduce the results from the original paper (see Table B.2).
SimCSE: Simple Contrastive Learning of Sentence Embeddings
For both BERT-flow and BERT-whitening, they have two variants of postprocessing: one takes the NLI data ("NLI") and one directly learns the embedding distribution on the target sets ("target"). We find that in our evaluation setting, "target" is generally worse than "NLI" (Table C.1), so we only report the NLI variant in the main results.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Normalization and temperature. We train Sim-CSE using both dot product and cosine similarity with different temperatures and evaluate them on the STS-B development set. As shown in Table D Zhang et al. (2020). We highlight the highest numbers among models with the same pre-trained encoder. MLM: adding MLM as an auxiliary task with λ = 0.1. flow/whitening) and find that they both hurt performance compared to their base models, showing that good uniformity of representations does not lead to better embeddings for transfer learning. As we argued earlier, we think that transfer tasks are not a major goal for sentence embeddings, and thus we take the STS results for main comparison.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Figure F.1 shows the singular value distribution of SimCSE together with other baselines. For both unsupervised and supervised cases, singular value drops the fastest for vanilla BERT or SBERT embeddings, while SimCSE helps flatten the spectrum distribution. Postprocessing-based methods such as BERT-flow or BERT-whitening flatten the curve even more since they directly aim for the goal of mapping embeddings to an isotropic distribution.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
To directly show the strengths of our approaches on STS tasks, we illustrate the cosine similarity distributions of STS-B pairs with different groups of human ratings in Figure G.1. Compared to all the baseline models, both unsupervised and supervised SimCSE better distinguish sentence pairs with different levels of similarities, thus leading to a better performance on STS tasks. In addition, we observe that SimCSE generally shows a more scattered distribution than BERT or SBERT, but also preserves a lower variance on semantically similar sentence pairs compared to whitened distribution. This observation further validates that SimCSE can achieve a better alignment-uniformity balance. 
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Supervised SimCSE-BERTbase Figure G.1: Density plots of cosine similarities between sentence pairs in STS-B. Pairs are divided into 5 groups based on ground truth ratings (higher means more similar) along the y-axis, and x-axis is the cosine similarity.
Self-Guided Contrastive Learning for BERT Sentence Representations
Pre-trained Transformer (Vaswani et al., 2017) language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been integral to achieving recent improvements in natural language understanding. However, it is not straightforward to directly utilize these models for sentencelevel tasks, as they are basically pre-trained to focus on predicting (sub)word tokens given context. The most typical way of converting the models into sentence encoders is to fine-tune them with supervision from a downstream task. In the process, as initially proposed by Devlin et al. (2019), a pre-defined token's (a.k.a. [CLS]) embedding from the last layer of the encoder is deemed as the representation of an input sequence. This simple but effective method is possible because, during supervised fine-tuning, the [CLS] embedding functions as the only communication gate between the pre-trained encoder Figure 1: BERT(-base)'s layer-wise performance with different pooling methods on the STS-B test set. We observe that the performance can be dramatically varied according to the selected layer and pooling strategy. Our self-guided training (SG / SG-OPT) assures much improved results compared to those of the baselines. and a task-specific layer, encouraging the [CLS] vector to capture the holistic information.
Self-Guided Contrastive Learning for BERT Sentence Representations
On the other hand, in cases where labeled datasets are unavailable, it is unclear what the best strategy is for deriving sentence embeddings from BERT. 1 In practice, previous studies (Reimers and Gurevych, 2019;Li et al., 2020;Hu et al., 2020) reported that naïvely (i.e., without any processing) leveraging the [CLS] embedding as a sentence representation, as is the case of supervised finetuning, results in disappointing outcomes. Currently, the most common rule of thumb for building BERT sentence embeddings without supervision is to apply mean pooling on the last layer(s) of BERT.
Self-Guided Contrastive Learning for BERT Sentence Representations
Yet, this approach can be still sub-optimal. In a preliminary experiment, we constructed sentence embeddings by employing various combinations of different BERT layers and pooling methods, and tested them on the Semantic Textual Similarity (STS) benchmark dataset (Cer et al., 2017). 2 We discovered that BERT(-base)'s performance, measured in Spearman correlation (× 100), can range from as low as 16.71 ([CLS], the 10 th layer) to 63.19 (max pooling, the 2 nd layer) depending on the selected layer and pooling method (see Figure 1). This result suggests that the current practice of building BERT sentence vectors is not solid enough, and that there is room to bring out more of BERT's expressiveness.
Self-Guided Contrastive Learning for BERT Sentence Representations
In this work, we propose a contrastive learning method that makes use of a newly proposed selfguidance mechanism to tackle the aforementioned problem. The core idea is to recycle intermediate BERT hidden representations as positive samples to which the final sentence embedding should be close. As our method does not require data augmentation, which is essential in most recent contrastive learning frameworks, it is much simpler and easier to use than existing methods (Fang and Xie, 2020;. Moreover, we customize the NT-Xent loss , a contrastive learning objective widely used in computer vision, for better sentence representation learning with BERT. We demonstrate that our approach outperforms competitive baselines designed for building BERT sentence vectors (Li et al., 2020;Wang and Kuo, 2020) in various environments. With comprehensive analyses, we also show that our method is more computationally efficient than the baselines at inference in addition to being more robust to domain shifts.
Self-Guided Contrastive Learning for BERT Sentence Representations
As BERT mostly requires some type of adaptation to be properly applied to a task of interest, it might not be desirable to derive sentence embeddings directly from BERT without fine-tuning. While Reimers and Gurevych (2019) attempt to alleviate this problem with typical supervised fine-tuning, we restrict ourselves to revising BERT in an unsupervised manner, meaning that our method only demands a bunch of raw sentences for training.
Self-Guided Contrastive Learning for BERT Sentence Representations
Among possible unsupervised learning strategies, we concentrate on contrastive learning which can inherently motivate BERT to be aware of similarities between different sentence embeddings. Considering that sentence vectors are widely used in computing the similarity of two sentences, the inductive bias introduced by contrastive learning can be helpful for BERT to work well on such tasks. The problem is that sentence-level contrastive learning usually requires data augmentation (Fang and Xie, 2020) or prior knowledge on training data, e.g., order information (Logeswaran and Lee, 2018), to make plausible positive/negative samples. We attempt to circumvent these constraints by utilizing the hidden representations of BERT, which are readily accessible, as samples in the embedding space.
Self-Guided Contrastive Learning for BERT Sentence Representations
We aim at developing a contrastive learning method that is free from external procedure such as data augmentation. A possible solution is to leverage (virtual) adversarial training (Miyato et al., 2018) in the embedding space. However, there is no assurance that the semantics of a sentence embedding would remain unchanged when it is added with a 
Self-Guided Contrastive Learning for BERT Sentence Representations
Layer ℓ Figure 2: Self-guided contrastive learning framework. We clone BERT into two copies at the beginning of training. BERT T (except Layer 0) is then fine-tuned to optimize the sentence vector c i while BERT F is fixed. random noise. As an alternative, we propose to utilize the hidden representations from BERT's intermediate layers, which are conceptually guaranteed to represent corresponding sentences, as pivots that BERT sentence vectors should be close to or be away from. We call our method as self-guided contrastive learning since we exploit internal training signals made by BERT itself to fine-tune it. We describe our training framework in Figure  2. First, we clone BERT into two copies, BERT F (fixed) and BERT T (tuned) respectively. BERT F is fixed during training to provide a training signal while BERT T is fine-tuned to construct better sentence embeddings. The reason why we differentiate BERT F from BERT T is that we want to prevent the training signal computed by BERT F from being degenerated as the training procedure continues, which often happens when BERT F = BERT T . This design decision also reflects our philosophy that our goal is to dynamically conflate the knowledge stored in BERT's different layers to produce sentence embeddings, rather than introducing new information via extra training. Note that in our setting, the [CLS] vector from the last layer of BERT T , i.e., c i , is regarded as the final sentence embedding we aim to optimize/utilize during/after fine-tuning.
Self-Guided Contrastive Learning for BERT Sentence Representations
Second, given b sentences in a mini-batch, say s 1 , s 2 , · · · , s b , we feed each sentence s i into BERT F and compute token-level hidden representations H i,k ∈ R len(s i )×d :
Self-Guided Contrastive Learning for BERT Sentence Representations
where 0 ≤ k ≤ l (0: the non-contextualized layer), l is the number of hidden layers in BERT, len(s i ) is the length of the tokenized sentence, and d is the size of BERT's hidden representations. Then, we apply a pooling function p to H i,k for deriving diverse sentence-level views h i,k ∈ R d from all layers, i.e., h i,k = p(H i,k ). Finally, we choose the final view to be utilized by applying a sampling function σ:
Self-Guided Contrastive Learning for BERT Sentence Representations
As we have no specific constraints in defining p and σ, we employ max pooling as p and a uniform sampler as σ for simplicity, unless otherwise stated. This simple choice for the sampler implies that each h i,k has the same importance, which is persuasive considering it is known that different BERT layers are specialized at capturing disparate linguistic concepts (Jawahar et al., 2019). 3 Third, we compute our sentence embedding c i for s i as follows:
Self-Guided Contrastive Learning for BERT Sentence Representations
where BERT(·) [CLS] corresponds to the [CLS] vector obtained from the last layer of BERT. Next, we collect the set of the computed vectors into X = {x|x ∈ {c i } ∪ {h i }}, and for all x m ∈ X, we compute the NT-Xent loss :
Self-Guided Contrastive Learning for BERT Sentence Representations
Note that τ is a temperature hyperparameter, f is a projection head consisting of MLP layers, 4 g(u, v) = u · v/ u v is the cosine similarity function, and µ(·) is the matching function defined as follows,
Self-Guided Contrastive Learning for BERT Sentence Representations
Lastly, we sum all L base m divided by 2b, and add a regularizer L reg = BERT F − BERT T As a result, the final loss L base is:
Self-Guided Contrastive Learning for BERT Sentence Representations
where the coefficient λ is a hyperparameter.
Self-Guided Contrastive Learning for BERT Sentence Representations
To summarize, our method refines BERT so that the sentence embedding c i has a higher similarity with h i , which is another representation for the sentence s i , in the subspace projected by f while being relatively dissimilar with c j,j =i and h j,j =i . After training is completed, we remove all the components except BERT T and simply use c i as the final sentence representation.
Self-Guided Contrastive Learning for BERT Sentence Representations
In Section 3.1, we relied on a simple variation of the general NT-Xent loss, which is composed of four factors. Given sentence s i and s j without loss of generality, the factors are as follows ( Figure 3):
Self-Guided Contrastive Learning for BERT Sentence Representations
The main component that mirrors our core motivation that a BERT sentence vector (c i ) should be consis-
Self-Guided Contrastive Learning for BERT Sentence Representations
Even though all the four factors play a certain role, some components may be useless or even cause a negative influence on our goal. For instance,  have recently reported that in image representation learning, only (1) is vital while others are nonessential. Likewise, we customize the training loss with three major modifications so that it can be more well-suited for our purpose. First, as our aim is to improve c i with the aid of h i , we re-define our loss focusing more on c i rather than considering c i and h i as equivalent entities:
Self-Guided Contrastive Learning for BERT Sentence Representations
. In other words, h i only functions as points that c i is encouraged to be close to or away from, and is not deemed as targets to be optimized. This revision naturally results in removing (4). Furthermore, we discover that (2) is also insignificant for improving performance, and thus derive L opt2 i :
Self-Guided Contrastive Learning for BERT Sentence Representations
). Lastly, we diversify signals from (1) and (3) by allowing multiple views {h i,k } to guide c i :
Self-Guided Contrastive Learning for BERT Sentence Representations
We expect with this refinement that the learning objective can provide more precise and fruitful training signals by considering additional (and freely available) samples being provided with. The final form of our optimized loss is:
Self-Guided Contrastive Learning for BERT Sentence Representations
In Section 5.1, we show the decisions made in this section contribute to improvements in performance.
Self-Guided Contrastive Learning for BERT Sentence Representations
In terms of pre-trained encoders, we leverage BERT (Devlin et al., 2019) for English datasets and MBERT, which is a multilingual variant of BERT, for multilingual datasets. We also employ RoBERTa (Liu et al., 2019) and SBERT (Reimers and Gurevych, 2019) in some cases to evaluate the generalizability of tested methods. We use the suffixes '-base' and '-large' to distinguish small and large models. Every trainable model's performance is reported as the average of 8 separate runs to reduce randomness. Hyperparameters are optimized on the STS-B validation set using BERTbase and utilized across different models. See Table  8 in Appendix A.1 for details. Our implementation is based on the HuggingFace's Transformers (Wolf et al., 2019) and SBERT (Reimers and Gurevych, 2019) library, and publicly available at https://github.com/galsang/SG-BERT.
Self-Guided Contrastive Learning for BERT Sentence Representations
We first evaluate our method and baselines on Semantic Textual Similarity (STS) tasks. Given two sentences, we derive their similarity score by computing the cosine similarity of their embeddings.
Self-Guided Contrastive Learning for BERT Sentence Representations
Datasets and Metrics. Following the literature, we evaluate models on 7 datasets in total, that is, STS-B (Cer et al., 2017), SICK-R (Marelli et al., 2014), and STS12-16 (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016. These datasets contain pairs of two sentences, whose similarity scores are labeled from 0 to 5. The relevance between gold annotations and the scores predicted by sentence vectors is measured in Spearman correlation (× 100).
Self-Guided Contrastive Learning for BERT Sentence Representations
Baselines and Model Specification. We first prepare two non-BERT approaches as baselines, i.e., Glove (Pennington et al., 2014) mean embeddings and Universal Sentence Encoder (USE; Cer et al. (2018)). In addition, various methods for BERT sentence embeddings that do not require supervision are also introduced as baselines:
Self-Guided Contrastive Learning for BERT Sentence Representations
• CLS token embedding: It regards the [CLS] vector from the last layer of BERT as a sentence representation. • Mean pooling: This method conducts mean pooling on the last layer of BERT and use the output as a sentence embedding. • WK pooling: This follows the method of Wang and Kuo (2020), which exploits QR decomposition and extra techniques to derive meaningful sentence vectors from BERT. • Flow: This is BERT-flow proposed by Li et al. (2020), which is a flow-based model that maps the vectors made by taking mean pooling on the last two layers of BERT to a Gaussian space. 6 • Contrastive (BT): Following Fang and Xie (2020), we revise BERT with contrastive learning. However, this method relies on back-translation to obtain positive samples, unlike ours. Details about this baseline are specified in Appendix A.2.
Self-Guided Contrastive Learning for BERT Sentence Representations
We make use of plain sentences from STS-B to fine-tune BERT using our approach, identical with Flow. 7 We name the BERT instances trained with our self-guided method as Contrastive (SG) and  Contrastive (SG-OPT), which utilize L base and L opt in Section 3 respectively.
Self-Guided Contrastive Learning for BERT Sentence Representations
Results. We report the performance of different approaches on STS tasks in Table 1 and Table 11 (Appendix A.6). From the results, we confirm the fact that our methods (SG and SG-OPT) mostly outperform other baselines in a variety of experimental settings. As reported in earlier studies, the naïve [CLS] embedding and mean pooling are turned out to be inferior to sophisticated methods. To our surprise, WK pooling's performance is even lower than that of mean pooling in most cases, and the only exception is when WK pooling is applied to SBERT-base. Flow shows its strength outperforming the simple strategies. Nevertheless, its performance is shown to be worse than that of our methods (although some exceptions exist in the case of SBERT-large). Note that contrastive learning becomes much more competitive when it is combined with our self-guidance algorithm rather than back-translation. It is also worth mentioning  that the optimized version of our method (SG-OPT) generally shows better performance than the basic one (SG), proving the efficacy of learning objective optimization (Section 3.2). To conclude, we demonstrate that our self-guided contrastive learning is effective in improving the quality of BERT sentence embeddings when tested on STS tasks.
Self-Guided Contrastive Learning for BERT Sentence Representations
We expand our experiments to multilingual settings by utilizing MBERT and cross-lingual zero-shot transfer. Specifically, we refine MBERT using only  English data and test it on datasets written in other languages. As in Section 4.2, we use the English STS-B for training. We consider two datasets for evaluation: (1) SemEval-2014 Task 10 (Spanish; Agirre et al. (2014)) and (2) SemEval-2017 Task 1 (Arabic, Spanish, and English; Cer et al. (2017)). Performance is measured in Pearson correlation (× 100) for a fair comparison with previous work. From Table 2, we see that MBERT with mean pooling already outperforms the best system (at the time of the competition was held) on SemEval-2014 and that our method further boosts the model's performance. In contrast, in the case of SemEval-2017 (Table 3), MBERT with mean pooling even fails to beat the strong Cosine baseline. 8 However, MBERT becomes capable of outperforming (in English/Spanish) or being comparable with (Arabic) the baseline by adopting our algorithm. We observe that while cross-lingual transfer using MBERT looks promising for the languages analogous to English (e.g., Spanish), its effectiveness may shrink on distant languages (e.g., Arabic). Compared against the best system which is trained on task-specific data, MBERT shows reasonable performance considering that it is never exposed to any labeled STS datasets. In summary, we demonstrate that MBERT fine-tuned with our method has a potential to be used as a simple but effective tool for multilingual (especially European) STS tasks.
Self-Guided Contrastive Learning for BERT Sentence Representations
We also evaluate BERT sentence vectors using the SentEval (Conneau and Kiela, 2018) toolkit. Given sentence embeddings, SentEval trains linear classifiers on top of them and estimates the quality of the vectors via their performance (accuracy) on down-8 The Cosine baseline computes its score as the cosine similarity of binary sentence vectors with each dimension representing whether an individual word appears in a sentence.  In Table 4, we compare our method (SG-OPT) with two baselines. 10 We find that our method is helpful over usual mean pooling in improving the performance of BERT-like models on SentEval. SG-OPT also outperforms WK pooling on BERTbase/large while being comparable on SBERT-base. From the results, we conjecture that self-guided contrastive learning and SBERT training suggest a similar inductive bias in a sense, as the benefit we earn by revising SBERT with our method is relatively lower than the gain we obtain when fine-tuning BERT. Meanwhile, it seems that WK pooling provides an orthogonal contribution that is effective in the focused case, i.e., SBERT-base.
Self-Guided Contrastive Learning for BERT Sentence Representations
In addition, we examine how our algorithm impacts on supervised fine-tuning of BERT, although it is not the main concern of this work. Briefly reporting, we identify that the original BERT(-base) and one tuned with SG-OPT show comparable performance on the GLUE (Wang et al., 2019) validation set, implying that our method does not influence much on BERT's supervised fine-tuning. We refer readers to Appendix A.4 for more details.
Self-Guided Contrastive Learning for BERT Sentence Representations
We here further investigate the working mechanism of our method with supplementary experiments. All the experiments conducted in this section follow the configurations stipulated in Section 4.1 and 4.2.  Table 5: Ablation study. Figure 4: Domain robustness study. The yellow bars indicate the performance gaps each method has according to which data it is trained with: in-domain (STS-B) or out-of-domain (NLI). Our method (SG-OPT) clearly shows its relative robustness compared to Flow.
Self-Guided Contrastive Learning for BERT Sentence Representations
We conduct an ablation study to justify the decisions made in optimizing our algorithm. To this end, we evaluate each possible variant on the test sets of STS tasks. From Table 5, we confirm that all our modifications to the NT-Xent loss contribute to improvements in performance. Moreover, we show that correct choices for hyperparameters are important for achieving the optimal performance, and that the projection head (f ) plays a significant role as in Chen et al. (2020).
Self-Guided Contrastive Learning for BERT Sentence Representations
Although our method in principle can accept any sentences in training, its performance might be varied with the training data it employs (especially depending on whether the training and test data share the same domain). To explore this issue, we apply SG-OPT on BERT-base by leveraging the mix of NLI datasets (Bowman et al., 2015;Williams et al., 2018) instead of STS-B, and observe the difference. From Figure 4, we confirm the fact  that no matter which test set is utilized (STS-B or all the seven STS tasks), our method clearly outperforms Flow in every case, showing its relative robustness to domain shifts. SG-OPT only loses 1.83 (on the STS-B test set) and 1.63 (on average when applied to all the STS tasks) points respectively when trained with NLI rather than STS-B, while Flow suffers from the considerable losses of 12.16 and 4.19 for each case. Note, however, that follow-up experiments in more diverse conditions might be desired as future work, as the NLI dataset inherently shares some similarities with STS tasks.
Self-Guided Contrastive Learning for BERT Sentence Representations
In this part, we compare the computational efficiency of our method to that of other baselines. For each algorithm, we measure the time elapsed during training (if required) and inference when tested on STS-B. All methods are run on the same machine (an Intel Xeon CPU E5-2620 v4 @ 2.10GHz and a Titan Xp GPU) using batch size 16. The experimental results specified in Table 6 show that although our method demands a moderate amount of time (< 8 min.) for training, it is the most efficient at inference, since our method is free from any post-processing such as pooling once training is completed.
Self-Guided Contrastive Learning for BERT Sentence Representations
We visualize a few variants of BERT sentence representations to grasp an intuition on why our method is effective in improving performance. Specifically, we sample 20 positive pairs (red, whose similarity scores are 5) and 20 negative pairs (blue, whose scores are 0) from the STS-B validation set. Then we compute their vectors and draw them on the 2D space with the aid of t-SNE. In Figure 5, we confirm that our SG-OPT encourages BERT sentence embeddings to be more well-aligned with their positive pairs while still being relatively far from their negative pairs. We also visualize embeddings from SBERT ( Figure 6 in Appendix A.5), and identify that our approach and the supervised fine-tuning   Red numbers correspond to positive sentence pairs and blue to negative pairs. used in SBERT provide a similar effect, making the resulting embeddings more suitable for calculating correct similarities between them.
Self-Guided Contrastive Learning for BERT Sentence Representations
In this section, we discuss a few weaknesses of our method in its current form and look into some possible avenues for future work.
Self-Guided Contrastive Learning for BERT Sentence Representations
First, while defining the proposed method in Section 3, we have made decisions on some parts without much consideration about their optimality, prioritizing simplicity instead. For instance, although we proposed utilizing all the intermediate layers of BERT and max pooling in a normal set-ting (indeed, it worked pretty well for most cases), a specific subset of the layers or another pooling method might bring better performance in a particular environment, as we observed in Section 4.4 that we could achieve higher numbers by employing mean pooling and excluding lower layers in the case of SentEval (refer to Appendix A.3 for details). Therefore, in future work, it is encouraged to develop a systematic way of making more optimized design choices in specifying our method by considering the characteristics of target tasks.
Self-Guided Contrastive Learning for BERT Sentence Representations
Second, we expect that the effectiveness of contrastive learning in revising BERT can be improved further by properly combining different techniques developed for it. As an initial attempt towards this direction, we conduct an extra experiment where we test the ensemble of back-translation and our self-guidance algorithm by inserting the original sentence into BERT T and its back-translation into BERT F when running our framework. In Table  7, we show that the fusion of the two techniques generally results in better performance, shedding some light on our future research direction.
Self-Guided Contrastive Learning for BERT Sentence Representations
In this paper, we have proposed a contrastive learning method with self-guidance for improving BERT sentence embeddings. Through extensive experiments, we have demonstrated that our method can enjoy the benefit of contrastive learning without relying on external procedures such as data augmentation or back-translation, succeeding in generating higher-quality sentence representations compared to competitive baselines. Furthermore, our method is efficient at inference because it does not require any post-processing once its training is completed, and is relatively robust to domain shifts.   In Table 9, we stipulate the hyperparameters of the SentEval toolkit used in our experiment. Additionally, we specify some minor modifications applied on our contrastive method (SG-OPT). First, we use the portion of the concatenation of SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) datasets as the training data instead of STS-B. Second, we do not leverage the first several layers of PLMs when making positive samples, similar to Wang and Kuo (2020), and utilize mean pooling instead of max pooling.  We here investigate the impact of our method on typical supervised fine-tuning of BERT models. Concretely, we compare the original BERT with one fine-tuned using our SG-OPT method on the GLUE (Wang et al., 2019) benchmark. Note that we use the first 10% of the GLUE validation set as the real validation set and the last 90% as the test set, as the benchmark does not officially provide its test data. We report experimental results tested on 5 sub-tasks in Table 10. The results show that our method brings performance improvements for 3 tasks (QNLI, SST2, and RTE). However, it seems that SG-OPT does not influence much on supervised fine-tuning results, considering that the absolute performance gap between the two models is not significant. We leave more analysis on this part as future work.
Self-Guided Contrastive Learning for BERT Sentence Representations
A.5 Representation Visualization (SBERT)  A.6 RoBERTa's Performance on STS Tasks
Self-Guided Contrastive Learning for BERT Sentence Representations
In Table 11, we additionally report the performance of sentence embeddings extracted from RoBERTa using different methods. Our methods, SG and SG-OPT, demonstrate their competitive performance  overall. Note that contrastive learning with backtranslation (BT) also shows its remarkable performance in the case of RoBERTa-base.
Debiased Contrastive Learning of Unsupervised Sentence Representations
As a fundamental task in the natural language processing (NLP) field, unsupervised sentence representation learning (Kiros et al., 2015;Hill et al., 2016) aims to derive high-quality sentence representations that can benefit various downstream tasks, especially for low-resourced domains or computationally expensive tasks, e.g., zero-shot text semantic matching (Qiao et al., 2016), large-scale semantic similarity comparison (Agirre et al., 2015), and document retrieval (Le and Mikolov, 2014).
Debiased Contrastive Learning of Unsupervised Sentence Representations
Recently, pre-trained language models (PLMs) (Devlin et al., 2019) have become a widely-used se- † † Corresponding author  Figure 1: The distribution of cosine similarity between an input sentence and 255 in-batch negatives from the commonly-used Wikipedia Corpus. It is evaluated by the SimCSE model (Gao et al., 2021). Almost half of the negatives have high similarities with the input. mantic representation approach, achieving remarkable performance on various NLP tasks. However, several studies have found that the native sentence representations derived by PLMs are not uniformly distributed with respect to directions, but instead occupy a narrow cone in the vector space (Ethayarajh, 2019), which largely limits their expressiveness. To address this issue, contrastive learning  has been adopted to refine PLM-derived sentence representations. It pulls semantically-close neighbors together to improve the alignment, while pushes apart non-neighbors for the uniformity of the whole representation space. In the learning process, both positive and negative examples are involved in contrast with the original sentence. For positive examples, previous works apply data augmentation strategies (Yan et al., 2021) on the original sentence to generate highly similar variations. While, negative examples are commonly sampled from the batch or training data (e.g., in-batch negatives (Gao et al., 2021)) at random, due to the lack of ground-truth annotations for negatives.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Although such a negative sampling way is simple and convenient, it may cause sampling bias and affect the sentence representation learning. First, the sampled negatives are likely to be false negatives that are indeed semantically close to the original sentence. As shown in Figure 1, given an input sentence, about half of in-batch negatives have a cosine similarity above 0.7 with the original sentence based on the SimCSE model (Gao et al., 2021). It is likely to hurt the semantics of the sentence representations by simply pushing apart these sampled negatives. Second, due to the anisotropy problem (Ethayarajh, 2019), the representations of sampled negatives are from the narrow representation cone spanned by PLMs, which cannot fully reflect the overall semantics of the representation space. Hence, it is sub-optimal to only rely on these representations for learning the uniformity objective of sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
To address the above issues, we aim to develop a better contrastive learning approach with debiased negative sampling strategies.The core idea is to improve the random negative sampling strategy for alleviating the sampling bias problem. First, in our framework, we design an instance weighting method to punish the sampled false negatives during training. We incorporate a complementary model to evaluate the similarity between each negative and the original sentence, then assign lower weights for negatives with higher similarity scores. In this way, we can detect semanticallyclose false negatives and further reduce their influence. Second, we randomly initialize new negatives based on random Gaussian noises to simulate sampling within the whole semantic space, and devise a gradient-based algorithm to optimize the noise-based negatives towards the most nonuniform points. By learning to contrast with the nonuniform noise-based negatives, we can extend the occupied space of sentence representations and improve the uniformity of the representation space.
Debiased Contrastive Learning of Unsupervised Sentence Representations
To this end, we propose DCLR, a general framework towards Debiased Contrastive Learning of unsupervised sentence Representations. In our approach, we first initialize the noise-based negatives from a Gaussian distribution, and leverage a gradient-based algorithm to update the new negatives by considering the uniformity of the representation space. Then, we adopt the complementary model to produce the weights for these noise-based negatives and randomly sampled negatives, where the false negatives will be punished. Finally, we augment the positive examples via dropout (Gao et al., 2021) and combine them with the above weighted negatives for contrastive learning. We demonstrate that our DCLR outperforms a number of competitive baselines on seven semantic textual similarity (STS) tasks using BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019).
Debiased Contrastive Learning of Unsupervised Sentence Representations
Our contributions are summarized as follows:
Debiased Contrastive Learning of Unsupervised Sentence Representations
(1) To our knowledge, our approach is the first attempt to reduce the sampling bias in contrastive learning of unsupervised sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(2) We propose DCLR, a debiased contrastive learning framework that incorporates an instance weighting method to punish false negatives and generates noise-based negatives to guarantee the uniformity of the representation space.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(3) Experimental results on seven semantic textual similarity tasks show the effectiveness of our framework.
Debiased Contrastive Learning of Unsupervised Sentence Representations
This work aims to make use of unlabeled corpus for learning effective sentence representations that can be directly utilized for downstream tasks, e.g., semantic textual similarity task (Agirre et al., 2015). Given a set of input sentences X = {x 1 , x 2 , . . . , x n }, our goal is to learn a representation h i ∈ R d for each sentence x i in an unsupervised manner. For simplicity, we denote this process with a parameterized function
Debiased Contrastive Learning of Unsupervised Sentence Representations
In this work, we mainly focus on using BERTbased PLMs (Devlin et al., 2019;Liu et al., 2019) to generate sentence representations. Following existing works (Li et al., 2020;Yan et al., 2021), we fine-tune PLMs on the unlabeled corpus via our proposed unsupervised learning approach. After that, for each sentence x i , we encode it by the fine-tuned PLMs and take the representation of the [CLS] token from the last layer as its sentence representation h i .
Debiased Contrastive Learning of Unsupervised Sentence Representations
Our proposed framework DCLR focuses on reducing the influence of sampling bias in the con-trastive learning of sentence representations. In this framework, we devise a noise-based negatives generation strategy to reduce the bias caused by the anisotropy PLM-derived representations, and an instance weighting method to reduce the bias caused by false negatives. Concretely, we initialize the noise-based negatives based on a Gaussian distribution and iteratively update these negatives towards non-uniformity maximization. Then, we utilize a complementary model to produce weights for all negatives (i.e., randomly sampled and the noisebased ones). Finally, we combine the weighted negatives and augmented positive examples for contrastive learning. The overview of our DCLR is presented in Figure 2.
Debiased Contrastive Learning of Unsupervised Sentence Representations
We aim to generate new negatives beyond the sentence representation space of PLMs during the training process, to alleviate the sampling bias derived from the anisotropy problem of PLMs (Ethayarajh, 2019). For each input sentence x i , we first initialize k noise vectors from a Gaussian distribution as the negative representations:
Debiased Contrastive Learning of Unsupervised Sentence Representations
where σ is the standard variance. Since these vectors are randomly initialized from such a Gaussian distribution, they are uniformly distributed within the whole semantic space. By learning to contrast with these new negatives, it is beneficial for the uniformity of sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
To further improve the quality of the new negatives, we consider iteratively updating the negatives to capture the non-uniformity points within the whole semantic space. Inspired by VAT (Miyato et al., 2017;Zhu et al., 2020), we design a non-uniformity loss maximization objective to produce gradients for improving these negatives. The non-uniformity loss is denoted as the contrastive loss between the noise-based negatives {ĥ j } and the positive representations of the original sentence (h i , h + i ) as:
Debiased Contrastive Learning of Unsupervised Sentence Representations
where τ u is a temperature hyper-parameter and
Debiased Contrastive Learning of Unsupervised Sentence Representations
Based on it, for each negativeĥ j ∈ {ĥ}, we opti-"Two dogs are running."
Debiased Contrastive Learning of Unsupervised Sentence Representations
Positive Augmentation "Two dogs are walking." "A kid is on a skateboard."
Debiased Contrastive Learning of Unsupervised Sentence Representations
Gaussian Distribution
Debiased Contrastive Learning of Unsupervised Sentence Representations
Pull mize it by t steps gradient ascent aŝ
Debiased Contrastive Learning of Unsupervised Sentence Representations
where β is the learning rate, || · || 2 is the L 2 -norm. g(ĥ j ) denotes the gradient ofĥ j by maximizing the non-uniformity loss between the positive representations and the noise-based negatives. In this way, the noise-based negatives will be optimized towards the non-uniform points of the sentence representation space. By learning to contrast with these negatives, the uniformity of the representation space can be further improved, which is essential for effective sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In addition to the above noise-based negatives, we also follow existing works (Yan et al., 2021;Gao et al., 2021) that adopt other in-batch representations as negatives {h − }. However, as discussed before, the sampled negatives may contain examples that have similar semantics with the positive example (i.e., false negatives).
Debiased Contrastive Learning of Unsupervised Sentence Representations
To alleviate this problem, we propose an instance weighting method to punish the false negatives. Since we cannot obtain the true labels or semantic similarities, we utilize a complementary model to produce the weights for each negative. In this paper, we adopt the state-of-the-art SimCSE (Gao et al., 2021) 
Debiased Contrastive Learning of Unsupervised Sentence Representations
where ϕ is a hyper-parameter of the instance weighting threshold, and sim C (h i , h − ) is the cosine similarity score evaluated by the complementary model. In this way, the negative that has a higher semantic similarity with the representation of the original sentence will be regarded as a false negative and will be punished by assigning the weight 0. Based on the weights, we optimize the sentence representations with a debiased crossentropy contrastive learning loss function as
Debiased Contrastive Learning of Unsupervised Sentence Representations
where τ is a temperature hyper-parameter. In our framework, we follow SimCSE (Gao et al., 2021) that utilizes dropout to augment positive examples h + i . Actually, we can utilize various positive augmentation strategies, and will investigate it in Section 6.1.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In this part, we present the overview and discussion of our DCLR approach.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Our framework DCLR contains three major steps. In the first step, we generate noise-based negatives to extend in-batch negatives. Concretely, we first initialize a set of new negatives via random Gaussian noises using Eq. 1. Then, we incorporate a gradient-based algorithm to adjust the noise-based negatives by maximizing the non-uniform objective using Eq. 3. After several iterations, we can obtain the noise-based negatives that correspond to the nonuniform points within the whole semantic space, and we mix up them with in-batch negatives to compose the negative set. In the second step, we adopt a complementary model (i.e., SimCSE) to compute the semantic similarity between the original sentence and each example from the negative set, and produce the weights using Eq. 5. Finally, we augment the positive examples via dropout and utilize the negatives with corresponding weights for contrastive learning using Eq. 6.
Debiased Contrastive Learning of Unsupervised Sentence Representations
As mentioned above, our approach aims to reduce the influence of the sampling bias about the negatives, and is agnostic to various positive data augmentation methods (e.g., token cutoff and dropout). Compared with traditional contrastive learning methods (Yan et al., 2021;Gao et al., 2021), our proposed DCLR expands the negative set by introducing noise-based negatives {ĥ}, and adds a weight term α h − to punish false negatives. Since the noise-based negatives are initialized from a Gaussian distribution and do not correspond to real sentences, they are highly confident negatives to broaden the representation space. By learning to contrast with them, the learning of the contrastive objective will not be limited by the anisotropy representations derived from PLMs. As a result, the sentence representations can span a broader semantic space, and the uniformity of the representation semantic space can be improved.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Besides, our instance weighting method also alleviates the false negative problem caused by the randomly sampling strategy. With the help of a complementary model, the false negatives with similar semantics as the original sentence will be detected and punished.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Following previous works (Kim et al., 2021;Gao et al., 2021), we conduct experiments on seven standard STS tasks. For all these tasks, we use the SentEval toolkit (Conneau and Kiela, 2018) for evaluation.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Semantic Textual Similarity Task. We evaluate our approach on 7 STS tasks: STS 2012-2016 (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016, STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). These datasets contain pairs of two sentences, whose similarity scores are labeled from 0 to 5. The relevance between gold annotations and the scores predicted by sentence representations is measured by the Spearman correlation. Following the suggestions from previous works (Gao et al., 2021;Reimers and Gurevych, 2019), we directly compute the cosine similarity between sentence embeddings for all STS tasks.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Baseline Methods. We compare DCLR with competitive unsupervised sentence representation learning methods, consisting of non-BERT and BERTbased methods:
Debiased Contrastive Learning of Unsupervised Sentence Representations
(1) GloVe (Pennington et al., 2014) averages GloVe embeddings of words as the sentence representation.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(2) USE (Cer et al., 2018) utilizes a Transformer model that learns the objective of reconstructing the surrounding sentences within a passage.
Debiased Contrastive Learning of Unsupervised Sentence Representations
( (4) Flow (Li et al., 2020) applies mean pooling on the layer representations and maps the outputs to the Gaussian space as sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(5) Whitening (Su et al., 2021) uses the whitening operation to refine representations and reduce dimensionality.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(6) Contrastive (BT) (Fang and Xie, 2020) uses contrastive learning with back-translation for data augmentation to enhance sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(7) ConSERT (Yan et al., 2021) explores various text augmentation strategies for contrastive learning of sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(8) SG-OPT (Kim et al., 2021) proposes a contrastive learning method with a self-guidance mechanism for improving the sentence embeddings of PLMs.
Debiased Contrastive Learning of Unsupervised Sentence Representations
(9) SimCSE (Gao et al., 2021) proposes a simple contrastive learning framework that utilizes dropout for data augmentation. Implementation Details. We implement our model based on Huggingface's transformers (Wolf et al., 2020). For BERT-base and RoBERTa-base, we start from the pre-trained checkpoints of their original papers. For BERT-large and RoBERTalarge, we utilize the checkpoints of SimCSE for stabilizing the convergence process. Following Sim-  Table 1: Sentence embedding performance on STS tasks (Spearman's correlation). The best performance and the second-best performance methods are denoted in bold and underlined fonts respectively. †: results from Kim et al. (2021); ‡: results from Gao et al. (2021); all other results are reproduced or reevaluated by ourselves.
Debiased Contrastive Learning of Unsupervised Sentence Representations
CSE (Gao et al., 2021), we use 1,000,000 sentences randomly sampled from Wikipedia as the training corpus. During training, we train our models for 3 epoch with temperature τ = 0.05 using an Adam optimizer (Kingma and Ba, 2015). For BERT-base and RoBERTa-base, the batch size is 128, the learning rate is 3e-5. For BERT-large and RoBERTalarge, the batch size is 256, the learning rate is 3e-5 and 1e-5, respectively. For the four backbone models, we set the instance weighting threshold ϕ as 0.9, 0.85, 0.9 and 0.85, respectively. For each batch, we generate k × batch_size noise-based negatives as the shared negatives of all instance within it, and k is 1, 2.5, 4 and 5 for BERT-base, RoBERTa-base, BERT-large and RoBERTa-large, respectively. The standard variance of the noise-based negatives is 1, and we update the noise-based negatives four times with the learning rate of 1e-3. We evaluate the model every 150 steps on the development set of STS-B and SICK-R and keep the best checkpoint for evaluation on test sets.
Debiased Contrastive Learning of Unsupervised Sentence Representations
To verify the effectiveness of our framework on PLMs, we select BERT-base and RoBERTa-base as the base model. Table 1 shows the results of different methods on seven STS tasks. Based on the results, we can find that the non-BERT methods (i.e., GloVe and USE) mostly outperform native PLM representation based baselines (i.e., CLS, Mean and First-Last AVG). The reason is that directly utilizing the PLM native representations is prone to be influenced by the anisotropy issue. Among non-BERT methods, USE outperforms Glove. A potential reason is that USE encodes the sentence using the Transformer model, which is more effective than simply averaging GloVe embeddings.
Debiased Contrastive Learning of Unsupervised Sentence Representations
For other PLM-based approaches, first, we can see that flow and whitening achieve similar results and outperform the native representations based methods by a margin. These two methods adopt specific improvement strategies to refine the representations of PLMs. Second, approaches based on contrastive learning outperform the other baselines in most cases. Contrastive learning can enhance both the alignment between semantically related positive pairs and the uniformity of the representation space using negative samples, resulting in better sentence representations. Furthermore, Sim-CSE performs the best among all the baselines. It indicates that dropout is a more effective positive augmentation method than others since it rarely hurts the semantics of the sentence.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Finally, DCLR performs better than all the baselines in most settings, including the approaches based on contrastive learning. Since these methods mostly utilize randomly sampled negatives (e.g., in-batch negatives) to learn the uniformity of all sentence representations, it may lead to sampling bias, such as false negatives and anisotropy representations. Different from these methods, our framework adopts an instance weighting method to punish false negatives and a gradient-based algorithm to generate noise-based negatives towards the nonuniform points. In this way, the sampling bias problem can be alleviated, and our model can better learn the uniformity to improve the quality of the sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In this section, we continue to study the effectiveness of our proposed DCLR.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Since our proposed DCLR is a general framework that mainly focuses on negative sampling for contrastive learning of unsupervised sentence representations, it can be applied to other methods that rely on different positive data augmentation strategies.   Thus, in this part, we conduct experiments to examine whether our framework can bring improvements with the following positive data augmentation strategies: (1) Token Shuffling that randomly shuffles the order of the tokens in the input sequences;
Debiased Contrastive Learning of Unsupervised Sentence Representations
(2) Feature/Token/Span Cutoff (Yan et al., 2021) that randomly erases features/tokens/token spans in the input;
Debiased Contrastive Learning of Unsupervised Sentence Representations
(3) Dropout that is similar to SimCSE (Gao et al., 2021). Note that we only revise the negative sampling strategies to implement these variants of our DCLR. As shown in Figure 3, our DCLR can boost the performance of all these augmentation strategies, it demonstrates the effectiveness of our framework with various augmentation strategies. Furthermore, the Dropout strategy leads to the best performance among all the variants. It indicates that dropout is a more effective approach to augment high-quality positives, and is also more appropriate for our approach.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Our proposed DCLR incorporates an instance weighting method to punish false negatives and also utilizes noise-based negatives to improve the uniformity of the whole sentence representation space. To verify their effectiveness, we conduct an ablation study for each of the two components on seven STS tasks and report the average value of the Spearman's correlation metric. As shown in Table 2, removing each component would lead to the performance degradation. It indicates that the instance weighting method and the noise-based negatives are both important in our framework. Besides, removing the instance weighting method results in a larger performance drop. The reason may be that the false negatives have a larger effect on sentence representation learning. Besides, we prepare three variants for further comparison: (1) Random Noise directly generates noise-based negatives without the gradient-based optimization; (2) Knowledge Distillation (Hinton et al., 2015) utilizes SimCSE as the teacher model to distill knowledge into the student model during training; (3) Self Instance Weighting adopts the model itself as the complementary model to generate the weights. From Table 2, we can see that these variations don't perform as well as the original DCLR. These results indicate the proposed designs in Section 4 are more suitable for our DCLR framework.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Uniformity is a desirable characteristic for sentence representations, describing how well the representations are uniformly distributed. To validate the improvement of the uniformity of our framework, we compare the uniformity loss curves of DCLR and SimCSE using BERT-base during training.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Following SimCSE (Gao et al., 2021), we utilize the following function to evaluate the uniformity:
Debiased Contrastive Learning of Unsupervised Sentence Representations
where p data is the distribution of all sentence representations, and a smaller value of this loss indicates a better uniformity. As shown in Figure 4, the uniformity loss of DCLR is much lower than that of SimCSE in almost the whole training process. Furthermore, we can see that the uniformity loss of DCLR decreases faster as training goes, while the one of SimCSE shows no significant decreasing trend. It might be because our DCLR samples noise-based negatives beyond the representation space, which can better improve the uniformity of sentence representations.
Debiased Contrastive Learning of Unsupervised Sentence Representations
To validate the reliability and the robustness of DCLR under the data scarcity scenarios, we conduct few-shot experiments using BERT-base as the backbone model. We train our model via different amounts of available training data from 100% to the extremely small size (i.e., 0.3%). We report the results evaluated on STS-B and SICK-R tasks.
Debiased Contrastive Learning of Unsupervised Sentence Representations
As shown in Figure 5, our approach achieves stable results under different proportions of the training data. Under the most extreme setting with 0.3% data proportion, the performance of our model drops by only 9 and 4 percent on STS-B and SICK-R, respectively. The results reveal the robustness and effectiveness of our approach under the data scarcity scenarios. Such characteristics are important in real-world application.
Debiased Contrastive Learning of Unsupervised Sentence Representations
For hyper-parameters analysis, we study the impact of instance weighting threshold ϕ and the proportion of noise-based negatives k. The ϕ is the threshold to punish false negatives, and k is the ratio of the noise-based negatives to the batch size. Both hyper-parameters are important in our framework. Concretely, we evaluate our model with varying values of ϕ and k on the STS-B and SICK-R tasks using the BERT-base model.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Weighting threshold. Figure 6(a) shows the influence of the instance weighting threshold ϕ. For the STS-B tasks, ϕ has a significant effect on the model performance. Too large or too small ϕ may lead to a performance drop. The reason is that a larger threshold cannot achieve effective punishment and a smaller one may cause misjudgment of true negatives. In contrast, the SICK-R is insensitive to the changes of ϕ. The reason may be that the problem of false negatives is not serious in this task.
Debiased Contrastive Learning of Unsupervised Sentence Representations
Negative proportion. As shown in Figure 6(b), our DCLR performs better when the number of noise-based negatives is close to the batch size. Under these circumstances, the noise-based negatives are more capable to enhance the uniformity of the whole semantic space without hurting the alignment, which is key why DCLR works well.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In this paper, we proposed DCLR, a debiased contrastive learning framework for unsupervised sentence representation learning. Our core idea is to alleviate the sampling bias caused by the random negative sampling strategy. To achieve it, in our framework, we incorporated an instance weighting method to punish false negatives during training and generated noise-based negatives to alleviate the influence of anisotropy PLM-derived representation. Experimental results on seven STS tasks have shown that our approach outperforms several competitive baselines.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In the future, we will explore other approaches to reducing the bias in contrastive learning of sentence representations (e.g., debiased pre-training). Besides, we will also consider to apply our method for multilingual or multimodal representation learning.
Debiased Contrastive Learning of Unsupervised Sentence Representations
In this section, we discuss the ethical considerations of this work from the following two aspects. First, for intellectual property protection, the code, data and pre-trained models adopted from previous works are granted for research-purpose usage. Second, since PLMs have been shown to capture certain biases from the data they have been pretrained on (Bender et al., 2021), there is a potential problem about biases that are from the use of PLMs in our approach. There are increasing efforts to address this problem in the community .
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Learning "universal" sentence representations that capture rich semantic information and are at the same time performant across a wide range of downstream NLP tasks without task-specific finetuning is an important open issue in the field (Conneau et al., 2017;Cer et al., 2018;Kiros et al., 2015;Logeswaran and Lee, 2018;Giorgi et al., 2020;Yan et al., 2021;Gao et al., 2021). Recent work has shown that finetuning pretrained language models with contrastive learning makes it possible to learn good sentence embeddings without any labeled data (Giorgi et al., 2020;Yan et al., 2021;Gao et al., 2021). Contrastive learning uses multiple augmentations on a single datum to construct positive pairs whose representations are trained to be more similar to one another than negative pairs. While different data augmentations (random cropping, color jitter, rotations, etc.) have been found to be crucial for pretraining vision models (Chen et al., 2020), such augmentations have generally been unsuccessful when applied to contrastive learning of sentence embeddings. Indeed, Gao et al. (2021) find that constructing positive pairs via a simple dropout-based augmentation works much better than more complex augmentations such as word deletions or replacements based on synonyms or masked language models. This is perhaps unsurprising in hindsight; while the training objective in contrastive learning encourages representations to be invariant to augmentation transformations, direct augmentations on the input (e.g., deletion, replacement) often change the meaning of the sentence. That is, ideal sentence embeddings should not be invariant to such transformations.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose to learn sentence representations that are aware of, but not necessarily invariant to, such direct surface-level augmentations. This is an instance of equivariant contrastive learning (Dangovski et al., 2021), which improves vision representation learning by using a contrastive loss on insensitive image transformations (e.g., grayscale) and a prediction loss on sensitive image transformations (e.g., rotations). We operationalize equivariant contrastive learning on sentences by using dropout-based augmentation as the insensitive transformation (as in SimCSE (Gao et al., 2021)) and MLM-based word replacement as the sensitive transformation. This results in an additional crossentropy loss based on the difference between the original and the transformed sentence.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We conduct experiments on 7 semantic textual similarity tasks (STS) and 7 transfer tasks from Sen-tEval (Conneau and Kiela, 2018) and find that this difference-based learning greatly improves over standard contrastive learning. Our DiffCSE approach can achieve around 2.3% absolute improve-"You [MASK] know what you're gonna [MASK] ." "You never know what you're gonna get . On the left-hand side is a standard SimCSE model trained with regular contrastive loss on dropout transformations. On the right hand side is a conditional difference prediction model which takes the sentence vector h as input and predict the difference between x and x ′′ . During testing we discard the discriminator and only use h as the sentence embedding. ment on STS datasets over SimCSE, the previous state-of-the-art model. We also conduct a set of ablation studies to justify our designed architecture. Qualitative study and analysis are also included to look into the embedding space of DiffCSE.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Learning universal sentence embeddings has been studied extensively in prior work, including unsupervised approaches such as Skip-Thought (Kiros et al., 2015), Quick-Thought (Logeswaran and Lee, 2018) and FastSent (Hill et al., 2016), or supervised methods such as InferSent (Conneau et al., 2017), Universal Sentence Encoder (Cer et al., 2018) and Sentence-BERT (Reimers and Gurevych, 2019). Recently, researchers have focused on (unsupervised) contrastive learning approaches such as Sim-CLR (Chen et al., 2020) to learn sentence embeddings. SimCLR (Chen et al., 2020) learns image representations by creating semantically close augmentations for the same images and then pulling these representations to be closer than representations of random negative examples. The same framework can be adapted to learning sentence embeddings by designing good augmentation methods for natural language. ConSERT (Yan et al., 2021) uses a combination of four data augmentation strategies: adversarial attack, token shuffling, cut-off, and dropout. DeCLUTR (Giorgi et al., 2020) uses overlapped spans as positive ex-amples and distant spans as negative examples for learning contrastive span representations. Finally, SimCSE (Gao et al., 2021) proposes an extremely simple augmentation strategy by just switching dropout masks. While simple, sentence embeddings learned in this manner have been shown to be better than other more complicated augmentation methods.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
DiffCSE is inspired by a recent generalization of contrastive learning in computer vision (CV) called equivariant contrastive learning (Dangovski et al., 2021). We now explain how this CV technique can be adapted to natural language.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Understanding the role of input transformations is crucial for successful contrastive learning. Past empirical studies have revealed useful transformations for contrastive learning, such as random resized cropping and color jitter for computer vision (Chen et al., 2020) and dropout for NLP (Gao et al., 2021). Contrastive learning encourages representations to be insensitive to these transformations, i.e. the encoder is trained to be invariant to a set of manually chosen transformations. The above studies in CV and NLP have also revealed transformations that are harmful for contrastive learning. For example, Chen et al. (2020) showed that making the representations insensitive to rotations decreases the ImageNet linear probe accuracy, and Gao et al. (2021) showed that using an MLM to replace 15% of the words drastically reduces performance on STS-B. While previous works simply omit these transformations from contrastive pre-training, here we argue that we should still make use of these transformations by learning representations that are sensitive (but not necessarily invariant) to such transformations.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
The notion of (in)sensitivity can be captured by the more general property of equivariance in mathematics. Let T be a transformation from a group G and let T (x) denote the transformation of a sentence x. Equivariance is the property that there is an induced group transformation T ′ on the output features (Dangovski et al., 2021):
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
In the special case of contrastive learning, T ′ 's target is the identity transformation, and we say that f is trained to be "invariant to T ." However, invariance is just a trivial case of equivariance, and we can design training objectives where T ′ is not the identity for some transformations (such as MLM), while it is the identity for others (such as dropout). Dangovski et al. (2021) show that generalizing contrastive learning to equivariance in this way improves the semantic quality of features in CV, and here we show that the complementary nature of invariance and equivariance extends to the NLP domain. The key observation is that the encoder should be equivariant to MLM-based augmentation instead of being invariant. We can operationalize this by using a conditional discriminator that combines the sentence representation with an edited sentence, and then predicts the difference between the original and edited sentences. This is essentially a conditional version of the ELEC-TRA model (Clark et al., 2020), which makes the encoder equivariant to MLM by using a binary discriminator which detects whether a token is from the original sentence or from a generator. We hypothesize that conditioning the ELECTRA model with the representation from our sentence encoder is a useful objective for encouraging f to be "equivariant to MLM."
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
To the best of our knowledge, we are the first to observe and highlight the above parallel between CV and NLP. In particular, we show that equivariant contrastive learning extends beyond CV, and that it works for transformations even without algebraic structures, such as diff operations on sentences. Further, insofar as the canonical set of useful transformations is less established in NLP than is in CV, DiffCSE can serve as a diagnostic tool for NLP researchers to discover useful transformations.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Our approach is straightforward and can be seen as combining the standard contrastive learning objective from SimCSE (Figure 1, left) with a difference prediction objective which conditions on the sentence embedding (Figure 1, right).
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Given an unlabeled input sentence x, SimCSE creates a positive example x + for it by applying different dropout masks. By using the BERT base encoder f , we can obtain the sentence embedding h = f (x) for x (see section 4 for how h is obtained). The training objective for SimCSE is:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
as we are using in-batch negative examples, sim(·, ·) is the cosine similarity function, and τ is a temperature hyperparameter.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
On the right-hand side of Figure 1 is a conditional version of the difference prediction objective used in ELECTRA (Clark et al., 2020), which contains a generator and a discriminator.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Given a sentence of length
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We use another pretrained MLM as the generator G to perform masked language modeling to recover randomly masked tokens in x ′ to obtain the edited sentence x ′′ = G(x ′ ). Then, we use a discriminator D to perform the Replaced Token Detection (RTD) task. For each token in the sentence, the model needs to predict whether it has been replaced or not. The cross-entropy loss for a single sentence x is:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
And the training objective for a batch is
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Finally we optimize these two losses together with a weighting coefficient λ:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
The difference between our model and ELECTRA is that our discriminator D is conditional, so it can use the information of x compressed in a fixeddimension vector h = f (x). The gradient of D can be backward-propagated into f through h. By doing so, f will be encouraged to make h informative enough to cover the full meaning of x, so that D can distinguish the tiny difference between x and x ′′ . This approach essentially makes the conditional discriminator perform a "diff operation", hence the name DiffCSE.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
When we train our DiffCSE model, we fix the generator G, and only the sentence encoder f and the discriminator D are optimized. After training, we discard D and only use f (which remains fixed) to extract sentence embeddings to evaluate on the downstream tasks.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
In our experiment, we follow the setting of unsupervised SimCSE (Gao et al., 2021) and build our model based on their PyTorch implementation. 3 We also use the checkpoints of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the initialization of our sentence encoder f . We add an MLP layer with Batch Normalization (Ioffe and Szegedy, 2015) (BatchNorm) on top of the [CLS] representation as the sentence embedding. We will compare the model with/without BatchNorm in section 5. For the discriminator D, we use the same model as the sentence encoder f (BERT/RoBERTa). For the generator G, we use the smaller DistilBERT and DistilRoBERTa (Sanh et al., 2019) for efficiency. Note that the generator is fixed during training unlike the ELECTRA paper (Clark et al., 2020). We will compare the results of using different size model for the generator in section 5. More training details are shown in Appendix A.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
For unsupervised pretraining, we use the same 10 6 randomly sampled sentences from English Wikipedia that are provided by the source code of SimCSE. 3 We evaluate our model on 7 semantic textual similarity (STS) and 7 transfer tasks in SentEval. 4 STS tasks includes STS 2012-2016 (Agirre et al., 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). All the STS experiments are fully unsupervised, which means no STS training datasets are used and all embeddings are fixed once they are trained. The transfer tasks are various sentence classification tasks, including MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). In these transfer tasks, we will use a logistic regression classifier trained on top of the frozen sentence embeddings, following the standard setup (Conneau and Kiela, 2018).
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Baselines We compare our model with many strong unsupervised baselines including Sim-CSE (Gao et al., 2021), IS-BERT (Zhang et al., 2020), CMLM , De-CLUTR (Giorgi et al., 2020), CT-BERT (Carlsson et al., 2021), SG-OPT (Kim et al., 2021) and some post-processing methods like BERT-flow (Li et al., 2020) and BERT-whitening (Su et al., 2021) along with some naive baselines like averaged GloVe embeddings (Pennington et al., 2014) and averaged first and last layer BERT embeddings.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Semantic Textual Similarity (STS) We show the results of STS tasks in Table 1 including BERT base (upper part) and RoBERTa base (lower part). We also reproduce the previous state-ofthe-art SimCSE (Gao et al., 2021). DiffCSE-BERT base can significantly outperform SimCSE-BERT base and raise the averaged Spearman's correlation from 76.25% to 78.49%. For the RoBERTa model, DiffCSE-RoBERTa base can also improve upon SimCSE-RoBERTa base from 76.57% to 77.80%.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Transfer Tasks We show the results of transfer tasks in Table 2. Compared with SimCSE-BERT base , DiffCSE-BERT base can improve the averaged scores from 85.56% to 86.86%. When applying it to the RoBERTa model, DiffCSE-RoBERTa base also improves upon SimCSE-RoBERTa base from 84.84% to 87.04%. Note that the CMLM-BERT base  can achieve even better performance than DiffCSE. However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2 
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
In the following sections, we perform an extensive series of ablation studies that support our model design. We use BERT base model to evaluate on the development set of STS-B and transfer tasks.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Removing Contrastive Loss In our model, both the contrastive loss and the RTD loss are crucial because they maintain what should be sensitive and what should be insensitive respectively. If we remove the RTD loss, the model becomes a SimCSE model; if we remove the contrastive loss, the performance of STS-B drops significantly by 30%, while the average score of transfer tasks also drops by 2% (see Table 3). This result shows that it is important to have insensitive and sensitive attributes that exist together in the representation space.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Next Sentence vs. Same Sentence Some methods for unsupervised sentence embeddings like Quick-Thoughts (Logeswaran and Lee, 2018) and CMLM  predict the next sentence as the training objective. We also experiment with a variant of DiffCSE by conditioning the ELECTRA loss based on the next sentence.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Note that this kind of model is not doing a "diff operation" between two similar sentences, and is not an instance of equivariant contrastive learning. As shown in Table 3 (use next sent. for x ′ ), the score of STS-B decreases significantly compared to DiffCSE while transfer performance remains similar. We also tried using the same sentence and the next sentence at the same time for conditioning the ELECTRA objective (use same+next sent. for x ′ ), and did not observe improvements.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Other Conditional Pretraining Tasks Instead of a conditional binary difference prediction loss, we can also consider other conditional pretraining tasks such as a conditional MLM objective proposed by , or corrective language modeling, 5 proposed by COCO-LM (Meng et al., 2021). We experiment with these objectives instead of the difference prediction objective in Table 3. We observe that conditional MLM on the same sentence does not improve the performance either on STS-B or transfer tasks compared with DiffCSE. Conditional MLM on the next sentence performs even worse for STS-B, but slightly better than using the same sentence on transfer tasks. Using both the same and the next sentence also does not improve the performance compared with DiffCSE. For the corrective LM objective, the performance of STS-B decreases significantly compared with DiffCSE.     randomly insert mask tokens to the sentence, and then use a generator to convert mask tokens into real tokens. The number of inserted masked tokens is 15% of the sentence length. The task is to predict whether a token is an inserted token or the original token. For deletion, we randomly delete 15% tokens in the sentence, and the task is to predict for each token whether a token preceding it has been deleted or not. The results are shown in Table 4. We can see that using either insertion or deletion achieves a slightly worse STS-B performance than using MLM replacement. For transfer tasks, their results are similar. Finally, we find that combining all three augmentations in the training process does not improve the MLM replacement strategy.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Pooler Choice In SimCSE, the authors use the pooler in BERT's original implementation (one linear layer with tanh activation function) as the final layer to extract features for computing contrastive loss. In our implementation (see details in Appendix A), we find that it is better to use a two-layer pooler with Batch Normalization (Batch-Norm) (Ioffe and Szegedy, 2015), which is commonly used in contrastive learning framework in computer vision (Chen et al., 2020;Grill et al., 2020;Chen and He, 2021;Hua et al., 2021). We show the ablation results in Table 5. We can observe that adding BatchNorm is beneficial for either DiffCSE or SimCSE to get better performance on STS-B and transfer tasks.   BERT's performance due to knowledge distillation.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We show our results in Table 6, we can see the performance of transfer tasks does not change much with different generators. However, the score of STS-B decreases as we switch from BERTmedium to BERT-tiny. This finding is not the same as ELECTRA, which works best with generators 1/4-1/2 the size of the discriminator. Because our discriminator is conditional on sentence vectors, it will be easier for the discriminator to perform the RTD task. As a result, using stronger generators (BERT base , DistilBERT base ) to increase the difficulty of RTD would help the discriminator learn better. However, when using a large model like BERT large , it may be a too-challenging task for the discriminator. In our experiment, using DistilBERT base , which has the ability close to but slightly worse than BERT base , gives us the best performance.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Masking Ratio In our conditional ELECTRA task, we can mask the original sentence in different ratios for the generator to produce MLM-based augmentations. A higher masking ratio will make more perturbations to the sentence. Our empirical result in Table 7 shows that the difference between difference masking ratios is small (in 15%-40% ), and a masking ratio of around 30% can give us the best performance.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Coefficient λ In Section 3, we use the λ coefficient to weight the ELECTRA loss and then add it with contrastive loss. Because the contrastive learning objective is a relatively easier task, the scale of contrastive loss will be 100 to 1000 smaller than  ELECTRA loss. As a result, we need a smaller λ to balance these two loss terms. In the Table 8 we show the STS-B result under different λ values. Note that when λ goes to zero, the model becomes a SimCSE model. We find that using λ = 0.005 can give us the best performance.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
6 Analysis
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
A very common application for sentence embeddings is the retrieval task. Here we show some retrieval examples to qualitatively explain why Dif-fCSE can perform better than SimCSE. In this study, we use the 2758 sentences from STS-B testing set as the corpus, and then use sentence query to retrieve the nearest neighbors in the sentence embedding space by computing cosine similarities. We show the retrieved top-3 examples in Table 9. The first query sentence is "you can do it, too.". The SimCSE model retrieves a very similar sentence but has a slightly different meaning ("you can use it, too.") as the rank-1 answer. In contrast, DiffCSE can distinguish the tiny difference, so it retrieves the ground truth answer as the rank-1 answer. The second query sentence is "this is not a problem". SimCSE retrieves a sentence with opposite meaning but very similar wording, while DiffCSE can retrieve the correct answer with less similar wording. We also provide a third example where both SimCSE and DiffCSE fail to retrieve the correct answer for a query sentence using double negation.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Besides the qualitative study, we also show the quantitative result of the retrieval task. Here we also use all the 2758 sentences in the testing set of STS-B as the corpus. There are 97 positive pairs in this corpus (with 5 out of 5 semantic similarity scores from human annotation). For each positive pair, we use one sentence to retrieve the other one, and see whether the other sentence is in the top-1/5/10 ranking. The recall@1/5/10 of the retrieval task are shown in 3) can you do it?
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Query: this is not a problem.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
1) this is a big problem. 1) i don 't see why this could be a problem. 2) you have a problem.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
2) i don 't see why that should be a problem. 3) i don 't see why that should be a problem.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
3) this is a big problem.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Query: i think that is not a bad idea.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
1) i do not think it's a good idea. 1) i do not think it's a good idea . 2) it's not a good idea .
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
2) it is not a good idea. 3) it is not a good idea .
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
3) but it is not a good idea.   recall@1/5/10, showing the effectiveness of using DiffCSE for the retrieval task.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
To look into the representation space of DiffCSE, we plot the cosine similarity distribution of sentence pairs from STS-B test set for both SimCSE and DiffCSE in Figure 2. We observe that both SimCSE and DiffCSE can assign cosine similarities consistent with human ratings. However, we also observe that under the same human rating, DiffCSE assigns slightly higher cosine similarities compared with SimCSE. This phenomenon  may be caused by the fact that ELECTRA and other Transformer-based pretrained LMs have the problem of squeezing the representation space, as mentioned by Meng et al. (2021). As we use the sentence embeddings as the input of ELECTRA to perform conditional ELECTRA training, the sentence embedding will be inevitably squeezed to fit the input distribution of ELECTRA. We follow prior studies (Wang and Isola, 2020;Gao et al., 2021) to use uniformity and alignment (details in Appendix C) to measure the quality of representation space for DiffCSE and SimCSE in 
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
In this paper, we present DiffCSE, a new unsupervised sentence embedding framework that is aware of, but not invariant to, MLM-based word replacement. Empirical results on semantic textual similarity tasks and transfer tasks both show the effectiveness of DiffCSE compared to current stateof-the-art sentence embedding methods. We also conduct extensive ablation studies to demonstrate the different modeling choices in DiffCSE. Qualitative study and the retrieval results also show that DiffCSE can produce a better embedding space for sentence retrieval. One limitation of our work is that we do not explore the supervised setting that uses human-labeled NLI datasets to further boost the performance. We leave this topic for future work. We believe that our work can provide researchers in the NLP community a new way to utilize augmentations for natural language and thus produce better sentence embeddings.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We use a single NVIDIA 2080Ti GPU for each experiment. The averaged running time for DiffCSE is 3-6 hours. We use gridsearch of batch size ∈ {64, 128} learning rate ∈ {2e-6, 3e-6, 5e-6, 7e-6, 1e-5} and masking ratio ∈ {0.15, 0.20, 0.30, 0.40} and λ ∈ {0.1, 0.05, 0.01, 0.005, 0.001}. The temperature τ in SimCSE is set to 0.05 for all the experiments. During the training process, we save the checkpoint with the highest score on the STS-B development set. And then we use STS-B development set to find the best hyperparameters (listed in Table 12) for STS task; we use the averaged score of the development sets of 7 transfer tasks to find the best hyperparameters (listed in Table 13) for transfer tasks. All numbers in Table 1 and Table 2      encoder, so the model size is the same as the Sim-CSE model.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Projector with BatchNorm In Section 5, we mention that we use a projector with BatchNorm as the final layer of our model. Here we provided the PyTorch code for its structure:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
class ProjectionMLP(nn.Module): def __init__(self, hidden_size): super().__init__() in_dim = hidden_size middle_dim = hidden_size * 2 out_dim = hidden_size self.net = nn.Sequential( nn.Linear(in_dim, middle_dim, bias=False), nn.BatchNorm1d(middle_dim), nn.ReLU(inplace=True), nn.Linear(middle_dim, out_dim, bias=False), nn.BatchNorm1d(out_dim, affine=False))
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
In Section 5, we try to use different augmentations (e.g. insertion, deletion, replacement) for learning equivariance. In Table 15 we provide the results of using these augmentations as additional positive or negative examples along with the SimCSE training paradigm. We can observe that using these augmentations as additional positives only decreases the performance. The only method that can improve the performance a little bit is to use MLM 15% replaced examples as additional negative examples. Overall, none of these results can perform better than our proposed method, e.g. using these augmentations to learn equivariance. Wang and Isola (2020) propose to use two properties, alignment and uniformity, to measure the quality of representations. Given a distribution of positive pairs p pos and the distribution of the whole dataset p data , alignment computes the expected distance between normalized embeddings of the paired sentences:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Uniformity measures how well the embeddings are uniformly distributed in the representation space:
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
The smaller the values of uniformity and alignment, the better the quality of the representation space is indicated.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We build our model using the PyTorch implementation of SimCSE 7 Gao et al. (2021), which is based on the HuggingFace's Transformers package. 8 We also upload our code 9 and pretrained models (links in README.md). Please follow the instructions in README.md to reproduce the results.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
On the risk side, insofar as our method utilizes pretrained language models, it may inherit and propagate some of the biases present in such models. Besides that, we do not see any other potential risks in our paper. 7 https://github.com/princeton-nlp/ SimCSE 8 https://github.com/huggingface/ transformers 9 https://github.com/voidism/DiffCSE
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Learning sentence representations, which encodes sentences into fixed-sized dense vectors such that semantically similar ones stay close, is a fundamental problem of natural language processing. It could benefit a wide range of downstream applications such as information retrieval, semantic similarity comparison, question answering, and so on.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Recently, with the great success of pre-trained Transformer-based language models (Devlin et al., 2019;Liu et al., 2019;Raffel et al., 2020;Brown et al., 2020;Liu et al., 2019) like BERT, they have been widely adopted for generating sentence representations. A straightforward way is by leveraging the [CLS] embedding (Devlin et al., 2019)  sb: He was born in Nazareth-Palestine, but immigrated to Lebanon with his parents.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
sc: He was born in Nazareth-Palestine, but immigrated to Lebanon. Figure 1: Sentence representations visualization. We generate the representations of three related sentences by passing them to BERT base , SimCSE-BERT base and ArcCSE-BERT base multiple times. With different dropout masks, we can generate different representations for each sentence. Then we normalize the embeddings and use t-SNE for dimensionality reduction.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
or applying mean pooling on the last layers of a BERT-like pre-trained language model (Reimers and Gurevych, 2019). However, the sentence embeddings coming from a pre-trained language model without further fine-tuning could not capture the semantic meaning of sentences very well as shown in Figure 1(a), and sometimes even underperform non-contextualized embeddings like GloVe (Pennington et al., 2014).
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
To make pre-trained language models more suitable for generating sentence embeddings, supervised methods like SBERT (Reimers and Gurevych, 2019) are proposed, which improve the performance by fine-tuning on a labeled dataset. As labeled data is not available or expensive to annotate in many tasks or domains, it is of great value for developing unsupervised/self-supervised approaches for learning sentence representations. So recent works like BERT-Flow  and BERT-Whitening (Su et al., 2021) propose postprocessing methods to improve the BERT-based sentence representation. They address that the nonsmooth anisotropic semantic space of BERT is a bottleneck and alleviate the problem through normalizing flows and whitening operation. To further improve the quality of sentence representations, several works (Kim et al., 2021;Yan et al., 2021;Giorgi et al., 2021;Carlsson et al., 2021;Gao et al., 2021) adopt self-supervised contrastive learning approach, which learns sentence representations by minimizing the distance of positive sentence representation pairs and maximizing the distance of negative pairs. In these works, positive pairs are often constituted through data augmentation or encoders with different structure or parameters, while negative pairs are derived from different sentences within the same batch. Then contrastive learning objective like normalized temperature-scaled crossentropy loss (NT-Xent) Gao et al., 2021) is used for optimizing. A typical example unsup-SimCSE (Gao et al., 2021) achieves state-of-the-art performance with a simple and effective idea of using standard dropout for data augmentation.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Though existing contrastive methods for learning sentence representation have shown promising results, most of them focus on the positive and negative pairs constitution, and the optimization objective itself is not fully exploited. The contrastive learning objective NT-Xent loss used in recent works (Yan et al., 2021;Giorgi et al., 2021;Gao et al., 2021) is a variation of cross-entropy loss with softmax function. Recent studies (Wang et al., 2018;Deng et al., 2019) have shown that the traditional softmax-based loss is insufficient to acquire the discriminating power, as shown in Figure 1(b) in which SimCSE-BERT base adopts the NT-Xent loss and could not separate s b and s c completely. In addition, the current optimization objectives only models sentence relations in a pairwise perspective, which tries to pull sentences with similar semantics closer and push dissimilar ones away from each other. However, there are different degrees of semantic similarity among related sentences. For example in Figure 1(d), s b is more similar to s a than s c is. The current optimization objectives lack the ability to model the partial order of semantics between sentences.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
To alleviate these problems, in this paper, we propose a new approach ArcCSE for sentence representation learning. For pairwise sentence relation modeling, we propose Additive Angular Margin Contrastive Loss (ArcCon Loss), which enhances the pairwise discriminative power by maximizing the decision margin in the angular space. Besides, in order to model the partial order of semantics between sentences, we propose a new self-supervised task that captures the entailment relation among triplet sentences. The task is implemented through automatically constituted triplet sentences with entailment relation among them. A visualization example of the generated representations through ArcCSE is shown in Figure 1(c). We evaluate our method on standard semantic textual similarity (STS) tasks and SentEval transfer tasks, and it outperforms the previous state-of-the-art approaches.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Early works usually learn sentence representations by augmenting the idea of word2vec (Mikolov et al., 2013), such as predicting surrounding sentences (Kiros et al., 2015;Hill et al., 2016;Logeswaran and Lee, 2018) or summing up n-gram embeddings (Pagliardini et al., 2018). With the rise of pre-trained language models, many works try to generate sentence representations through BERT-like models. A common way is leveraging the [CLS] embedding or applying mean pooling on the last layers of BERT (Reimers and Gurevych, 2019;. Instead of using BERT embeddings directly, BERT-Flow  and BERT-Whitening (Su et al., 2021) further improve sentence representation through post-processing. Recently, several works adopt the contrastive learning framework for sentence representation learning. They propose different strategies to constitute contrastive pairs, either through different data transforming methods (Zhang et al., 2020;Yan et al., 2021;Giorgi et al., 2021), or through encoders with different structures or parameters (Carlsson et al., 2021;Kim et al., 2021;Gao et al., 2021). A typical example SimCSE (Gao et al., 2021) uses dropout as data augmentation strategy and achieves state-of-the-art performance. However, most existing works pay little attention to the training objective and use the traditional contrastive  Figure 2: The framework of ArcCSE. ArcCSE models pairwise and triple-wise sentence relations simultaneously. For pairwise sentence relation modeling, we pass sentences to a BERT-like encoder with dropout turn on twice. Then we feed the representations into ArcCon loss which is more discriminative than NT-Xent loss. Triplet sentences are constituted through masking. We pass them to the same BERT-like encoder with dropout turn off and use a triplet loss to model their relations.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
loss directly, which is insufficient in discrimination and unable to model the partial order of semantics between sentences. So, in our work, we propose a new approach that jointly models the pairwise and triple-wise sentence relations and further improves the sentence representations' quality.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
The goal of Deep Metric Learning (DML) is to learn a function that maps objects into an embedded space, in which similar objects stay close and dissimilar ones are far away. In order to achieve this goal, many approaches have been proposed, and designing appropriate loss functions plays a key role in it. Contrastive training objectives like Contrastive Loss (Chopra et al., 2005), N-Pair Loss (Sohn, 2016), Structured Loss (Song et al., 2016) and Triplet Margin Loss (Ma et al., 2021) apply the definition of metric learning directly. These objectives are among the earliest training objectives used for deep metric learning. Later, softmax-based losses which learn a center for each class and penalize the distances between deep features and their corresponding class centers achieve more promis-ing results in supervised metric learning. Typical examples like Center Loss (Wen et al., 2016), SphereFace (Liu et al., 2017), CosFace (Wang et al., 2018) and ArcFace (Deng et al., 2019) are widely adopted in deep learning applications such as face recognition and sentence classification (Coria et al., 2020). However, these losses need class labels and are not suitable for learning sentence representations. So inspired by ArcFace, we propose a new training objective ArcCon that does not need class labels and can model pairwise sentence relations with more discriminative power than traditional contrastive training objectives.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
In this section, we present ArcCSE, an angular based contrastive sentence representation learning framework, which could generate superior sentence embeddings from unlabeled data. Given a pretrained language model M and an unlabeled text dataset D, the task is fine-tuning M on D so that the sentence representations generated through M could be more semantic discriminative. Our framework consists of two components that model pairwise and triple-wise sentence relations simultaneously, as shown in Figure 2. We start with angular margin based contrastive learning in Section 3.1, which models pairwise relations between sentences by pulling semantic similar ones closer while pushing dissimilar ones away. Then we introduce the method which models the partial order of semantics between automatically constituted triplet sentences in Section 3.2.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
To model the positive/negative pairwise relations between sentences, we first need to generate sentence representations and group them into positive and negative pairs. Then we feed these pairs to a training objective for optimizing. Given a collection of sentences D = {s i } N i=1 , we generate the sentence representations through a BERT-like pre-trained language model M. Following SimCSE, we use dropout as the data augmentation method. For each sentence s i , we generate two different representations h i and h * i from s i by passing s i to M twice with independently sampled dropout masks. These two representations with the same semantics constitute a positive pair, while the negative pairs are derived from the representations of different sentences within the same batch.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
After getting the positive and negative sentence pairs, we put them into a training objective for model fine-tune. The most widely adopted training objective is NT-Xent loss Gao et al., 2021), which has been used in previous sentence and image representation learning methods and can be formulated as follows:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
τ is a temperature hyperparameter and n is the number of sentences within a batch. Though the training objective tries to pull representations with similar semantics closer and push dissimilar ones away from each other, these representations may still not be sufficiently discriminative and not very robust to noise. Let us denote angular θ i,j as follows: For sentence representation h i , we try to make θ i,i * smaller and θ i,j larger, so the optimization direction follows the arrow. With an extra margin m, ArcCon is more discriminative and noise-tolerant. Figure 3. Due to lack of decision margin, a small perturbation around the decision boundary may lead to an incorrect decision.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
To overcome the problem, we propose a new training objective for sentence representation learning by adding an additive angular margin m between positive pair h i and h * i . We named it Additive Angular Margin Contrastive Loss (ArcCon Loss), which can be formulated as follows:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
(3) In this loss, the decision boundary for h i is θ i,i * + m = θ i,j , as show in Figure 3. Compared with NT-Xent, it further pushed h i towards to the area where θ i,i * get smaller and θ i,j get larger, by increasing the compactness of sentence representations with the same semantics and enlarging the discrepancy of different semantic representations. This help enhance the alignment and uniformity properties (Wang and Isola, 2020), which are two key measures of representation quality related to contrastive learning, indicating how close between positive pair embeddings and how well the embeddings are uniformly distributed. The quantitative analysis is illustrated in Section 4.5. Besides, the decision boundary leaves an extra margin m to boundary θ i,i * = θ i,j which is often used during inference, making it more tolerant to noise and more robust. All these properties make ArcCon loss more discriminative than traditional training objectives like NT-Xent. Compared with Arcface (Deng et al., 2019) which is often used in largescale fine-grained categorization in computer vision community, ArcCon loss does not need clas-sification labels, and could handle contrastive task properly.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Previously the training objectives for sentence representation learning like NT-Xent loss only considered pairwise sentence relations, in which sentences are either similar or dissimilar in semantics. But in fact, there are varying degrees of semantic similarity. For example, sentence s 2 could be more similar to sentence s 1 than sentence s 3 to s 1 . Existing methods lack the ability to model such partial order of semantics between sentences. In order to distinguish the slight differences in semantics between different sentences, we propose a new self-supervised task which models the entailment relation of automatically generated triplet sentences. For each sentence s i in the text dataset D, we first generate an external sentence s ′ i by masking contiguous segments of s i with a masking rate of 20%. Then we enlarge the masking area and get a new sentence s ′′ i with a masking rate of 40% to s i . The masking rates are set up experimentally, and an ablation study about the effect of masking rates is illustrated in Section 4.4. An example of the masking procedure is shown as follows:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
s i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
s ′ i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
s ′′ i Al Jaber's first long distance travel was of 800km which he covered by circling Qatar.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
We can constitute a triplet (s i , s ′ i , s ′′ i ) with entailment relation among them. Though in rare cases, the strategy may generate sentences that do not exhibit the desired relationship and introduce some noise, the entailment relation holds true most of the time. We expect encountering enough data will reinforce the correct ones whereas the impact of incorrect ones will diminish.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Since the s i , s ′ i and s ′′ i are similar literally and semantically, generating their representations with dropout noise may obscure their entailment relation and add inaccurate signals to the representation learning process. So we turn off the dropout of the encoder when modeling the triplet relation.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
As s ′ i is more similar to s i in semantics than s ′′ i is, we could model such relation with a triplet objective:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
4) in whichh i is the sentence representation of s i generated without dropout noise and sim(i, j) is the cosine similarity between i and j. As the semantic difference between s ′ i and s ′′ i may be subtle depending on the original sentence s i and the masked words, here we set m to zero.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Combine formula (3) and formula (4), the final form of our training objective is:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
in which λ is a coefficient.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Evaluation Tasks We evaluate our method on two kinds of sentence related tasks:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
• Unsupervised Semantic Textual Similarity (STS): These tasks measure the model's ability to estimate the semantic similarities between sentences.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
• SentEval Transfer Tasks: These tasks measure the effectiveness of sentence embeddings used in downstream transfer tasks.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Baselines We compare ArcCSE to several representative methods on STS and SentEval tasks, such as average GloVe embeddings (Pennington et al., 2014), Skip-thought (Kiros et al., 2015), average BERT embeddings from the last layer (Devlin et al., 2019), BERT-Flow , and BERT-Whitening (Su et al., 2021). We also include the recently proposed contrastive learning methods, such as ISBERT (Zhang et al., 2020), CT-BERT (Carlsson et al., 2021), ConSERT (Yan et al., 2021), and the current state-of-the-art method SimCSE (Gao et al., 2021).
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Implementation Details We train ArcCSE with the pre-trained checkpoints of BERT base and BERT large (Devlin et al., 2019). We also employ our method to SBERT (Reimers and Gurevych, 2019), which has been trained on NLI datasets, to verify the generalizability of our method.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Following SimCSE (Gao et al., 2021) the sentence representation during training, and use the [CLS] output without MLP layer for evaluation. The dropout rate is set to 0.1. For ArcCon loss, we set the angular margin m to 10 degrees and the temperature τ to 0.05. When modeling the entailment relation of triplet sentences, we set the masking ratios as 20% and 40% respectively. Since the semantic difference between triplet sentences is more obvious for long sentences, we filter out sentences with less than 25 words and use the left ones for the triplet loss. The loss coefficient λ is set to 0.1 experimentally. We use one million random sampled sentences from English Wikipedia for training, which has been used in previous work (Gao et al., 2021) 1 . During training, the sentences are sampled by length. We set different maximum sentence lengths for ArcCon loss and triplet loss to save memory. The length is set to 32 for the ArcCon loss in large models, and to the maximum length within a batch for all other cases. We train our model for one epoch and the learning rate is set to 3e-5 for base models and 1e-5 for large models. We search the batch size within {8, 16, 32} and always update the parameters every 64 steps. The model is optimized by the AdamW with Sharpness-Aware Minimization (Foret et al., 2021) and default configurations. We evaluate our model every 125 training steps on the development set of STS-B, and the best checkpoint is used for the final evaluation on test sets. Our implementation is based on Hugging-Face's Transformers (Wolf et al., 2020).
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
We conduct experiments on 7 semantic textual similarity (STS) tasks, including STS tasks 2012-2016 (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016, STS Benchmark (Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014). Within these datasets, each sample contains two sentences and a gold score between 0 and 5 which indicates their semantic similarity. We use SentEval toolkit (Conneau and Kiela, 2018) for evaluation and report the Spearman's correlation following previous works (Reimers and Gurevych, 2019;Gao et al., 2021).
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
The evaluation results are shown in Table 1  from which we can see that ArcCSE outperforms the previous approaches. Compared with the previous state-of-the-art method SimCSE, ArcCSE-BERT base raises the average Spearman's correlation from 76.25% to 78.11%, and ArcCSE-BERT large further pushes the results to 79.37%. The performance is even better than strong supervised method SBERT, which has already been trained on NLI datasets. Furthermore, we can also employ our method to SBERT and improve its performance to 79.06% and 80.69% for the base and large models respectively, which is more effective than SimCSE. We also explore the improvements made by the ArcCon loss and triplet loss independently based on BERT base . From Table 1 we can see that with ArcCon loss alone, the average Spearman's correlation is 77.25%. When combining the traditional NT-Xent loss with our proposed triplet loss, the average Spearman's correlation is 77.02%. Both of them outperform the previous state-of-the-art method SimCSE, whose average Spearman's correlation is 76.25%. This demonstrates the effectiveness of ArcCon and triplet loss we proposed.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
We evaluate our model with SentEval toolkit on several supervised transfer tasks, including: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). For each task, SentEval trains a logistic regression classifier on top of the sentence embeddings and tests the performance on the downstream task. For a fair comparison, we do not include models with auxiliary tasks like masked language modeling.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
The results are shown in Table 2. We can see that ArcCSE performs on par or better than baseline methods in both BERT base and BERT large level. This demonstrates the effectiveness of our method in learning domain-specific sentence embeddings.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Effect of Angular Margin The angular margin m in ArcCon loss affects the discriminative power directly. To investigate the effect of m, we conduct an experiment by varying m from 0 degrees to 20 degrees, increased by 2 degrees at each step. We tune the hyper-parameter based on Spearman's correlation on the development set of STS-B following previous works (Kim et al., 2021;Gao et al., 2021). The results are shown in Figure 4. We can see that the best performance is achieved when m = 10, either larger or smaller margin degrade the performance. This matches our intuition since small m may have little effect, and large m may negatively influence the positive pair relation modeling.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Effect of Temperature The temperature τ in ArcCon Loss affects its effectiveness, so we carry out an experiment with τ varying from 0.01 to 0.1, increased by 0.01 at each step. The results are shown in Figure 5. We can see that the model  Table 3: Effect of on-off Switching of Dropout. We use different dropout settings to generate sentence embeddings used for ArcCon loss and triplet loss respectively. The "on", "off" and "mix" mean turn dropout on, turn dropout off and use different settings for two passes separately.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
performs best when τ = 0.05, so we use this value throughout our experiments. Effect of Masking Ratios The masking ratios determine the sentences generated for the entailment relation modeling and their differences in semantics, so we conduct an experiment to explore the effect of different masking ratios. The first masking ratio r 1 is varied from 10% to 25%, increased by 5% for each step. The second masking ratio r 2 is derived by adding an extra value r d to r 1 . r d is varied from 10% to 35%, increased by 5% for each step. The results are shown in Figure 6. We can see that large differences between the two masking ratios tend to lead lower Spearman's correlation compared to the smaller ones. The reason may be that the larger the semantic difference is, the easier for the model to estimate the entailment relations among the triplet sentences, which makes the triplet loss less helpful. The best performance is achieved when r 1 is 20% and r 2 is 40%, and the corresponding Spearman's correlation is 0.847. We use them as our hyper-parameters.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Effect of on-off Switching of Dropout The onoff switching of dropout in the BERT-like sentence encoder affects the generated sentence representations directly. Since dropout performs a kind of averaging over the ensemble of possible subnetworks, an embedding generated with dropout turned off can be seen as a kind of "averaging" representation, while an embedding generated with dropout turned on can be seen as generated through a subnetwork.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
In ArcCSE, we use the embeddings generated with the encoder dropout turned on as input for Arc-Con loss, which regularizes the network by making representations generated through different subnetworks similar. When modeling the entailment relation, we generate "averaging" representations with dropout turn-off to avoid inaccurate signals. In order to verify our intuition, we conduct two experiments with different dropout settings. In the first experiment, we feed ArcCon two sentence representations generated with dropout turns on and off respectively. We carry out this experiment with angular margins ranging between 2 degrees to 12 degrees and report the best result. In the second one, we feed the triplet loss representations that are generated with dropout turns on and maintain the other settings. The results are shown in Table  3. We can see that the original settings that turn dropout on for ArcCon and turn dropout off for triplet loss achieve the best performance, which confirms our intuition.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Effect of Coefficient in the Training Objective The coefficient λ in the final optimization objective adjusts the relative weights between ArcCon and the triplet loss, as shown in formula (5). To find the most suitable λ, we conduct an experiment by varying λ from 0 to 1.2 and increased by 0.1 at each step. The results are shown in Figure 7.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
We can see that the best performance is achieved when λ = 0.1, and the corresponding Spearman's correlation is 0.847. This demonstrates that we can get the best performance by combining ArcCon and the triplet loss with proper λ.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
Alignment and uniformity are two properties closely related to contrastive learning and could be used to measure the quality of representations (Wang and Isola, 2020). Alignment favors encoders that generate similar representations for similar instances. It could be defined with the expected distance between embeddings of the positive paired instances:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
where p pos denotes the distribution of positive paired instances. Uniformity prefers uniformly distributed representations, which helps preserve maximal information. It could be defined as:
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
where p data denotes whole data distribution.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
To justify the inner workings of our approach, we calculate the alignment and uniformity metrics every 10 steps during training on the STS-B development set. We compare our approach with Sim-CSE and visualize the results in Figure 8. We can see that compared to the original BERT checkpoint, both ArcCSE and SimCSE improve the alignment and uniformity measures during training. ArcCSE performs better on the alignment measure and on par with SimCSE on the uniformity measure. This verifies the intuition of our approach and demonstrates that ArcCSE could help improve the quality of sentence representations.
A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space
In this work, we propose ArcCSE, a self-supervised contrastive learning framework for learning sen-  tence representation. We propose a new optimizing objective ArcCon loss to model pairwise sentence relations with enhanced discriminating power, and a new self-supervised task to model the partial order of semantics between sentences. Experimental results on semantic textual similarity tasks (STS) and SentEval tasks demonstrate that both techniques bring substantial improvements and our method outperforms previous state-of-theart method for sentence representation learning.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Due to the limited amount of labelled training data available for many natural language processing (NLP) tasks, transfer learning has become ubiquitous (Ruder et al., 2019). For some time, transfer learning in NLP was limited to pretrained word embeddings (Mikolov et al., 2013;Pennington et al., 1 https://github.com/JohnGiorgi/DeCLUTR 2014). Recent work has demonstrated strong transfer task performance using pretrained sentence embeddings. These fixed-length vectors, often referred to as "universal" sentence embeddings, are typically learned on large corpora and then transferred to various downstream tasks, such as clustering (e.g. topic modelling) and retrieval (e.g. semantic search). Indeed, sentence embeddings have become an area of focus, and many supervised (Conneau et al., 2017), semi-supervised (Subramanian et al., 2018;Phang et al., 2018;Cer et al., 2018;Reimers and Gurevych, 2019) and unsupervised (Le and Mikolov, 2014;Jernite et al., 2017;Kiros et al., 2015;Hill et al., 2016;Logeswaran and Lee, 2018) approaches have been proposed. However, the highest performing solutions require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. Therefore, closing the performance gap between unsupervised and supervised universal sentence embedding methods is an important goal.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Pretraining transformer-based language models has become the primary method for learning textual representations from unlabelled corpora (Radford et al., 2018;Devlin et al., 2019;Dai et al., 2019;Yang et al., 2019;Liu et al., 2019;Clark et al., 2020). This success has primarily been driven by masked language modelling (MLM). This selfsupervised token-level objective requires the model to predict the identity of some randomly masked tokens from the input sequence. In addition to MLM, some of these models have mechanisms for learning sentence-level embeddings via self-supervision. In BERT (Devlin et al., 2019), a special classification token is prepended to every input sequence, and its representation is used in a binary classifi-cation task to predict whether one textual segment follows another in the training corpus, denoted Next Sentence Prediction (NSP). However, recent work has called into question the effectiveness of NSP (Conneau and Lample, 2019;You et al., 1904;Joshi et al., 2020). In RoBERTa (Liu et al., 2019), the authors demonstrated that removing NSP during pretraining leads to unchanged or even slightly improved performance on downstream sentencelevel tasks (including semantic text similarity and natural language inference). In ALBERT (Lan et al., 2020), the authors hypothesize that NSP conflates topic prediction and coherence prediction, and instead propose a Sentence-Order Prediction objective (SOP), suggesting that it better models inter-sentence coherence. In preliminary evaluations, we found that neither objective produces good universal sentence embeddings (see Appendix A). Thus, we propose a simple but effective self-supervised, sentence-level objective inspired by recent advances in metric learning.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Metric learning is a type of representation learning that aims to learn an embedding space where the vector representations of similar data are mapped close together, and vice versa (Lowe, 1995;Mika et al., 1999;Xing et al., 2002). In computer vision (CV), deep metric learning (DML) has been widely used for learning visual representations (Wohlhart and Lepetit, 2015;Wen et al., 2016;Zhang and Saligrama, 2016;Bucher et al., 2016;Leal-Taixé et al., 2016;Tao et al., 2016;Yuan et al., 2020;He et al., 2018;Grabner et al., 2018;Yelamarthi et al., 2018;. Generally speaking, DML is approached as follows: a "pretext" task (often self-supervised, e.g. colourization or inpainting) is carefully designed and used to train deep neural networks to generate useful feature representations. Here, "useful" means a representation that is easily adaptable to other downstream tasks, unknown at training time. Downstream tasks (e.g. object recognition) are then used to evaluate the quality of the learned features (independent of the model that produced them), often by training a linear classifier on the task using these features as input. The most successful approach to date has been to design a pretext task for learning with a pair-based contrastive loss function. For a given anchor data point, contrastive losses attempt to make the distance between the anchor and some positive data points (those that are similar) smaller than the distance between the anchor and some neg-ative data points (those that are dissimilar) (Hadsell et al., 2006). The highest-performing methods generate anchor-positive pairs by randomly augmenting the same image (e.g. using crops, flips and colour distortions); anchor-negative pairs are randomly chosen, augmented views of different images (Bachman et al., 2019;Tian et al., 2020;He et al., 2020;. In fact, Kong et al., 2020 demonstrate that the MLM and NSP objectives are also instances of contrastive learning.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Inspired by this approach, we propose a selfsupervised, contrastive objective that can be used to pretrain a sentence encoder. Our objective learns universal sentence embeddings by training an encoder to minimize the distance between the embeddings of textual segments randomly sampled from nearby in the same document. We demonstrate our objective's effectiveness by using it to extend the pretraining of a transformer-based language model and obtain state-of-the-art results on SentEval (Conneau and Kiela, 2018) -a benchmark of 28 tasks designed to evaluate universal sentence embeddings. Our primary contributions are:
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• We propose a self-supervised sentence-level objective that can be used alongside MLM to pretrain transformer-based language models, inducing generalized embeddings for sentence-and paragraph-length text without any labelled data (subsection 5.1).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• We perform extensive ablations to determine which factors are important for learning highquality embeddings (subsection 5.2).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• We demonstrate that the quality of the learned embeddings scale with model and data size. Therefore, performance can likely be improved simply by collecting more unlabelled text or using a larger encoder (subsection 5.3).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• We open-source our solution and provide detailed instructions for training it on new data or embedding unseen text. 2
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Our method learns textual representations via a contrastive loss by maximizing agreement between textual segments (referred to as "spans" in the rest of the paper) sampled from nearby in the same document. Illustrated in Figure 1, this approach comprises the following components:
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• A data loading step randomly samples paired anchor-positive spans from each document in a minibatch of size N . Let A be the number of anchor spans sampled per document, P be the number of positive spans sampled per anchor and i ∈ {1 . . . AN } be the index of an arbitrary anchor span. We denote an anchor span and its corresponding p ∈ {1 . . . P } positive spans as s i and s i+pAN respectively. This procedure is designed to maximize the chance of sampling semantically similar anchor-positive pairs (see subsection 3.2).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• An encoder f (·) maps each token in the input spans to an embedding. Although our method places no constraints on the choice of encoder, we chose f (·) to be a transformer-based language model, as this represents the state-ofthe-art for text encoders (see subsection 3.3).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• A pooler g(·) maps the encoded spans f (s i ) and f (s i+pAN ) to fixed-length embeddings e i = g(f (s i )) and its corresponding mean positive embedding
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Similar to Reimers and Gurevych 2019, we found that choosing g(·) to be the mean of the token-level embeddings (referred to as "mean pooling" in the rest of the paper) performs well (see Appendix, Table 4). We pair each anchor embedding with the mean of multiple positive embeddings. This strategy was proposed by Saunshi et al. 2019, who demonstrated theoretical and empirical improvements compared to using a single positive example for each anchor.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• A contrastive loss function defined for a contrastive prediction task. Given a set of embedded spans {e k } including a positive pair of examples e i and e i+AN , the contrastive prediction task aims to identify e i+AN in {e k } k =i for a given e i
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
where sim(u, v) = u T v/||u|| 2 ||v|| 2 denotes the cosine similarity of two vectors u and v, 1 [i =k] ∈ {0, 1} is an indicator function evaluating to 1 if i = k, and τ > 0 denotes the temperature hyperparameter.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
During training, we randomly sample minibatches of N documents from the train set and define the contrastive prediction task on anchorpositive pairs e i , e i+AN derived from the N documents, resulting in 2AN data points. As proposed in (Sohn, 2016), we treat the other 2(AN − 1) instances within a minibatch as negative examples. The cost function takes the following form
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
This is the InfoNCE loss used in previous works (Sohn, 2016;Wu et al., 2018;Oord et al., 2018) and denoted normalized temperature-scale crossentropy loss or "NT-Xent" in .
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
To embed text with a trained model, we simply pass batches of tokenized text through the model, without sampling spans. Therefore, the computational cost of our method at test time is the cost of the encoder, f (·), plus the cost of the pooler, g(·), which is negligible when using mean pooling.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We start by choosing a minimum and maximum span length; in this paper, min = 32 and max = 512, the maximum input size for many pretrained transformers. Next, a document d is tokenized to produce a sequence of n tokens x d = (x 1 , x 2 . . . x n ). To sample an anchor span s i from x d , we first sample its length anchor from a beta distribution and then randomly (uniformly) sample its starting position s start
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We then sample p ∈ {1 . . . P } corresponding positive spans s i+pAN independently following a similar procedure
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
where p anchor ∼ Beta(α = 4, β = 2), which skews anchor sampling towards longer spans, and p positive ∼ Beta(α = 2, β = 4), which skews positive sampling towards shorter spans (Figure 1, C). In practice, we restrict the sampling of anchor spans from the same document such that they are a minimum of 2 * max tokens apart. In Appendix B, we show examples of text that has been sampled by our method. We note several carefully considered decisions in the design of our sampling procedure:
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Sampling span lengths from a distribution clipped at min = 32 and max = 512 encourages the model to produce good embeddings for text ranging from sentence-to paragraphlength. At test time, we expect our model to be able to embed up-to paragraph-length texts.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• We found that sampling longer lengths for the anchor span than the positive spans improves performance in downstream tasks (we did not find performance to be sensitive to the specific choice of α and β). The rationale for this is twofold. First, it enables the model to learn global-to-local view prediction as in ( • Sampling positives nearby to the anchor exploits the distributional hypothesis and increases the chances of sampling valid (i.e. semantically similar) anchor-positive pairs.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• By sampling multiple anchors per document, each anchor-positive pair is contrasted against both easy negatives (anchors and positives sampled from other documents in a minibatch) and hard negatives (anchors and positives sampled from the same document).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
In conclusion, the sampling procedure produces three types of positives: positives that partially overlap with the anchor, positives adjacent to the anchor, and positives subsumed by the anchor (Figure 1, B) and two types of negatives: easy negatives sampled from a different document than the anchor, and hard negatives sampled from the same document as the anchor. Thus, our stochastically generated training set and contrastive loss implicitly define a family of predictive tasks which can be used to train a model, independent of any specific encoder architecture.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We use our objective to extend the pretraining of a transformer-based language model (Vaswani et al., 2017), as this represents the state-of-the-art encoder in NLP. We implement the MLM objective as described in (Devlin et al., 2019) on each anchor span in a minibatch and sum the losses from the MLM and contrastive objectives before backpropagating Implementation We implemented our model in PyTorch (Paszke et al., 2017) using AllenNLP (Gardner et al., 2018). We used the NT-Xent loss function implemented by the PyTorch Metric Learning library (Musgrave et al., 2019) and the pretrained transformer architecture and weights from the Transformers library (Wolf et al., 2020). All models were trained on up to four NVIDIA Tesla V100 16 or 32GB GPUs. Training Unless specified otherwise, we train for one to three epochs over the 497,868 documents with a minibatch size of 16 and a temperature τ = 5 × 10 −2 using the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate (LR) of 5 × 10 −5 and a weight decay of 0.1. For every document in a minibatch, we sample two anchor spans (A = 2) and two positive spans per anchor (P = 2). We use the Slanted Triangular LR scheduler (Howard and Ruder, 2018) with a number of train steps equal to training instances and a cut fraction of 0.1. The remaining hyperparameters of the underlying pretrained transformer (i.e. DistilRoBERTa or RoBERTa-base) are left at their defaults. All gradients are scaled to a vector norm of 1.0 before backpropagating. Hyperparameters were tuned on the SentEval validation sets.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We evaluate all methods on the SentEval benchmark, a widely-used toolkit for evaluating generalpurpose, fixed-length sentence representations. SentEval is divided into 18 downstream tasks -representative NLP tasks such as sentiment analysis, natural language inference, paraphrase detection and image-caption retrieval -and ten probing tasks, which are designed to evaluate what linguistic properties are encoded in a sentence representation. We report scores obtained by our model and the relevant baselines on the downstream and probing tasks using the SentEval toolkit 3 with default parameters (see Appendix C for details). Note that all the supervised approaches we compare to are trained on the SNLI corpus, which is included as a downstream task in SentEval. To avoid train-test contamination, we compute average downstream scores without considering SNLI when comparing to these approaches in Table 2.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We compare to the highest performing, most popular sentence embedding methods: InferSent, Google's USE and Sentence Transformers. For In-ferSent, we compare to the latest model. 4 We use the latest "large" USE model 5 , as it is most similar in terms of architecture and number of parameters to DeCLUTR-base. For Sentence Transformers, we compare to "roberta-base-nli-mean-tokens" 6 , which, like DeCLUTR-base, uses the RoBERTabase architecture and pretrained weights. The only difference is each method's extended pretraining strategy. We include the performance of averaged GloVe 7 and fastText 8 word vectors as weak baselines. Trainable model parameter counts and sentence embedding dimensions are listed in Table 1. Despite our best efforts, we could not evaluate the pretrained QuickThought models against the full SentEval benchmark. We cite the scores from the paper directly. Finally, we evaluate the pretrained transformer model's performance before it is subjected to training with our contrastive objective, denoted "Transformer-*". We use mean pooling on the pretrained transformers token-level output to produce sentence embeddings -the same pooling strategy used in our method.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
In subsection 5.1, we compare the performance of our model against the relevant baselines. In the remaining sections, we explore which components contribute to the quality of the learned embeddings.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Downstream task performance Compared to the underlying pretrained models DistilRoBERTa  and RoBERTa-base, DeCLUTR-small and DeCLUTR-base obtain large boosts in average downstream performance, +4% and +6% respectively (Table 2). DeCLUTR-base leads to improved or equivalent performance for every downstream task but one (SST5) and DeCLUTR-small for all but three (SST2, SST5 and TREC). Compared to existing methods, DeCLUTR-base matches or even outperforms average performance without using any hand-labelled training data. Surprisingly, we also find that DeCLUTR-small outperforms Sentence Transformers while using ∼34% less trainable parameters.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Probing task performance With the exception of InferSent, existing methods perform poorly on the probing tasks of SentEval (Table 3). Sentence Transformers, which begins with a pretrained transformer model and fine-tunes it on NLI datasets, scores approximately 10% lower on the probing tasks than the model it fine-tunes. In contrast, both DeCLUTR-small and DeCLUTR-base perform comparably to the underlying pretrained model in terms of average performance. We note that the purpose of the probing tasks is not the development of ad-hoc models that attain top performance on them (Conneau et al., 2018). However, it is still interesting to note that high downstream task performance can be obtained without sacrificing probing task performance. Furthermore, these results suggest that fine-tuning transformer-based language models on NLI datasets may discard some of the linguistic information captured by the pretrained model's weights. We suspect that the inclusion of MLM in our training objective is responsible for DeCLUTR's relatively high performance on the probing tasks.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
The downstream evaluation of SentEval includes supervised and unsupervised tasks. In the unsupervised tasks, the embeddings of the method to evaluate are used as-is without any further training (see Appendix C for details). Interestingly, we find that USE performs particularly well across the unsupervised evaluations in SentEval (tasks marked with a * in Table 2). Given the similarity of the USE architecture to Sentence Transformers and DeCLUTR and the similarity of its supervised NLI training objective to InferSent and Sentence Transformers, we suspect the most likely cause is one or more of its additional training objectives. These include a conversational response prediction task (Henderson et al., 2017) and a Skip-Thoughts (Kiros et al., 2015) like task.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
We ablate several components of the sampling procedure, including the number of anchors sampled per document A, the number of positives sampled per anchor P , and the sampling strategy for those positives (Figure 2). We note that when A = 2, the model is trained on twice the number of spans and twice the effective batch size (2AN , where N is the number of documents in a minibatch) as compared to when A = 1. To control for this, all experi- We find that sampling multiple anchors per document has a large positive impact on the quality of learned embeddings. We hypothesize this is because the difficulty of the contrastive objective increases when A > 1. Recall that a minibatch is composed of random documents, and each anchor-positive pair sampled from a document is contrasted against all other anchor-positive pairs in the minibatch. When A > 1, anchor-positive pairs will be contrasted against other anchors and positives from the same document, increasing the difficulty of the contrastive objective, thus leading to better representations. We also find that a positive sampling strategy that allows positives to be adjacent to and subsumed by the anchor outperforms a strategy that only allows adjacent or subsuming views, suggesting that the information captured by these views is complementary. Finally, we note that sampling multiple positives per anchor (P > 1) has minimal impact on performance. This is in contrast to (Saunshi et al., 2019), who found both theoretical and empirical improvements when multiple positives are averaged and paired with a given anchor.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
To determine the importance of the training objectives, train set size, and model capacity, we trained two sizes of the model with 10% to 100% (1 full epoch) of the train set ( Figure 3). Pretraining the model with both the MLM and contrastive objectives improves performance over training with either objective alone. Including MLM alongside the contrastive objective leads to monotonic improvement as the train set size is increased. We hypothesize that including the MLM loss acts as a form of regularization, preventing the weights of the pretrained model (which itself was trained with an MLM loss) from diverging too dramatically, a phenomenon known as "catastrophic forgetting" (McCloskey and Cohen, 1989;Ratcliff, 1990). These results suggest that the quality of embeddings learned by our approach scale in terms of model capacity and train set size; because the training method is completely self-supervised, scaling the train set would simply involve collecting more unlabelled text.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
In this paper, we proposed a self-supervised objective for learning universal sentence embeddings. Our objective does not require labelled training data and is applicable to any text encoder. We demonstrated the effectiveness of our objective by evaluating the learned embeddings on the SentE-val benchmark, which contains a total of 28 tasks designed to evaluate the transferability and linguistic properties of sentence representations. When used to extend the pretraining of a transformerbased language model, our self-supervised objective closes the performance gap with existing methods that require human-labelled training data. Our experiments suggest that the learned embeddings' quality can be further improved by increasing the model and train set size. Together, these results demonstrate the effectiveness and feasibility of replacing hand-labelled data with carefully designed self-supervised objectives for learning universal sentence embeddings. We release our model and code publicly in the hopes that it will be extended to new domains and non-English languages. 
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Certain pretrained transformers, such as BERT and ALBERT, have mechanisms for learning sequencelevel embeddings via self-supervision. These models prepend every input sequence with a special classification token (e.g. "[CLS]"), and its representation is learned using a simple classification task, such as Next Sentence Prediction (NSP) or Sentence-Order Prediction (SOP) (see Devlin et al. 2019 andLan et al. 2020 respectively for details on these tasks). However, during preliminary experiments, we noticed that these models are not good universal sentence encoders, as measured by their performance on the SentEval benchmark (Conneau and Kiela, 2018). As a simple experiment, we evaluated three pretrained transformer models on SentEval: one trained with the NSP loss (BERT), one trained with the SOP loss (ALBERT) and one trained with neither, RoBERTa (Liu et al., 2019). We did not find that the CLS embeddings produced by models trained against the NSP or SOP losses to outperform that of a model trained without either loss and sometimes failed to outperform a bag-ofwords (BoW) baseline (Table 4). Furthermore, we find that pooling token embeddings via averaging (referred to as "mean pooling" in our paper) outperforms pooling via the CLS classification token. Our results are corroborated by Liu et al. 2019, who find that removing NSP loss leads to the same or better results on downstream tasks and Reimers and Gurevych 2019, who find that directly using the output of BERT as sentence embeddings leads to poor performances on the semantic similarity tasks of SentEval.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
In Table 5, we present examples of anchor-positive and anchor-negative pairs generated by our sampling procedure. We show one example for each possible view of a sampled positive, e.g. positives adjacent to, overlapping with, or subsumed by the anchor. For each anchor-positive pair, we show examples of both a hard negative (derived from the same document) and an easy negative (derived from another document). Recall that a minibatch is composed of random documents, and each anchor-positive pair sampled from a document is contrasted against all other anchor-positive pairs in the minibatch. Thus, hard negatives, as we have described them here, are generated only when sampling multiple anchors per document (A > 1).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
SentEval is a benchmark for evaluating the quality of fixed-length sentence embeddings. It is divided into 18 downstream tasks, and 10 probing tasks. Sentence embedding methods are evaluated on these tasks via a simple interface 9 , which standardizes training, evaluation and hyperparameters. For most tasks, the method to evaluate is used to produce fix-length sentence embeddings, and a simple logistic regression (LR) or multi-layer perception (MLP) model is trained on the task using these embeddings as input. For other tasks (namely Table 4: Results on the downstream and probing tasks from the validation set of SentEval. We compare models trained with the Next Sentence Prediction (NSP) and Sentence-Order Prediction (SOP) losses to a model trained with neither, using two different pooling strategies: "*-CLS", where the special classification token is used as its sentence representation, and "*-mean", where each sentence is represented by the mean of its token embeddings. several semantic text similarity tasks), the embeddings are used as-is without any further training. Note that this setup is different from evaluations on the popular GLUE benchmark (Wang et al., 2019), which typically use the task data to fine-tune the parameters of the sentence embedding model. In subsection C.1, we present the individual tasks of the SentEval benchmark. In subsection C.2, we explain our method for computing the average downstream and average probing scores presented in our paper.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
The downstream tasks of SentEval are representative NLP tasks used to evaluate the transferability of fixed-length sentence embeddings. We give a brief overview of the broad categories that divide the tasks below (see Conneau and Kiela 2018 for more details):
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Binary and multi-class classification:
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
These tasks cover various types of sentence classification, including sentiment analysis (MR Lee 2005, SST2 andSST5 Socher et al. 2013), question-type (TREC) (Voorhees and Tice, 2000), product reviews (CR) (Hu and Liu, 2004), subjectivity/objectivity (SUBJ) (Pang and Lee, 2004) and opinion polarity (MPQA) (Wiebe et al., 2005).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Entailment and semantic relatedness: These tasks cover multiple entailment datasets (also known as natural language inference or NLI), including SICK-E (Marelli et al., 2014)   to such feelings. Fawkes cried out and flew ahead, and Albus Dumbledore followed. Further along the Dementors' path, people were still alive to be fought for. And no matter how much he himself was hurting, while there were still people who needed him he would go on. For
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Caption-Image retrieval This task is comprised of two sub-tasks: ranking a large collection of images by their relevance for some given query text (Image Retrieval) and ranking captions by their relevance for some given query image (Caption Retrieval). Both tasks are evaluated on data from the COCO dataset (Lin et al., 2014). Each image is represented by a pretrained, 2048-dimensional embedding produced by a ResNet-101 (He et al., 2016).
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
The probing tasks are designed to evaluate what linguistic properties are encoded in a sentence representation. All tasks are binary or multi-class classification. We give a brief overview of each task below (see Conneau et al. 2018 for more details):
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Sentence length (SentLen): A multi-class classification task where a model is trained to predict the length of a given input sentence, which is binned into six possible length ranges.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Word content (WC): A multi-class classification task where, given 1000 words as targets, the goal is to predict which of the target words appears in a given input sentence. Each sentence contains a single target word, and the word occurs exactly once in the sentence.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Tree depth (TreeDepth): A multi-class classification task where the goal is to predict the maximum depth (with values ranging from 5 to 12) of a given input sentence's syntactic tree.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Bigram Shift (BShift): A multi-class classification task where the goal is to predict whether two consecutive tokens within a given sentence have been inverted.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Top Constituents (TopConst): A multi-class classification task where the goal is to predict the top constituents (from a choice of 19) immediately below the sentence (S) node of the sentence's syntactic tree.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Tense: A binary classification task where the goal is to predict the tense (past or present) of the main verb in a sentence.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Subject number (SubjNum): A binary classification task where the goal is to predict the number (singular or plural) of the subject of the main clause.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Object number (ObjNum): A binary classification task, analogous to SubjNum, where the goal is to predict the number (singular or plural) of the direct object of the main clause.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Semantic odd man out (SOMO): A binary classification task where the goal is to predict whether a sentence has had a single randomly picked noun or verb replaced with another word with the same part-of-speech.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Coordinate inversion (CoordInv): A binary classification task where the goal is to predict whether the order of two coordinate clauses in a sentence has been inverted.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
In our paper, we present averaged downstream and probing scores. Computing averaged probing scores was straightforward; each of the ten probing tasks reports a simple accuracy, which we averaged.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
To compute an averaged downstream score, we do the following:
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• If a task reports Spearman correlation (i.e. SICK-R, STS-B), we use this score when computing the average downstream task score. If the task reports a mean Spearman correlation for multiple subtasks (i.e. STS12, STS13, STS14, STS15, STS16), we use this score.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• If a task reports both an accuracy and an F1score (i.e. MRPC), we use the average of these two scores.
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• For the Caption-Image Retrieval task, we report the average of the Recall@K, where K ∈ {1, 5, 10} for the Image and Caption retrieval tasks (a total of six scores). This is the default behaviour of SentEval.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Sentence representation learning plays a vital role in natural language processing tasks (Kiros et al., 2015;Hill et al., 2016;Conneau et al., 2017;Cer et al., 2018). Good sentence representations benefit a wide range of downstream tasks, especially for computationally expensive ones, including largescale semantic similarity comparison and information retrieval.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Recently, BERT-based pre-trained language models have achieved high performance on many * Work done during internship at Meituan Inc. The first two authors contribute equally. Weiran Xu is the corresponding author.  downstream tasks with additional supervision. However, the native sentence representations derived from BERT 1 are proved to be of low-quality (Reimers and Gurevych, 2019;Li et al., 2020). As shown in Figure 1a, when directly adopt BERTbased sentence representations to semantic textual similarity (STS) tasks, almost all pairs of sentences achieved a similarity score between 0.6 to 1.0 , even if some pairs are regarded as completely unrelated by the human annotators. In other words, the BERT-derived native sentence representations are somehow collapsed (Chen and He, 2020), which means almost all sentences are mapped into a small area and therefore produce high similarity.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Such phenomenon is also observed in several previous works (Gao et al., 2019;Li et al., 2020). They find the word representation space of BERT is anisotropic, the high-frequency words are clustered and close to the origin, while low-frequency words disperse sparsely. When averaging token embeddings, those high-frequency words dominate the sentence representations, inducing biases against their real semantics 2 . As a result, it is inappropriate to directly apply BERT's native sentence representations for semantic matching or text retrieval. Traditional methods usually fine-tune BERT with additional supervision. However, human annotation is costly and often unavailable in real-world scenarios.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
To alleviate the collapse issue of BERT as well as reduce the requirement for labeled data, we propose a novel sentence-level training objective based on contrastive learning Chen et al., 2020a,b). By encouraging two augmented views from the same sentence to be closer while keeping views from other sentences away, we reshape the BERT-derived sentence representation space and successfully solve the collapse issue (shown in Figure 1b). Moreover, we propose multiple data augmentation strategies for contrastive learning, including adversarial attack (Goodfellow et al., 2014;Kurakin et al., 2016), token shuffling, cutoff (Shen et al., 2020) and dropout (Hinton et al., 2012), that effectively transfer the sentence representations to downstream tasks. We name our approach ConSERT, a Contrastive Framework for SEntence Representation Transfer.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
ConSERT has several advantages over previous approaches. Firstly, it introduces no extra structure or specialized implementation during inference. The parameter size of ConSERT keeps the same as BERT, making it easy to use. Secondly, compared with pre-training approaches, ConSERT is more efficient. With only 1,000 unlabeled texts drawn from the target distribution (which is easy to collect in real-world applications), we achieve 35% relative performance gain over BERT, and the training stage takes only a few minutes (1-2k steps) on a single V100 GPU. Finally, it includes several effective and convenient data augmentation methods with minimal semantic impact. Their effects are validated and analyzed in the ablation studies.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Our contributions can be summarized as follows: 1) We propose a simple but effective sentence-level training objective based on contrastive learning. It mitigates the collapse of BERT-derived representations and transfers them to downstream tasks. 2) We explore various effective text augmentation strategies to generate views for contrastive learning and analyze their effects on unsupervised sentence representation transfer. 3) With only fine-tuning on unsupervised target datasets, our approach achieves significant improvement on STS tasks. When further incorporating with NLI supervision, our ap-proach achieves new state-of-the-art performance. We also show the robustness of our approach in data scarcity scenarios and intuitive analysis of the transferred representations. 3 2 Related Work 2.1 Sentence Representation Learning Supervised Approaches Several works use supervised datasets for sentence representation learning. Conneau et al. (2017) finds the supervised Natural Language Inference (NLI) task is useful to train good sentence representations. They use a BiLSTM-based encoder and train it on two NLI datasets, Stanford NLI (SNLI) (Bowman et al., 2015) and Multi-Genre NLI (MNLI) (Williams et al., 2018). Universal Sentence Encoder (Cer et al., 2018) adopts a Transformer-based architecture and uses the SNLI dataset to augment the unsupervised training. SBERT (Reimers and Gurevych, 2019) proposes a siamese architecture with a shared BERT encoder and is also trained on SNLI and MNLI datasets.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Self-supervised Objectives for Pre-training BERT (Devlin et al., 2019) proposes a bidirectional Transformer encoder for language model pre-training. It includes a sentence-level training objective, namely next sentence prediction (NSP), which predicts whether two sentences are adjacent or not. However, NSP is proved to be weak and has little contribution to the final performance . After that, various self-supervised objectives are proposed for pre-training BERT-like sentence encoders. Cross-Thought  and CMLM  are two similar objectives that recover masked tokens in one sentence conditioned on the representations of its contextual sentences. SLM (Lee et al., 2020) proposes an objective that reconstructs the correct sentence ordering given the shuffled sentences as the input. However, all these objectives need document-level corpus and are thus not applicable to downstream tasks with only short texts.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Unsupervised Approaches BERT-flow (Li et al., 2020) proposes a flow-based approach that maps BERT embeddings to a standard Gaussian latent space, where embeddings are more suitable for comparison. However, this approach introduces extra model structures and need specialized implementation, which may limit its application.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Contrastive Learning for Visual Representation Learning Recently, contrastive learning has become a very popular technique in unsupervised visual representation learning with solid performance (Chen et al., 2020a;Chen et al., 2020b). They believe that good representation should be able to identify the same object while distinguishing itself from other objects. Based on this intuition, they apply image transformations (e.g. cropping, rotation, cutout, etc.) to randomly generate two augmented versions for each image and make them close in the representation space. Such approaches can be regarded as the invariance modeling to the input samples. Chen et al. (2020a) proposes SimCLR, a simple framework for contrastive learning. They use the normalized temperature-scaled cross-entropy loss (NT-Xent) as the training loss, which is also called InfoNCE in the previous literature (Hjelm et al., 2018).
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Contrastive Learning for Textual Representation Learning Recently, contrastive learning has been widely applied in NLP tasks. Many works use it for language model pre-training. IS-BERT (Zhang et al., 2020) proposes to add 1-D convolutional neural network (CNN) layers on top of BERT and train the CNNs by maximizing the mutual information (MI) between the global sentence embedding and its corresponding local contexts embeddings. CERT (Fang and Xie, 2020) adopts a similar structure as MoCo  and uses back-translation for data augmentation. However, the momentum encoder needs extra memory and back-translation may produce false positives. BERT-CT (Carlsson et al., 2021) uses two individual encoders for contrastive learning, which also needs extra memory. Besides, they only sample 7 negatives, resulting in low training efficiency. De-CLUTR (Giorgi et al., 2020) adopts the architecture of SimCLR and jointly trains the model with contrastive objective and masked language model objective. However, they only use spans for contrastive learning, which is fragmented in semantics. CLEAR  uses the same architecture and objectives as DeCLUTR. Both of them are used to pre-train the language model, which needs a large corpus and takes a lot of resources. 
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
In this section, we present ConSERT for sentence representation transfer. Given a BERT-like pretrained language model M and an unsupervised dataset D drawn from the target distribution, we aim at fine-tuning M on D to make the sentence representation more task-relevant and applicable to downstream tasks. We first present the general framework of our approach, then we introduce several data augmentation strategies for contrastive learning. Finally, we talk about three ways to further incorporate supervision signals.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Our approach is mainly inspired by SimCLR (Chen et al., 2020a). As shown in Figure 2, there are three major components in our framework:
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• A data augmentation module that generates different views for input samples at the token embedding layer.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• A shared BERT encoder that computes sentence representations for each input text. During training, we use the average pooling of the token embeddings at the last layer to obtain sentence representations.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• A contrastive loss layer on top of the BERT encoder. It maximizes the agreement between one representation and its corresponding version that is augmented from the same sentence while keeping it distant from other sentence representations in the same batch. For each input text x, we first pass it to the data augmentation module, in which two transformations T 1 and T 2 are applied to generate two versions of token embeddings:
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
where e i , e j ∈ R L×d , L is the sequence length and d is the hidden dimension. After that, both e i and e j will be encoded by multi-layer transformer blocks in BERT and produce the sentence representations r i and r j through average pooling.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Following Chen et al. (2020a), we adopt the normalized temperature-scaled cross-entropy loss (NT-Xent) as the contrastive objective. During each training step, we randomly sample N texts from D to construct a mini-batch, resulting in 2N representations after augmentation. Each data point is trained to find out its counterpart among 2(N − 1) in-batch negative samples:
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
, where sim(·) indicates the cosine similarity function, τ controls the temperature and 1 is the indicator. Finally, we average all 2N in-batch classification losses to obtain the final contrastive loss L con .
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
We explore four different data augmentation strategies to generate views for contrastive learning, including adversarial attack (Goodfellow et al., 2014;Kurakin et al., 2016), token shuffling, cutoff (Shen et al., 2020) and dropout (Hinton et al., 2012), as illustrated in Figure 3. Adversarial Attack Adversarial training is generally used to improve the model's robustness. They generate adversarial samples by adding a worst-case perturbation to the input sample. We implement this strategy with Fast Gradient Value (FGV) (Rozsa et al., 2016), which directly uses the gradient to compute the perturbation and thus is faster than two-step alternative methods. Note that this strategy is only applicable when jointly training with supervision since it relies on supervised loss to compute adversarial perturbations.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Token Shuffling In this strategy, we aim to randomly shuffle the order of the tokens in the input sequences. Since the bag-of-words nature in the transformer architecture, the position encoding is the only factor about the sequential information. Thus, similar to Lee et al. (2020), we implement this strategy by passing the shuffled position ids to the embedding layer while keeping the order of the token ids unchanged.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Cutoff Shen et al. (2020) proposes a simple and efficient data augmentation strategy called cutoff. They randomly erase some tokens (for token cutoff), feature dimensions (for feature cutoff), or token spans (for span cutoff) in the L × d feature matrix. In our experiments, we only use token cutoff and feature cutoff and apply them to the token embeddings for view generation.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Dropout Dropout is a widely used regularization method that avoids overfitting. However, in our experiments, we also show its effectiveness as an augmentation strategy for contrastive learning. For this setting, we randomly drop elements in the token embedding layer by a specific probability and set their values to zero. Note that this strategy is different from Cutoff since each element is considered individually.  
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Besides unsupervised transfer, our approach can also be incorporated with supervised learning. We take the NLI supervision as an example. It is a sentence pair classification task, where the model are trained to distinguish the relation between two sentences among contradiction, entailment and neutral. The classification objective can be expressed as following:
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
, where r 1 and r 2 denote two sentence representations.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
We propose three ways for incorporating additional supervised signals:
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Joint training (joint) We jointly train the model with the supervised and unsupervised objectives L joint = L ce + αL con on NLI dataset. α is a hyper-parameter to balance two objectives.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Supervised training then unsupervised transfer (sup-unsup) We first train the model with L ce on NLI dataset, then use L con to finetune it on the target dataset.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Joint training then unsupervised transfer (joint-unsup) We first train the model with the L joint on NLI dataset, then use L con to fine-tune it on the target dataset.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
To verify the effectiveness of our proposed approach, we conduct experiments on Semantic Textual Similarity (STS) tasks under the unsupervised and supervised settings.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Dataset Following previous works (Reimers and Gurevych, 2019;Li et al., 2020;Zhang et al., 2020), we evaluate our approach on multiple STS datasets, including STS tasks 2012 -2016 (STS12 -STS16) (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016, STS benchmark (STSb) (Cer et al., 2017) and SICK-Relatedness (SICK-R) (Marelli et al.). Each sample in these datasets contains a pair of sentences as well as a gold score between 0 and 5 to indicate their semantic similarity. For our unsupervised experiments, we mix the unlabeled texts from these datasets to fine-tune our model. We obtain all 7 datasets through the SentEval toolkit (Conneau and Kiela, 2018). The statistics is shown in Table 1.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
For supervised experiments, we use the combination of SNLI (570k samples) (Bowman et al., 2015) and MNLI (430k samples) (Williams et al., 2018) to train our model. In the joint training setting, the NLI texts are also used for contrastive objectives.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Baselines To show our effectiveness on unsupervised sentence representation transfer, we mainly select BERT-flow (Li et al., 2020) for comparison, since it shares the same setting as our approach. Evaluation When evaluating the trained model, we first obtain the representation of sentences by averaging the token embeddings at the last two layers 4 , then we report the spearman correlation between the cosine similarity scores of sentence representations and the human-annotated gold scores. When calculating spearman correlation, we merge all sentences together (even if some STS datasets have multiple splits) and calculate spearman correlation for only once 5 . 4 As shown in Li et al. (2020), averaging the last two layers of BERT achieves slightly better results than averaging the last one layer. 5 Note that such evaluation procedure is different from Method STS12 STS13 STS14 STS15 STS16 STSb SICK-R Avg.  Implementation Details Our implementation is based on the Sentence-BERT 6 (Reimers and Gurevych, 2019). We use both the BERT-base and BERT-large for our experiments. The max sequence length is set to 64 and we remove the default dropout layer in BERT architecture considering the cutoff and dropout data augmentation strategies used in our framework. The ratio of token cutoff and feature cutoff is set to 0.15 and 0.2 respectively, as suggested in Shen et al. (2020). The ratio of dropout is set to 0.2. The temperature τ of NT-Xent loss is set to 0.1, and the α is set to 0.15 for the joint training setting. We adopt Adam optimizer and set the learning rate to 5e-7. We use a linear learning rate warm-up over 10% of the training steps. The batch size is set to 96 in most of our experiments. We use the dev set of STSb to tune the hyperparameters (including the augmentation strategies) and evaluate the model every 200 steps during training. The best checkpoint on the dev set of STSb is saved for test. We further discuss the influence of the batch size and the temperature in the subsequent section.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
For unsupervised evaluation, we load the pretrained BERT to initialize the BERT encoder in our framework. Then we randomly mix the unlabeled texts from 7 STS datasets and use them to fine-tune our model. SentEval toolkit, which calculates spearman correlation for each split and reports the mean or weighted mean scores. 6 https://github.com/UKPLab/ sentence-transformers
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
The results are shown in Table 2. We can observe that both BERT-flow and ConSERT can improve the representation space and outperform the GloVe and BERT baselines with unlabeled texts from target datasets. However, ConSERT large achieves the best performance among 6 STS datasets, significantly outperforming BERT large -flow with an 8% relative performance gain on average (from 70.76 to 76.45). Moreover, it is worth noting that ConSERT large even outperforms several supervised baselines (see Figure 3) like InferSent (65.01) and Universal Sentence Encoder (71.72), and keeps comparable to the strong supervised method SBERT large -NLI (76.55). For the BERT base architecture, our approach ConSERT base also outperforms BERT base -flow with an improvement of 3.17 (from 69.57 to 72.74).
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
For supervised evaluation, we consider the three settings described in Section 3.3. Note that in the joint setting, only NLI texts are used for contrastive learning, making it comparable to SBERT-NLI. We use the model trained under the joint setting as the initial checkpoint in the joint-unsup setting. We also re-implement the SBERT-NLI baselines and use them as the initial checkpoint in the sup-unsup setting.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
The results are illustrated in Table 3. For the models trained with NLI supervision, we find that ConSERT joint consistently performs better than SBERT, revealing the effectiveness of our proposed contrastive objective as well as the data augmentation strategies. On average, ConSERT base joint  Table 3: The performance comparison of ConSERT with other methods in a supervised setting. We report the spearman correlation ρ × 100 on 7 STS datasets. Methods with † indicate that we directly report the scores from the corresponding paper, while methods with ‡ indicate our implementation.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
achieves a performance gain of 2.88 over the reimplemented SBERT base -NLI, and ConSERT large joint achieves a performance gain of 2.70. When further performing representation transfer with STS unlabeled texts, our approach achieves even better performance. On average, ConSERT large joint-unsup outperforms the initial checkpoint ConSERT large joint with 1.84 performance gain, and outperforms the previous state-ofthe-art BERT large -flow with 2.92 performance gain. The results demonstrate that even for the models trained under supervision, there is still a huge potential of unsupervised representation transfer for improvement.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
To prove the hypothesis that the collapse issue is mainly due to the anisotropic space that is sensitive to the token frequency, we conduct experiments that mask the embeddings of several most frequent tokens when applying average pooling to calculate the sentence representations. The relation between the number of removed top-k frequent tokens and the average spearman correlation is shown in Figure 4.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
We can observe that when removing a few top frequent tokens, the performance of BERT improves sharply on STS tasks. When removing 34 most frequent tokens, the best performance is achieved (61.66), and there is an improvement of 7.8 from the original performance (53.86). For ConSERT, we find that removing a few most frequent tokens only results in a small improvement of less than 0.3. The results show that our approach reshapes the BERT's original embedding space, reducing the influence of common tokens on sentence representations.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
In this section, we study the effect of data augmentation strategies for contrastive learning. We consider 5 options for each transformation, including None (i.e. doing nothing), Shuffle, Token Cutoff, Average spearman correlation Feature Cutoff, and Dropout, resulting in 5×5 combinations. Note that the Adversarial Attack strategy is not considered here, since it needs additional supervision to generate adversarial samples. All these experiments follow the unsupervised setting and use the BERT base architecture.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
The results can be found in Figure 5. We can make the following observations. First, Shuffle and Token Cutoff are the two most effective strategies (where Shuffle is slightly better than Token Cutoff), significantly outperforming Feature Cutoff and Dropout. This is probably because Shuffle and Token Cutoff are more related to the downstream STS tasks since they are directly operated on the token level and change the structure of the sentence to produce hard examples.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Secondly, Feature Cutoff and Dropout also improve performance by roughly 4 points when compared with the None-None baseline. Moreover, we find they work well as a complementary strategy. Combining with another strategy like Shuffle may further improve the performance. When combined Shuffle with Feature Cutoff, we achieve the best result. We argue that Feature Cutoff and Dropout are useful in modeling the invariance of the internal noise for the sentence encoder, and thus improve the model's robustness.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Finally, we also observe that even without any data augmentation (the None-None combination), our contrastive framework can improve BERT's performance on STS tasks (from 53.86 to 63.84). This None-None combination has no effect on maximizing agreement between views since the repre-  Figure 6: The few-shot experiments under the unsupervised and supervised settings. We report the average spearman correlation on STS datasets with 1, 10, 100, 1,000, and 10,000 unlabeled texts available, respectively. The full dataset indicates all 89192 unlabeled texts from 7 STS datasets.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
sentations of augmented views are exactly the same. On the contrary, it tunes the representation space by pushing each representation away from others. We believe that the improvement is mainly due to the collapse phenomenon of BERT's native representation space. To some extent, it also explains why our method works.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
To validate the reliability and the robustness of ConSERT under the data scarcity scenarios, we conduct the few-shot experiments. We limit the number of unlabeled texts to 1, 10, 100, 1000, and 10000 respectively, and compare their performance with the full dataset. Figure 6 presents the results. For both the unsupervised and the supervised settings, our approach can make a huge improvement over the baseline with only 100 samples available. When the training samples increase to 1000, our approach can basically achieve comparable results with the models trained on the full dataset. The results reveal the robustness and effectiveness of our approach under the data scarcity scenarios, which is common in reality. With only a small amount of unlabeled texts drawn from the target data distribution, our approach can also tune the representation space and benefit the downstream tasks.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
The temperature τ in NT-Xent loss (Equation 1) is used to control the smoothness of the distribution normalized by softmax operation and thus influences the gradients when backpropagation. A large temperature smooths the distribution while a small temperature sharpens the distribution. In our experiments, we explore the influence of temperature   and present the result in Figure 7.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
As shown in the figure, we find the performance is extremely sensitive to the temperature. Either too small or too large temperature will make our model perform badly. And the optimal temperature is obtained within a small range (from about 0.08 to 0.12). This phenomenon again demonstrates the collapse issue of BERT embeddings, as most sentences are close to each other, a large temperature may make this task too hard to learn. We select 0.1 as the temperature in most of our experiments.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
In some previous works of contrastive learning, it is reported that a large batch size benefits the final performance and accelerates the convergence of the model since it provides more in-batch negative samples for contrastive learning (Chen et al., 2020a). Those in-batch negative samples improve the training efficiency. We also analyze the influence of the batch size for unsupervised sentence representation transfer.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
The results are illustrated in Table 4. We show both the spearman correlation and the corresponding training steps. We find that a larger batch size does achieve better performance. However, the improvement is not so significant. Meanwhile, a larger batch size does speed up the training process, but it also needs more GPU memories at the same time.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
In this paper, we propose ConSERT, a selfsupervised contrastive learning framework for transferring sentence representations to downstream tasks. The framework does not need extra structure and is easy to implement for any encoder. We demonstrate the effectiveness of our framework on various STS datasets, both our unsupervised and supervised methods achieve new state-of-the-art performance. Furthermore, few-shot experiments suggest that our framework is robust in the data scarcity scenarios. We also compare multiple combinations of data augmentation strategies and provide fine-grained analysis for interpreting how our approach works. We hope our work will provide a new perspective for future researches on sentence representation transfer. 2019YFF0303302, DOCOMO Beijing Communications Laboratories Co., Ltd, MoE-CMCC "Artifical Intelligence" Project No. MCM20190701.
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Sentence representation learning is a basic task in natural language processing and benefits many downstream tasks. This work proposes a contrastive learning based framework to solve the collapse issue of BERT and transfer BERT sentence representations to target data distribution. Our approach not only provides a new perspective about BERT's representation space, but is also useful in practical applications, especially for data scarcity scenarios. When applying our approach, the user should collect a few unlabeled texts from target data distribution and use our framework to finetune BERT encoder in a self-supervised manner. Since our approach is self-supervised, no bias will be introduced from human annotations. Moreover, our data augmentation strategies also have little probability to introduce extra biases since they are all based on random sampling. However, it is still possible to introduce data biases from the unlabeled texts. Therefore, users should pay special attention to ensure that the training data is ethical, unbiased, and closely related to downstream tasks.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Sentence embedding learning, which aims at deriving semantically meaningful fixed-sized vectors for sentences, is a natural language processing (NLP) technique of great significance, especially for time-sensitive downstream tasks (Reimers and Gurevych, 2019). Recently, contrastive learning (CL) is proven effective to learn representation (Wu et al., 2018;Tian et al., 2020; and * Work done during the internship at Microsoft. † Corresponding author. 1 Our implementation is available at https://github. com/qiyuw/PeerCL.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Order N-gram Bag-of-words substantially improve its performance (Yan et al., 2021;Gao et al., 2021) when coupling with pretrained language models (PLMs). The main idea of contrastive learning for sentence embedding is pulling semantic neighbors together and pushing semantic non-neighbors apart (Hadsell et al., 2006), which naturally requires effective contrastive pairs. As effective contrastive pairs are usually scarce and require much human effort to collect, how to learn sentence embeddings in an fully unsupervised manner has become a challenging yet attractive research area Giorgi et al., 2021).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
The key to unsupervised contrastive learning for sentence embedding is to augment a given anchor sentence with an effective positive instance to construct the pairs. Hence, many efforts have been made to design augmentation methods by adding noises or using heuristics, which mainly fall into two categories in terms of augmentation formatdiscrete and continuous. The former operates directly on words or n-grams in the sentence, e.g., synonym substitution (Su et al., 2021b), shuffling and word deletion (Yan et al., 2021). The latter operates on latent embeddings derived by neural encoder(s), e.g., SimCSE with twice dropouts (Gao et al., 2021). However, these existing approaches usually depend on a mono-augmenting format (i.e., either discrete or continuous) with a limited number of augmenting strategies, which suffer from learning shortcuts (Ilyas et al., 2019;Du et al., 2021)  towards the augmenting biases and thus corrupt the quality of learned embeddings. For example, learning shortcuts caused by discrete augmenting biases are shown in Table 1, and SimCSE based solely on dropout in continuous format is biased towards the sentence length (Wu et al., 2021b).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
To prevent the learning shortcuts caused by the potential biases from the mono-augmenting strategy, one straightforward solution coming into our mind is that we can consider more diverse augmentations for a given sentence in both continuous and discrete formats. Besides learning from diverse instances for inherent anti-bias ability, it can also bring a great opportunity for more effective learning. In particular, controlling the qualities of the noisy and heuristic augmentations for different sentences is almost impossible 2 . As illustrated in Figure 1(a), the resulting contrastive instances may become ineffective and even poisonous for conventional CL. Nonetheless, diverse instances from various augmentation strategies can notably improve the possibility of at-least-one effective positives in the contrastive instances, so how to leverage the rich relations among the diverse augmentations for more effective CL is worth exploiting.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
To this end, we propose a brand-new Peer-Contrastive Learning (PCL) with diverse augmentations, and an illustration of its overall framework is shown in Figure 1(b). Firstly, PCL not only per-2 It may cause poisonous positive. For example, given a sentence "A dog is chasing a cat.", a possible shuffled sentence is "A cat is chasing a dog.", which is semantically different. forms the vanilla positive-negative contrasts but also takes the opportunity to learn the rich structured relations among the diverse positives (i.e., peer-positive) to highlight the more possibly effective ones. Then, to learn the structured relations in a fully unsupervised manner, we propose a cooperative learning framework consisting of two peer embedding networks (i.e., peer-network). The two networks learn from each other to prevent error reinforcement in sole-network and achieve a common agreement from different views (as shown in Figure 1(c)). Consequently, the sentence embedding network is equipped with (i) anti-bias abilities by CL on the diverse augmentations and (ii) improved effectiveness by the unsupervised PCL, leading to a high quality of sentence embeddings.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We conduct experiments on 7 standard semantic textual similarity (STS) tasks (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016Cer et al., 2017;Marelli et al., 2014) to evaluate PCL. Results demonstrate that PCL significantly outperforms state of the art on 7 STS tasks. Typically, PCL achieves a 2.85% improvement over the previous best approach in the averaged Spearman's correlation of 7 STS tasks in the BERT base setting. PCL also outperforms previous approaches across different PLMs initialization and model sizes. Moreover, ablation study and analysis show that the two proposed components, i.e., peer-positive contrasts and peer-network cooperation, are both capable of improving unsupervised sentence embedding learning.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
This section begins with a formal definition of unsupervised sentence embeddings, followed by detailed formulations of our PCL with diverse augmentations ( §2.1 and §2.2). Lastly, we will elaborate on our training and inference procedure for unsupervised sentence embeddings ( §2.3).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Unsupervised Sentence Embedding. Given a sentence x i ∼ X, the target of this task is to learn a neural network f θ (parameterized by θ) without any human-labeled data. Then, the network can be applied to x and derive a dense real-valued vector representation, i.e., h i = f θ (x i ) ∈ R d . Consequently, h i can be used to represent the semantics of x i and fulfill downstream sentence-related tasks, e.g., semantic textual similarity. Thereby, this task depends on the designs of unsupervised (a.k.a self-supervised) objectives based on X to learn f θ effectively.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
The recent common practice of representation learning in an unsupervised manner is contrastive learning (Zhang et al., 2020;Yan et al., 2021;Gao et al., 2021), which aims to learn effective representations to pull similar instances together and push apart the other instances. Thereby, compared to supervised contrastive learning that has already offered contrastive pairs, how to augment the given anchor (e.g., an image and sentence) with effective positive and negative instances to construct the contrastive pairs is critical in the unsupervised scenario. More recently, a simple contrastive learning framework (SimCLR) is proposed in visual representation learning , which constructs positive instances by different views (e.g., chop) of an image then learn to minimize the following InfoNCE loss.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
δ(·) denotes using a different view of the image x i as the positive instance during visual contrastive learning, x j denotes negative instances against x i to construct contrastive pairs with δ(·), and s[·, ·] denotes a similarity metric between two dense vectors. And L (c) θ denotes this loss function is optimized w.r.t the subscript θ. It is also noteworthy that x j ∼ X is usually implemented by using other in-batch instances during mini-batch SGD (a.k.a in-batch negatives).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
However, when switching to unsupervised sentence embedding, augmenting an input sentence by a fully random chop or permutation may become very intractable. This is because these operations are most likely to destroy the original sentence in both semantics and syntax and cause trivial positive augmentations. Hence, many research efforts have been made to design δ for effective positive augmentations in the NLP community. These efforts mainly fall into two categories in terms of augmentation format -'discrete' and 'continuous'. Discrete augmentation format denotes operating directly on the inputted sentence, where δ is defined as word deletion, shuffling (Yan et al., 2021), back translation , etc. In contrast, continuous ones operate on hidden states or network parameters, where δ is defined as network twice dropout (Gao et al., 2021), etc.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Nonetheless, compared to visual contrastive learning that barely introduces new data distribution for the positive instances, such heuristic augmentation methods in the text field cause shortcut learning (Ilyas et al., 2019;Du et al., 2021) -each method exposes the learning procedure to potential biases towards the augmented instances and thus corrupts the quality of learned embeddings. Therefore, existing unsupervised contrastive sentence embedding works usually depend on the limitedor even mono-augmenting methods for their positive instances and inevitably suffer from the biases in the positive instances.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
To prevent learning shortcuts caused by the potential biases from mono-augmenting strategy and exploit rich relations among diverse augmentations for more effective positives, we propose a brand-new contrastive learning method, called peercontrastive learning (PCL). Besides the vanilla contrastive objective, it contains a novel 'contrastcooperation' learning mechanism, which we will detail in the following.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
First, we adopt a multi-augmenting strategy for extensively diverse augmentations. Given a sentence x i ∼ X, it considers extensive augmentation methods from both continuous and discrete perspectives.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
This can be formally written as
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
where ∆ denotes a set of multiple augmentation methods from both the continuous ∆ (c) set and discrete ∆ (d) set, and |∆| = K. Then, we can obtain diverse augmented positives by applying ∆ to a sentence x i ∼ X, i.e.,
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
The contrastive sentence embedding based on diverse positives can mitigate the biases towards mono-augmenting strategy, but it comes with a double-edged sword. That is, thex i k ∼X i varies a lot with many factors (e.g., input sentence x i and augmentation method δ k ), making it hard to control the quality of each x i k . To one extreme, one augmentation can become ineffective and even poisonous if its semantics is largely changed and thus corrupt the model.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
To effectively learn from the uneven qualities of the augmented positives, we propose a brand-new peer-contrastive learning framework that not only performs the vanilla positive-negative contrast but a positive-positive contrast. This is because our diverse augmentations provide a great opportunity to model rich structured relations among the positives and improve the probability of 'at-least-one' effective positive inX i . And the positive-positive contrast can mimic 'peer-competition' to highlight more likely effective positives but weaken the others' effects by suppressing them.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Formally, we first derive a group-wise probability distribution by contrasting the anchor x i with both diverse positivesX i and in-batch negatives
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
where '+' here denotes a union of two sets. Identical to the vanilla contrastive sentence embedding (Gao et al., 2021), we also leverage a softmax normalization to fulfill peer-contrast among augmented positives. Please note we introduce θ 1 and θ 2 for clear deliveries in the remaining sections, and the two parameters here can be either tied (i.e., θ 1 = θ 2 = θ) or not. Although using the augmented positives to 'compete' each peer sounds attractive for contrastive learning, one critical question remains about how to learn merely from effective positives and guide the positive-peer contrasts p P-Cf θ 1 ,θ 2 (x i ) in a fully unsupervised way.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We propose a cooperative learning framework to learn contrasts among the augmented positives. It contains two peer embedding networks, and the two networks learn from each other to prevent error reinforcement in sole-network and achieve a common agreement from different views. Specifically, we first build a peer network θ ′ which acts like a momentum encoder  to cooperatively learn with θ. Here, θ and θ ′ can be untied or even heterogeneous. Then, we present the loss of the momentum-like cooperative learning, which is a combination of two Kullback-Leibler divergence losses. That is
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We call this 'momentum-like' since θ ′ is not strictly a history of θ for more different views but depends on the second KL term to prevent significant divergence from θ. Meanwhile, the first KL term is to reach an agreement between the main network θ and its peer network θ ′ . This 'learning-from-agreement' paradigm, including mutual-distillation (Zhang et al., 2018) and denosing-by-agreement (Wei et al., 2020), is proven effective in improving performance and learning from label noises by prior supervised works. In contrast, we hold a distinct motivation that the implicit relations in a group of diverse positives and in-batch negatives are expected to unsupervisedly match each peer embedding network with another view (e.g., structures and parameters).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Remark. The meaning of 'peer' has two folds: (i) It denotes that we want to learn the rich structured relations among the diverse augmented positives to highlight the effective ones; (ii) It involves a cooperative learning framework based on peer networks for modeling positive-positive contrasts to achieve PCL with diverse augmentations.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Training Objective. We write the loss as a combination of (i) our proposed contrast-cooperation learning for both θ and θ ′ simultaneously to highlight more effective positives and (ii) vanilla contrastive learning that is applied to θ and θ ′ separately and based on our diverse augmentations ∆ for their strong anti-bias initializations. That is,
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
where β is a hyperparameter to control if the training inclines to vanilla CL for the anti-bias purpose. Hence, β could be annealing to provide strong unbiased initializations for different views at the beginning and then focus on contrast-cooperation with peers for more effective learning.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Inference. Due to the symmetrical learning paradigm, we empirically found θ and θ ′ achieve comparable performance in our pilot experiments. Nonetheless, we only use the main embedding network θ rather than ensembles them to encode each sentence for fair comparisons with its competitors.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We train and evaluate our model in a fully unsupervised manner. Following Gao et al. (2021), we train our model on 10 6 randomly sampled sentences from Wikipedia English. We evaluate our model on the semantic textual similarity (STS) tasks without using any STS training data. We report results on 7 datasets, namely the STS benchmark (STSb) (Cer et al., 2017) the SICK-Relatedness (SICK-R) dataset (Marelli et al., 2014) and the STS tasks 2012 -2016 (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016) (STS12-STS16). These datasets provide a gold standard semantic similarity between 0 and 5 for each sentence pair, which include texts from various domains, and we obtain them from the SentEval toolkit (Conneau and Kiela, 2018).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Augmentation Strategies In this paper we utilize five unsupervised augmentation strategies that are commonly adopted in previous works (Wei and Zou, 2019;Yan et al., 2021;Gao et al., 2021).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Augmentations from discrete perspectives ∆ (d) includes: 1) Shuffled Sentence (SS) shuffles the position of words in the sentence; 2) Inverted Sentence (IS) inverts the original sentence as the augmented sample; 3) Words Repetition (WR) duplicates part of words and randomly insert them into the original sentences; 4) Words Deletion (WD) deletes part of words in the sentences. The augmentations from the continuous perspective ∆ (c) include Dropout (DP). It generates augmentation instances in the embedding level by passing the original sentence again into the encoder with different dropout masks.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
More implementation details about augmentation are presented in § A due to the page limit.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Network Implementation We initialize the networks θ and θ ′ with the PLMs checkpoint downloaded from Huggingface's Transformers 3 of BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019). The encoder consists of 12 and 24 Transformer layers for the base and large model, respectively. The hidden size is set to 768 and 1024, and the number of attention heads is set to 12 and 16 for base and large models, respectively. We choose the representation of the [CLS] token as the embedding of the input sentence. The hyperparameter β is set to 1 for training simplicity without tuning, and the number of augmentations K is 9 for base models. Due to the computation resource limitation, particularly for large models, we set K to 4, and the two networks θ and θ ′ are tied, in which the cooperative learning is performed between the two passes through the network.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Training Setups. We follow common practices and carry out preliminary grid search on the development set of STSb to decide the hyper-parameter configuration. The learning rate is set to 3e-5 for base models and 1e-5 for large models, respectively. Except for learning rate, We use the same training hyper-parameters for all experiments with the batch size of 64 and the maximum length of 32. The temperature parameter τ is set to 0.05, and the dropout probability is set to 0.1. We train our model for 1 epoch and evaluate the model on the STSb development set every 125 steps, and keep the best checkpoint by following Gao et al. (2021).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Evaluation Setups. We evaluate PCL on 7 STS tasks, including STS12-STS16, STSb, and SICK-R as introduced in § 3.  We highlight the highest numbers among models with the same pre-trained encoder. † : Our models. * : We also run our models five times with different random seeds and report the average of these five results on each column as the final number.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
et al. (2021) has studied the evaluation settings for sentence embedding. We adopt their suggestions and follow the standard settings in Sentence-BERT (Reimers and Gurevych, 2019). Specifically, we do not train an additional regressor for STSb and SICK-R, use Spearman's correlation as the metric, concatenate all tasks and report the overall Spearman's correlation. To fairly compare with previous approaches, we use the evaluation scripts released by Gao et al. (2021) 4 . Moreover, we train our model for five times with different random seeds and report the average of these five results.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We also evaluate PCL on 7 transfer tasks (Conneau and Kiela, 2018). PCL achieves competitive performance and the detailed results are presented in Appendix C.2. As mentioned in previous works (Reimers and Gurevych, 2019;Gao et al., 2021), the main goal of sentence embeddings is to cluster semantically similar sentences. Hence we only take STS as the main results in this paper. 4 https://github.com/princeton-nlp/SimCSE
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We compare our model with previous state-of-theart unsupervised sentence embedding approaches, including basic embedding approaches (e.g. average of GloVe (Pennington et al., 2014),  (Liu et al., 2021b) and SimCSE (Gao et al., 2021)). SGPT (Muennighoff, 2022) and Sentence-T5 (Ni et al., 2021) are proposed with new paradigm and far larger models, which underperform with comparable model size. Trans-Encoder  proposes a cooperative method with in-domain pairwise data for mutual benefits of bi-and crossencoder, making the results incomparable. Please refer to § B for more details.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Experimental results on STS tasks are shown in Table 2. We can find that our PCL significantly outperforms the previous best result on all seven tasks as well as the average STS score with a large margin compared to the baseline methods based on BERT base or RoBERTa base PLMs. Specifically, PCL improves the previous best result on average STS score from 76.25 to 78.42 for BERT base and 76.57 to 77.91 for RoBERTa base , respectively. As SimCSE (Gao et al., 2021) did not report their performance on BERT large , we compare all large models together, and the results are shown in the last rows of the table. We can observe that PCL outperforms the best result on all tasks apart from the STS16. Despite this, our PCL still obtain an improvement from 78.90 to 80.14 on the average STS score. Our PCL achieves more significant improvement over the base models than the large models, And even so, PCL still outperforms SimCSE on almost all tasks in large models and all tasks in base models, which shows that PCL is effective across different model sizes and different types of PLMs.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Diversity and the number of augmentations are two crucial factors of PCL. In this section, we test the performance of PCL with varying K and diversity. For PCL and all variants of PCL in this section, we train them for 5 times with different random seeds, and take the average as the final results.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
The number of augmentations. To mitigate the model bias towards the mono-augmenting strategy, we propose to augment the input sentence with a group of positive instances. The number of augmentations K is a crucial hyper-parameter in this framework. To check if the performance of PCL is sensitive to K, we conduct experiments on PCL-BERT base with varying K on 7 STS tasks and report the average STS score. We keep the diversity of augmentations ∆ as much as possible when K > 1. As shown in Figure 2, the performance of PCL maintains an upward trend with increasing K. This indicates that multiple augmentation strategy improves unsupervised sentence embeddings compared with learning with mono-augmenting strategy. This supports our motivation that contrastive learning with mono-augmenting strategy causes learning shortcuts. More detailed results on all 7 STS tasks are presented in § C.3. The diversity of augmentations. Another critical factor is the diversity of augmentations. We fix K = 9 and reduce the diversity of augmentations ∆ to check if PCL is sensitive to the diversity. We conduct experiments on PCL-BERT base with K = 9 but only use one type of augmentation strategy from discrete and continuous perspective, respectively. In other words, we keep at least one DP augmentation for all variants. We compare PCL with five mono-augmentation variants that are denoted as PCL DP , PCL SS , PCL IS , PCL WR and PCL WD , respectively. The details of augmentation strategies are introduced in § A. Average Spearman's correlation scores of 7 STS tasks are shown in the Figure 3. Experimental results show that PCL significantly outperforms its monoaugmenting variants, even keeping the K constant, indicating a better generalization. This supports our motivation that PCL with diverse augmentations can mitigate the shortcut learning biased towards mono-augmenting strategy. Particularly, SimCSE utilizes dual-dropout to construct the contrastive pairs, hence the PCL DP variant (9 positive instances generated by the dropout) can be regarded as Sim-CSE w/ 9 augmented positive samples. Our proposed contrasts and cooperation among peers improve SimCSE from 76.25 to 77.14, but the score is still lower than PCL with a large margin. This is another piece of evidence that shows the advantage of the diversity of augmentations. The detailed results on all 7 STS tasks are presented in § C.3.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We first check the impact of the proposed two components of PCL, peer-network cooperation and peer-positive contrast, i.e., the two terms in Equation 6 respectively. We designed two variants of PCL on PCL-BERT base by removing the cooperation loss and contrast loss, which are denoted as PCL noP and PCL noC respectively. To ensure the networks have the essential ability to learn sentence embeddings, we keep the contrastive loss with a   single DP augmentation for PCL noC , which is equal to the setting in Figure 2 when K = 1. We train each variant for five times with different random seeds and take the average of these seeds as the final results. As in Table 3, the average scores of PCL drop by 1.02 and 1.08 when removing the cooperation loss and contrast loss, respectively. This indicates that our proposed peer cooperation and peer contrast are both beneficial to unsupervised learning of sentence embeddings. Among the two components, the peer cooperation loss plays a more important role as it incorporates contrasts among peer positives and peer networks, enabling an inherent anti-bias ability and an effective way to learn from diverse augmentations.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Fixed peer encoder vs. trainable peer encoder. Particularly, We are also curious about the impact of 'learning from agreement' (i.e., the second term in Equation 5) in the cooperative learning objective. Therefore, we further test additional variants of PCL with a fixed peer encoder, denoted as PCL FixedP . Specifically, we download a checkpoint of SimCSE as the peer encoder but fix its parameters while training. Experimental results show that the performance of PCL FixedP drops with a large margin on STS12, STS13, STS14, STS15, STSb, and the average STS. The reason may be that although SimCSE is by far the best practice S T S 1 2 S T S 1 3 S T S 1 4 S T S 1 5 S T S 1 6 S T S b S I C K -R A v g . of sentence embeddings, it is still biased towards a mono-augmenting strategy. Hence, cooperative learning with a biased peer network can be harmful to the network with diverse augmentations. This also indicates that it is necessary to simultaneously update the two peer networks and learn the agreement between them in PCL. Furthermore, there can be a considerable discrepancy between the embedding spaces produced by two methods, which hinders the cooperative training of two networks.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Unsupervised Sentence Embedding. Common practice of unsupervised sentence embedding is to take the average of pre-trained word embeddings (Mikolov et al., 2013;Pennington et al., 2014) PLMs, like BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019), Wu et al. (2021a) takes the average of word embeddings as context embedding to enhance the language pre-training. Other works also take the [CLS] embedding from the last layer of PLMs with post-processing (Li et al., 2020;Su et al., 2021a). Some works (Kiros et al., 2015;Logeswaran and Lee, 2018;Hill et al., 2016) directly train a deep model for sentence embeddings using co-occurrence information. Recent approaches couple PLMs with CL (Zhang et al., 2020;Yan et al., 2021;Kim et al., 2021;Carlsson et al., 2021;Giorgi et al., 2021;Gao et al., 2021;Xie et al., 2022) with a particular single strategy to construct contrastive pairs. It is straightforward to extend mono-augmentation into multi-augmentation to learn expressive representations. For example, CLEAR  uses various token/span manipulations for noise-invariant representations while Mirror-BERT (Liu et al., 2021b) employs several fast augmentation strategies. However, these methods usually take the augmented positives equally, regardless of the uncontrollable qualities.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Thereby, we take a step further to consider the contrasts among the augmented positives to figure out which augmentation is relatively reasonable. This is achieved by our novel cooperative learning method with peer networks. More related to our work, ESimCSE (Wu et al., 2021b) found learning on dual-dropout causes sentence length bias so it employs another augmentation strategy, i.e., word repetition, to prevent the length bias. However, word repetition introduces learning shortcut by itself, not to mention it makes the sentence unnatural and even semantics-wrong. To circumvent this dilemma, we propose exhaustive augmentations to ensure "at-least-one" true positive and reduce learning shortcuts by complementary augmenting strategies. By doing so, PCL achieve a better performance on STS tasks. Please refer to § D.1 for more discussion details.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Contrastive Learning. The main idea of CL is to pull semantic close neighbors close and push non-neighbors apart (Hadsell et al., 2006;Zbontar et al., 2021). It is shown to be a successful way to learn representation. Approaches in computer vision (CV) (Chen et al., 2017;Wu et al., 2018;Tian et al., 2020;Zbontar et al., 2021) try to make an image to be invariant to transformations on itself, while remaining discriminative to other images. More references in CV are discussed in the recent survey (Jaiswal et al., 2021). CL is also coupled with PLMs to learn sentence embeddings. But, recent works (Xiao et al., 2021) argue that learning invariance to particular transformations may be harmful to the robustness of the model. This also supports our idea of leveraging diverse augmentations to improve unsupervised sentence embeddings from another angle.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Learning from Agreement. Another line of work close to ours is learning from agreement, e.g., Decoupling (Malach and Shalev-Shwartz, 2017), Co-teaching (Han et al., 2018;Yu et al., 2019), and mutual CL . This paradigm has been proven effective in improving model performance and learning with label noises by prior fully-supervised works. As text data is discrete and compositional, qualities of multiple augmentations can be uneven, which may corrupt the generalization of sentence embeddings. Besides widely used regularization like dropout (Srivastava et al., 2014) and weight decay (Krogh and Hertz, 1991), we consider learning from agreement paradigm to offer a robust way to learn from our diverse positives.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
In this paper, we propose a brand-new contrastive learning framework, dubbed as peer-contrastive learning (PCL), to capture rich relations among diverse positive peers and highlight effective positives, which are learned by cooperative learning by peer networks. Besides inherent anti-bias ability by diverse augmentations, it can learn from unsupervised corpus more effectively than vanilla contrastive in the text field. Experiments show that the number and diversity of augmentations are crucial to PCL. Ablation study also shows that the two components of PCL, i.e., peer-positive contrast and peer-network cooperation, are both beneficial to unsupervised CL for sentence embeddings.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Limitation. We also recognize that our PCL framework has its certain limitations: (i) Due to peer positives and encoders, our framework needs higher (~3×, i.e., 2.5 GPU-hours for base models) computation overheads compared to vanilla CL. Nonetheless, the acceptable extra overhead and same inference makes our framework still practical and scalable. (ii) As a work addressing the general shortcut learning problem in a fundamental task, the proposed PCL is only evaluated on resources in English. It can be further extend to more applications such as in low-resource or other languages. (iii) The performance of our framework relies on the choice of augmentation methods, and it is hard to strictly claim which combination of the methods is optimal except experimental verification. Although we have analysed the effect of varying combinations of augmentations with extensive experiments, we can only select several widely-adopted augmentations to evaluate the general effectiveness of our framework.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
This paper investigates unsupervised contrastive learning for sentence embedding. There will not be any ethical problems or negative social consequences from the research. The data in this paper are all publicly available and are widely adopted by researchers. The proposed method does not introduce ethical bias in the data.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We propose diverse augmentation strategies for each sentence. In this paper we utilize five unsupervised augmentation strategies that are commonly adopted in previous works (Wei and Zou, 2019;Yan et al., 2021;Gao et al., 2021). Augmentations from discrete perspectives ∆ (d) includes: 1) Shuffled Sentence (SS) shuffles the position of words in the sentence. SS corrupts the order of the original sentence but preserves the semantic information of words; 2) Inverted Sentence (IS) inverts the original sentence as the augmented sample. Apart from the reading order, IS preserves all language properties even including n-gram statistics (Dufter and Schütze, 2020); 3) Words Repetition (WR) duplicates part of words and randomly insert them into the original sentences; 4) Words Deletion (WD) deletes part of words in the sentences. WD and WR change the length and words of the original sentence but roughly preserve the reading order. The deletion and repetition ratio are empirically set to 0.2. The augmentations from the continuous perspective ∆ (c) include Dropout (DP). It generates augmentation instances in the embedding level by passing the original sentence again into the encoder with different dropout masks. The above five strategies can be repeatedly applied in practice. As there is randomness in the processes of augmentation and encoding, repeatedly generated instances with the same strategy can be regarded as diverse positives. But the diversity may accordingly decline. Note that the primary goal of this paper is to verify the effectiveness of our PCL framework, hence all of the chosen augmentations are common and simple. We speculate that our PCL can be further improved with more fine-tuned augmentation strategies.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We compare PCL with previous state-of-the-art unsupervised sentence embedding approaches. Basic approaches include taking the average of GloVe, BERT, or RoBERTa embeddings. Besides, BERTwhitening and BERT-flow post-process the embeddings distribution of BERT. We also compare PCL with recent approaches using contrastive learning, including IS-BERT, ConSERT, SG-OPT, Contrastive Tension, DeCLUTR, and SimCSE. The following are the details of these baselines,
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• GloVe (Pennington et al., 2014) maps words into a meaningful space where the distance be-tween words is related to semantic similarity. The results of the average of GloVe embeddings are from Reimers and Gurevych (2019).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• Su et al. (2021a) takes the average of the first and last layers of BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) embeddings. We report the results from Gao et al. (2021).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• BERT-whitening (Su et al., 2021a) and BERTflow (Li et al., 2020) post-process the embeddings distribution of BERT. We report the results from Gao et al. (2021) for a fair comparison.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• IS-BERT (Zhang et al., 2020) encourages the representation of a specific sentence to encode all aspects of its local context information, using local contexts derived from other input sentences as negative examples for contrastive learning. We report the results from the original paper.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• ConSERT (Yan et al., 2021) contrasts a pair of sentences augmented by different augmentation methods. We report the results in the original paper.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• SG-OPT (Kim et al., 2021) is a contrastive learning method using self-guidance. The results are from the original paper.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• Contrastive Tension (CT) (Carlsson et al., 2021) propose a training objective that aligns the embeddings of the same sentence encoded by two different encoders. We report the results from Gao et al. (2021).
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• DeCLUTR (Giorgi et al., 2021) is a contrastive approach that takes different spans from the same document as contrastive pairs. The results are from Gao et al. (2021) • Mirror-BERT (Liu et al., 2021b) employs several fast augmentation strategies for effective representations. The results are from the original paper.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• SimCSE (Gao et al., 2021) contrasts a pair of embeddings of one sentence encoded with different dropout masks. The results are from the original paper. We had re-run SimCSE with same setups and it performs worse than the numbers reported in the original paper (e.g., 75.36 averaged over 5 seeds on BERT base ).  We reported higher numbers for a fair comparison.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Due to the surge of this topic, many concurrent works emerge with two trends: new model structure and more in-domain data. SGPT (Muennighoff, 2022) and Sentence-T5 (Ni et al., 2021) are proposed with new paradigm and far larger models, which underperform with comparable model size.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Trans-Encoder  proposes a cooperative method with in-domain pairwise data for mutual benefits of bi-and cross-encoder, making the results incomparable.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We can also find some variants of SimCSE in our controlled experiments. For example, PCL noP in Table 3 is regarded as SimCSE w/ multiaugmentations. PCL K=1 in Figure 2 can be regarded as SimCSE w/ peer-network cooperation, and PCL DP in Figure 3 is regarded as SimCSE w/ 9 dropout augmented positive samples. As we discussed in the § 3.5 and § 3.6, the comparison between the variants of SimCSE and PCL show the advantages and importance of our proposed peer-contrast and peer-cooperation.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
We also evaluate PCL on 7 transfer tasks (Conneau and Kiela, 2018). As the Table 4 shows, PCL achieves competitive performance compared with baselines. Note that as mentioned in previous works (Reimers and Gurevych, 2019;Gao et al., 2021), the main goal of sentence embeddings is to cluster semantically similar sentences. Hence we only take STS as the main results in this paper.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
In this section, we present detailed results of experiments of diverse augmentations on all 7 STS tasks. We test the performance of PCL with varying K and diversity. Experimental results are shown in Table 5. As the results and our analysis in § 3.5 show, the performance of PCL maintains an upward trend with increasing K. Besides, it is also shown that PCL significantly outperforms its monoaugmenting variants, even keeping the K of them constant, which indicates a better generalization. As a result, PCL with more diverse augmentations performs better. We also speculate that our PCL can be further improved with larger K and more fine-tuned augmentation strategies.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
D.1 Distinction between PCL and other contemporary methods.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Unsupervised sentence embedding w/ multiple positive augmentations. It's straightforward to extend mono-augmentation into multiaugmentation to learn expressive representations. For example, CLEAR  uses various token/span manipulations for noise-invariant representations while Mirror-BERT (Liu et al., Table 5: Effect of the number and diversity of augmentations. We report the Spearman's correlation ρ (%) on 7 STS datasets. All variants are run for five times with different random seeds and the average of these five results on each column is reported as the final number.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
2021b) employs several fast augmentation strategies for effective representations. However, these methods usually take the augmented positives equally, regardless of the uncontrolable qualities. For example, given "Two men are wrestling on the floor", we get "Two men are squirming on the floor" and "Two persons are wrestling on the floor" by word replacement, but only the 2nd is reasonable. Thereby, we take a step further to consider the contrasts among the augmented positives to figure out which augmentation is relatively reasonable. This is achieved by our novel cooperative learning method with peer networks.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Unsupervised sentence embedding for anti-bias. More related to our work, ESimCSE (Wu et al., 2021b) found learning on dual-dropout causes sentence length bias so it employs another augmentation strategy, i.e., word repetition, to prevent the length bias. However, word repetition introduces learning shortcut (i.e., order and BoW as in Table  1) by itself, not to mention it makes the sentence unnatural and even semantics-wrong (e.g., repeating "no"). To circumvent this dilemma, we propose exhaustive augmentations to ensure "at-least-one" true positive and reduce learning shortcuts by complementary augmenting strategies (See Figure 1 and 3: if we employ every strategy, the shortcuts can be blocked). Nonetheless, PCL still has a better performance (78.42 vs. 78.27) compared with ESimCSE on STS tasks.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Cooperative learning has more parameters, is it the reason leading better performance? One of the advantages of our cooperative learning is to ef-fectively learn from the positives with uncontrolled qualities, or it can be also interpreted as 'noisy labels'. In the noisy circumstance, more parameters not necessarily lead to better performance. And it can be anticipated that it possibly leads to worse performance because the over fitting in the noisy positives.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Efficiency Due to peer positives and encoders, our framework needs higher (~3×, i.e., 2.5 GPU-hours for base models) computation overheads compared to vanilla CL (Gao et al., 2021). Nonetheless, the same inference makes our framework still practical and scalable. Since PCL contains more loss items, we do not see any significant difference in convergence time.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Impact of augmentations The performance of our framework relies on the augmentation methods, and it is hard to claim which combination of the methods is optimal except experimental verification. In this work, we only intuitively select several methods without extensive trials. We have illustrated the performance of mono-augmentation in Figure 3.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Sentence embeddings aim to capture rich semantic information to be applied in many downstream tasks . Recently, researchers have started to use contrastive learning to learn better unsupervised sentence embeddings Yan et al., 2021;Wu et al., 2021a;Zhou et al., 2022;Wu et al., 2021b;Chuang et al., 2022). Contrastive learning methods assume that effective sentence embeddings should bring similar sentences closer and dissimilar sentences farther. These methods use various data augmentation methods to randomly generate different * The first two authors contribute equally. † Corresponding author.   . The masked language model (MLM) objective brings a consistent drop to the SimCSE model in semantic textual similarity tasks. "w/" means "with", λ is the balance hyperparameter for MLM loss.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
views for each sentence and constrain one sentence semantically to be more similar to its augmented counterpart than any other sentence. SimCSE  is the representative work of contrastive sentence embedding, which uses dropout acts as minimal data augmentation. SimCSE encodes the same sentence twice into embeddings to obtain "positive pairs" and takes other sentence embeddings in the same mini-batch as "negatives". There have been many improvements to SimCSE, including enhancing positive and negative sample building methods (Wu et al., 2021a), alleviating the influence of improper mini-batch negatives (Zhou et al., 2022), and learning sentence representations that are aware of direct surface-level augmentations (Chuang et al., 2022). Although SimCSE and its variants have achieved good results and can learn sentence embeddings that can distinguish different sentences, this is not enough to indicate sentence embeddings already contain the semantics of sentences well. If a sentence embedding is sufficiently equivalent to the semantics of the sentence, it should also be able to reconstruct the original sentence to a large extent (Lu et al., 2021). However, as shown in Table 1 from , experiments show that "the masked language model objective brings a consistent drop in semantic textual similarity tasks" in SimCSE. This is due to the gradients of the MLM Figure 1: Comparison of InfoCSE and SimCSE structures. SimCSE learns sentence representations through contrastive learning on the [CLS] output embeddings of the BERT model. In addition to contrastive learning, InfoCSE designs an auxiliary network for sentence reconstruction with the [CLS] embeddings, enabling to learn better sentence representations. objective optimization will easily over-update the parameters of the encoder network, thus causing disturbance to the contrastive learning task. Therefore, it is not an easy job to incorporate sentence reconstruction task in contrastive sentence embedding learning.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
To improve contrastive sentence embedding learning with the sentence reconstruction task, we propose an information-aggregated contrastive learning framework, termed InfoCSE. The previous work  shares the encoder when jointly optimizing and contrastive learning objective and MLM objective. Unlike , we design an auxiliary network to optimize the MLM objective, as shown in Figure 1- (b). The auxiliary network is an 8-layer transformer network consisting of a frozen lower six layers and an additional two layers. The auxiliary network takes two inputs. One is the sentence embedding of the original text, and the other is the masked text. The sentence embedding is a vector of the [CLS] position encoded by 12 layers of BERT, abbreviated as 12-cls. The lower six Transformer layers encode the masked text to output hidden states at each position. We collectively refer to non-[CLS] positions' representations as 6-hiddens. Then, we feed the concatenation of the 12-cls and 6-hiddens into the additional two layers to perform token prediction for the masked positions. Such a design brings two benefits.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
• Since the auxiliary network only contains 8
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Transformer layers and the frozen lower six lay-ers cannot be optimized, the sentence reconstruction ability is limited. Also, the non- [CLS] embeddings are the outputs of the 6th Transformer layer, with insufficient semantic information learned. So the MLM task is forced to rely more on the 12-cls embedding, encouraging the 12-cls embedding to encode richer semantic information. • The gradient update of the auxiliary network is only back-propagated to the BERT network through the 12-cls embedding. Compared to perfTorming the MLM task directly on the BERT, the effect of gradient updates using the 12-cls embedding will be much smaller and will not cause large perturbations to the contrastive learning task.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Therefore, under the InfoCSE framework, the 12cls embedding learned can be distinguished from other sentence embeddings through contrastive learning and reconstructing sentences through auxiliary MLM training, while avoiding the disadvantage that the gradient of MLM objective will overupdate the parameters of the encoder network.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Experiments on the semantic text similarity (STS) tasks show that InfoCSE outperforms Sim-CSE by an average Spearman correlation of 2.60%, 1.77% on the base of BERT-base, BERT-large, respectively. InfoCSE also significantly outperforms SimCSE on the open-domain information retrieval benchmark BEIR. We also conduct a set of ablation studies to illustrate the soundness of InfoCSE design.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Definitions Let's define some symbols first. Suppose we have a set of sentences x ∈ X, and a 12layer Transformer blocks BERT model Enc. For a sentence x of length l, we append a special [CLS] token to it, and then feed it into BERT for encoding. The output of each layer is a vector list of length l + 1, also called hidden states. We use H to denote the last layer's hidden states and h to refer to the vector at [CLS] position. In addition, we use M to represent the hidden state of the 6th layer, M >0 to represent the other hidden states of the 6th layer except the [CLS] position, SimCSE As shown in subfigure (a) of Figure  1, we plot the structure diagram of SimCSE-BERT base . The model's output is the [CLS] position's vector, which is used as the semantic representation of the sentence. SimCSE uses the same sentence to construct semantically related positive pairs x, x + , i.e. x + = x. Specifically, SimCSE uses dropout as the minimum data augmentation, feeding the same input x twice to the encoder with different dropout masks z and z + , and outputs the hidden states of the last layer:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
A pooler layer P ooler is further applied to the hidden states as follows:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Then, for a mini-batch B, the contrastive learning objective w.r.t x is formulated as:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
, where τ is a temperature hyperparameter and sim (h, h + ) is the similarity metric, which is typically the cosine similarity function.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
BERT's MLM objective MLM randomly masks out a subset of input tokens and requires the model to predict them. Given a sentence x, following BERT (Devlin et al., 2018), we randomly replace 15% of the tokens with [MASK] and get a masked sentence x. Then x will be feed into the BERT, and the hidden states of the last layer H will be projected through a matrix W to predict the original token of each masked position.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
The process uses the cross entropy loss function CE for optimization:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
, where masked denotes the masked positions.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
3 InfoCSE: Information-aggregated Contrastive Learning
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
In this section, we first introduce the MLM pretraining of the auxiliary network, and then describe how to jointly train the contrastive learning objective and the MLM objective with the auxiliary network.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
As shown in Figure 1 -(b). the auxiliary network is an 8-layer transformer network consisting of a lower six layers and an additional two layers. In the pre-training phase of the auxiliary network, the auxiliary network and the BERT network share the lower six layers. We optimize two MLM objectives simultaneously using the same input. The first is BERT's MLM objective, which we have already covered. The second is the MLM objective of the auxiliary network. We concatenate the vector at [CLS] position of BERT's 12th layer (h) and the hidden states of the 6th layer except for the [CLS] position (M >0 ):
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Then ∼ H is fed into the additional two layers, and the output hidden state will be used to calculate the cross-entropy loss:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Therefore, the pre-training loss of the entire auxiliary network is defined as the sum of the two MLM losses:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
The output projection matrix W is shared between the two MLM losses.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
When jointly training the contrastive learning and MLM objectives, the auxiliary network no longer shares the lower six layers with BERT. But they are both initialized with the parameters from the first stage. While we still optimize the MLM objective for the auxiliary network, the lower six layers' parameters are frozen. For a sentence x, it will be copied twice. One copy is used to form a positive pair x, x + for contrastive learning. The other is used to generate the masked input x, which will be used in the MLM task. The loss of contrastive learning is the same as SimCSE, which we have already introduced. The MLM loss of the auxiliary network has changed a little, here the M >0 is the output from the lower six layers of the auxiliary network:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
, where Detach means that the gradient will not back-propagate through M >0 . The loss for joint training is defined as the sum of the contrastive learning loss and the auxiliary network's MLM loss:
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
, where λ is the balanced hyperparameter for the MLM loss of the auxiliary network.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
When pretraining the auxiliary network, following (Gao and Callan, 2021) we use Bookcorpus (Zhu et al., 2015) and Wikipedia 1 as our datasets. When jointly optimizing contrastive objective and the auxiliary MLM objective, for a fair comparison, our experimental setup mainly follows SimCSE. We use 1-million sentences randomly drawn from English Wikipedia for training 2 . All evaluations are directly based on the sentence embeddings output by the BERT network, and the auxiliary network will not be used.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
The semantic textual similarity task measures the capability of sentence embeddings, and we conduct our experiments on seven standard semantic textual similarity (STS) datasets. STS12-STS16 datasets (Agirre et al., 2012(Agirre et al., , 2013(Agirre et al., , 2014(Agirre et al., , 2015(Agirre et al., , 2016 do not have train or development sets, and thus we evaluate the models on the development set of STS-B (Cer et al., 2017) to search for better settings of the hyper-parameters. The SentEval toolkit 3 is used for evaluation, and Spearman correlation coefficient 4 is used to report the model performance.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Open-domain Retrieval Tasks Sentence embeddings are also commonly used on the retrieval tasks, so we evaluate the zero-shot performance of embeddings on the information retrieval benchmark BEIR (Thakur et al., 2021). BEIR contains 9 open-domain information retrieval tasks (fact checking, citation prediction, duplicate question retrieval, parameter retrieval, news retrieval, question answering, tweet retrieval, biomedical IR, entity retrieval) from 18 different datasets. We evaluate InfoCSE on the 14 publicly available datasets.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
We start from the publicly pre-trained checkpoint of BERT-base or BERT-large. The hyperparameters basically follow Condenser (Gao and Callan, 2021) and SimCSE . When pretraining the auxiliary network, following (Gao and Callan, 2021), we train 8 epochs using the Adam optimizer with learning rate = 1e − 4, global batch size = 1024 on 8 Nvidia V100 GPUs 5 . The hyperparameter λ mlm = 1.0 to balance the two MLM losses. When jointly optimizing contrastive objective and the auxiliary MLM objective, following , we train one epoch using the Adam optimizer with learning rate = 3e − 5, batch size = 64 on a single Nvidia 3090 GPU. The hyperparameter λ mlm = 0.005 to balance the MLM loss and contrastive learning loss. We evaluate the model every 125 training steps on the development set of STS-B and keep the best checkpoint for the final evaluation on test sets.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Baselines We compare our model with many strong unsupervised baselines including SimCSE    .♥ : results from (Yan et al., 2021). ♠ : results from (Kim et al., 2021). ♦ : results from (Li et al., 2020). : results are reproduced and reevaluated by . : results from . : results from (Wu et al., 2021a). ¶ : results from (Chuang et al., 2022). †: The original paper does not report the results of BERT-large, so we use the official public code to perform a grid search on important hyperparameters for the best results. Semantic Textual Similarity (STS) Table 2 shows different methods' performances on seven semantic textual similarity (STS) test sets. It can be seen that InfoCSE improves the measurement of semantic textual similarity in different parameter scale settings over previous methods. Specifically, InfoCSE-BERT base outperforms SimCSE-BERT base by +2.60%, InfoCSE-BERT large outperforms SimCSE-BERT large by +1.77%. Compared with other recent improvements to SimCSE, In-foCSE also achieves better results. Task Table 3 shows the zero-shot performance of different methods on the BEIR benchmark. On the average performance of 14 datasets, InfoCSE outperforms other methods substantially, and InfoCSE achieves the best performance on 10 datasets. Although the recent improvements to SimCSE have achieved better results than SimCSE on STS tasks, their performance on open-domain information retrieval is similar to SimCSE. In contrast, InfoCSE significantly improves on both the STS and open-domain information retrieval tasks over SimCSE, which shows that InfoCSE has better generalization ability.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
We perform an extensive series of ablation studies to InfoCSE with BERT base scale on the development set of STS-B.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Pre-training of The Auxiliary Network We compared the effect of directly using the parameters of the lower six layers and the upper two layers of BERT as the auxiliary network without pre-training. As shown in Table 4, the joint training effect using the auxiliary network without pretraining is significantly worse, illustrating the importance of the pre-training process. Pre-training enables the additional two-layer transformer to fit the MLM task well, avoiding large gradient oscillations during the joint learning.  Table 3: Zero-shot evaluation results on the BEIR benchmark. All scores denote nDCG@10. The best score on a given dataset is marked in bold, and the second best is underlined.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
We remove different loss terms to study their importance in joint training. We list the results in Table 5. When we remove the MLM loss, the In-foCSE model degenerates the SimCSE model, and the performance on STS-B drops by 3%. When we remove the contrastive loss, the InfoCSE model degenerates the pre-trained auxiliary network, the performance drops sharply. This result shows that both the contrastive loss and the auxiliary MLM loss are crucial in InfoCSE.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
During joint training, we freeze the parameters of the lower six layers of the auxiliary network through the gradient detach operation. In other words, the lower six layers are only used as feature extractors for the non-[CLS] positions. The additional two transformer layers have a limited number of parameters, and the modeling ability is weak. As shown in Table 6, the joint training effect without gradient separation operation is reduced by 1%. Therefore, the gradient detach operation forces the MLM task to rely more on the sentence embedding of 12-cls embedding, which is beneficial to the learning of sentence embedding.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
InfoCSE 85.49 w/o pre-training 83.73  
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
InfoCSE 85.49 w/o gradient detach 84.41 Table 6: Development set results of STS-B for InfoCSE with or without gradient detach. "w/o" denotes without. (Wettig et al., 2022) shows that 15% may not always be the optimal mask rate. Therefore, we fur-    ther explore the impact of different mask rates for MLM task in joint training. We vary the mask rate from 10% to 45%. As shown in 7, when the mask rate increase from 10% to 25%, the model performance steadily increases. When the mask rate is over 25%, the model performance begins to fluctuate, and the optimal mask rate is 40%.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Section 3.2 uses the λ coefficient to weight the auxiliary MLM loss and add it with contrastive loss. Because the contrastive learning objective is a relatively easier task, the scale of contrastive loss will be thousands of times smaller than MLM loss. As a result, we need a smaller λ to balance these two loss terms. In the Table 8 we show the STS-B result under different λ values. Note that when λ goes to zero, the model becomes a SimCSE model. We find that using λ = 1e-5 can give us the best performance.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
There are two different pooler methods in SimCSE. One is "cls_before_pooler", which is the [CLS] representation of BERT's last layer. The other is "cls", which is "cls_before_pooler" with additional linear and activation. Here we compare the effects of using two poolers in different models. As shown in Table 9, "cls_before_pooler" is always better than "cls", and InfoCSE is significantly better than other models when using either pooler.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
In this section, we further analyze the performance of InfoCSE on transfer tasks and in-domain information retrieval task. We also explore whether different auxiliary objectives can coexist.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Dataset The transfer tasks include: MR (movie review) (Pang and Lee, 2005), CR (product review) (Hu and Liu, 2004), SUBJ (subjectivity status) (Pang and Lee, 2004) , MPQA (opinion-polarity) (Wiebe et al., 2005), SST-2 (binary sentiment analysis) (Socher et al., 2013), TREC (questiontype classification) (Voorhees and Tice, 2000) and MRPC (paraphrase detection) (Dolan and Brockett, 2005). In these transfer tasks, we will use a logistic regression classifier trained on top of the frozen sentence embeddings, following the standard setup 6 . We show the results of transfer tasks in Table 10. Compared with SimCSE, InfoCSE can improve the average scores from 85.81% to 86.63%. In SimCSE, the authors also propose directly using BERT's MLM task to further boost the performance of transfer tasks, but it brings a consistent drop in STS tasks. Compared with SimCSE with MLM, InfoCSE achieves comparable results on transfer tasks without a drop in STS tasks. This shows that the proposed auxiliary MLM network can simultaneously improve contrastive sentence embedding's performance on semantic text similarity tasks and downstream tasks.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Following (Chuang et al., 2022)   .♥ : results from (Yan et al., 2021). : results are reproduced and reevaluated by . ¶ : results from (Chuang et al., 2022).  query s 1 to retrieve its nearest neighbours from the corpus in the sentence embedding space and see whether its corresponding s 2 is ranked in the top-1/5/10 order. We show quantitative results in Table  11. InfoCSE outperforms SimCSE and DiffCSE on recall@1/5/10 results, demonstrating InfoCSE's effectiveness for in-domain retrieval tasks.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Likewise, DiffCSE also introduces an auxiliary objective, replaced token detection (RTD), which brings non-trivial improvements to the ability of sentence embeddings. The difference is that RTD is a discriminative objective, while MLM is a generative objective. Therefore, we further explore whether these two different auxiliary objectives can coexist. Specifically, we simultaneously apply the [CLS] sentence embedding to the auxiliary MLM objective and RTD objective for joint optimization. As shown in  if properly designed.
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
In this paper, we present InfoCSE, an informationaggregated contrastive learning framework for learning unsupervised sentence embeddings. Empirical improvements on different datasets show the effectiveness and transferability of InfoCSE. We also conduct extensive ablation studies to demonstrate the different modeling choices in InfoCSE. We believe that improving sentence embeddings by optimizing auxiliary objectives is a very promising direction. In the future, we will further explore its potential on more tasks, such as dense passage retrieval (Wu et al., 2022).