Is it possible to extract the data used to train a language model and if so, how to do it?
Existing works on training data extraction mainly
focus on membership inference attack or untar-
geted training data extraction. For membership
inference attack, adversaries need to judge whether
a given example is contained in the training data
of the attacked model. Shokri et al. (2017); Song 
and Shmatikov (2019) train several shadow models
that mimic the attacked models’ behaviors to help
train an auditing model that can predict whether
an example is contained in the training dataset.
Hisamoto et al. (2020) perform membership infer-
ence attacks on machine translation systems. They
find it is harder to attack sequence generation mod-
els than classification models. Song and Raghu-
nathan (2020) show that the encoded dense repre-
sentations can leak information under membership
inference attack. Mireshghallah et al. (2022) fo-
cuses on attacking masked language models that
are pre-trained on possibly sensitive data (e.g., clin-
ical notes). They introduce an additional refer-
ence masked language model besides the original
attacked model and compute the ratio of the like-
lihood measured by the attacked model and the
reference model, which is better than solely relying
on the attacked model.
For untargeted training data extraction, adver-
saries first generate various samples using the at-
tacked model and then predict whether they are con-
tained in its training set. Carlini et al. (2021) extract
hundreds of verbatim sequences from the popular
pre-trained language model GPT-2 (Radford et al.,
2019b). And there is privacy information such as
names, phone numbers, and email addresses in the
extracted sequences. Lehman et al. (2021) try to
extract sensitive information from BERT (Devlin
et al., 2019) pre-trained on clinical notes. How-
ever, they are mostly unable to meaningfully ex-
pose Personal Health Information by simply using
templates. Different from the existing works, we
focus on targeted training data extraction that aims
to recover the suffix when given a prefix, which is
more security-critical and easier to evaluate
Extracting Training Data from Large Language Models
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.
[{"middle":[],"last":"Carlini","first":"Nicholas"},{"middle":[],"last":"Tramer","first":"Florian"},{"middle":[],"last":"Wallace","first":"Eric"},{"middle":[],"last":"Jagielski","first":"Matthew"},{"middle":[],"last":"Herbert-Voss","first":"Ariel"},{"middle":[],"last":"Lee","first":"Katherine"},{"middle":[],"last":"Roberts","first":"Adam"},{"middle":[],"last":"Brown","first":"Tom"},{"middle":[],"last":"Song","first":"Dawn"},{"middle":[],"last":"Erlingsson","first":"Ulfar"},{"middle":[],"last":"Oprea","first":"Alina"},{"middle":[],"last":"Raffel","first":"Colin"}]
Auditing Data Provenance in Text-Generation Models
To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new model auditing technique that helps users check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, including word prediction and dialog generation. These models are at the core of popular online services and are often trained on personal data such as users' messages, searches, chats, and comments. We design and evaluate a black-box auditing method that can detect, with very few queries to a model, if a particular user's texts were used to train it (among thousands of other users). We empirically show that our method can successfully audit well-generalized models that are not overfitted to the training data. We also analyze how text-generation models memorize word sequences and explain why this memorization makes them amenable to auditing.
[{"first":"Congzheng","last":"Song","middle":[]},{"first":"Vitaly","last":"Shmatikov","middle":[]}]
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Large Transformers pretrained over clinical notes from Electronic Health Records (EHR) have afforded substantial gains in performance on predictive clinical tasks. The cost of training such models (and the necessity of data access to do so) coupled with their utility motivates parameter sharing, i.e., the release of pretrained models such as ClinicalBERT. While most efforts have used deidentified EHR, many researchers have access to large sets of sensitive, non-deidentified EHR with which they might train a BERT model (or similar). Would it be safe to release the weights of such a model if they did? In this work, we design a battery of approaches intended to recover Personal Health Information (PHI) from a trained BERT. Specifically, we attempt to recover patient names and conditions with which they are associated. We find that simple probing methods are not able to meaningfully extract sensitive information from BERT trained over the MIMIC-III corpus of EHR. However, more sophisticated “attacks” may succeed in doing so: To facilitate such research, we make our experimental setup and baseline probing models available at https://github.com/elehman16/exposing_patient_data_release.
[{"first":"Eric","last":"Lehman","middle":[]},{"first":"Sarthak","last":"Jain","middle":[]},{"first":"Karl","last":"Pichotta","middle":[]},{"first":"Yoav","last":"Goldberg","middle":[]},{"first":"Byron","last":"Wallace","middle":[]}]
Membership Inference Attacks against Machine Learning Models
We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on. We empirically evaluate our inference techniques on classification models trained by commercial"machine learning as a service"providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.
[{"first":"Reza","last":"Shokri","middle":[]},{"first":"Marco","last":"Stronati","middle":[]},{"first":"Congzheng","last":"Song","middle":[]},{"first":"Vitaly","last":"Shmatikov","middle":[]}]
Information Leakage in Embedding Models
Embeddings are functions that map raw input data to low-dimensional vector representations, while preserving important semantic information about the inputs. Pre-training embeddings on a large amount of unlabeled data and fine-tuning them for downstream tasks is now a de facto standard in achieving state of the art learning in many domains. We demonstrate that embeddings, in addition to encoding generic semantics, often also present a vector that leaks sensitive information about the input data. We develop three classes of attacks to systematically study information that might be leaked by embeddings. First, embedding vectors can be inverted to partially recover some of the input data. As an example, we show that our attacks on popular sentence embeddings recover between 50\%--70\% of the input words (F1 scores of 0.5--0.7). Second, embeddings may reveal sensitive attributes inherent in inputs and independent of the underlying semantic task at hand. Attributes such as authorship of text can be easily extracted by training an inference model on just a handful of labeled embedding vectors. Third, embedding models leak moderate amount of membership information for infrequent training data inputs. We extensively evaluate our attacks on various state-of-the-art embedding models in the text domain. We also propose and evaluate defenses that can prevent the leakage to some extent at a minor cost in utility.
[{"first":"Congzheng","last":"Song","middle":[]},{"first":"Ananth","last":"Raghunathan","middle":[]}]
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named Ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is availabel at https://github.com/thu-coai/Targeted-Data-Extraction.
[{"first":"Zhexin","last":"Zhang","middle":[]},{"first":"Jiaxin","last":"Wen","middle":[]},{"first":"Minlie","last":"Huang","middle":[]}]
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Data privacy is an important issue for “machine learning as a service” providers. We focus on the problem of membership inference attacks: Given a data sample and black-box access to a model’s API, determine whether the sample existed in the model’s training data. Our contribution is an investigation of this problem in the context of sequence-to-sequence models, which are important in applications such as machine translation and video captioning. We define the membership inference problem for sequence generation, provide an open dataset based on state-of-the-art machine translation models, and report initial results on whether these models leak private information against several kinds of membership inference attacks.
[{"first":"Sorami","last":"Hisamoto","middle":[]},{"first":"Matt","last":"Post","middle":[]},{"first":"Kevin","last":"Duh","middle":[]}]
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The wide adoption and application of Masked language models (MLMs) on sensitive data (from legal to medical) necessitates a thorough quantitative investigation into their privacy vulnerabilities. Prior attempts at measuring leakage of MLMs via membership inference attacks have been inconclusive, implying potential robustness of MLMs to privacy attacks.In this work, we posit that prior attempts were inconclusive because they based their attack solely on the MLM’s model score. We devise a stronger membership inference attack based on likelihood ratio hypothesis testing that involves an additional reference MLM to more accurately quantify the privacy risks of memorization in MLMs. We show that masked language models are indeed susceptible to likelihood ratio membership inference attacks: Our empirical results, on models trained on medical notes, show that our attack improves the AUC of prior membership inference attacks from 0.66 to an alarmingly high 0.90 level.
[{"first":"Fatemehsadat","last":"Mireshghallah","middle":[]},{"first":"Kartik","last":"Goyal","middle":[]},{"first":"Archit","last":"Uniyal","middle":[]},{"first":"Taylor","last":"Berg-Kirkpatrick","middle":[]},{"first":"Reza","last":"Shokri","middle":[]}]
Extracting Training Data from Large Language Models
Language models (LMs)-statistical models which assign a probability to a sequence of words-are fundamental to many natural language processing tasks. Modern neural-networkbased LMs use very large model architectures (e.g., 175 billion parameters [7]) and train on massive datasets (e.g., nearly a terabyte of English text [55]). This scaling increases the ability of LMs to generate fluent natural language [53,74,76], and also allows them to be applied to a plethora of other tasks [29,39,55], even without updating their parameters [7].
Extracting Training Data from Large Language Models
At the same time, machine learning models are notorious for exposing information about their (potentially private) training data-both in general [47,65] and in the specific case of language models [8,45]. For instance, for certain models it is known that adversaries can apply membership inference attacks [65] to predict whether or not any particular example was in the training data.
Extracting Training Data from Large Language Models
East Stroudsburg Stroudsburg... Given query access to a neural network language model, we extract an individual person's name, email address, phone number, fax number, and physical address. The example in this figure shows information that is all accurate so we redact it to protect privacy.
Extracting Training Data from Large Language Models
Such privacy leakage is typically associated with overfitting [75]-when a model's training error is significantly lower than its test error-because overfitting often indicates that a model has memorized examples from its training set. Indeed, overfitting is a sufficient condition for privacy leakage [72] and many attacks work by exploiting overfitting [65].
Extracting Training Data from Large Language Models
The association between overfitting and memorization haserroneously-led many to assume that state-of-the-art LMs will not leak information about their training data. Because these models are often trained on massive de-duplicated datasets only for a single epoch [7,55], they exhibit little to no overfitting [53]. Accordingly, the prevailing wisdom has been that "the degree of copying with respect to any given work is likely to be, at most, de minimis" [71] and that models do not significantly memorize any particular training example.
Extracting Training Data from Large Language Models
Contributions. In this work, we demonstrate that large language models memorize and leak individual training examples. In particular, we propose a simple and efficient method for extracting verbatim sequences from a language model's training set using only black-box query access. Our key insight is that, although training examples do not have noticeably lower losses than test examples on average, certain worstcase training examples are indeed memorized.
Extracting Training Data from Large Language Models
In our attack, we first generate a large, diverse set of highlikelihood samples from the model, using one of three generalpurpose sampling strategies. We then sort each sample using one of six different metrics that estimate the likelihood of each sample using a separate reference model (e.g., another LM), and rank highest the samples with an abnormally high likelihood ratio between the two models.
Extracting Training Data from Large Language Models
Our attacks directly apply to any language model, including those trained on sensitive and non-public data [10,16]. We use the GPT-2 model [54] released by OpenAI as a representative language model in our experiments. We choose to attack GPT-2 to minimize real-world harm-the GPT-2 model and original training data source are already public.
Extracting Training Data from Large Language Models
To make our results quantitative, we define a testable definition of memorization. We then generate 1,800 candidate memorized samples, 100 under each of the 3 × 6 attack configurations, and find that over 600 of them are verbatim samples from the GPT-2 training data (confirmed in collaboration with the creators of GPT-2). In the best attack configuration, 67% of candidate samples are verbatim training examples. Our most obviously-sensitive attack extracts the full name, physical address, email address, phone number, and fax number of an individual (see Figure 1). We comprehensively analyze our attack, including studying how model size and string frequency affects memorization, as well as how different attack configurations change the types of extracted data.
Extracting Training Data from Large Language Models
We conclude by discussing numerous practical strategies to mitigate privacy leakage. For example, differentially-private training [1] is theoretically well-founded and guaranteed to produce private models if applied at an appropriate record level, but it can result in longer training times and typically degrades utility. We also make recommendations, such as carefully de-duplicating documents, that empirically will help to mitigate memorization but cannot prevent all attacks.
Extracting Training Data from Large Language Models
To begin, we introduce the relevant background on large (billion-parameter) neural network-based language models (LMs) as well as data privacy attacks.
Extracting Training Data from Large Language Models
Language models are a fundamental building block of current state-of-the-art natural language processing pipelines [12,31,50,52,55]. While the unsupervised objectives used to train these models vary, one popular choice is a "next-step prediction" objective [5,31,44,52]. This approach constructs a generative model of the distribution Pr(x 1 , x 2 , . . . , x n ), where x 1 , x 2 , . . . , x n is a sequence of tokens from a vocabulary V by applying the chain rule of probability
Extracting Training Data from Large Language Models
Pr(x 1 , x 2 , . . . , x n ) = Π n i=1 Pr(x i | x 1 , . . . , x i−1 ). State-of-the-art LMs use neural networks to estimate this probability distribution. We let f θ (x i | x 1 , . . . , x i−1 ) denote the likelihood of token x i when evaluating the neural network f with parameters θ. While recurrent neural networks (RNNs) [26,44] used to be a common choice for the neural network architecture of LMs, attention-based models [4] have recently replaced RNNs in state-of-the-art models. In particular, Transformer LMs [70] consist of a sequence of attention layers and are the current model architecture of choice. Because we believe our results are independent of the exact architecture used, we will not describe the Transformer architecture in detail here and instead refer to existing work [3].
Extracting Training Data from Large Language Models
Training Objective. A language model is trained to maximize the probability of the data in a training set X . In this paper, each training example is a text document-for example, a specific news article or webpage from the internet. Formally, training involves minimizing the loss function
Extracting Training Data from Large Language Models
over each training example in the training dataset X . Because of this training setup, the "optimal" solution to the task of language modeling is to memorize the answer to the question "what token follows the sequence x 1 , . . . , x i−1 ?" for every prefix in the training set. However, state-of-the-art LMs are trained with massive datasets, which causes them to not exhibit significant forms of memorization: empirically, the training loss and the test loss are nearly identical [7,53,55].
Extracting Training Data from Large Language Models
Generating Text. A language model can generate new text (potentially conditioned on some prefix x 1 , . . . , x i ) by iteratively samplingx i+1 ∼ f θ (x i+1 |x 1 , . . . , x i ) and then feedingx i+1 back into the model to samplex i+2 ∼ f θ (x i+2 |x 1 , . . . ,x i+1 ). This process is repeated until a desired stopping criterion is reached. Variations of this text generation method include deterministically choosing the most-probable token rather than sampling (i.e., "greedy" sampling) or setting all but the top-n probabilities to zero and renormalizing the probabilities before sampling (i.e., top-n sampling 1 [18]).
Extracting Training Data from Large Language Models
GPT-2. Our paper focuses on the GPT variant of Transformer LMs [7,52,54]. Specifically, we demonstrate our training data extraction attacks on GPT-2, a family of LMs that were all trained using the same dataset and training algorithm, but with varying model sizes. GPT-2 uses a word-pieces [61] vocabulary with a byte pair encoder [22].
Extracting Training Data from Large Language Models
GPT-2 XL is the largest model with 1.5 billion parameters. For the remainder of this paper, the "GPT-2" model refers to this 1.5 billion parameter model or, when we specifically indicate this, its Small and Medium variants with 124 million and 334 million parameters, respectively.
Extracting Training Data from Large Language Models
The GPT-2 model family was trained on data scraped from the public Internet. The authors collected a dataset by following outbound links from the social media website Reddit. The webpages were cleaned of HTML, with only the document text retained, and then de-duplicated at the document level. This resulted in a final dataset of 40GB of text data, over which the model was trained for approximately 12 epochs. 2 As a result, GPT-2 does not overfit: the training loss is only roughly 10% smaller than the test loss across all model sizes.
Extracting Training Data from Large Language Models
It is undesirable for models to remember any details that are specific to their (potentially private) training data. The field of training data privacy develops attacks (to leak training data details) and defenses (to prevent leaks).
Extracting Training Data from Large Language Models
Privacy Attacks. When models are not trained with privacy-preserving algorithms, they are vulnerable to numerous privacy attacks. The least revealing form of attack is the membership inference attack [28,47,65,67]: given a trained model, an adversary can predict whether or not a particular example was used to train the model. Separately, model inversion attacks [21] reconstruct representative views of a subset of examples (e.g., a model inversion attack on a face recognition classifier might recover a fuzzy image of a particular person that the classifier can recognize).
Extracting Training Data from Large Language Models
Training data extraction attacks, like model inversion attacks, reconstruct training datapoints. However, training data extraction attacks aim to reconstruct verbatim training examples and not just representative "fuzzy" examples. This makes them more dangerous, e.g., they can extract secrets such as verbatim social security numbers or passwords. Training data extraction attacks have until now been limited to small LMs trained on academic datasets under artificial training setups (e.g., for more epochs than typical) [8,66,68,73], or settings where the adversary has a priori knowledge of the secret they want to extract (e.g., a social security number) [8,27].
Extracting Training Data from Large Language Models
Privacy. An approach to minimizing memorization of training data is to apply differentially-private training techniques [1,9,43,60,64]. Unfortunately, training models with differentially-private mechanisms often reduces accuracy [34] because it causes models to fail to capture the long 2 Personal communication with the GPT-2 authors. tails of the data distribution [19,20,67]. Moreover, it increases training time, which can further reduce accuracy because current LMs are limited by the cost of training [35,38,55]. As a result, state-of-the-art LMs such as GPT-2 [53], GPT-3 [7], and T5 [55] do not apply these privacy-preserving techniques.
Extracting Training Data from Large Language Models
Training data extraction attacks are often seen as theoretical or academic and are thus unlikely to be exploitable in practice [71]. This is justified by the prevailing intuition that privacy leakage is correlated with overfitting [72], and because stateof-the-art LMs are trained on large (near terabyte-sized [7]) datasets for a few epochs, they tend to not overfit [53].
Extracting Training Data from Large Language Models
Our paper demonstrates that training data extraction attacks are practical. To accomplish this, we first precisely define what we mean by "memorization". We then state our threat model and our attack objectives. Finally, we discuss the ethical considerations behind these attacks and explain why they are likely to be a serious threat in the future.
Extracting Training Data from Large Language Models
There are many ways to define memorization in language modeling. As mentioned earlier, memorization is in many ways an essential component of language models because the training objective is to assign high overall likelihood to the training dataset. LMs must, for example, "memorize" the correct spelling of individual words.
Extracting Training Data from Large Language Models
Indeed, there is a research direction that analyzes neural networks as repositories of (memorized) knowledge [51,59]. For example, when GPT-2 is prompted to complete the sentence "My address is 1 Main Street, San Francisco CA", it generates "94107": a correct zip code for San Francisco, CA. While this is clearly memorization in some abstract form,we aim to formalize our definition of memorization in order to restrict it to cases that we might consider "unintended" [8].
Extracting Training Data from Large Language Models
We define eidetic memorization as a particular type of memorization. 3 Informally, eidetic memorization is data that has been memorized by a model despite only appearing in a small set of training instances. The fewer training samples that contain the data, the stronger the eidetic memorization is.
Extracting Training Data from Large Language Models
To formalize this notion, we first define what it means for a model to have knowledge of a string s. Our definition is loosely inspired by knowledge definitions in interactive proof systems [24]: a model f θ knows a string s if s can be extracted by interacting with the model. More precisely, we focus on black-box interactions where the model generates s as the most likely continuation when prompted with some prefix c:
Extracting Training Data from Large Language Models
A string s is extractable 4 from an LM f θ if there exists a prefix c such that:
Extracting Training Data from Large Language Models
We abuse notation slightly here to denote by f θ (s | c) the likelihood of an entire sequence s . Since computing the most likely sequence s is intractable for large N, the arg max in Definition 1 can be replaced by an appropriate sampling strategy (e.g., greedy sampling) that reflects the way in which the model f θ generates text in practical applications. We then define eidetic memorization as follows:
Extracting Training Data from Large Language Models
Definition 2 (k-Eidetic Memorization) A string s is keidetic memorized (for k ≥ 1) by an LM f θ if s is extractable from f θ and s appears in at most k examples in the training data X: |{x ∈ X : s ⊆ x}| ≤ k.
Extracting Training Data from Large Language Models
Key to this definition is what "examples" means. For GPT-2, each webpage is used (in its entirety) as one training example. Since this definition counts the number of distinct training examples containing a given string, and not the total number of times the string occurs, a string may appear multiple times on one page while still counting as k = 1 memorization.
Extracting Training Data from Large Language Models
This definition allows us to define memorization as a spectrum. While there is no definitive value of k at which we might say that memorization is unintentional and potentially harmful, smaller values are more likely to be so. For any given k, memorizing longer strings is also "worse" than shorter strings, although our definition omits this distinction for simplicity.
Extracting Training Data from Large Language Models
For example, under this definition, memorizing the correct spellings of one particular word is not severe if the word occurs in many training examples (i.e., k is large). Memorizing the zip code of a particular city might be eidetic memorization, depending on whether the city was mentioned in many training examples (e.g., webpages) or just a few. Referring back to Figure 1, memorizing an individual person's name and phone number clearly (informally) violates privacy expectations, and also satisfies our formal definition: it is contained in just a few documents on the Internet-and hence the training data.
Extracting Training Data from Large Language Models
Adversary's Capabilities. We consider an adversary who has black-box input-output access to a language model. This allows the adversary to compute the probability of arbitrary sequences f θ (x 1 , . . . , x n ), and as a result allows the adversary to obtain next-word predictions, but it does not allow the adversary to inspect individual weights or hidden states (e.g., attention vectors) of the language model. This threat model is highly realistic as many LMs are available through black-box APIs. For example, the GPT-3 model [7] created by OpenAI is available through black-box API access. Auto-complete models trained on actual user data have also been made public, although they reportedly use privacy-protection measures during training [10].
Extracting Training Data from Large Language Models
Adversary's Objective. The adversary's objective is to extract memorized training data from the model. The strength of an attack is measured by how private (formalized as being k-eidetic memorized) a particular example is. Stronger attacks extract more examples in total (both more total sequences, and longer sequences) and examples with lower values of k.
Extracting Training Data from Large Language Models
We do not aim to extract targeted pieces of training data, but rather indiscriminately extract training data. While targeted attacks have the potential to be more adversarially harmful, our goal is to study the ability of LMs to memorize data generally, not to create an attack that can be operationalized by real adversaries to target specific users.
Extracting Training Data from Large Language Models
Attack Target. We select GPT-2 [54] as a representative LM to study for our attacks. GPT-2 is nearly a perfect target. First, from an ethical standpoint, the model and data are public, and so any memorized data that we extract is already public. 5 Second, from a research standpoint, the dataset (despite being collected from public sources) was never actually released by OpenAI. Thus, it is not possible for us to unintentionally "cheat" and develop attacks that make use of knowledge of the GPT-2 training dataset.
Extracting Training Data from Large Language Models
Training data extraction attacks present numerous privacy risks. From an ethical standpoint, most of these risks are mitigated in our paper because we attack GPT-2, whose training data is public. However, since our attacks would apply to any LM, we also discuss potential consequences of future attacks on models that may be trained on private data. Data Secrecy. The most direct form of privacy leakage occurs when data is extracted from a model that was trained on confidential or private data. For example, GMail's autocomplete model [10] is trained on private text communications between users, so the extraction of unique snippets of training data would break data secrecy.
Extracting Training Data from Large Language Models
Contextual Integrity of Data. The above privacy threat corresponds to a narrow view of data privacy as data secrecy.
Extracting Training Data from Large Language Models
A broader view of the privacy risks posed by data extraction stems from the framework of data privacy as contextual integrity [48]. That is, data memorization is a privacy infringement if it causes data to be used outside of its intended context. An example violation of contextual integrity is shown in Figure 1. This individual's name, address, email, and phone number are not secret-they were shared online in a specific context of intended use (as contact information for a software project)-but are reproduced by the LM in a separate context. Due to failures such as these, user-facing applications that use LMs may inadvertently emit data in inappropriate contexts, e.g., a dialogue system may emit a user's phone number in response to another user's query.
Extracting Training Data from Large Language Models
Small-k Eidetic Risks. We nevertheless focus on k-eidetic memorization with a small k value because it makes extraction attacks more impactful.While there are cases where large-k memorization may still matter (for example, a company may refer to the name of an upcoming product multiple times in private-and even though it is discussed often the name itself may still be sensitive) we study the small-k case.
Extracting Training Data from Large Language Models
Moreover, note that although we frame our paper as an "attack", LMs will output memorized data even in the absence of an explicit adversary. We treat LMs as black-box generative functions, and the memorized content that we extract can be generated through honest interaction with the LM. Indeed, we have even discovered at least one memorized training example among the 1,000 GPT-3 samples that OpenAI originally released in its official repository [49].
Extracting Training Data from Large Language Models
In this paper, we will discuss and carefully examine specific memorized content that we find in our extraction attacks. This raises ethical considerations as some of the data that we extract contains information about individual users.
Extracting Training Data from Large Language Models
As previously mentioned, we minimize ethical concerns by using data that is already public. We attack the GPT-2 model, which is available online. Moreover, the GPT-2 training data was collected from the public Internet [54], and is in principle available to anyone who performs the same (documented) collection process as OpenAI, e.g., see [23].
Extracting Training Data from Large Language Models
However, there are still ethical concerns even though the model and data are public. It is possible-and indeed we find it is the case-that we might extract personal information for individuals from the training data. For example, as shown in Figure 1, we recovered a person's full name, address, and phone number. In this paper, whenever we succeed in extracting personally-identifying information (usernames, phone numbers, etc.) we partially mask out this content with the token . We are aware of the fact that this does not provide complete mediation: disclosing that the vulnerability exists allows a malicious actor to perform these attacks on their own to recover this personal information.
Extracting Training Data from Large Language Models
Just as responsible disclosure still causes some (limited) harm, we believe that the benefits of publicizing these attacks outweigh the potential harms. Further, to make our attacks public, we must necessarily reveal some sensitive information.
Extracting Training Data from Large Language Models
We contacted the individual whose information is partially shown in Figure 1 to disclose this fact to them in advance and received permission to use this example. Our research findings have also been disclosed to OpenAI.
Extracting Training Data from Large Language Models
Unfortunately, we cannot hope to contact all researchers who train large LMs in advance of our publication. We thus hope that this publication will spark further discussions on the ethics of memorization and extraction among other companies and research teams that train large LMs [2,36,55,63].
Extracting Training Data from Large Language Models
We begin with a simple strawman baseline for extracting training data from a language model in a two-step procedure.
Extracting Training Data from Large Language Models
• Generate text. We generate a large quantity of data by unconditionally sampling from the model (Section 4.1). • Predict which outputs contain memorized text. We next remove the generated samples that are unlikely to contain memorized text using a membership inference attack (Section 4.2).
Extracting Training Data from Large Language Models
These two steps correspond directly to extracting model knowledge (Definition 1), and then predicting which strings might be k-eidetic memorization (Definition 2).
Extracting Training Data from Large Language Models
To generate text, we initialize the language model with a onetoken prompt containing a special start-of-sentence token and then repeatedly sample tokens in an autoregressive fashion from the model (see Section 2.1 for background). We hope that by sampling according to the model's assigned likelihood, we will sample sequences that the model considers "highly likely", and that likely sequences correspond to memorized text. Concretely, we sample exactly 256 tokens for each trial using the top-n strategy from Section 2.1 with n = 40.
Extracting Training Data from Large Language Models
Given a set of samples from the model, the problem of training data extraction reduces to one of membership inference: predict whether each sample was present in the training data [65]. In their most basic form, past membership inference attacks rely on the observation that models tend to assign higher confidence to examples that are present in the training data [46]. Therefore, a potentially high-precision membership inference classifier is to simply choose examples that are assigned the highest likelihood by the model.
Extracting Training Data from Large Language Models
Since LMs are probabilistic generative models, we follow prior work [8] and use a natural likelihood measure: the per-plexity of a sequence measures how well the LM "predicts" the tokens in that sequence. Concretely, given a sequence of tokens x 1 , . . . , x n , the perplexity is defined as
Extracting Training Data from Large Language Models
That is, if the perplexity is low, then the model is not very "surprised" by the sequence and has assigned on average a high probability to each subsequent token in the sequence.
Extracting Training Data from Large Language Models
We generate 200,000 samples using the largest version of the GPT-2 model (XL, 1558M parameters) following the text generation scheme described in Section 4.1. We then sort these samples according to the model's perplexity measure and investigate those with the lowest perplexity.
Extracting Training Data from Large Language Models
This simple baseline extraction attack can find a wide variety of memorized content. For example, GPT-2 memorizes the entire text of the MIT public license, as well as the user guidelines of Vaughn Live, an online streaming site. While this is "memorization", it is only k-eidetic memorization for a large value of k-these licenses occur thousands of times.
Extracting Training Data from Large Language Models
The most interesting (but still not eidetic memorization for low values of k) examples include the memorization of popular individuals' Twitter handles or email addresses (omitted to preserve user privacy). In fact, all memorized content we identify in this baseline setting is likely to have appeared in the training dataset many times.
Extracting Training Data from Large Language Models
This initial approach has two key weaknesses that we can identify. First, our sampling scheme tends to produce a low diversity of outputs. For example, out of the 200,000 samples we generated, several hundred are duplicates of the memorized user guidelines of Vaughn Live.
Extracting Training Data from Large Language Models
Second, our baseline membership inference strategy suffers from a large number of false positives, i.e., content that is assigned high likelihood but is not memorized. The majority of these false positive samples contain "repeated" strings (e.g., the same phrase repeated multiple times). Despite such text being highly unlikely, large LMs often incorrectly assign high likelihood to such repetitive sequences [30].
Extracting Training Data from Large Language Models
The proof-of-concept attack presented in the previous section has low precision (high-likelihood samples are not always in the training data) and low recall (it identifies no k-memorized content for low k). Here, we improve the attack by incorporating better methods for sampling from the model (Section 5.1) and membership inference (Section 5.2).
Extracting Training Data from Large Language Models
The first step in our attack is to randomly sample from the language model. Above, we used top-n sampling and conditioned the LM on the start-of-sequence token as input. This strategy has clear limitations [32]: it will only generate sequences that are likely from beginning to end. As a result, top-n sampling from the model will cause it to generate the same (or similar) examples several times. Below we describe two alternative techniques for generating more diverse samples from the LM.
Extracting Training Data from Large Language Models
As described in Section 2.1, an LM outputs the probability of the next token given the prior tokens Pr(x i | x 1 , . . . , x i−1 ). In practice, this is achieved by evaluating the neural network z = f θ (x 1 , . . . , x i−1 ) to obtain the "logit" vector z, and then computing the output probability distribution as y = softmax(z) defined by softmax(z) i = exp (z i )/ ∑ n j=1 exp (z j ). One can artificially "flatten" this probability distribution to make the model less confident by replacing the output softmax(z) with softmax(z/t), for t > 1. Here, t is called the temperature. A higher temperature causes the model to be less confident and more diverse in its output.
Extracting Training Data from Large Language Models
However, maintaining a high temperature throughout the generation process would mean that even if the sampling process began to emit a memorized example, it would likely randomly step off the path of the memorized output. Thus, we use a softmax temperature that decays over time, starting at t = 10 and decaying down to t = 1 over a period of the first 20 tokens (≈10% of the length of the sequence). This gives a sufficient amount of time for the model to "explore" a diverse set of prefixes while also allowing it to follow a high-confidence paths that it finds.
Extracting Training Data from Large Language Models
Even when applying temperature sampling, there are still some prefixes that are unlikely to be sampled but nevertheless occur in actual data. As a final strategy, our third sampling strategy seeds the model with prefixes from our own scrapes of the Internet. This sampling strategy ensures that we will generate samples with a diverse set of prefixes that are similar in nature to the type of data GPT-2 was trained on.
Extracting Training Data from Large Language Models
We follow a different data collection process as used in GPT-2 (which follows Reddit links) in order to reduce the likelihood that our dataset has any intersection with the model's training data. In particular, we select samples from a subset of Common Crawl 6 to feed as context to the model.  Figure 2: Workflow of our extraction attack and evaluation. 1) Attack. We begin by generating many samples from GPT-2 when the model is conditioned on (potentially empty) prefixes. We then sort each generation according to one of six metrics and remove the duplicates. This gives us a set of potentially memorized training examples. 2) Evaluation. We manually inspect 100 of the top-1000 generations for each metric. We mark each generation as either memorized or not-memorized by manually searching online, and we confirm these findings by working with OpenAI to query the original training data. An open-source implementation of our attack process is available at https://github.com/ftramer/LM_Memorization.
Extracting Training Data from Large Language Models
As in prior work [55], we perform basic data-sanitization by removing HTML and JavaScript from webpages, and we de-duplicate data on a line-by-line basis. This gives us a dataset of 50MB of text. We randomly sample between 5 and 10 tokens of context from this scraped data and then continue LM generation with top-n sampling as in Section 4.1.
Extracting Training Data from Large Language Models
Performing membership inference by filtering out samples with low likelihood has poor precision due to failures in the underlying language model: there are many samples that are assigned spuriously high likelihood. There are predominantly two categories of such samples:
Extracting Training Data from Large Language Models
• Trivial memorization. We identify many cases where GPT-2 outputs content that is uninteresting because of how common the text is. For example, it repeats the numbers from 1 to 100 with high probability. • Repeated substrings. One common failure mode of LMs is their propensity to repeatedly emit the same string over and over [30,37]. We found many of the high-likelihood samples that are not memorized are indeed repeated texts (e.g., "I love you. I love you. . . ").
Extracting Training Data from Large Language Models
Our insight is that we can filter out these uninteresting (yet still high-likelihood samples) by comparing to a second LM. Given a second model that accurately captures text likelihood, we should expect it will also assign high likelihood to these forms of memorized content. Therefore, a natural strategy for finding more diverse and rare forms of memorization is to filter samples where the original model's likelihood is "unexpectedly high" compared to a second model. Below we discuss four methods for achieving this.
Extracting Training Data from Large Language Models
Comparing to Other Neural Language Models. Assume that we have access to a second LM that memorizes a different set of examples than GPT-2. One way to achieve this would be to train a model on a disjoint set of training data, in which case it is unlikely that the two models will memorize the same data for small k. An alternate strategy is to take a much smaller model trained on the same underlying dataset: because smaller models have less capacity for memorization, we conjecture that there are samples that are k-eidetic memorized (for small k) by the largest GPT-2 model, but which are not memorized by smaller GPT-2 models. Specifically, we use the Small (117M parameters) and Medium (345M parameters) models.
Extracting Training Data from Large Language Models
Comparing to zlib Compression. It is not necessary that we compare to another neural LM; any technique that quantifies some notion of "surprise" for a given sequence can be useful. As a simple baseline method, we compute the zlib [41] entropy of the text: the number of bits of entropy when the sequence is compressed with zlib compression. We then use the ratio of the GPT-2 perplexity and the zlib entropy as our membership inference metric. Although text compressors are simple, they can identify many of the examples of trivial memorization and repeated patterns described above (e.g., they are excellent at modeling repeated substrings).
Extracting Training Data from Large Language Models
Comparing to Lowercased Text. Instead of detecting memorization by comparing one model to another model, another option detects memorization by comparing the perplexity of the model to the perplexity of the same model on a "canonicalized" version of that sequence. Specifically, we measure the ratio of the perplexity on the sample before and after lowercasing it, which can dramatically alter the perplexity of memorized content that expects a particular casing.
Extracting Training Data from Large Language Models
Perplexity on a Sliding Window. Sometimes a model is not confident when the sample contains one memorized substring surrounded by a block of non-memorized (and high perplexity) text. To handle this, we use the minimum perplexity when averaged over a sliding window of 50 tokens. 8 
Extracting Training Data from Large Language Models
We now evaluate the various data extraction methods and study common themes in the resulting memorized content.
Extracting Training Data from Large Language Models
An overview of our experimental setup is shown in Figure 2. We first build three datasets of 200,000 generated samples (each of which is 256 tokens long) using one of our strategies: For each of these 3 × 6 = 18 configurations, we select 100 samples from among the top-1000 samples according to the chosen metric. 9 This gives us 1,800 total samples of potentially memorized content. In real-world attacks, adversaries will look to uncover large amounts of memorized content and thus may generate many more samples. We focus on a smaller set as a proof-of-concept attack.
Extracting Training Data from Large Language Models
Data De-Duplication. To avoid "double-counting" memorized content, we apply an automated fuzzy de-duplication step when we select the 100 samples for each configuration.
Extracting Training Data from Large Language Models
Given a sample s, we define the trigram-multiset of s, denoted tri(s) as a multiset of all word-level trigrams in s (with words split on whitespace and punctuation characters). For example, the sentence "my name my name my name" has two trigrams ("my name my" and "name my name") each of multiplicity 2. We mark a sample s 1 as a duplicate of another sample s 2 , if their trigram multisets are similar, specifically if |tri(s 1 ) ∩ tri(s 2 )| ≥ |tri(s 1 )|/2.
Extracting Training Data from Large Language Models
For each of the 1,800 selected samples, one of four authors manually determined whether the sample contains memorized text. Since the training data for GPT-2 was sourced from the public Web, our main tool is Internet searches. We mark a sample as memorized if we can identify a non-trivial substring that returns an exact match on a page found by a Google search.
Extracting Training Data from Large Language Models
Validating Results on the Original Training Data. Finally, given the samples that we believe to be memorized, we work with the original authors of GPT-2 to obtain limited query access to their training dataset. To do this we sent them all 1, 800 sequences we selected for analysis. For efficiency, they then performed a fuzzy 3-gram match to account for memorization with different possible tokenizations. We marked samples as memorized if all 3-grams in the memorized sequence occurred in close proximity in the training dataset. This approach eliminates false negatives, but has false positives. It can confirm that our samples are memorized but cannot detect cases where we missed memorized samples.
Extracting Training Data from Large Language Models
In some experiments below, we report exact counts for how often a particular sequence occurs in the training data. We obtained these counts by asking the GPT-2 authors to perform a separate grep over the entire dataset to get an exact count.
Extracting Training Data from Large Language Models
In total across all strategies, we identify 604 unique memorized training examples from among the 1,800 possible candidates, for an aggregate true positive rate of 33.5% (our best variant has a true positive rate of 67%). Below, we categorize what types of content is memorized by the model, and also study which attack methods are most effective.
Extracting Training Data from Large Language Models
Categories of Memorized Content. We manually grouped the memorized samples into different categories (a description of these categories is in Appendix A). The results are shown in Table 1. Most memorized content is fairly canonical text from news headlines, log files, entries from forums or wikis, or religious text. However, we also identify a significant amount of unique data, containing 128-bit UUIDs, (correctlyresolving) URLs containing random substrings, and contact information of individual people and corporations. In Section 6.3, we study these cases in more detail. Sampling conditioned on Internet text is the most effective way to identify memorized content, however, all generation schemes reveal a significant amount of memorized content. For example, the baseline strategy of generating with top-n sampling yields 191 unique memorized samples, whereas conditioning on Internet text increases this to 273. As discussed earlier, looking directly at the LM perplexity is a poor membership inference metric when classifying data generated with top-n or temperature sampling: just 9% and 3% of inspected samples are memorized, respectively. The comparison-based metrics are significantly more effective at predicting if content was memorized. For example, 67% of Internet samples marked by zlib are memorized. Figure 3 compares the zlib entropy and the GPT-2 XL perplexity for each sample, with memorized examples highlighted. Plots for the other strategies are shown in Figure 4 in Appendix B. Observe that most samples fall along a diagonal, i.e., samples with higher likelihood under one model also have higher likelihood under another model. However, there are numerous outliers in the top left: these samples correspond to those that GPT-2 assigns a low perplexity (a high likelihood) but zlib is surprised by. These points, especially those which are extreme outliers, are more likely to be memorized than those close to the diagonal.
Extracting Training Data from Large Language Models
The different extraction methods differ in the type of memorized content they find. A complete breakdown of the data is given in Appendix A; however, to briefly summarize: 1. The zlib strategy often finds non-rare text (i.e., has a high k-memorization). It often finds news headlines, license files, or repeated strings from forums or wikis, and there is only one "high entropy" sequence this strategy finds.
Extracting Training Data from Large Language Models
2. Lower-casing finds content that is likely to have irregular capitalization, such as news headlines (where words are capitalized) or error logs (with many uppercase words).
Extracting Training Data from Large Language Models
3. The Small and Medium strategies often find rare content. There are 13 and 10 high entropy examples found by using the Small and Medium GPT-2 variants, respectively (compared to just one with zlib).  We find 46 examples that contain individual peoples' names. When counting occurrences of named individuals, we omit memorized samples that relate to national and international news (e.g., if GPT-2 emits the name of a famous politician, we do not count this as a named individual here). We further find 32 examples that contain some form of contact information (e.g., a phone number or social media handle). Of these, 16 contain contact information for businesses, and 16 contain private individuals' contact details.
Extracting Training Data from Large Language Models
Some of this memorized content is exclusive to just a few documents. For example, we extract the usernames of six users participating in an IRC conversation that appeared in exactly one training document.
Extracting Training Data from Large Language Models
URLs. We identify 50 examples of memorized URLs that correctly resolve to live webpages. Many of these URLs contain uncommon pieces of text, such as random numbers or base-64 encoded strings. We also identify several URLs that resolve correctly but we cannot identify their source (and we thus do not count them as "memorized" in our evaluation).
Extracting Training Data from Large Language Models
Code. We identify 31 generated samples that contain snippets of memorized source code. Despite our ability to recover the source code verbatim, we are almost always unable to recover the original authorship notices or terms of use. Often, this information is given either before the code itself or in a LICENSE file that appears separately. For many of these samples, we can also extend their length and recover thousands of lines of (near verbatim) source code (see Section 6.4).
Extracting Training Data from Large Language Models
Unnatural Text. Memorization is not limited to naturallooking text. We find 21 instances of random number sequences with at least 50 bits of entropy. 10 For example, we 10 We estimate the entropy through manual analysis by guessing the entropy space given the format of the string.  Table 3: Examples of k = 1 eidetic memorized, highentropy content that we extract from the training data. Each is contained in just one document. In the best case, we extract a 87-characters-long sequence that is contained in the training dataset just 10 times in total, all in the same document.
Extracting Training Data from Large Language Models
extract the following UUID: 1e4bd2a8-e8c8-4a62-adcd-40a936480059 from the model; a Google search for this string identifies just 3 documents containing this UUID, and it is contained in just one GPT-2 training document (i.e., it is 1-eidetic memorized). Other memorized random number sequences include UUIDs contained in only a few documents (not listed to preserve privacy), git commit hashes, random IDs used for ad tracking, and product model numbers. Table 3 gives nine examples of k = 1 eidetic memorized content, each of which is a random sequences between 10 and 87 characters long. In each of these cases, the memorized example is contained in exactly one training document, and the total number of occurrences within that single document varies between just 10 and 311.
Extracting Training Data from Large Language Models
Data From Two Sources. We find samples that contain two or more snippets of memorized text that are unrelated to one another. In one example, GPT-2 generates a news article about the (real) murder of a woman in 2013, but then attributes the murder to one of the victims of a nightclub shooting in Orlando in 2016. Another sample starts with the memorized Instagram biography of a pornography producer, but then goes on to incorrectly describe an American fashion model as a pornography actress. This type of generation is not k-eidetic memorization (these independent pieces of information never appear in the same training documents), but it is an example of a contextual integrity violation.
Extracting Training Data from Large Language Models
Removed Content. Finally, GPT-2 memorizes content that has since been removed from the Internet, and is thus now primarily accessible through GPT-2. We are aware of this content as it is still cached by Google search, but is no longer present on the linked webpage. Some of this data is not particularly interesting in its own right, e.g., error logs due to a misconfigured webserver that has since been fixed. However, the fact that this type of memorization occurs highlights that LMs that are trained entirely on (at-the-time) public data may end up serving as an unintentional archive for removed data.
Extracting Training Data from Large Language Models
In our previous experiments, we extract strings of 256 tokens in length. Here, we briefly investigate if we can extract longer sequences. In particular, we extend the length of some of the memorized sequences by seeding the model with each sample and continuing to generate. To do this, we apply a beamsearch-like decoding method introduced in prior work [8] instead of greedy decoding which often fails to generate long verbatim sequences.
Extracting Training Data from Large Language Models
We can extend many of the memorized samples. For example, we identify a piece of source code taken from a repository on GitHub. We can extend this snippet to extract an entire file, namely 1450 lines of verbatim source code. We can also extract the entirety of the MIT, Creative Commons, and Project Gutenberg licenses. This indicates that while we have extracted 604 memorized examples, we could likely extend many of these to much longer snippets of memorized content.
Extracting Training Data from Large Language Models
Consistent with recent work on constructing effective "prompts" for generative LMs [7,62], we find that the memorized content is highly dependent on the model's context.
Extracting Training Data from Large Language Models
For example, GPT-2 will complete the prompt "3.14159" with the first 25 digits of π correctly using greedy sampling. However, we find that GPT-2 "knows" (under Definition 2) more digits of π because using the beam-search-like strategy introduced above extracts 500 digits correctly.
Extracting Training Data from Large Language Models
Interestingly, by providing the more descriptive prompt "pi is 3.14159", straight greedy decoding gives the first 799 digits of π-more than with the sophisticated beam search. Further providing the context "e begins 2.7182818, pi begins 3.14159", GPT-2 greedily completes the first 824 digits of π.
Extracting Training Data from Large Language Models
This example demonstrates the importance of the context: in the right setting, orders of magnitude more extraction is feasible than when the context is just slightly suboptimal. We find that this holds true for our memorized examples as well. None of the 273 extracted samples found using Internet conditioning can be reliably reproduced when using the same prefix initially provided to GPT-2 that produced this sample. However, nearly all can be reproduced with high probability if we provided the entire sequence of data up to (but not including) the beginning of the memorized content.
Extracting Training Data from Large Language Models
The important lesson here is that our work vastly underestimates the true amount of content that GPT-2 memorized.
Extracting Training Data from Large Language Models
There are likely prompts that would identify much more memorized content, but because we stick to simple prompts we do not find this memorized content.
Extracting Training Data from Large Language Models
Thus far, we have shown that language models can memorize verbatim training strings, even when they are trained for few epochs and achieve small train-test accuracy gaps. A natural question is how many times a string must appear for it to be memorized (i.e., k in Definition 2). Prior work has investigated LM memorization by varying the number of times particular "canary" tokens were inserted into a training dataset [8]. The main limitation of this approach is that it is synthetic: canaries are inserted artificially after the dataset has been collected and may not be representative of natural data.
Extracting Training Data from Large Language Models
Here, we study how well GPT-2 memorizes naturally occurring canaries in the training data. In particular, we consider a piece of memorized content with the following prefix:
Extracting Training Data from Large Language Models
{"color":"fuchsia","link":"https://www. reddit.com/r/The_Donald/comments/ The reddit.com URL above is completed by a specific 6-character article ID and a title. We located URLs in this specific format in a single document on pastebin.com. Each URL appears a varying number of times in this document, and hence in the GPT-2 training dataset. 11 Table 4 shows a subset of the URLs that appear more than once, and their respective counts in the document. 12 This allows us to ask the question: how many times must an example appear in the training dataset for us to extract it?
Extracting Training Data from Large Language Models
Methods. We attempt two approaches to extract URLs of this format, and run three variants of GPT-2 (XL, Medium, and Small). The two approaches vary the "difficulty" of the attack, so even if the more difficult fails the easier may succeed.
Extracting Training Data from Large Language Models
First, we directly prompt each variant of GPT-2 with the prefix above, and use top-n sampling to generate 10,000 possible extensions. Then, we test whether any of the URLs in the training document were among those that were emitted by GPT-2. We count a URL as emitted if it matches verbatim with one of the 10,000 generations.
Extracting Training Data from Large Language Models
Some URLs are not extractable with this technique, and so we make the problem easier for GPT-2 by additionally providing GPT-2 the 6-character random token that begins each URL. Given this additional prefix, we then sample from  Table 4: We show snippets of Reddit URLs that appear a varying number of times in a single training document. We condition GPT-2 XL, Medium, or Small on a prompt that contains the beginning of a Reddit URL and report a if the corresponding URL was generated verbatim in the first 10,000 generations. We report a 1 /2 if the URL is generated by providing GPT-2 with the first 6 characters of the URL and then running beam search.
Extracting Training Data from Large Language Models
the model using the beam search procedure. This task is easier in two ways: we have first provided more context and additionally use a higher recall sampling strategy.
Extracting Training Data from Large Language Models
Results. Table 4 summarizes the key results. Under the more difficult of the two approaches, the full-sized 1.5 billion parameter GPT-2 model emits all examples that are inserted 33 times or more, the medium-sized 345 million parameter memorizes half of the URLs, and the smallest 117 million parameter model memorizes none of these URLs.
Extracting Training Data from Large Language Models
When given the additional context and using beam search, the medium model can emit four more URLs, and the small model only emits the one URL that was inserted 359 times.
Extracting Training Data from Large Language Models
These results illustrate two fundamental lessons in LM memorization. First, larger models memorize significantly more training data: even hundreds of millions of parameters are not enough to memorize some of the training points. The ability of LMs to improve with model size has been extensively studied [35,38]; we show a negative trend where these improvements come at the cost of decreased privacy. Second, for the largest LM, complete memorization occurs after just 33 insertions. This implies that any potentially sensitive information that is repeated a non-trivial amount of times is at risk for memorization, even if it was only repeated multiple times in a single training document.
Extracting Training Data from Large Language Models
Now that we have shown that memorized training data can be extracted from LMs, a natural question is how to mitigate these threats. Here we describe several possible strategies.
Extracting Training Data from Large Language Models
Training With Differential Privacy. Differential privacy (DP) [13,14] is a well-established notion of privacy that offers strong guarantees on the privacy of individual records in the training dataset. Private machine learning models can be trained with variants of the differentially private stochastic gradient descent (DP-SGD) algorithm [1] which is widely implemented [17,25]. Large companies have even used DP in production machine learning models to protect users' sensitive information [15,69]. The tradeoffs between privacy and utility of models have been studied extensively: differentially-private training typically prevents models from capturing the long tails of the data distribution and thus hurts utility [19,20,67].
Extracting Training Data from Large Language Models
In the content of language modeling, recent work demonstrates the privacy benefits of user-level DP models [56]. Unfortunately, this work requires labels for which users contributed each document; such labels are unavailable for data scraped from the open Web. It may instead seem natural to aim for DP guarantees at the granularity of individual webpages, but rare snippets of text (e.g., an individual's name and contact information as in Figure 1) might appear in more than one webpage. It is thus unclear how to apply DP in a principled and effective way on Web data.
Extracting Training Data from Large Language Models
Curating the Training Data. One cannot manually vet the extremely large training datasets used for training LMs. However, there are methods to limit the amount of sensitive content that is present, e.g., by identifying and filtering personal information or content with restrictive terms of use [11,58].
Extracting Training Data from Large Language Models
Aside from attempting to remove sensitive content, it is also important to carefully de-duplicate the data. Many language modeling datasets are de-duplicated at the documentor paragraph-level, which means that a single document can still contain many repeated occurrences of a sensitive piece of content. We envision more sophisticated strategies to deduplicate the training data, or limit the contribution of any single source of training data.
Extracting Training Data from Large Language Models
It is also vital to carefully source the training data. Many of the potentially-sensitive training examples that we extracted (e.g., individuals' personal information) came from websites that are known to host sensitive content, e.g., pastebin is the 12th most popular domain in GPT-2's training set.
Extracting Training Data from Large Language Models
Overall, sanitizing data is imperfect-some private data will always slip through-and thus it serves as a first line of defense and not an outright prevention against privacy leaks.
Extracting Training Data from Large Language Models
Limiting Impact of Memorization on Downstream Applications. In many downstream applications, e.g., dialogue systems [76] and summarization models [29], LMs are finetuned on task-specific data. On the positive side, this finetuning process may cause the LM to "forget" [42,57] some of the data that is memorized during the pre-training stage. On the negative side, fine-tuning may introduce its own privacy leakages if the task-specific data also contains private information. An interesting direction for future work is to explore how memorization is inherited by fine-tuned models.
Extracting Training Data from Large Language Models
Downstream applications built on top of language models could also attempt to filter out generated text that contains memorized content, if such content can be reliably detected (e.g., using various membership inference strategies).
Extracting Training Data from Large Language Models
Auditing ML Models for Memorization. Finally, after mitigating privacy leaks, it is vital to audit models to empirically determine the privacy level they offer in practice [33]. Auditing is important even when using differential privacy, as it can complement theoretical upper bounds on privacy leakage [1]. We envision using our proposed methods, as well as existing attacks [8,33,65,72], to audit LMs.
Extracting Training Data from Large Language Models
Extraction Attacks Are a Practical Threat. Prior work shows that (100× to 1000× smaller) language models potentially memorize training data in semi-realistic settings [8,73]. Our results show that state-of-the-art LMs do memorize their training data in practice, and that adversaries can extract this data with simple techniques. Our attacks are practical even when the data contains a given sequence only a few times.
Extracting Training Data from Large Language Models
As our attacks interact with a language model as a blackbox, our results approximate the worst-case behavior of language models when interacting with benign users. In particular, among 600,000 (honestly) generated samples, our attacks find that at least 604 (or 0.1%) contain memorized text.
Extracting Training Data from Large Language Models
Note that this is likely an extremely loose lower bound. We only manually inspected 1,800 potential candidate memorized samples; if we had started with more candidates we would likely have identified significantly more memorized content. Developing improved techniques for extracting memorized data, including attacks that are targeted towards specific content, is an interesting area for future work.
Extracting Training Data from Large Language Models
Memorization Does Not Require Overfitting. It is often believed that preventing overfitting (i.e., reducing the traintest generalization gap) will prevent models from memorizing training data. However, large LMs have no significant traintest gap, and yet we still extract numerous examples verbatim from the training set. The key reason is that even though on average the training loss is only slightly lower than the validation loss, there are still some training examples that have anomalously low losses. Understanding why this happens is an important problem for future work [6,40].
Extracting Training Data from Large Language Models
Larger Models Memorize More Data. Throughout our experiments, larger language models consistently memorized more training data than smaller LMs. For example, in one setting the 1.5 billion parameter GPT-2 model memorizes over 18× as much content as the 124 million parameter model (Section 7). Worryingly, it is likely that as LMs become bigger (in fact they already are 100× larger than the GPT-2 model we study [7]), privacy leakage will become even more prevalent.
Extracting Training Data from Large Language Models
Memorization Can Be Hard to Discover. Much of the training data that we extract is only discovered when prompting the LM with a particular prefix. Currently, we simply attempt to use high-quality prefixes and hope that they might elicit memorization. Better prefix selection strategies [62] might identify more memorized data.
Extracting Training Data from Large Language Models
Adopt and Develop Mitigation Strategies. We discuss several directions for mitigating memorization in LMs, including training with differential privacy, vetting the training data for sensitive content, limiting the impact on downstream applications, and auditing LMs to test for memorization. All of these are interesting and promising avenues of future work, but each has weaknesses and are incomplete solutions to the full problem. Memorization in modern LMs must be addressed as new generations of LMs are emerging and becoming building blocks for a range of real-world applications.
Extracting Training Data from Large Language Models
For large language models to be widely adopted, they must address the training data memorization problems that we have identified. Our extraction attacks are practical and efficient, and can recover hundreds of training examples from a model, even when they are contained in just one training document.
Extracting Training Data from Large Language Models
Our analysis is best viewed as a cautionary tale of what could happen when training large LMs on sensitive data. Even though our attacks target GPT-2 (which allows us to ensure that our work is not harmful), the same techniques apply to any LM. Moreover, because memorization gets worse as LMs become larger, we expect that these vulnerabilities will become significantly more important in the future.
Extracting Training Data from Large Language Models
There will therefore need to be techniques developed to specifically address our attacks. Training with differentiallyprivate techniques is one method for mitigating privacy leakage, however, we believe that it will be necessary to develop new methods that can train models at this extreme scale (e.g., billions of parameters) without sacrificing model accuracy or training time. More generally, there are many open questions that we hope will be investigated further, including why models memorize, the dangers of memorization, and how to prevent memorization.   Figure 4 shows the distribution of the perplexities of samples generated with each of our three text generation strategies and ordered based on our six membership inference strategies.
Extracting Training Data from Large Language Models
Here we present additional results from our manual analysis of the memorized content.
Extracting Training Data from Large Language Models
We identify several memorized URLs that originated from the leaked Podesta Emails available on WikiLeaks 13 . There is only one training document that contains these memorized URLs. Due to the nature of email, the text of one message is often included in subsequent replies to this email. As a result, a URL that is used (intentionally) only once can be included in the dataset tens of times due to the replies. 13 https://en.wikipedia.org/wiki/Podesta_emails Memorized Donald Trump Quotes and Tweets. The GPT-2 training dataset was collected when the 2016 US Presidential election was often in the news. As a result, we find several instances of memorized quotes from Donald Trump, both in the form of official remarks made as President (found in the official government records), as well as statements made on Twitter.
Extracting Training Data from Large Language Models
Memorized Promotional Content. We extract memorized samples of promotional content, such as advertisements for books, beauty products, software products. One of these samples includes a link to an author's valid Patreon account, along with a list of named and pseudonymous prior donors.
Extracting Training Data from Large Language Models
Memorized Number Sequences. We identify many examples where GPT-2 emits common number sequences. Nearly ten examples contain the integers counting up from some specific value. We also find examples of GPT-2 counting the squares Memorized News Headlines. Numerous memorized text snippets are verbatim copies of news articles and headlines. A large number of these memorized samples are attributed to a single source: thehill.com, an American news website. Interestingly, most of these samples follow the exact same template: (1) they contain a list of different news headlines separated by a "pipe" symbol (|), (2) the sample begins with two merged words, e.g., "TrumpJesuit", (3) the headline list ends with the all-caps word "MORE", and (4) the sample contains the all-caps word "ADVERTISEMENT".
Extracting Training Data from Large Language Models
We indeed find pages on the Web that contain copies of headlines from thehill.com under this exact template. The peculiarities of these snippets likely contributed to their memorization. For example, the token TrumpJesuit does not appear in any other context on the entire Web.
Extracting Training Data from Large Language Models
Memorized Base-64 Content. One particularly interesting form of memorization that we identify is the ability of GPT-2 to emit base-64 encoded content. For example, we extract out of the model the following sequence:  Figure 4: For each of our three text generation strategies (Top-n, Internet and Temperature), we generate 200,000 samples using GPT-2 and apply a de-duplication procedure. The two left-most plots show the distribution of perplexities for the full sample, and the most likely window of 50 tokens. The remaining plots compare the distribution of perplexities of GPT-2 to other measure of sample likelihood: zlib entropy, perplexity under GPT-2 Small and GPT-2 Medium, and perplexity of lower-cased samples. Each plot highlights the 100 samples we selected for manual inspection (red) and the subset that was confirmed as memorized (blue).    Table 7: Memorized content found using our six membership inference strategies. We show categories with at least 5 samples.
Auditing Data Provenance in Text-Generation Models
Data-protection policies and regulations such as the European Union's General Data Protection Regulation (GDPR) [9] give users the right to know how their data is processed. As machine learning (ML) becomes a core component of data processing in many offline and online services, and incidents such as DeepMind's unauthorized use of NHS patients' data to train ML models [3] illustrate the resulting privacy risks, it is essential to be able to audit the provenance of personal data used for model training.
Auditing Data Provenance in Text-Generation Models
In this paper, we design and evaluate a technology that can help users audit ML models to determine if their data was used to train these models. We focus specifically on auditing models Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. that generate natural-language text. Text-generation models for tasks such as next-word prediction (the basis of query autocompletion and predictive virtual keyboards) and dialog generation (the basis of chatbots and automated customer service) are extensively trained on personal data, including users' messages, documents, chats, comments, and search queries. Our technology can help users audit a publicly available text-generation model and see if their words were used, perhaps without their permission, to create this model. Furthermore, our work sheds new light on how deep learning-based, text-generation models memorize their training data-a topic that has important implications for both data privacy and natural language processing.
Auditing Data Provenance in Text-Generation Models
The problem of auditing is closely related to the problem of membership inference (see Section 7), but auditing text-generation models requires new technical machinery vs. membership inference in image-classification and categorical models.
Auditing Data Provenance in Text-Generation Models
First, we assume a very restrictive auditing scenario, which we believe matches how an individual user may audit a deployed MLbased service in practice. The auditor has only black-box access to the model and can query it only on a limited number of inputs. We assume that the model's output does not include numeric probabilities or confidence values (deployed models rarely release these values). Furthermore, we consider scenarios where the model's output is restricted to a relatively small list of words or even a single word. This precludes the application of most previously proposed membership inference methods.
Auditing Data Provenance in Text-Generation Models
Second, we work with text-generation models that are trained on the data of hundreds or thousands of users and are well-generalized, i.e., their accuracy on test inputs is not substantially different from their accuracy on training inputs. This precludes the application of membership inference methods that exploit the test-train accuracy gap exhibited by overfitted models.
Auditing Data Provenance in Text-Generation Models
Third, state-of-the-art text-generation models are based on recurrent neural networks (RNNs). We investigate how these models overfit to their training data, what signal this overfitting creates in their outputs, and how to exploit this signal for effective auditing. We show that overfitting in text-generation models appears to manifest primarily via shifted probability distributions over the models' output space. Specifically, we show that these models tend to assign significantly higher rank to relatively rare words when they appear in a familiar context (e.g., in a sentence seen during training). This does not affect the top-ranked, likeliest word generated by the model and therefore-in contrast to "conventional" overfitting-does not manifest in reduced test accuracy.
Auditing Data Provenance in Text-Generation Models
Fourth, we show how to use auxiliary public datasets and crossdomain training when the auditor does not know the distribution from which the training data for the target model was drawn.
Auditing Data Provenance in Text-Generation Models
Fifth, we focus on user-level auditing (vs. inferring membership of individual inputs in the training dataset) and measure how many queries are needed to determine if the user's data was used-possibly in combination with the data from thousands of other users-to train the model. We quantitatively show that sequences that include relatively rare words are more effective for auditing than word sequences randomly selected from the user's data. We also measure the robustness of our auditing methodology to noise and errors in the test inputs used for auditing. This is important because the user may not know exactly which of his chats or online comments were used, or when the model creator may have started training on the user's data.
Auditing Data Provenance in Text-Generation Models
Our black-box auditing methodology is very effective. In our experiments on the Reddit, SATED, and Dialogs tasks for, respectively, word prediction, translation, and dialog generation, it performs perfectly (i.e., its AUC score is 1) when the models are trained on the data of hundreds of users and the models' outputs cover the entire vocabulary. Furthermore, it requires surprisingly few queries. If the auditor selects query sequences that include relatively rare words, a single query achieves AUC between 0.8 and 0.9 depending on the task, and 8 queries achieve almost perfect AUC.
Auditing Data Provenance in Text-Generation Models
If the word-prediction and dialog-generation models are restricted to generate and rank only the 500 likeliest words, AUC score of our auditor remains above 0.9. If the translation model generates a single word (as opposed to a ranked list of words), the auditor can still infer with a much-better-than-random probability if the model was trained on the word sequences of a particular user. For the Reddit word-prediction model, the auditor's AUC score remains close to 0.9 even if the model was trained on the data of over 4,000 users. Furthermore, we empirically show that our auditing is robust to a significant amount of noise and errors in the audit queries. These results demonstrate that auditing modern text-generation models is feasible in realistic scenarios.
Auditing Data Provenance in Text-Generation Models
Finally, to explain why auditing works, we provide new insights into memorization in different types of text-generation models. For example, we demonstrate that deep learning-based translation models are more prone than the word-prediction models to memorize training sequences in their inner units.
Auditing Data Provenance in Text-Generation Models
A deep learning model is a function f θ : X → Y parameterized by θ , where X is the input space and Y is the output space. Supervised training of a model f θ aims to find the best set of parameters θ using a labeled training dataset D = {(x i , y i )} n i=1 and a loss function L. For ML tasks where the input space is discrete and sparse (e.g., text or location data), the standard approach is to transform discrete inputs into a lower-dimensional continuous vector representation. For a text corpus with vocabulary V , an embedding is a function E : V → R d emb where d emb , the dimension of the embedding, is a hyper-parameter. In many NLP tasks, the input is a variable-length sequence of tokens x = [x 1 , . . . , x l ] in the embedding space. The output y can be either a class label (e.g., for sentiment analysis), or a token (e.g., for next-word prediction), or a sequence of tokens (e.g., for machine translation).
Auditing Data Provenance in Text-Generation Models
Recurrent neural networks (RNNs) are a common architecture for text-generation tasks such as next-word prediction. An RNN maps the input sequence to a sequence of hidden representations a = [a 1 , . . . , a l ], where the computation of a j is recursively dependent on the previous hidden representation a j−1 and the current input token x j , and feeds these hidden representations to a classifier.
Auditing Data Provenance in Text-Generation Models
Sequence-to-sequence models are a common architecture for textgeneration tasks where both the input x = [x 1 , . . . , x l ] and the output y = [y 1 , . . . , y t ] are sequences of tokens. A typical sequenceto-sequence model consists of an encoder RNN and a decoder RNN. The encoder learns the representation for the input texts, then passes this representation as the initial state for the decoder, which makes word predictions one at a time. Translation models are similar: the decoder predicts words in the target language by feeding its hidden representations to a classifier.
Auditing Data Provenance in Text-Generation Models
Next-word prediction is used in many natural-language applications, including predictive virtual keyboards and query autocompletion. Given an input sequence x = [x 1 , . . . , x l ], the task is to predict the next token x j from the context [x 1 , . . . , x j−1 ]. RNNs are commonly used for this task. RNN feeds the last hidden representation a j−1 in the context sequence to a |V |-way classifier to predict the next token, where V is the vocabulary.
Auditing Data Provenance in Text-Generation Models
Neural machine translation (NMT) models based on RNNs reach near-human performance on many language pairs [34]. The input to these models is a sequence of tokens from the source language, the output is a sequence of tokens from the target language. NMT models use the sequence-to-sequence framework. The input text is encoded as a hidden representation, and the decoder RNN predicts translated tokens based on this representation.
Auditing Data Provenance in Text-Generation Models
Dialog generation aims to generate replies in a conversation. It is a common component of chatbots and question-answering services. The input is a sentence, the output is the next sentence in the same conversation. Dialog-generation models can also employ a sequence-to-sequence architecture [18,33]. Similar to NMT, the model encodes the input sentence to a hidden representation, then generates the reply by passing this representation to the decoder.
Auditing Data Provenance in Text-Generation Models
Loss functions. For the next-word prediction task, given an input sequence x = [x 1 , . . . , x l ], the RNN models the conditional probability Pr(x j |x 1 , . . . , x j−1 ) = f (x 1 , . . . , x j−1 ) and aims to maximize the probability for the sequence Pr(x) = l j=1 Pr(x j |x 1 , . . . , x j−1 ). The loss function used when training the model is thus the negative log likelihood:
Auditing Data Provenance in Text-Generation Models
. For the machine translation and dialog-generation tasks where the input is x and the target is y = [y 1 , . . . , y t ], the sequence-tosequence model computes the probability Pr(y j |y 1 , . . . , y j−1 ; x) as f (y 1 , . . . , y j−1 ; x). Similar to the next-word prediction task, the loss function is the negative log probability on the target sequence.
Auditing Data Provenance in Text-Generation Models
Consider a training dataset D train where each row is associated with an individual user, and let U train be the set of all users in D train . The target model f is trained on D train using a training protocol T target , which includes the learning algorithm and the hyper-parameters that govern the training regime. As described in Section 2.2, a text-generation model f takes as input a sequence of tokens x and outputs a prediction f (x) for a single token (if the task is next-word prediction) or a sequence of tokens (if the task is machine translation or dialog generation). The prediction f (x) is a probability distribution or a sequence of distributions over the training vocabulary V or a subset of V . We assume that the tokens in the model's output space are ranked (i.e., the output distribution imposes an order on all possible tokens) but do not assume that the numeric probabilities from which the ranks are computed are available as part of the model's output.
Auditing Data Provenance in Text-Generation Models
The goal of auditing is to infer user-level membership against the target model f , i.e., to decide whether a user u ∈ U train or not.
Auditing Data Provenance in Text-Generation Models
We assume that the auditor has black-box access to f : given an input query x, the auditor can observe f (x). In realistic deployments of text-generation models, the auditor may not be able to observe the entire vector of ranked words f (x) but only several top-ranked predictions. In our experiments in Section 4.3, we vary the size of the model's output and show how it affects the accuracy of auditing.
Auditing Data Provenance in Text-Generation Models
We assume that the auditor knows the learning algorithm used to create f but he may or may not know the training hyper-parameters (see Section 4.3). The auditor also needs an auxiliary dataset D ref to train shadow models that perform the same task as f . Fig 1 outlines the auditing process. Similar to standard membership inference [28], the auditor's goal is to learn to distinguish the outputs produced by the target model on sequences that it trained on and its outputs on sequences that it did not see during training. For this purpose, the auditor builds a binary user-level membership classifier f audit that takes as input a (processed) list of predictions obtained by querying f with a subset of the user's dataset D u and outputs a decision on u ∈ U train . In Section 4.3, we show that a small subset of D u is sufficient for this purpose.
Auditing Data Provenance in Text-Generation Models
Training shadow models. To collect the data for training f audit , the auditor first trains k shadow models f ′ 1 . . . f ′ k (that "simulate" f ) using the same protocol T target as f with the same hyper-parameters (if known) or varying the hyper-parameters as in Section 4.3.
Auditing Data Provenance in Text-Generation Models
The training data for each shadow is a random user subset
Auditing Data Provenance in Text-Generation Models
Our shadow training technique is inspired by [28], but one essential distinction is that in our case the shadow-training data does not need to be drawn from the same distribution as the training data of the target model In Section 4.3, we show that public sources can be used for D ref and the loss in audit accuracy is negligible when D train and D ref are drawn from different domains. This is important for real-world auditing because in practice the auditor may not know the entire distribution of the target model's training data, and API limits may prevent the auditor from querying the target model repeatedly to extract sufficient data for training shadow models as in [28].
Auditing Data Provenance in Text-Generation Models
The auditor then queries the shadow models with D ref,u for each u in U ref and labels the resulting outputs as "member" if u was part of the shadow's training data, "non-member" otherwise. The next step is to use these labeled predictions to train a binary membership classifier.
Auditing Data Provenance in Text-Generation Models
Training the audit model. Record-level membership inference typically uses the output probability distribution directly as the feature to distinguish between members and non-members. Userlevel membership inference in text-generation models calls for a different approach. Each user is associated with multiple sequences, each of which has multiple words. Therefore, the auditor can obtain a collection of output predictions. On the negative side, the actual probabilities associated with each prediction may not be available.
Auditing Data Provenance in Text-Generation Models
As mentioned before, the output prediction f (x) for a input x is a probability distribution across the entire training vocabulary V , i.e., a |V |-dimensional probability vector. |V | is generally large and the probability values are noisy. Instead of the raw probability values, we use the ranks of the target words in the output distributions as signals for inferring user-level membership. As we will show in Section 5, even for a well-generalized model (i.e., whose test-train accuracy gap is small), there is a substantial gap in the predicted rank of the same word when it appears in a training text and a test text. Specifically, the model ranks relatively rare words much higher when it sees them during testing in the same context as it saw them during training.
Auditing Data Provenance in Text-Generation Models
Given a user u's data D ref,u , the auditor queries the shadow model on each data point (x, y) ∈ D ref,u and collects the ranks of y in f (x) into a rank set R u . Taking English-to-French machine translation task as an example where (x, y) = (I love you, Je t'aime),
Auditing Data Provenance in Text-Generation Models
is a sequence of two probability vectors for tokens "Je" and "t'aime. " The auditor collects the rank of the probability of "Je" in f (x) 1 (e.g., 2) and the rank of the probability of "t'aime" in f (x) 2 (e.g., 213), and adds {2, 213} to the rank set R u . Rank 2 means that the word is the second likeliest prediction in the entire vocabulary. After collecting the ranks for all (x, y) ∈ D ref,u , the auditor builds a histogram for R u with a fixed number of bins d. The final feature vector h u is a d-way count vector where each entry is the count of the ranks in that bin.
Auditing Data Provenance in Text-Generation Models
The auditor extracts features h u and labels them as 1 if u ∈ U train ref and 0 otherwise. The auditor repeats this procedure for each user in each shadow model and obtains a collection of labeled feature vectors D audit . Finally, the auditor trains a binary membership classifier f audit on D audit . We refer to f audit as the audit model.
Auditing Data Provenance in Text-Generation Models
Auditing membership in the training data. At inference (i.e., audit) time, the auditor queries the target model f with the user's data D u . If the number of queries to f is limited, only a sample from D u is used. It can be random, but we show in Section 4.3 that it is more effective to select test inputs that have the smallest frequency counts in their labels y, i.e., sequences with relatively rare words are more useful for auditing. After querying f , the auditor processes the corresponding outputs and obtains a feature vector h u that describes the distribution of the predicted ranks for each word in D u . Finally, the auditor feeds h u to f audit , which decides whether u ∈ U train or not.
Auditing Data Provenance in Text-Generation Models
The Reddit comments dataset (Reddit) is a randomly chosen month (November 2017) from the public Reddit dataset. 1 We filtered it to retain only the users with at least 150 but no more than 500 posts, for a total of 83,293 users with 247 posts each on average. We use the resulting dataset for the next-word prediction task.
Auditing Data Provenance in Text-Generation Models
The speaker annotated TED talks dataset (SATED) consists of transcripts from TED talks, 2 totaling 2,324 talks with roughly 271K sentences in each language [24]. The dataset contains English-French (en-fr), English-German (en-de) and English-Spanish (en-es) language pairs and speaker annotation. We use the data from the en-fr pair for the machine translation task.
Auditing Data Provenance in Text-Generation Models
The Cornell movie dialogs corpus (Dialogs) is a collection of fictional conversations extracted from movie scripts [7]. There are a total of 220,579 exchanges between pairs of characters engaging in at least 5 exchanges, involving 9,035 characters from 617 movies. We use this dataset for the dialogue-generation task.
Auditing Data Provenance in Text-Generation Models
Cross-domain reference datasets. The auditor may not know the distribution on which the target model was trained and thus needs a reference dataset to train its shadow models. In our experiments, we use public datasets for this purpose. As the cross-domain reference dataset for word prediction, we use the Wikitext-103 corpus 3 obtained by a Wikipedia crawl. For translation, we use the English-French pair in the Europarl dataset [15], a parallel language corpus extracted from the proceedings of the European Parliament. For dialog generation, we use the Ubuntu dialogs dataset [20], which contains two-person technical support chat logs. These datasets are not labeled with individual users, thus we split them into random n u subsets, each corresponding to an artificial "user." Our experiments show that we can produce effective audit models even with this artificial separation into users and even though the topics of the reference datasets are very different from the target models' training datasets (e.g., technical support chats vs. conversations between movie characters).
Auditing Data Provenance in Text-Generation Models
We use standard architectures and hyper-parameters to train target models (see Section B.1) and evaluate their performance using word prediction accuracy = 1 The lower the perplexity, the better the model fits the data. Table 1 shows the results for models trained on 300 users, with the test data sampled from 300 disjoint users from the training set. These results match the literature. On Reddit, test accuracy of word prediction is 20%, similar to [23]. On SATED, test perplexity is 10, close to [21]. Low test perplexity shows that the models are learning a meaningful language-generation process. Test-train accuracy gaps are below 5%, indicating that the models are not overfitted. Perplexity gaps are within 15, which is relatively small.
Auditing Data Provenance in Text-Generation Models
To train shadow models, we sample a set of "shadow users" disjoint from both the training and test users. The number of shadow users is twice the number of training users. We use one half of the shadow users to train shadow models and the other half to collect the shadow models' outputs on the non-members of their training datasets (see Section 3). We train 10 shadow models for all tasks and use a linear SVM as the audit classifier.
Auditing Data Provenance in Text-Generation Models
Our metrics are precision (the percentage of users classified by the audit model as "members" who are indeed members), recall (the  percentage of members who are classified as "members"), accuracy (the percentage of all users who are classified correctly), and AUC, the area under the ROC curve that shows the gap between the scores (i.e., distances to the decision hyperplane of SVM) given by the audit model to members and non-members. We use 300 members and non-members. Therefore, the baseline for all metrics is 0.5, corresponding to random guessing. Our audit model achieves the perfect score (i.e., 1) on all metrics for all datasets and models when there is no restriction on the output size of the target models (i.e., they produce predictions over the entire vocabulary) and the auditor can query the target models any number of times.
Auditing Data Provenance in Text-Generation Models
Effect of different hyper-parameters. To demonstrate that knowledge of the target model's hyper-parameters is not essential for successful auditing, we train 10 shadow models for each task with different training configurations (detailed in Appendix B.2). Table 2 shows the results. Auditing scores are still above 0.95 on nearly all metrics for all tasks and models.
Auditing Data Provenance in Text-Generation Models
Effect of the number of users. To evaluate how the number of users in the training dataset affects the auditor's ability to infer the presence of a single user, we train word-prediction models on 100, 500, 1,000, 2000, 4,000, and 10,000 users from the Reddit dataset. Test users and shadow users are disjoint samples of the same size. Fig. 2 shows the results. When the number of users is under 1,000, all metrics are at least 0.95. With 4,000 users, precision drops below 0.8 while AUC is still around 0.9. Audit performance drops more significantly when the number of users is 10,000.
Auditing Data Provenance in Text-Generation Models
Effect of the number and selection of audit queries. To measure the performance of auditing when the auditor is restricted to only a few queries, we vary the number of audit queries between 1, 2, 4, 8, 16, and 32 word sequences. audit performance is low with fewer than 8 queries. If the auditor queries the target with the user's word sequences whose summary word-frequency counts are the lowest, even with a single query, the auditor can accurately determine if the user's data was used to train the model on the Reddit or Dialogs dataset. This remarkable result demonstrates the extent to which text-generation models memorize word sequences they were trained on, especially those that contain relatively rare words.
Auditing Data Provenance in Text-Generation Models
Effect of the size of the model's output. In a realistic deployment of a text-generation model, its output may be limited to a few top-ranked words rather than the entire ranked vocabulary. We constrain the model's output to the top-ranked 1, 5, 50, 500, and 1000 words, while the other hyper-parameters remain as in Section B.1. When building the histogram feature vector for training the audit model (see Section 3), we add an additional feature that counts how many times the ground-truth words are not among the top predictions output by the model. Table 3 shows the results. On Reddit and Dialogs, the auditor's performance is close to random guessing when the model's outputs are limited to the top 50 or fewer words, increasing to above 0.9  when the output size is the top 500 words (only 10% of the entire vocabulary)-regardless of whether the shadow models are trained on the same domain as the target model or a different domain. For the translation task, audit performance is much higher than random guessing even if the model outputs just one top-ranked word and exceeds 0.9 when the model outputs 50 topranked words (1% of the vocabulary). These results demonstrate the remarkable extent to which translation models memorize specific word sequences encountered in training.
Auditing Data Provenance in Text-Generation Models
Effect of noise and errors in the queries. D u may be noisy or partially erroneous (e.g., if not all of D u was used to train the target model f ). To evaluate how this affects auditing, for each training user, we use part of his data to train f and hold out the remaining fraction to represent noise during auditing. We vary this fraction between 0.1, 0.2, . . . , 0.5. No obfuscation: i see so many adults that could benefit from this going around having themselves a big fat sugar snack or soda pop as a treat it 's so sad Google: i saw so many adults who can benefit from cherishing big fat sugar snacks and soda pop and going around, it is very sad Yandex: i think a lot of adults have benefited over your big fat candy and and handling of grief  Fig. 4 shows the results. For SATED and Dialogs, recall drops significantly, close to 0 for SATED when the fraction of noise is 0.5. Increasing the amount of noise biases the audit model towards misclassifying most training users as "non-members. " Precision and AUC remain high when noise increases. This may indicate that the scores of the membership classifier at the heart of the audit model still have a distinguishable gap between members and nonmembers, which is however not learned from the outputs of the shadow models queried with clean data (see Section 3).
Auditing Data Provenance in Text-Generation Models
Auditing obfuscated data. Finally, we evaluate the effect of obfuscation on the success of auditing. This is the first step towards determining whether text-generation models memorize specific word sequences (which would not be preserved by obfuscation) rather than higher-level linguistic features (which might be).
Auditing Data Provenance in Text-Generation Models
We use an obfuscation technique, previously considered for evading author attribution [4], that machine-translates the text to a different language and back. We obfuscate the training and test users' Reddit comments using Google 4 and Yandex 5 translation APIs to translate English to Japanese and back to English. Table 4 shows examples of obfuscated text. Table 5 reports the results of auditing on obfuscated texts. For both Google-and Yandex-based obfuscation, audit accuracy drops to near random and recall is very low. AUC scores are still around 0.8, which is much higher than random guessing. This indicates there is some useful signal in the model's outputs on obfuscated texts, but the auditor's membership classifier-which was trained on non-obfuscated texts-fails to capture this signal. This is a remarkable result given the poor quality of translation. Even if the user's text has been garbled almost to the point of incomprehensibility, in some cases there is still enough information left to detect its presence in the training data.
Auditing Data Provenance in Text-Generation Models
In this section, we analyze why auditing works so well for textgeneration models that are not overfitted as measured by their test-train accuracy gap (see Section 4.2). Word frequency and probability. The loss function for the textgeneration models is the sum of the negative log probabilities of the words in the input sequence (see Section 2.2). By its very construction, this loss function "encourages" the model to memorize sequences that occur in the training data. Fig. 5 shows the histograms of the log probabilities of the more and less frequent words in the training ("train") and test ("unseen") sequences. For the more frequent words, the histograms for the training and test sequences are almost identical. For the less frequent words, the model fits worse for both the training and test sequences as modes focus on smaller log probability values. Most importantly, there is a gap between the less frequent words in the training sequences and those in the test sequences. This gap indicates that the model assigns higher probabilities to words in the training sequences, producing a strong signal that can be used for membership inference and consequently auditing.
Auditing Data Provenance in Text-Generation Models
These histograms also demonstrate that our text-generation models are not overfitted to their training datasets in terms of the loss value. The 20% most frequent words account for 86.9% of the training data and 88.1% of the test data in Reddit, 89.5% and 90.4% in SATED, and 93.1% and 94.1% in Dialogs. Consequently, these words dominate the training and test loss value. Not surprisingly, text-generation models typically generate words from the top 20% of the word-frequency distribution. As long as the log probabilities remain similar for the top 20% words in both the training and test datasets, the training and test losses of the model will be similar.
Auditing Data Provenance in Text-Generation Models
Word frequency and predicted rank. Memorization of training sequences produces a much stronger signal in the relative rank assigned by the model to the candidate words in the model's output vocabulary. Fig. 6 shows the relationship between a word's rank in the frequency table of the training corpus and its rank in the model's predictions. A smaller rank number indicates that the word is ranked higher in the vocabulary, i.e., more frequent in the corpus or more likely to be predicted by the model. On all datasets, less frequent words exhibit a much bigger gap between the rank predicted by the model when the word appears in a training sequence and when it appears in a test sequence. This explains why our auditing algorithm is more successful when it queries the target model with sequences consisting of the less-frequent words (see Section 4.3).
Auditing Data Provenance in Text-Generation Models
Ablation analysis. We have shown that probabilities and ranks produced by text-generation models exhibit a gap between the training and test sequences for the less-frequent words but not for the most-frequent words. We hypothesize that these models learn generalizable patterns for the most-frequent words while hardmemorizing the sequences consisting of the less-frequent words.
Auditing Data Provenance in Text-Generation Models
To gather evidence for this hypothesis, we carried out an experiment based on ablation analysis that was recently proposed to detect memorization in deep-learning models [25]. As more hidden units are ablated, accuracy on the training data degrades quicker for models that are hard-memorizing the training data.
Auditing Data Provenance in Text-Generation Models
We train target models without dropout (since dropout ablates the hidden units during training) on Reddit and SATED, keeping the other hyper-parameters the same as in Section B.1. We randomly set a fraction of the model's hidden representations to zero and evaluate the accuracy of word prediction on the training data. We vary the fraction from 0.1 to 0.5 on Reddit and 0.1 to 0.9 on SATED and report the accuracy score separately for the 10% most frequent words and the remaining 90% in Fig. 7.
Auditing Data Provenance in Text-Generation Models
When no hidden units are ablated, accuracy is similar for the most-frequent words and the rest. As the fraction of ablated units increases, accuracy on the less-frequent words drops more significantly than on the most-frequent words. This indicates that predicting less-frequent words is more dependent on specific hidden units in the model and thus involves more memorization.
Auditing Data Provenance in Text-Generation Models
Models trained on a very large number of users. In some industrial implementation of text-generation models [22,23], the number of users is on the scale of millions. Performance of our auditor starts to drop when the number of users reaches 10,000 (Section 4.3). We expect that our black-box algorithm will not be able to audit models trained on a very large number (dozens or hundreds of thousands) of users. That said, (a) many state-of-the-art models are trained on fewer than 10,000 users [17,24,37], and (b) white-box auditing techniques may be effective even against models trained on dozens of thousands of users. This is a topic for future work.
Auditing Data Provenance in Text-Generation Models
Deeper models. In our experiments, both the target and shadow models are one-layer LSTMs or GRUs. We have not experimented with auditing deeper and more sophisticated models. We expect that such models are even more susceptible to memorization, but this is another topic for future research.
Auditing Data Provenance in Text-Generation Models
Differentially private models. In theory, user-level differential privacy (DP) is a direct countermeasure to user-level membership inference. We used federated learning with differential privacy [23] to train a next-word prediction model on the Reddit dataset, setting the number of users to 5,000, user sampling rate to 0.04 per round, Frequency rank
Auditing Data Provenance in Text-Generation Models
Frequency Train Unseen Figure 6: Ranks of words in the frequency table of the training corpus and in the models' predictions (lower rank means that the word is more likely). Shaded area is the 95% confidence interval for all occurrences of the word in the data. These charts demonstrate that the models assign much higher rank to words when they appear in training sequences vs. when they appear in test sequences, especially for the less-frequent words. L 2 bound on a single user's contribution to 10.0, and the other hyperparameters as in [23]. After 300 rounds of training, this produced an (ϵ, δ )-DP model with ϵ = 4.129 and δ = 1e − 4 which achieves 15% word prediction accuracy, similar to [23]. By contrast, the accuracy of our non-DP model is 20% when trained on only 100 users, i.e., the DP model is significantly less accurate than the non-DP one. Our auditing algorithm fails against the DP model, with performance scores near 0.5 (equivalent to random guessing).
Auditing Data Provenance in Text-Generation Models
To further investigate the predictive power of the DP model, Fig. 8 plots the ranks of words in the vocabulary (based on their frequencies) and in the model's predictions. The predicted rank is larger than the frequency rank for the 50% most frequent words and remains around 3,000 for the other 50%. The predicted rank is very similar for the words in the training and test sequences, which explains why auditing fails.
Auditing Data Provenance in Text-Generation Models
The plot also suggests that the differentially private model will almost always predict common words and hardly ever predict relatively rare words. While it does not appear that the model memorizes its training data, it is not clear to what extent it generalizes.
Auditing Data Provenance in Text-Generation Models
Membership inference. Membership inference attacks involve observing the output of some computations over a hidden dataset D and determining whether a specific data point is a member of D. Membership inference attacks against aggregate statistics have been demonstrated in the context of genomic studies [13], location time-series [26], and noisy statistics in general [8].
Auditing Data Provenance in Text-Generation Models
Shokri et al. [28] develop black-box membership inference techniques against ML models which perform best when the target model is overfitted to the training data. Truex et al. [32] extend and generalize this work to white-box and federated-learning settings. Rahman et al. [27] use membership inference to evaluate the tradeoff between test accuracy and membership privacy in differentially private ML models. Hayes et al. [11] study membership inference against generative models. Long et al. [19] show that well-generalized models can leak membership information, but the adversary must first identify a handful of vulnerable records in the training dataset. Yeom et al. [35] formalize membership inference and theoretically show that overfitting is sufficient but not necessary.
Auditing Data Provenance in Text-Generation Models
Memorization in ML models. Zhang et al. [36] show that deep learning models can achieve perfect accuracy even on randomly labeled training data. Song et al. [29] present algorithms that intentionally encode the training data in the model. By contrast, we demonstrate that popular text-generation models unintentionally memorize their training data.
Auditing Data Provenance in Text-Generation Models
Carlini et al. [5] show that a black-box adversary can extract specific numbers that occur in the training data of a generative model, given some prior knowledge about the format (e.g., a credit card number). For a text-generation model, numbers are essentially random data, thus this is another illustration that models memorize random data. By contrast, we show that text-generation models memorize even words and sentences that are directly related to their primary task and leverage this into an effective auditing method. User-level differential privacy. User-level differential privacy (DP) bounds the influence of any single user on the model. McMahan et al. propose a DP federated learning algorithm for language models [23]. With the current state of the art, a massive number of users (at least 10,000) is needed to create DP models that achieve reasonable accuracy. How to build accurate DP models with fewer users remains an open question.
Auditing Data Provenance in Text-Generation Models
Auditing ML models. Much recent work aims to understand the behavior of ML models with black-box access [2,16]. These approaches improve the interpretability of the model by showing how features or training data points influence the model's predictions.
Auditing Data Provenance in Text-Generation Models
Other model-auditing research focuses on detecting bias and discrimination [30,31]. We are not aware of any prior work that aims to audit the use of specific data sources to train a model.
Auditing Data Provenance in Text-Generation Models
Deep learning-based, text-generation models for word prediction, translation, and dialog generation are core components of many popular online services. We demonstrated that these models memorize their training data. This memorization does not appear to manifest in reduced test accuracy, which is a symptom of "conventional" overfitting, but is reflected instead in how they rank the candidate words they generate. We developed a black-box auditing method that enables users to check if their chats, messages, or comments have been used to train someone else's model. Our auditing method, based on a new flavor of membership inference that exploits memorization in text-generation models, is very effective. More powerful auditing algorithms may be possible if the auditor has access to the model's parameters and can observe its internal representations rather than just output predictions. This is a topic for future work.
Auditing Data Provenance in Text-Generation Models
We view the results of this paper as essentially positive, demonstrating how memorization in ML models can help detect unauthorized uses of sensitive personal data and ensure compliance with GDPR and other data-protection policies and regulations.
Auditing Data Provenance in Text-Generation Models
In this appendix, we provide the pseudo-code of the auditing algorithm and describe all model architectures, configurations, and hyper-parameters needed to reproduce the results.
Auditing Data Provenance in Text-Generation Models
Algorithm 1 is the pseudo-code for the auditing process. Function AuditMembership is used to audit if the user's data D u was included in the training data of the target model f . The auditor first trains an audit model using function TrainAuditModel. He then extracts histogram features h u on (possibly sampled) user's data using function HistogramFeature. Finally, the audit model predicts if the user's data was used to train f given the extracted features h u .
Auditing Data Provenance in Text-Generation Models
Function TrainAuditModel is the procedure for training the audit model. The auditor first trains k shadow models with a randomly sampled set of users, collects their outputs, extracts feature vectors for the training and test users, and labels these feature vectors accordingly. The auditor then trains a binary classifier based on the labeled feature vectors.
Auditing Data Provenance in Text-Generation Models
Function HistogramFeatureextracts features from the predicted ranks, as described in Section 3. The feature vector is a d-dimensional vector representing the histogram of predicted ranks. The ith entry of the vector is the count of ranks in the range [(i − 1) · b, i · b], where b is the histogram bin size.
Auditing Data Provenance in Text-Generation Models
When the number of queries to the target model is limited to m, the auditor uses function SampleQueries to sample a subset of the user's data. The auditor can choose either m texts at random, or m texts with the smallest word frequencies.
Auditing Data Provenance in Text-Generation Models
B EXPERIMENT SETUP B.1 Target models Next-word prediction. We use a one-layer long short-term memory (LSTM) [12] as the target model. LSTM is a more complicated RNN that can capture the long-term dependency in the sequence. The input sequence of tokens is first mapped to a sequence of embeddings. The embedding is then fed to the LSTM that learns a hidden representation for the context for predicting the next word.
Auditing Data Provenance in Text-Generation Models
Neural machine translation. We use a sequence-to-sequence target model with the attention module as described in [24]. Both the encoder and the decoder are one-layer LSTMs that operate on the embedding of source tokens and target tokens. The attention module adds an additional layer that operates on all hidden representations in the encoder LSTM and helps the decoder determine where to pay attention in the source texts when predicting a token in the target language.
Auditing Data Provenance in Text-Generation Models
Dialog generation. We use a sequence-to-sequence model without the attention module. The encoder and the decoder are one-layer LSTMs.
Auditing Data Provenance in Text-Generation Models
Target models. We train the word-prediction model on the comments of 300 randomly selected users from the Reddit dataset. We
Auditing Data Provenance in Text-Generation Models
We train the translation and dialog-generation models on 300 randomly selected users from SATED and Dialogs, respectively. We set both the embedding dimension and LSTM hidden-representation size in the encoder and decoder to 128. We use the Adam optimizer with the learning rate set to 1e-3, batch size to 20, and the number of training epochs to 30. For all datasets, we fix the vocabulary to the most frequent 5,000 tokens in the training texts. Tokens not in the vocabulary are replaced with a special <UNK> token. To prevent overfitting, we add dropout with 0.5 rate to all hidden layers of all models.
Auditing Data Provenance in Text-Generation Models
Shadow models. For the experiments in Section 4.3, we construct shadow models using different hyper-parameters than the target models. On all tasks, we used Gated Recurrent Units (GRU) [6] instead of LSTM. The size of hidden units and embedding is set to 64, 96, 128, 160, . . . , 352 for the shadow models. We optimize the shadow models using momentum SGD with the learning rate set to 0.01, momentum set to 0.9, and number of training epochs to 50.
Auditing Data Provenance in Text-Generation Models
Target and shadow models. All target and shadow models were implemented with Keras 6 using TensorFlow [1] backend.
Auditing Data Provenance in Text-Generation Models
Audit model. We use linear SVM implemented in LIBLINEAR [10] to train the audit model with the default hyper-parameters. 7 Hardware. All models were trained on a machine with 3 NVIDIA Titan X GPUs, 8-core Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz and 94 GBs of RAM.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Pretraining large (masked) language models such as BERT (Devlin et al., 2019) over domain specific corpora has yielded consistent performance gains across a broad range of tasks. In biomedical NLP, this has often meant pretraining models over collections of Electronic Health Records (EHRs) (Alsentzer et al., 2019). For example, Huang et al. (2019) showed that pretraining models over EHR data improves performance on clinical predictive tasks. Given their empirical utility, and the fact that pretraining large networks requires a nontrivial amount of compute, there is a natural desire to equal contribution. 1 https://github.com/elehman16/ exposing_patient_data_release.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
share the model parameters for use by other researchers in the community.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
However, in the context of pretraining models over patient EHR, this poses unique potential privacy concerns: Might the parameters of trained models leak sensitive patient information? In the United States, the Health Insurance Portability and Accountability Act (HIPAA) prohibits the sharing of such text if it contains any reference to Protected Health Information (PHI). If one removes all reference to PHI, the data is considered "deidentified", and is therefore legal to share.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
While researchers may not directly share nondeidentified text, 2 it is unclear to what extent models pretrained on non-deidentified data pose privacy risks. Further, recent work has shown that general purpose large language models are prone to memorizing sensitive information which can subsequently be extracted (Carlini et al., 2020). In the context of biomedical NLP, such concerns have been cited as reasons for withholding direct publication of trained model weights (McKinney et al., 2020). These uncertainties will continue to hamper dissemination of trained models among the broader biomedical NLP research community, motivating a need to investigate the susceptibility of such models to adversarial attacks.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
This work is a first step towards exploring the potential privacy implications of sharing model weights induced over non-deidentified EHR text. We propose and run a battery of experiments intended to evaluate the degree to which Transformers (here, BERT) pretrained via standard masked language modeling objectives over notes in EHR might reveal sensitive information (Figure 1). 3 Figure 1: Overview of this work. We explore initial strategies intended to extract sensitive information from BERT model weights estimated over the notes in Electronic Health Records (EHR) data.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We find that simple methods are able to recover associations between patients and conditions at rates better than chance, but not with performance beyond that achievable using baseline condition frequencies. This holds even when we enrich clinical notes by explicitly inserting patient names into every sentence. Our results using a recently proposed, more sophisticated attack based on generating text (Carlini et al., 2020) are mixed, and constitute a promising direction for future work.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We use the Medical Information Mart for Intensive Care III (MIMIC-III) English dataset to conduct our experiments (Johnson et al., 2016). We follow prior work (Huang et al., 2019) and remove all notes except for those categorized as 'Physician', 'Nursing', 'Nursing/Others', or 'Discharge Summary' note types. The MIMIC-III database was deidentified using a combination of regular expressions and human oversight, successfully removing almost all forms of PHI (Neamatullah et al., 2008). All patient first and last names were replaced with [Known First Name ...] and [Known Last Name ...] pseudo-tokens respectively.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We are interested in quantifying the risks of releasing contextualized embedding weights trained on non-deidentified text (to which one working at hospitals would readily have access). To simulate the existence of PHI in the MIMIC-III set, we randomly select new names for all patients (Stubbs et al., 2015). 4 Specifically, we replaced [Known First Name] and [Known Last Name] with names sampled from US Census data, randomly sampling first names (that appear at least 10 times in census data) and last names (that appear at least 400 times). 5 This procedure resulted in 11.5% and 100% of patients being assigned unique first and last names, respectively. While there are many forms of PHI, we are primarily interested in recovering name and condition pairs, as the ability to infer with some certainty the specific conditions that a patient has is a key privacy concern. This is also consistent with prior work on static word embeddings learned from EHR (Abdalla et al., 2020).
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Notes in MIMIC-III do not consistently explicitly reference patient names. First or last names are mentioned in at least one note for only 27,906 (out of 46,520) unique patients. 6 Given that we cannot reasonably hope to recover information regarding tokens that the model has not observed, in this work we only consider records corresponding to these 27,906 patients. Despite comprising 61.3% of the total number of patients, these 27,906 patients are associated with the majority (82.6%) of all notes (1,247,291 in total). Further, only 10.2% of these notes contain at least one mention of a patient's first or last name.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Of the 1,247,291 notes considered, 17,044 include first name mentions, and 220,782 feature last name mentions. Interestingly, for records corresponding to the 27,906 patients, there are an additional 18,345 false positive last name mentions and 29,739 false positive first name mentions; in these cases the name is also an English word (e.g., 'young'). As the frequency with which patient names are mentioned explicitly in notes may vary by hospital conventions, we also present semisynthetic results in which we insert names into notes such that they occur more frequently.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
As a first attempt to evaluate the risk of BERT leaking sensitive information, we define the following task: Given a patient name that appears in the set of EHR used for pretraining, query the model for the conditions associated with this patient. Operationally this requires defining a set of conditions against which we can test each patient. We consider two general ways of enumerating conditions: (1) Using International Classification of Diseases, revision 9 (ICD-9) codes attached to records, and (2) Extracting condition strings from the free-text within records. 7 Specifically, we experiment with the following variants.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
[ICD-9 Codes] We collect all ICD-9 codes associated with individual patients. ICD-9 is a standardized global diagnostic ontology maintained by the World Health Organization. Each code is also associated with a description of the condition that it represents. In our set of 27,906 patients, we observe 6,841 unique ICD-9 codes. We additionally use the short ICD-9 code descriptions, which comprise an average of 7.03 word piece tokens per description (under the BERT-Base tokenizer). On average, patient records are associated with 13.6 unique ICD-9 codes.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
[MedCAT] ICD-9 codes may not accurately reflect patient status, and may not be the ideal means of representing conditions. Therefore, we also created lists of conditions to associate with patients by running the MedCAT concept annotation tool (Kraljevic et al., 2020) over all patient notes. We only keep those extracted entities that correspond to a Disease / Symptom, which we use to normalize condition mentions and map them to their UMLS (Bodenreider, 2004) CUI and description. This yields 2,672 unique conditions from the 27,906 patient set. On average, patients are associated with an average of 29.5 unique conditions, and conditions comprise 5.37 word piece tokens.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Once we have defined a set of conditions to use for an experiment, we assign binary labels to patients indicating whether or not they are associated with each condition. We then aim to recover the conditions associated with individual patients.  [SEP]". This overrepresentation of names should make it easier to recover information about patients.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We also explore whether PHI from the MIMIC database can be retrieved using static word embeddings derived via CBoW and skip-gram word2vec models (Mikolov et al., 2013). Here, we follow prior work (Abdalla et al. 2020; this was conducted on a private set of EHR, rather than MIMIC). We induce embeddings for (multi-word) patient names and conditions by averaging constituent word representations. We then calculate cosine similarities between these patient and condition embeddings (See Section 6.3). 8 https://github.com/kexinhuang12345/ clinicalBERT/blob/master/notebook/ pretrain.ipynb 9 https://github.com/google-research/ bert
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We first test the degree to which we are able to retrieve conditions associated with a patient, given their name. (We later also consider a simpler task: Querying the model as to whether or not it observed a particular patient name during training.) All results presented are derived over the set of 27,906 patients described in Section 4.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
The following methods output scalars indicating the likelihood of a condition, given a patient name and learned BERT weights. We compute metrics with these scores for each patient, measuring our ability to recover patient/condition associations. We aggregate metrics by averaging over all patients. We report AUCs and accuracy at 10 (A@10), i.e., the fraction of the top-10 scoring conditions that the patient indeed has (according to the reference set of conditions for said patient).
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We attempt to reveal information memorized during pretraining using masked template strings. The idea is to run such templates through BERT, and observe the rankings induced over conditions (or names). 10 This requires specifying templates.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Generic Templates We query the model to fill in the masked tokens in the following sequence:
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Here, Mr. and Mrs. are selected according to the gender of the patient as specified in the MIMIC corpus. 11 The [MASK] + above is actually a sequence of [MASK] tokens, where the length of this sequence depends on the length of the tokenized condition for which we are probing.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Given a patient name and condition, we compute the perplexity (PPL) for condition tokens as candidates to fill the template mask. For example, if we wanted to know whether a patient ("John Doe") was associated with a particular condition ("MRSA"), we would query the model with the following (populated) template: "[CLS] Mr. John Doe is a yo patient with [MASK] [SEP]" and measure the perplexity of "MRSA" assuming the [MASK] input token position. For multiword conditions, we first considered taking an average PPL over constituent words, but this led to  counterintuitive results: longer conditions tend to yield lower PPL. In general, multi-word targets are difficult to assess as PPL is not well-defined for masked language models like BERT (Jiang et al., 2020a;Salazar et al., 2020). Therefore, we bin conditions according to their wordpiece length and compute metrics for bins individually. This simplifies our analysis, but makes it difficult for an attacker to aggregate rankings of conditions with different lengths.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We use the generic template method to score ICD-9 or MedCAT condition descriptions for each patient. We report the performance (averaged across length bins) achieved by this method in Table 2, with respect to AUC and A@10. This straightforward approach fares better than chance, but worse than a baseline approach of assigning scores equal to the empirical frequencies of conditions. 12 Perhaps this is unsurprising for MIMIC- 12 We note that these frequencies are derived from the MIMIC data, which affords an inherent advantage, although it seems likely that condition frequencies derived from other data sources would be similar. We also note that some very common conditions are associated with many patients -see Appendix Figures A1 and A2 -which may effectively 'inflate' the AUCs achieved by the frequency baseline. III, as only 0.3% of sentences explicitly mention a patient's last name.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
If patient names appeared more often in the notes, would this approach fare better? To test this, we present results for the Name Insertion and Template Only variants in Table 2. Recall that for these we have artificially increased the number of patient names that occur in the training data; this should make it easier to link conditions to names. The Template Only variant yields better performance for MedCAT labels, but still fares worse than ranking conditions according to empirical frequencies. However, it may be that the frequency baseline performs so well simply due to many patients sharing a few dominating conditions. To account for this, we additionally calculate performance using the Template Only model on MedCAT conditions that fewer than 50 patients have. We find that the AUC is 0.570, still far lower than the frequency baseline of 0.794 on this restricted condition set.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Other templates, e.g., the most common phrases in the train set that start with a patient name and end with a condition, performed similarly.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Masking the Condition (Only) Given the observed metrics achieved by the 'frequency' baseline, we wanted to establish whether models are effectively learning to (poorly) approximate condition frequencies, which might in turn allow for the better than chance AUCs in Table 2 Table 3, we report AUCs, A@10 scores, and Spearman correlations with frequency scores (again, averaged across length bins). The latter are low, suggesting that the model rankings differ from overall frequencies.  Table 3: Average AUC, A@10 and Spearman correlations over conditions binned by description length.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Correlations are w/r/t empirical condition frequencies.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
The above token prediction infill setup attacks the model only via fixed templates. But the induced representations might implicitly encode sensitive information that happens to not be readily exposed by the template. We therefore also investigate a probing setup (Alain and Bengio, 2017; Bouraoui et al., 2020), in which a representation induced by a pretrained model is provided to a second probing model which is trained to predict attributes of interest. Unlike masked token prediction, probing requires that the adversary have access to a subset of training data to associate targets with representations.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We train an MLP binary classifier on top of the encoded CLS token from the last layer of BERT. The probe is trained to differentiate positive instances (conditions the patient has) from negative examples (conditions the patient does not have) on a randomly sampled subset of 5000 patients (we downsample the negative class for balancing). We use the following template to encode the patient- We run experiments on the Base, Large, and Name Insertion models. These models achieve strong AUCs, nearly matching the frequency baseline performance in Table 2. 13 However, it appears that removing the patient's name and simply encoding the condition to make a binary prediction yields similar (in fact, slightly better) per-13 Though the AUCs for the probing are calculated over a randomly sampled test subset of the full data used in Table 2  formance. This suggests that the model is mostly learning to approximate condition frequencies.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
The standard probing setup encourages the model to use the frequency of target conditions to make predictions. To address this, we also consider a variant in which we probe for only individual conditions, rather than defining a single model probing for multiple conditions, as above. This means we train independent models per condition, which can then be used to score patients with respect to said conditions. To train such models we upsample positive examples such that we train on balanced sets of patients for each condition. 14 This approach provides results for each condition which vary in frequency. To assess the comparative performance of probes over conditions of different prevalence, we group conditions into mutually exclusive bins reflecting frequency (allowing us to analyze differences in performance, e.g., on rare conditions). We group conditions by frequencies, from rarest (associated with 2-5 patients) to most common (associated with >20 patients). We randomly sample 50 conditions from each of these groups, and train an MLP classifier on top of the encoded CLS token from the last layer in BERT (this results in 50 different models per group, i.e., 200 independent models). We measure, in terms of AUC and A@10, whether the probe for a condition return comparatively higher scores for patients that have that condition.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We report results in Table 5. Except for the rarest conditions (associated with <5 patients), these models achieve AUCs that are at best modestly better than chance, with all A@10 metrics  Table 5: Probing results (AUCs) for conditions with different frequencies. We make predictions for conditions using independent models based on BERTencoded CLS tokens. We use a 50/50 train/test split over patients (results are over the test set). Columns correspond to conditions of different frequencies, with respect to the number of patients with whom they are associated (headers provide ranges). All A@10 ≈ 0.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
≈0. In sum, these models do not meaningfully recover links between patients and conditions.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Prior work (Abdalla et al., 2020) has demonstrated that static word vectors can leak information: The cosine similarities between learned embeddings of patient names and conditions are on average significantly smaller than the similarities between patient names and conditions they do not have. We run a similar experiment to investigate whether contextualized embeddings similarly leak information (and also to assess the degree to which this holds on the MIMIC corpus as a point of comparison). We calculate the average cosine similarity between learned embeddings of patient names and those of positive conditions (conditions that the patient has) minus negative conditions (those that they do not have). Conditions and names span multiple tokens; we perform mean pooling over these to induce embeddings. Here again we evaluate on the aforementioned set of 27,906 patients. We report results for BERT and word2vec (CBoW and SkipGram; Mikolov et al. 2013) in Table 6. 15 Values greater than zero here suggest leakage, as this implies that patient names end up closer to conditions that patients have, relative to those that they do not. Even when trained over the Name Insertion data (which we manipulated to frequently mention names), we do not observe leakage from the contextualized embeddings. 15 We provide additional results in the Appendix, including results for alternative pooling strategies and results on the original MIMIC dataset; all yield qualitatively similar results.  Table 6: Differences in (a) similarities between patient names and conditions they have, and (b) similarities between patient names and conditions they do not have. Static embeddings are 200 dimensional; we train these for 10 epochs. For BERT models, we use 10k patients rather than the ∼28k due to compute constraints.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Here we try something even more basic: We attempt to determine whether a pretrained model has seen a particular patient name in training. The ability to reliably recover individual patient names (even if not linked to specific conditions) from BERT models trained over EHR data would be concerning if such models were to be made public.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We consider a number of approaches to this task.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
[NAME] [SEP]) using BERT and train a Logistic Regression classifier that consumes resultant CLS representations and predicts whether the corresponding patient has been observed in training.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
As mentioned above, patient names are explicitly mentioned in notes for 27,906 patients; these constitute our positive examples, and the remaining patients (of the 46,520) are negative examples. We split the data into equally sized train and test sets. We report results in Table 7. To contextualize these results, we also run this experiment on the standard BERT base model (which is not trained on this EHR data). We observe that the AUCs are near chance, and that the performance of the standard BERT base model is relatively similar to that of the Regular and Large base models, despite the fact that the standard BERT base model has not seen any notes from MIMIC.   
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Given a first name, can we predict whether we have seen a corresponding last name? More specifically, we mask out a patient's last name (but not their first) in the template "[CLS] [First Name] [MASK] + [SEP]" and record the perplexity of the target last name. We take as the set of outputs all 46,520 patient names in the corpus. We can also flip this experiment, masking only first names. This is intuitively quite difficult, as only 10K / 77M sentences (0.013%) contain both the patient's first and last name. This number includes first and last name mentions that are also other English words (e.g. "young"). Results are reported in Table 8. We do observe reasonable signal in the semi-synthetic Name Insertion and Template Only variants.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Recent work by Carlini et al. (2020) showed that GPT-2 (Radford et al., 2019) memorizes training data, and proposed techniques to efficiently recover sensitive information from this model (e.g., email addresses). They experimented only with large, auto-regressive language models (i.e., GPT-2), but their techniques are sufficiently general for us to use here. More specifically, to apply their approaches to a BERT-based model 16 we must be able to sample text from BERT, which is complicated by the fact that it is not a proper (autoregressive) language model. To generate outputs from BERT we therefore followed a method proposed in prior work (Wang and Cho, 2019). This entails treating BERT as a Markov random field language model and using a Gibbs sampling procedure to generate outputs. We then analyze these outputs from (a) our regular BERT-based model trained on MIMIC; (b) the Name Insertion model, and; (c) a standard BERT Base model (Devlin et al., 2019). We generate 500k samples from each, each sample consisting of 100 wordpiece tokens.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Comparator Model Perplexity Following Carlini et al. (2020), we attempt to identify which pieces of generated text are most likely to contain memorized names (in this case, from EHR). To this end, we examine segments of the text in which the difference in likelihood of our trained BERT model versus the standard BERT-base model (Devlin et al., 2019) is high. For the samples generated from the standard BERT-base model (not trained on MIMIC), we use our ClinicalBERT model as the comparator. 17 Using an off-the-shelf NER tagger (Honnibal et al., 2020), we identify samples containing name tokens.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
For each sample, we mask name tokens individually and calculate their perplexity under each of the the respective models. We take the difference between these to yield a score (sequences with high likelihood under the trained model and low likelihood according to the general-domain BERT may contain vestiges of training data) and use it to rank our extracted names; we then use this to calculate A@100.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
As expected, the Name Insertion model produced more names than the Base model, with approximately 60% of all sentences containing a name (not necessarily in MIMIC). Additionally, the A@100 of the Name Insertion model substantially outperforms the Base model. However, when we use spaCy to examine sentences that contain both a condition and a patient's name (of the 27,906), we find that 23.5% of the time the pa-  Table 9: Results over texts generated by the Base and Name Insertion models. The 'Sent. with Name' column is percentage of extracted sentences that contain a name token. The First and Last name columns show what percent of unique names produced are in the MIMIC dataset. After re-ranking all unique names, we report the percentage of top 100 names that belong to a reidentified patient. Finally, The Name + Positive Condition displays what percent of sentences with a patient's name also contain one of their true (MedCAT) conditions. tient does indeed have a condition produced by the Base model. It is unclear to what extent this reflects memorization of concrete patient-condition pairs per se, as opposed to learning more diffused patient-agnostic distributions of conditions in the MIMIC dataset. The corresponding statistic for the Name Insertion variant (4.17%) may be low because this tends to produce poor quality outputs with many names, but not many conditions. This is an intriguing result that warrants further research. However, we caution that these generation experiments are affected by the accuracy of NER taggers used. For example, many of the extracted names tend to also be generic words (e.g., 'young', 'date', 'yo', etc.) which may artificially inflate our scores. In addition, MedCAT sometimes uses abbreviations as conditions, which may also yield 'false positives' for conditions.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
This work has important limitations. We have considered only relatively simple "attacks", based on token in-filling and probing. Our preliminary results using the more advanced generation approach (inspired by Carlini et al. 2020) is a promising future direction, although the quality of generation from BERT -which is not naturally a language model -may mitigate this. This highlights a second limitation: We have only considered BERT, as it is currently the most common choice of pretrained Transformer in the bioNLP community. Auto-regressive models such as GPT-2 may be more prone to memorization. Larger models (e.g., T5  or GPT-3 (Brown et al., 2020)) are also likely to heighten the risk of data leakage if trained over EHR.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Another limitation is that we have only considered the MIMIC-III corpus here, and the style in which notes are written in this dataset -names appear very infrequently -likely renders it particularly difficult for BERT to recover implicit as-sociations between patient names and conditions. We attempted to address this issue with the semisynthetic Name Insertion variant, where we artificially inserted patient names into every sentence; this did not yield qualitatively different results for most experiments. Nonetheless, it is possible that experiments on EHR datasets from other hospitals (with different distributions over tokens and names) would change the degree to which one is able to recover PHI.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Finally, these results for BERT may change under different masking strategies -for example, dynamic masking (Liu et al., 2019) or choice of tokenizer. Both of these may affect memorization and extraction method performance.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
We have performed an initial investigation into the degree to which large Transformers pretrained over EHR data might reveal sensitive personal health information (PHI). We ran a battery of experiments in which we attempted to recover such information from BERT model weights estimated over the MIMIC-III dataset (into which we artificially reintroduced patient names, as MIMIC is deidentified). Across these experiments, we found that we were mostly unable to meaningfully expose PHI using simple methods. Moreover, even when we constructed a variant of data in which we prepended patient names to every sentence prior to pretraining BERT, we were still unable to recover sensitive information reliably. Our initial results using more advanced techniques based on generation (Carlini et al. 2020; Table 9) are intriguing but inconclusive at present.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Our results certainly do not rule out the possibility that more advanced methods might reveal PHI. But, these findings do at least suggest that doing so is not trivial. To facilitate further research, we make our experimental setup and baseline probing models available: https://github.com/ elehman16/exposing_patient_data_release.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
This work has ethical implications relevant to patient privacy. HIPAA prohibits the distribution of PHI, for good reason. Without this type of privacy law, patient information, for example, could be passed on to a lender and be used to deny a patient's application for mortgages or credit card. It is therefore essential that patient information remain private. This raises an important practical concerning methods in NLP that we have sought to address: Does releasing models pretrained over sensitive data pose a privacy risk? While we were unable to reliably recover PHI in this work, we hope that this effort encourages the community to develop more advanced attacks to probe this potential vulnerability. We would still advise researchers to err on the side of caution and only consider releasing models trained over fully deidentified data (e.g. MIMIC). Figure A1: A distribution of ICD-9 codes and how many patients (of the 27K) have each condition. All bin end values are not inclusive.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
As mentioned previously, we follow most of the hyperparameters stated in (Huang et al., 2019). The code presented in Huang et al. (2019) accidentally left out all notes under the category 'Nursing/Other'; we added these back in, in addition to any notes that fell under the 'Discharge Summaries' summary category. Our dataset consists of approximately 400M words (ignoring wordpieces). The number of epochs (following Devlin et al. 2019) can be calculated as num steps · batch size · tokens per seq total number of tokens , which at batch size of 128 and sequence length of 128, comes out to 40 epochs if trained for 1M steps (in the ++ models). For standard models, it comes out to 29 epochs. We used cloud TPUs (v2 and v3) to train our models. All experiments are run on a combination of V100, Titan RTX and Quadro RTX 8000 GPUs.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
In Appendix Figures A1 and A2, we can see the distribution of ICD-9 and MedCAT conditions across patients. With respect to the ICD-9 codes, there are only 4 conditions that are shared across 10,000+ patients. This number is 32 for MedCAT conditions.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
In addition to the results in Table 2, we report all Spearman coefficients, relative to the frequency of conditions (in Appendix Table 10). We additionally report results for Base++, Large++, and   Pubmed-Base models. With respect to AUC, these models all perform worse than the Regular Large model. Additionally, in Appendix Figure  A3, we can see how experiment results change with respect to the length of conditions (owing, as we mentioned in the main text, to complications in computing likelihoods of varying length sequences under MLMs).
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
In addition to the results in Table 3, we show results for Base++, Large++, and Pubmed-Base models. Interestingly, the Large and Pubmed-Base model's perform better when names are not included. We see the biggest difference between Appendix Table 10 and 11 in the Templates Only model, suggesting that this model is memorizing the relationship between patients and conditions.   
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
In this experiment, we randomly sample 10,000 patients from our 27,906 patient set (due to computational constraints), of which we keep 5,000 for training and 5,000 for testing. For each of these patient names and every condition in our universe of conditions, we construct the previously specified template and assign it a binary label indicating whether the patient have that condition or not.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Since the negative class is over-represented by a large amount in this training set, we use downsampling to balance our data. We map each of these templates to their corresponding CLS token embedding. We use the embeddings for templates associated with training set patients to train a MLP classifier implemented in Scikit-Learn (Pedregosa et al., 2011) (Note we did not use on a validation set here). We used a hidden layer size of 128 with default hyperparameters.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
At test time, for each of the 5000 patients in test set and each condition, we calculate the score using this MLP probe and compute our metrics with respect to the true label associated with that patient-condition pair.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
In this experiment, we samples 50 conditions from each of the 4 frequency bins. For each condition, we trained a probe to distinguish between patients that have that condition vs those that do not. This experiment differs from the preceding fill-in-theblank and probing experiments: Here we compute an AUC for each condition (indicating whether the probe discriminates between patients that have a particular condition and those that do not),whereas in the fill-in-the-blank experiments we computed AUCs per patient.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
For probing individual conditions, we used an MLP classifier implemented in Scikit-Learn (Pedregosa et al., 2011). We did not evaluate on a validation set. We used a hidden layer size of 128 with default hyperparameters. All experiments were only run once. For the Regular BERT model, we additionally experimented with backpropagating through the BERT weights, but found that this made no difference in predictive performance.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
All versions of Skipgram and CBoW (Mikolov et al., 2013) were trained for 10 epochs using gensim library (Řehůřek and Sojka, 2010), used a vector size of 200, and a window size of 6. We only trained one variant of each W2V model. For BERT models, we used the last layer wordpiece embeddings. For word embedding models, we ran this experiment on whole reidentified patient set, whereas for BERT models, we sampled 10K patients. We report averages over the patients. In addition to the mean-pool collapsing of conditions, we also try 'Max-Pooling' and a variant we label as 'All Pairs Pooling'. We present results from all cosine-similarity experiments in Appendix Ta- Table 12: Similarity for Positive Conditions -Negative Conditions. All experiments are performed using ICD-9 codes. Max and Average refer to max-pooling and average-pooling over multiple embeddings, respectively. "All" entails the following: For every word piece in the name, find the cosine similarity for every word piece in the condition; then, use the largest cosine similarity. All word embedding models are trained for 10 epochs, with dimensionality 200. ble 12. The mean pooling results in Table 6 seem to outperform the alternative pooling mechanisms presented here.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
To see if our BERT models are able to recognize the patient names that appear in training data, we train a linear probe on top of names encoded via BERT. We train this Linear Regression classifier using all default parameters from Scikit-Learn (10,000 max steps) (Pedregosa et al., 2011). We did not evaluate on a validation set. Each experiment was only run once.  A.9 Does observing part of a name reveal more information?
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Similar to the results in Table 8, we report results on the Base++, Large++, and Pubmed-Base models (Appendix Table 13). We find no significant difference between these results and the ones reported in Table 8.
Membership Inference Attacks against Machine Learning Models
Abstract-We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on.
Membership Inference Attacks against Machine Learning Models
We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.
Membership Inference Attacks against Machine Learning Models
Machine learning is the foundation of popular Internet services such as image and speech recognition and natural language translation. Many companies also use machine learning internally, to improve marketing and advertising, recommend products and services to users, or better understand the data generated by their operations. In all of these scenarios, activities of individual users-their purchases and preferences, health data, online and offline transactions, photos they take, commands they speak into their mobile phones, locations they travel to-are used as the training data.
Membership Inference Attacks against Machine Learning Models
Internet giants such as Google and Amazon are already offering "machine learning as a service." Any customer in possession of a dataset and a data classification task can upload this dataset to the service and pay it to construct a model. The service then makes the model available to the customer, typically as a black-box API. For example, a mobile-app maker can use such a service to analyze users' activities and query the resulting model inside the app to promote in-app purchases to users when they are most likely to respond. Some machinelearning services also let data owners expose their models to external users for querying or even sell them.
Membership Inference Attacks against Machine Learning Models
Our contributions. We focus on the fundamental question known as membership inference: given a machine learning model and a record, determine whether this record was used as * This research was performed while the author was at Cornell Tech. part of the model's training dataset or not. We investigate this question in the most difficult setting, where the adversary's access to the model is limited to black-box queries that return the model's output on a given input. In summary, we quantify membership information leakage through the prediction outputs of machine learning models.
Membership Inference Attacks against Machine Learning Models
To answer the membership inference question, we turn machine learning against itself and train an attack model whose purpose is to distinguish the target model's behavior on the training inputs from its behavior on the inputs that it did not encounter during training. In other words, we turn the membership inference problem into a classification problem.
Membership Inference Attacks against Machine Learning Models
Attacking black-box models such as those built by commercial "machine learning as a service" providers requires more sophistication than attacking white-box models whose structure and parameters are known to the adversary. To construct our attack models, we invented a shadow training technique. First, we create multiple "shadow models" that imitate the behavior of the target model, but for which we know the training datasets and thus the ground truth about membership in these datasets. We then train the attack model on the labeled inputs and outputs of the shadow models.
Membership Inference Attacks against Machine Learning Models
We developed several effective methods to generate training data for the shadow models. The first method uses black-box access to the target model to synthesize this data. The second method uses statistics about the population from which the target's training dataset was drawn. The third method assumes that the adversary has access to a potentially noisy version of the target's training dataset. The first method does not assume any prior knowledge about the distribution of the target model's training data, while the second and third methods allow the attacker to query the target model only once before inferring whether a given record was in its training dataset.
Membership Inference Attacks against Machine Learning Models
Our inference techniques are generic and not based on any particular dataset or model type. We evaluate them against neural networks, as well as black-box models trained using Amazon ML and Google Prediction API. All of our experiments on Amazon's and Google's platforms were done without knowing the learning algorithms used by these services, nor the architecture of the resulting models, since Amazon and Google don't reveal this information to the customers. For our evaluation, we use realistic classification tasks and standard model-training procedures on concrete datasets of images, retail purchases, location traces, and hospital inpatient stays. In addition to demonstrating that membership inference attacks are successful, we quantify how their success relates to the classification tasks and the standard metrics of overfitting.
Membership Inference Attacks against Machine Learning Models
Inferring information about the model's training dataset should not be confused with techniques such as model inversion that use a model's output on a hidden input to infer something about this input [17] or to extract features that characterize one of the model's classes [16]. As explained in [27] and Section IX, model inversion does not produce an actual member of the model's training dataset, nor, given a record, does it infer whether this record was in the training dataset. By contrast, the membership inference problem we study in this paper is essentially the same as the well-known problem of identifying the presence of an individual's data in a mixed pool given some statistics about the pool [3], [15], [21], [29]. In our case, however, the goal is to infer membership given a black-box API to a model of unknown structure, as opposed to explicit statistics.
Membership Inference Attacks against Machine Learning Models
Our experimental results show that models created using machine-learning-as-a-service platforms can leak a lot of information about their training datasets. For multi-class classification models trained on 10,000-record retail transaction datasets using Google's and Amazon's services in default configurations, our membership inference achieves median accuracy of 94% and 74%, respectively. Even if we make no prior assumptions about the distribution of the target model's training data and use fully synthetic data for our shadow models, the accuracy of membership inference against Google-trained models is 90%. Our results for the Texas hospital discharge dataset (over 70% accuracy) indicate that membership inference can present a risk to health-care datasets if these datasets are used to train machine learning models and access to the resulting models is open to the public. Membership in such datasets is highly sensitive.
Membership Inference Attacks against Machine Learning Models
We discuss the root causes that make these attacks possible and quantitatively compare mitigation strategies such as limiting the model's predictions to top k classes, decreasing the precision of the prediction vector, increasing its entropy, or using regularization while training the model.
Membership Inference Attacks against Machine Learning Models
In summary, this paper demonstrates and quantifies the problem of machine learning models leaking information about their training datasets. To create our attack models, we developed a new shadow learning technique that works with minimal knowledge about the target model and its training dataset. Finally, we quantify how the leakage of membership information is related to model overfitting.
Membership Inference Attacks against Machine Learning Models
Machine learning algorithms help us better understand and analyze complex data. When the model is created using unsupervised training, the objective is to extract useful features from the unlabeled data and build a model that explains its hidden structure. When the model is created using supervised training, which is the focus of this paper, the training records (as inputs of the model) are assigned labels or scores (as outputs of the model). The goal is to learn the relationship between the data and the labels and construct a model that can generalize to data records beyond the training set [19]. Modeltraining algorithms aim to minimize the model's prediction error on the training dataset and thus may overfit to this dataset, producing models that perform better on the training inputs than on the inputs drawn from the same population but not used during the training. Many regularization techniques have been proposed to prevent models from becoming overfitted to their training datasets while minimizing their prediction error [19].
Membership Inference Attacks against Machine Learning Models
Supervised training is often used for classification and other prediction tasks. For example, a retailer may train a model that predicts a customer's shopping style in order to offer her suitable incentives, while a medical researcher may train a model to predict which treatment is most likely to succeed given a patient's clinical symptoms or genetic makeup.
Membership Inference Attacks against Machine Learning Models
Machine learning as a service. Major Internet companies now offer machine learning as a service on their cloud platforms. Examples include Google Prediction API, 1 Amazon Machine Learning (Amazon ML), 2 Microsoft Azure Machine Learning (Azure ML), 3 and BigML. 4 These platforms provide simple APIs for uploading the data and for training and querying models, thus making machine learning technologies available to any customer. For example, a developer may create an app that gathers data from users, uploads it into the cloud platform to train a model (or update an existing model with new data), and then uses the model's predictions inside the app to improve its features or better interact with the users. Some platforms even envision data holders training a model and then sharing it with others through the platform's API for profit. 5 The details of the models and the training algorithms are hidden from the data owners. The type of the model may be chosen by the service adaptively, depending on the data and perhaps accuracy on validation subsets. Service providers do not warn customers about the consequences of overfitting and provide little or no control over regularization. For example, Google Prediction API hides all details, while Amazon ML provides only a very limited set of pre-defined options (L1-or L2-norm regularization). The models cannot be downloaded and are accessed only through the service's API. Service providers derive revenue mainly by charging customers for queries through this API. Therefore, we treat "machine learning as a service" as a black box. All inference attacks we demonstrate in this paper are performed entirely through the services' standard APIs.
Membership Inference Attacks against Machine Learning Models
Before dealing with inference attacks, we need to define what privacy means in the context of machine learning or, alternatively, what it means for a machine learning model to breach privacy.
Membership Inference Attacks against Machine Learning Models
A. Inference about members of the population A plausible notion of privacy, known in statistical disclosure control as the "Dalenius desideratum," states that the model should reveal no more about the input to which it is applied than would have been known about this input without applying the model. This cannot be achieved by any useful model [14].
Membership Inference Attacks against Machine Learning Models
A related notion of privacy appears in prior work on model inversion [17]: a privacy breach occurs if an adversary can use the model's output to infer the values of unintended (sensitive) attributes used as input to the model. As observed in [27], it may not be possible to prevent this "breach" if the model is based on statistical facts about the population. For example, suppose that training the model has uncovered a high correlation between a person's externally observable phenotype features and their genetic predisposition to a certain disease. This correlation is now a publicly known scientific fact that allows anyone to infer information about the person's genome after observing that person.
Membership Inference Attacks against Machine Learning Models
Critically, this correlation applies to all members of a given population. Therefore, the model breaches "privacy" not just of the people whose data was used to create the model, but also of other people from the same population, even those whose data was not used and whose identities may not even be known to the model's creator (i.e., this is "spooky action at a distance"). Valid models generalize, i.e., they make accurate predictions on inputs that were not part of their training datasets. This means that the creator of a generalizable model cannot do anything to protect "privacy" as defined above because the correlations on which the model is based-and the inferences that these correlations enable-hold for the entire population, regardless of how the training sample was chosen or how the model was created from this sample.
Membership Inference Attacks against Machine Learning Models
To bypass the difficulties inherent in defining and protecting privacy of the entire population, we focus on protecting privacy of the individuals whose data was used to train the model. This motivation is closely related to the original goals of differential privacy [13].
Membership Inference Attacks against Machine Learning Models
Of course, members of the training dataset are members of the population, too. We investigate what the model reveals about them beyond what it reveals about an arbitrary member of the population. Our ultimate goal is to measure the membership risk that a person incurs if they allow their data to be used to train a model.
Membership Inference Attacks against Machine Learning Models
The basic attack in this setting is membership inference, i.e., determining whether a given data record was part of the model's training dataset or not. When a record is fully known to the adversary, learning that it was used to train a particular model is an indication of information leakage through the model. In some cases, it can directly lead to a privacy breach. For example, knowing that a certain patient's clinical record was used to train a model associated with a disease (e.g, to determine the appropriate medicine dosage or to discover the genetic basis of the disease) can reveal that the patient has this disease.
Membership Inference Attacks against Machine Learning Models
We investigate the membership inference problem in the black-box scenario where the adversary can only supply inputs to the model and receive the model's output(s). In some situations, the model is available to the adversary indirectly. For example, an app developer may use a machine-learning service to construct a model from the data collected by the app and have the app make API calls to the resulting model. In this case, the adversary would supply inputs to the app (rather than directly to the model) and receive the app's outputs (which are based on the model's outputs). The details of internal model usage vary significantly from app to app. For simplicity and generality, we will assume that the adversary directly supplies inputs to and receives outputs from the black-box model.
Membership Inference Attacks against Machine Learning Models
Consider a set of labeled data records sampled from some population and partitioned into classes. We assume that a machine learning algorithm is used to train a classification model that captures the relationship between the content of the data records and their labels.
Membership Inference Attacks against Machine Learning Models
For any input data record, the model outputs the prediction vector of probabilities, one per class, that the record belongs to a certain class. We will also refer to these probabilities as confidence values. The class with the highest confidence value is selected as the predicted label for the data record. The accuracy of the model is evaluated by measuring how it generalizes beyond its training set and predicts the labels of other data records from the same population.
Membership Inference Attacks against Machine Learning Models
We assume that the attacker has query access to the model and can obtain the model's prediction vector on any data record. The attacker knows the format of the inputs and outputs of the model, including their number and the range of values they can take. We also assume that the attacker either (1) knows the type and architecture of the machine learning model, as well as the training algorithm, or (2) has black-box access to a machine learning oracle (e.g., a "machine learning as a service" platform) that was used to train the model. In the latter case, the attacker does not know a priori the model's structure or meta-parameters.
Membership Inference Attacks against Machine Learning Models
The attacker may have some background knowledge about the population from which the target model's training dataset was drawn. For example, he may have independently drawn samples from the population, disjoint from the target model's training dataset. Alternatively, the attacker may know some general statistics about the population, for example, the marginal distribution of feature values.
Membership Inference Attacks against Machine Learning Models
The setting for our inference attack is as follows. The attacker is given a data record and black-box query access to the target model. The attack succeeds if the attacker can correctly determine whether this data record was part of the model's training dataset or not. The standard metrics for attack accuracy are precision (what fraction of records inferred as members are indeed members of the training dataset) and The attacker queries the target model with a data record and obtains the model's prediction on that record. The prediction is a vector of probabilities, one per class, that the record belongs to a certain class. This prediction vector, along with the label of the target record, is passed to the attack model, which infers whether the record was in or out of the target model's training dataset.
Membership Inference Attacks against Machine Learning Models
Target Model recall (what fraction of the training dataset's members are correctly inferred as members by the attacker).
Membership Inference Attacks against Machine Learning Models
Our membership inference attack exploits the observation that machine learning models often behave differently on the data that they were trained on versus the data that they "see" for the first time. Overfitting is a common reason but not the only one (see Section VII). The objective of the attacker is to construct an attack model that can recognize such differences in the target model's behavior and use them to distinguish members from non-members of the target model's training dataset based solely on the target model's output.
Membership Inference Attacks against Machine Learning Models
Our attack model is a collection of models, one for each output class of the target model. This increases accuracy of the attack because the target model produces different distributions over its output classes depending on the input's true class.
Membership Inference Attacks against Machine Learning Models
To train our attack model, we build multiple "shadow" models intended to behave similarly to the target model. In contrast to the target model, we know the ground truth for each shadow model, i.e., whether a given record was in its training dataset or not. Therefore, we can use supervised training on the inputs and the corresponding outputs (each labeled "in" or "out") of the shadow models to teach the attack model how to distinguish the shadow models' outputs on members of their training datasets from their outputs on non-members.
Membership Inference Attacks against Machine Learning Models
Formally, let f target () be the target model, and let D train target be its private training dataset which contains labeled data Let f attack () be the attack model. Its input x attack is composed of a correctly labeled record and a prediction vector of size c target . Since the goal of the attack is decisional membership inference, the attack model is a binary classifier with two output classes, "in" and "out." Figure 1 illustrates our end-to-end attack process. For a labeled record (x, y), we use the target model to compute the prediction vector y = f target (x). The distribution of y (classification confidence values) depends heavily on the true class of x. This is why we pass the true label y of x in addition to the model's prediction vector y to the attack model. Given how the probabilities in y are distributed around y, the attack model computes the membership probability Pr{(x, y) ∈ D train target }, i.e., the probability that ((x, y), y) belongs to the "in" class or, equivalently, that x is in the training dataset of f target ().
Membership Inference Attacks against Machine Learning Models
The main challenge is how to train the attack model to distinguish members from non-members of the target model's training dataset when the attacker has no information about the internal parameters of the target model and only limited query access to it through the public API. To solve this conundrum, we developed a shadow training technique that lets us train the attack model on proxy targets for which we do know the training dataset and can thus perform supervised training.
Membership Inference Attacks against Machine Learning Models
The attacker creates k shadow models f i shadow (). Each shadow model i is trained on a dataset D train shadow i of the same format as and distributed similarly to the target model's training dataset. These shadow training datasets can be generated using one of methods described in Section V-C. We assume that the datasets used for training the shadow models are disjoint from the private dataset used to train the target model (∀i, D train shadow i ∩ D train target = ∅). This is the worst case for the attacker; the attack will perform even better if the training datasets happen to overlap.
Membership Inference Attacks against Machine Learning Models
The shadow models must be trained in a similar way to the target model. This is easy if the target's training algorithm Algorithm 1 Data synthesis using the target model 1: procedure SYNTHESIZE(class : c) 2:
Membership Inference Attacks against Machine Learning Models
x ← RANDRECORD(.) initialize a record randomly 3:
Membership Inference Attacks against Machine Learning Models
for iteration = 1 · · · iter max do 7:
Membership Inference Attacks against Machine Learning Models
query the target model 8: if y c ≥ y * c then accept the record 9: if y c > conf min and c = arg max(y) then 10: if rand() < y c then x ← RANDRECORD(x * , k) randomize k features 25: end for 26: return ⊥ failed to synthesize 27: end procedure (e.g., neural networks, SVM, logistic regression) and model structure (e.g., the wiring of a neural network) are known. Machine learning as a service is more challenging. Here the type and structure of the target model are not known, but the attacker can use exactly the same service (e.g., Google Prediction API) to train the shadow model as was used to train the target model-see Figure 2. The more shadow models, the more accurate the attack model will be. As described in Section V-D, the attack model is trained to recognize differences in shadow models' behavior when these models operate on inputs from their own training datasets versus inputs they did not encounter during training. Therefore, more shadow models provide more training fodder for the attack model.
Membership Inference Attacks against Machine Learning Models
To train shadow models, the attacker needs training data that is distributed similarly to the target model's training data. We developed several methods for generating such data.
Membership Inference Attacks against Machine Learning Models
Model-based synthesis. If the attacker does not have real training data nor any statistics about its distribution, he can generate synthetic training data for the shadow models using the target model itself. The intuition is that records that are classified by the target model with high confidence should be statistically similar to the target's training dataset and thus provide good fodder for shadow models.
Membership Inference Attacks against Machine Learning Models
The synthesis process runs in two phases: (1) search, using a hill-climbing algorithm, the space of possible data records to find inputs that are classified by the target model with high confidence; (2) sample synthetic data from these records. After this process synthesizes a record, the attacker can repeat it until the training dataset for shadow models is full.
Membership Inference Attacks against Machine Learning Models
See Algorithm 1 for the pseudocode of our synthesis procedure. First, fix class c for which the attacker wants to generate synthetic data. The first phase is an iterative process. Start by randomly initializing a data record x. Assuming that the attacker knows only the syntactic format of data records, sample the value for each feature uniformly at random from among all possible values of that feature. In each iteration, propose a new record. A proposed record is accepted only if it increases the hill-climbing objective: the probability of being classified by the target model as class c.
Membership Inference Attacks against Machine Learning Models
Each iteration involves proposing a new candidate record by changing k randomly selected features of the latest accepted record x * . This is done by flipping binary features or resampling new values for features of other types. We initialize k to k max and divide it by 2 when rej max subsequent proposals are rejected. This controls the diameter of search around the accepted record in order to propose a new record. We set the minimum value of k to k min . This controls the speed of the search for new records with a potentially higher classification probability y c .
Membership Inference Attacks against Machine Learning Models
The second, sampling phase starts when the target model's probability y c that the proposed data record is classified as belonging to class c is larger than the probabilities for all other classes and also larger than a threshold conf min . This ensures that the predicted label for the record is c, and that the target model is sufficiently confident in its label prediction. We select such record for the synthetic dataset with probability y * c and, if selection fails, repeat until a record is selected.
Membership Inference Attacks against Machine Learning Models
This synthesis procedure works only if the adversary can efficiently explore the space of possible inputs and discover inputs that are classified by the target model with high confidence. For example, it may not work if the inputs are highresolution images and the target model performs a complex image classification task.
Membership Inference Attacks against Machine Learning Models
Statistics-based synthesis. The attacker may have some statistical information about the population from which the target model's training data was drawn. For example, the attacker may have prior knowledge of the marginal distributions of different features. In our experiments, we generate synthetic training records for the shadow models by independently sampling the value of each feature from its own marginal distribution. The resulting attack models are very effective.
Membership Inference Attacks against Machine Learning Models
Noisy real data. The attacker may have access to some data that is similar to the target model's training data and can be considered as a "noisy" version thereof. In our experiments with location datasets, we simulate this by flipping the (binary) values of 10% or 20% randomly selected features, then (data record, class label) predict(data) (prediction, class label, "in" / "out")  Fig. 3: Training the attack model on the inputs and outputs of the shadow models. For all records in the training dataset of a shadow model, we query the model and obtain the output. These output vectors are labeled "in" and added to the attack model's training dataset. We also query the shadow model with a test dataset disjoint from its training dataset. The outputs on this set are labeled "out" and also added to the attack model's training dataset. Having constructed a dataset that reflects the black-box behavior of the shadow models on their training and test datasets, we train a collection of ctarget attack models, one per each output class of the target model.
Membership Inference Attacks against Machine Learning Models
training our shadow models on the resulting noisy dataset. This scenario models the case where the training data for the target and shadow models are not sampled from exactly the same population, or else sampled in a non-uniform way.
Membership Inference Attacks against Machine Learning Models
The main idea behind our shadow training technique is that similar models trained on relatively similar data records using the same service behave in a similar way. This observation is empirically borne out by our experiments in the rest of this paper. Our results show that learning how to infer membership in shadow models' training datasets (for which we know the ground truth and can easily compute the cost function during supervised training) produces an attack model that successfully infers membership in the target model's training dataset, too.
Membership Inference Attacks against Machine Learning Models
We query each shadow model with its own training dataset and with a disjoint test set of the same size. The outputs on the training dataset are labeled "in," the rest are labeled "out." Now, the attacker has a dataset of records, the corresponding outputs of the shadow models, and the in/out labels. The objective of the attack model is to infer the labels from the records and corresponding outputs. Figure 3 shows how to train the attack model. For all (x, y) ∈ D train shadow i , compute the prediction vector y = f i shadow (x) and add the record (y, y, in) to the attack training set D train attack . Let D test shadow i be a set of records disjoint from the training set of the ith shadow model. Then, ∀(x, y) ∈ D test shadow i compute the prediction vector y = f i shadow (x) and add the record (y, y, out) to the attack training set D train attack . Finally, split D train attack into c target partitions, each associated with a different class label. For each label y, train a separate model that, given y, predicts the in or out membership status for x. If we use model-based synthesis from Section V-C, all of the raw training data for the attack model is drawn from the records that are classified by the target model with high confidence. This is true, however, both for the records used in the shadow models' training datasets and for the test records left out of these datasets. Therefore, it is not the case that the attack model simply learns to recognize inputs that are classified with high confidence. Instead, it learns to perform a much subtler task: how to distinguish between the training inputs classified with high confidence and other, non-training inputs that are also classified with high confidence.
Membership Inference Attacks against Machine Learning Models
In effect, we convert the problem of recognizing the complex relationship between members of the training dataset and the model's output into a binary classification problem. Binary classification is a standard machine learning task, thus we can use any state-of-the-art machine learning framework or service to build the attack model. Our approach is independent of the specific method used for attack model training. For example, in Section VI we construct the attack model using neural networks and also using the same black-box Google Prediction API that we are attacking, in which case we have no control over the model structure, model parameters, or training metaparameters-but still obtain a working attack model.
Membership Inference Attacks against Machine Learning Models
We first describe the datasets that we use for evaluation, followed by the description of the target models and our experimental setup. We then present the results of our membership inference attacks in several settings and study in detail how and why the attacks work against different datasets and machine learning platforms.
Membership Inference Attacks against Machine Learning Models
CIFAR. CIFAR-10 and CIFAR-100 are benchmark datasets used to evaluate image recognition algorithms [24]. CIFAR-10 is composed of 32×32 color images in 10 classes, with 6, 000 images per class. In total, there are 50, 000 training images and 10, 000 test images. CIFAR-100 has the same format as CIFAR-10, but it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. We use different fractions of this dataset in our attack experiments to show the effect of the training dataset size on the accuracy of the attack.
Membership Inference Attacks against Machine Learning Models
Purchases. Our purchase dataset is based on Kaggle's "acquire valued shoppers" challenge dataset that contains shopping histories for several thousand individuals. 6 The purpose of the challenge is to design accurate coupon promotion strategies. Each user record contains his or her transactions over a year. The transactions include many fields such as product name, store chain, quantity, and date of purchase.
Membership Inference Attacks against Machine Learning Models
For our experiments, we derived a simplified purchase dataset (with 197, 324 records), where each record consists of 600 binary features. Each feature corresponds to a product and represents whether the user has purchased it or not. To design our classification tasks, we first cluster the records into multiple classes, each representing a different purchase style. In our experiments, we use 5 different classification tasks with a different number of classes {2, 10, 20, 50, 100}. The classification task is to predict the purchase style of a user given the 600-feature vector. We use 10, 000 randomly selected records from the purchase dataset to train the target model. The rest of the dataset contributes to the test set and (if necessary) the training sets of the shadow models.
Membership Inference Attacks against Machine Learning Models
Locations. We created a location dataset from the publicly available set of mobile users' location "check-ins" in the Foursquare social network, restricted to the Bangkok area and collected from April 2012 to September 2013 [36]. 7 The check-in dataset contains 11, 592 users and 119, 744 locations, for a total of 1, 136, 481 check-ins. We filtered out users with fewer than 25 check-ins and venues with fewer than 100 visits, which left us with 5, 010 user profiles. For each location venue, we have the geographical position as well as its location type (e.g., Indian restaurant, fast food, etc.). The total number of location types is 128. We partition the Bangkok map into areas of size 0.5km × 0.5km, yielding 318 regions for which we have at least one user check-in.
Membership Inference Attacks against Machine Learning Models
Each record in the resulting dataset has 446 binary features, representing whether the user visited a certain region or location type, i.e., the user's semantic and geographical profile. The classification task is similar to the purchase dataset. We cluster the location dataset into 30 classes, each representing a different geosocial type. The classification task is to predict the user's geosocial type given his or her record. We use 1, 600 randomly selected records to train the target model. The rest Texas hospital stays. This dataset is based on the Hospital Discharge Data public use files with information about inpatients stays in several health facilities, 8 released by the Texas Department of State Health Services from 2006 to 2009. Each record contains four main groups of attributes: the external causes of injury (e.g., suicide, drug misuse), the diagnosis (e.g., schizophrenia, illegal abortion), the procedures the patient underwent (e.g., surgery) and some generic information such as the gender, age, race, hospital id, and length of stay.
Membership Inference Attacks against Machine Learning Models
Our classification task is to predict the patient's main procedure based on the attributes other than secondary procedures. We focus on the 100 most frequent procedures. The resulting dataset has 67, 330 records and 6, 170 binary features. We use 10, 000 randomly selected records to train the target model.
Membership Inference Attacks against Machine Learning Models
Note that our experiments do not involve re-identification of known individuals and fully comply with the data use agreement for the original Public Use Data File.
Membership Inference Attacks against Machine Learning Models
MNIST. This is a dataset of 70, 000 handwritten digits formatted as 32 × 32 images and normalized so that the digits are located at the center of the image. 9 We use 10, 000 randomly selected images to train the target model.
Membership Inference Attacks against Machine Learning Models
UCI Adult (Census Income). This dataset includes 48, 842 records with 14 attributes such as age, gender, education, marital status, occupation, working hours, and native country. The (binary) classification task is to predict if a person makes over $50K a year based on the census attributes. 10 We use 10, 000 randomly selected records to train the target model.
Membership Inference Attacks against Machine Learning Models
We evaluated our inference attacks on three types of target models: two constructed by cloud-based "machine learning as a service" platforms and one we implemented locally. In all cases, our attacks treat the models as black boxes. For the cloud services, we do not know the type or structure of the models they create, nor the values of the hyper-parameters used during the training process.
Membership Inference Attacks against Machine Learning Models
Machine learning as a service. The first cloud-based machine learning service in our study is Google Prediction API. With this service, the user uploads a dataset and obtains an API for querying the resulting model. There are no configuration parameters that can be changed by the user.
Membership Inference Attacks against Machine Learning Models
The other cloud service is Amazon ML. The user cannot choose the type of the model but can control a few metaparameters. In our experiments, we varied the maximum number of passes over the training data and L2 regularization amount. The former determines the number of training epochs and controls the convergence of model training; its default value is 10. The latter tunes how much regularization is performed on the model parameters in order to avoid overfitting. We used the platform in two configurations: the default setting (10, 1e − 6) and (100, 1e − 4).
Membership Inference Attacks against Machine Learning Models
Neural networks. Neural networks have become a very popular approach to large-scale machine learning. We use Torch7 and its nn packages, 11 a deep-learning library that has been used and extended by major Internet companies such as Facebook. 12 On CIFAR datasets, we train a standard convolutional neural network (CNN) with two convolution and max pooling layers plus a fully connected layer of size 128 and a SoftMax layer. We use Tanh as the activation function. We set the learning rate to 0.001, the learning rate decay to 1e − 07, and the maximum epochs of training to 100.
Membership Inference Attacks against Machine Learning Models
On the purchase dataset (see Section VI-A), we train a fully connected neural network with one hidden layer of size 128 and a SoftMax layer. We use Tanh as the activation function. We set the learning rate to 0.001, the learning rate decay to 1e − 07, and the maximum epochs of training to 200. 11 https://github.com/torch/nn 12 https://github.com/facebook/fblualib
Membership Inference Attacks against Machine Learning Models
The training set and the test set of each target and shadow model are randomly selected from the respective datasets, have the same size, and are disjoint. There is no overlap between the datasets of the target model and those of the shadow models, but the datasets used for different shadow models can overlap with each other.
Membership Inference Attacks against Machine Learning Models
We set the training set size to 10, 000 for the purchase dataset as well as the Texas hospital-stay dataset, Adult dataset and the MNIST dataset. We set it to 1, 200 for the location dataset. We vary the size of the training set for the CIFAR datasets, to measure the difference in the attack accuracy. For the CIFAR-10 dataset, we choose 2, 500; 5, 000; 10, 000; and 15, 000. For the CIFAR-100 dataset, we choose 4, 600; 10, 520; 19, 920; and 29, 540.
Membership Inference Attacks against Machine Learning Models
The experiments on the CIFAR datasets were run locally, against our own models, so we can vary the model's configuration and measure the impact on the attack accuracy. The experiments on the other datasets (purchases with {2, 10, 20, 50, 100} classes, Texas hospital stays, locations, Adult, and MNIST) were run against models trained using either Google or Amazon services, where we have no visibility into their choice of the model type and structure and little control over the training process (see Section VI-B).
Membership Inference Attacks against Machine Learning Models
For the purchase dataset, we built target models on all platforms (Google, Amazon, local neural networks) employing the same training dataset, thus enabling us to compare the leakage from different models. We used similar training architectures for the attack models across different platforms: either a fully connected neural network with one hidden layer of size 64 with ReLU (rectifier linear units) activation functions and a SoftMax layer, or a Google-trained black-box model.
Membership Inference Attacks against Machine Learning Models
We set the number of shadow models to 100 for the CIFAR datasets, 20 for the purchase dataset, 10 for the Texas hospitalstay dataset, 60 for the location dataset, 50 for the MNIST dataset, and 20 for the Adult dataset. Increasing the number of shadow models would increase the accuracy of the attack but also its cost.
Membership Inference Attacks against Machine Learning Models
The attacker's goal is to determine whether a given record was part of the target model's training dataset. We evaluate this attack by executing it on randomly reshuffled records from the target's training and test datasets. In our attack evaluation, we use sets of the same size (i.e, equal number of members and non-members) in order to maximize the uncertainty of inference, thus the baseline accuracy is 0.5.
Membership Inference Attacks against Machine Learning Models
We evaluate the attack using the standard precision and recall metrics. Precision is the fraction of the records inferred as members of the training dataset that are indeed members. Recall measures coverage of the attack, i.e., the fraction of the training records that the attacker can correctly infer as members. Most measurements are reported per class because the accuracy of the attack can vary considerably for different classes. This is due to the difference in size and composition of the training data belonging to each class and highly depends on the dataset.
Membership Inference Attacks against Machine Learning Models
The test accuracy of our target neural-network models with the largest training datasets (15, 000 and 29, 540 records, respectively) is 0.6 and 0.2 for CIFAR-10 and CIFAR-100, respectively. The accuracy is low, indicating that the models are heavily overfitted on their training sets. Figure 4 shows the results of the membership inference attack against the CIFAR models. For both CIFAR-10 and CIFAR-100, the attack performs much better than the baseline, with CIFAR-100 especially vulnerable. Table I shows the training and test accuracy of the models constructed using different machine learning platforms for the purchase dataset with 100 classes. Large gaps between training and test accuracy indicate overfitting. Larger test accuracy indicates better generalizability and higher predictive power. Figure 5 shows the results of the membership inference attack against the black-box models trained by Google's and Amazon's machine learning platforms. Figure 7 compares precision of the attacks against these models with the attacks against a neural-network model trained on the same data. Models trained using Google Prediction API exhibit the biggest leakage.
Membership Inference Attacks against Machine Learning Models
For the Texas hospital-stay dataset, we evaluated our attack against a Google-trained model. The training accuracy of the target model is 0.66 and its test accuracy is 0.51. Figure 6 shows the accuracy of membership inference. Precision is mostly above 0.6, and for half of the classes, it is above 0.7. Precision is above 0.85 for more than 20 classes.
Membership Inference Attacks against Machine Learning Models
For the location dataset, we evaluated our attacks against a Google-trained model. The training accuracy of the target model is 1 and its test accuracy is 0.66. Figure 8 shows the accuracy of membership inference. Precision is between 0.6 and 0.8, with an almost constant recall of 1. Figure 8 reports precision of the attacks trained on the shadow models whose training datasets are noisy versions of the real data (disjoint from the target model's training dataset but sampled from the same population). Precision drops as the amount of noise increases, but the attack still outperforms the   Results are shown for the shadow models trained on real data and for the shadow models trained on noisy data with 10% and 20% noise (i.e., x% of features are replaced with random values). Precision of the attack over all classes is 0.678 (real data), 0.666 (data with 10% noise), and 0.613 (data with 20% noise). The corresponding recall of the attack is 0.98, 0.99, and 1.00, respectively.
Membership Inference Attacks against Machine Learning Models
baseline and, even with 10% of the features in the shadows' training data replaced by random values, matches the original attack. This demonstrates that our attacks are robust even if the attacker's assumptions about the distribution of the target model's training data are not very accurate. Figure 9 reports precision of the attacks when the attacker has no real data (not even noisy) for training his shadow models. Instead, we used the marginal distributions of individual features to generate 187, 300 synthetic purchase records, then trained 20 shadow models on these records.
Membership Inference Attacks against Machine Learning Models
We also generated 30, 000 synthetic records using the model-based approach presented in Algorithm 1. In our experiments with the purchase dataset where records have 600 binary features, we initialize k to k max = 128 and divide it by 2 when rej max = 10 subsequent proposals are rejected. We set its minimum value k min = 4. In the sampling phase, we set the minimum confidence threshold conf min to 0.2.
Membership Inference Attacks against Machine Learning Models
For our final set of sampled records, the target model's confidence in classifying the records is 0.24 on average (just a bit over our threshold conf min = 0.2). On average, each synthetic record needed 156 queries (of proposed records) during our hill-climbing two-phase process (see Section V-C). We trained 8 shadow models on this data. Figure 9 compares precision of the attacks when shadow models are trained on real data versus shadow models trained on synthetic data. The overall precision is 0.935 on real data compared to 0.795 for marginal-based synthetics and 0.895 for model-based synthetics. The accuracy of the attack using marginal-based synthetic data is noticeably reduced versus real data, but is nevertheless very high for most classes. The attack using model-based synthetic data exhibits dual behavior. For most classes its precision is high and close to the attacks that use real data for shadow training, but for a few classes precision is very low (less than 0.1).
Membership Inference Attacks against Machine Learning Models
The reason for the attack's low precision on some classes is that the target classifier cannot confidently model the distribution of data records belonging to these classes-because it has not seen enough examples. These classes are underrepresented in the target model's training dataset. For example, each of the classes where the attack has less than 0.1 precision contributes under 0.6% of the target model's training dataset. Some of these classes have fewer than 30 training records (out of 10, 000). This makes it very difficult for our algorithm to synthesize representatives of these classes when searching the high-dimensional space of possible records.
Membership Inference Attacks against Machine Learning Models
For the majority of the target model's classes, our attack achieves high precision. This demonstrates that a membership inference attack can be trained with only black-box access to the target model, without any prior knowledge about the distribution of the target model's training data if the attacker can efficiently generate inputs that are classified by the target model with high confidence. are better for CIFAR-100 than for CIFAR-10. The CIFAR-100 model is also more overfitted to its training dataset. For the same number of training records per class, the attack performs better against CIFAR-100 than against CIFAR-10. For example, compare CIFAR-10 when the size of the training dataset is 2, 000 with CIFAR-100 when the size of the training dataset is 20, 000. The average number of data records per class is 200 in both cases, but the attack accuracy is much better (close to 1) for CIFAR-100.
Membership Inference Attacks against Machine Learning Models
To quantify the effect that the number of classes has on the accuracy of the attack, we trained target models using Google Prediction API on the purchase dataset with {2, 10, 20, 50, 100} classes. Figure 10 shows the distribution of attack precision for each model. Models with fewer classes leak less information about their training inputs. As the number of classes increases, the model needs to extract more distinctive features from the data to be able to classify inputs with high accuracy. Informally, models with more output classes need to remember more about their training data, thus they leak more information. Figure 11 shows the relationship between the amount of training data per class and the accuracy of membership inference. This relationship is more complex, but, in general, the more data in the training dataset is associated with a given class, the lower the attack precision for that class. Table II shows the precision of membership inference against Google-trained models. For the MNIST dataset, the training accuracy of the target model is 0.984 and its test accuracy is 0.928. The overall precision of the membership inference attack is 0.517, which is just slightly above random guessing. The lack of randomness in the training data for each class and the small number of classes contribute to the failure of the attack.
Membership Inference Attacks against Machine Learning Models
For the Adult dataset, the training accuracy of the target model is 0.848 and its test accuracy is 0.842. The overall precision of the attack is 0.503, which is equivalent to random  guessing. There could be two reasons for why membership inference fails against this model. First, the model is not overfitted (its test and train accuracies are almost the same). Second, the model is a binary classifier, which means that the attacker has to distinguish members from non-members by observing the behavior of the model on essentially 1 signal, since the two outputs are complements of each other. This is not enough for our attack to extract useful membership information from the model.
Membership Inference Attacks against Machine Learning Models
The more overfitted a model, the more it leaks-but only for models of the same type. For example, the Amazontrained (100, 1e − 4) model that, according to Table I, is more overfitted leaks more than the Amazon-trained (10, 1e − 6) model. However, they both leak less than the Google-trained model, even though the Google model is less overfitted than one of the Amazon models and has a much better predictive power (and thus generalizability) than both Amazon models. Therefore, overfitting is not the only factor that causes a model to be vulnerable to membership inference. The structure and type of the model also contribute to the problem.
Membership Inference Attacks against Machine Learning Models
In Figure 11, we look deeper into the factors that contribute to attack accuracy per class, including how overfitted the model is and what fraction of the training data belongs to each class. The (train-test) accuracy gap is the difference between the accuracy of the target model on its training and test data. Similar metrics are used in the literature to measure how overfitted a model is [18]. We compute this metric for each class. Bigger gaps indicate that the model is overfitted on its training data for that class. The plots show that, as expected, bigger (train-test) accuracy gaps are associated with higher precision of membership inference. Table II shows the relationship between the accuracy of our membership inference attack and the (train-test) gap of the target models. Figure 12 also illustrates how the target models' outputs distinguish members of their training datasets from the non-members. This is the information that our attack exploits.
Membership Inference Attacks against Machine Learning Models
Specifically, we look at how accurately the model predicts the correct label as well as its prediction uncertainty. The accuracy for class i is the probability that the model classifies an  Fig. 11: Relationship between the precision of the membership inference attack on a class and the (train-test) accuracy gap of the target model, as well as the fraction of the training dataset that belongs to this class. Each point represent the values for one class. The (train-test) accuracy gap is a metric for generalization error [18] and an indicator of how overfitted the target model is.  The difference between the member and non-member output distributions is among the factors that our attack exploits to infer membership. The accuracy of our attack is higher for the models where the two distributions are more distinguishable (See Table II). input with label i as i. Prediction uncertainty is the normalized entropy of the model's prediction vector: −1 log(n)
Membership Inference Attacks against Machine Learning Models
where p i is the probability that the input belongs to class i, and n is the number of classes. The plots show that there is an observable difference between the output (both accuracy and uncertainty) of the model on the member inputs versus the non-member inputs in the cases where our attack is successful.
Membership Inference Attacks against Machine Learning Models
Success of membership inference is directly related to the (1) generalizability of the target model and (2) diversity of its training data. If the model overfits and does not generalize well to inputs beyond its training data, or if the training data is not representative, the model leaks information about its training inputs. We quantify this relationship in Fig. 11. From the machine learning perspective, overfitting is harmful because it produces models that lack predictive power. In this paper, we show another harm of overfitting: the leakage of sensitive information about the training data.
Membership Inference Attacks against Machine Learning Models
As we explained in Section VI, overfitting is not the only reason why our inference attacks work. Different machine learning models, due to their different structures, "remember" different amounts of information about their training datasets. This leads to different amounts of information leakage even if the models are overfitted to the same degree (see Table I).
Membership Inference Attacks against Machine Learning Models
As explained in Section VII, overfitting is an important (but not the only) reason why machine learning models leak information about their training datasets. Of course, overfitting is a canonical problem in machine learning because it limits the predictive power and generalizability of models. This means that instead of the usual tradeoff between utility and privacy, machine learning research and privacy research have similar objectives in this case. Regularization techniques such as dropout [31] can help defeat overfitting and also strengthen privacy guarantees in neural networks [23]. Regularization is also used for objective perturbation in differentially private machine learning [9].
Membership Inference Attacks against Machine Learning Models
(Ideal) well-regularized models should not leak much information about their training data, and our attack can serve as a metric to quantify this. Also, models with a trivial structure (e.g., XOR of some input features) generalize to the entire universe and do not leak information.
Membership Inference Attacks against Machine Learning Models
If the training process is differentially private [12], the probability of producing a given model from a training dataset that includes a particular record is close to the probability of producing the same model when this record is not included. Differentially private models are, by construction, secure against membership inference attacks of the kind developed in this paper because our attacks operate solely on the outputs of the model, without any auxiliary information. One obstacle is that differentially private models may significantly reduce the model's prediction accuracy for small values. In Section IX, we survey some of the related work in this area.
Membership Inference Attacks against Machine Learning Models
In the case of machine learning as a service, platform operators such as Google and Amazon have significant responsibility to the users of their services. In their current form, these services simply accept the data, produce a model of unknown type and structure, and return an opaque API to this model that data owners use as they see fit, without any understanding that by doing so, they may be leaking out their data. Machine learning services do not inform their customers about the risks of overfitting or the harm that may result from models trained on inadequate datasets (for example, with unrepresentative records or too few representatives for certain classes).
Membership Inference Attacks against Machine Learning Models
Instead, when adaptively choosing a model for a customersupplied dataset, services such as Google Prediction API and Amazon ML should take into account not only the accuracy of the model but also the risk that it will leak information about its training data. Furthermore, they need to explicitly warn customers about this risk and provide more visibility into the model and the methods that can be used to reduce this leakage. Our inference attacks can be used as metrics to quantify leakage from a specific model, and also to measure the effectiveness of future privacy protection techniques deployed by machine-learning services.
Membership Inference Attacks against Machine Learning Models
We quantitatively evaluate several defenses against membership inference. Restrict the prediction vector to top k classes. When the number of classes is large, many classes may have very small probabilities in the model's prediction vector. The model will still be useful if it only outputs the probabilities of the most likely k classes. To implement this, we add a filter to the last layer of the model. The smaller k is, the less information the model leaks. In the extreme case, the model returns only the label of the most likely class without reporting its probability. Coarsen precision of the prediction vector. To implement this, we round the classification probabilities in the prediction  vector down to d floating point digits. The smaller d is, the less information the model leaks.
Membership Inference Attacks against Machine Learning Models
Increase entropy of the prediction vector. One of the signals that membership inference exploits is the difference between the prediction entropy of the target model on its training inputs versus other inputs. As a mitigation technique for neuralnetwork models, we can modify (or add) the softmax layer and increase its normalizing temperature t > 0. The softmax layer converts the logits computed for each class into probabilities. For the logits vector z, the i th output of the softmax function with temperature t is e z i /t j e z j /t . This technique, also used in knowledge distillation and information transfer between models [20], would increase the entropy of the prediction vector. Note that for a very large temperature, the output becomes almost uniform and independent of its input, thus leaking no information.
Membership Inference Attacks against Machine Learning Models
Use regularization. Regularization techniques are used to overcome overfitting in machine learning. We use L 2 -norm standard regularization that penalizes large parameters by adding λ i θ 2 i to the model's loss function, where θ i s are model's parameters. We implement this technique with various values for the regularization factor λ. The larger λ is, the stronger the effect of regularization during the training.
Membership Inference Attacks against Machine Learning Models
To evaluate the effectiveness of different mitigation strategies, we implemented all of them in locally trained mod-els over which we have full control. The inference attack, however, still assumes only black-box access to the resulting models. The baseline model for these experiments is a neural network with one hidden layer with 256 units (for the purchase dataset) and 1,000 units (for the Texas hospital-stay dataset). We use Tanh as the activation function. Table III shows the results of our evaluation. It compares different mitigation strategies based on how they degrade the accuracy of our attack relative to the attack on a model that does not use any mitigation. The mitigation strategies that we implemented did not impose any cost on the target model's prediction accuracy, and in the case of regularization, the target model's prediction accuracy increased as expected. Note that more regularization (by increasing λ even further) would potentially result in a significant reduction of the target model's test accuracy, even if it foils membership inference. This is shown in the table for λ = 1e − 2 on the purchase dataset, and for λ = 5e − 3 on the Texas hospital stay dataset.
Membership Inference Attacks against Machine Learning Models
Overall, our attack is robust against these mitigation strategies. Filtering out low-probability classes from the prediction vector and limiting the vector to the top 1 or 3 most likely classes does not foil the attack. Even restricting the prediction vector to a single label (most likely class), which is the absolute minimum a model must output to remain useful, is not enough to fully prevent membership inference. Our attack can still exploit the mislabeling behavior of the target model because members and non-members of the training dataset are mislabeled differently (assigned to different wrong classes). If the prediction vector contains probabilities in addition to the labels, the model leaks even more information that can be used for membership inference.
Membership Inference Attacks against Machine Learning Models
Some of the mitigation methods are not suitable for machine-learning-as-service APIs used by general applications and services. Regularization, however, appears to be necessary and useful. As mentioned above, it (1) generalizes the model and improves its predictive power and (2) decreases the model's information leakage about its training dataset. However, regularization needs to be deployed carefully to avoid damaging the model's performance on the test datasets.
Membership Inference Attacks against Machine Learning Models
Attacks on statistical and machine learning models. In [2], knowledge of the parameters of SVM and HMM models is used to infer general statistical information about the training dataset, for example, whether records of a particular race were used during training. By contrast, our inference attacks work in a black-box setting, without any knowledge of the model's parameters, and infer information about specific records in the training dataset, as opposed to general statistics.
Membership Inference Attacks against Machine Learning Models
Homer et al. [21] developed a technique, which was further studied in [3], [15], for inferring the presence of a particular genome in a dataset, based on comparing the published statistics about this dataset (in particular, minor allele frequencies) to the distribution of these statistics in the general population. By contrast, our inference attacks target trained machine learning models, not explicit statistics. Fig. 13: Images produced by model inversion on a trained CIFAR-10 model. Top: airplane, automobile, bird, cat, deer. Bottom: dog, frog, horse, ship, truck. The images do not correspond to any specific image from the training dataset, are not human-recognizable, and at best (e.g., the truck class image) are vaguely similar to the average image of all objects in a given class.
Membership Inference Attacks against Machine Learning Models
Other attacks on machine learning include [7], where the adversary exploits changes in the outputs of a collaborative recommender system to infer inputs that caused these changes. These attacks exploit temporal behavior specific to the recommender systems based on collaborative filtering.
Membership Inference Attacks against Machine Learning Models
Model inversion. Model inversion [16], [17] uses the output of a model applied to a hidden input to infer certain features of this input. See [27] for a detailed analysis of this attack and an explanation of why it does not necessarily entail a privacy breach. For example, in the specific case of pharmacogenetics analyzed in [17], the model captures the correlation between the patient's genotype and the dosage of a certain medicine. This correlation is a valid scientific fact that holds for all patients, regardless of whether they were included in the model's training dataset or not. It is not possible to prevent disclosure due to population statistics [14].
Membership Inference Attacks against Machine Learning Models
In general, model inversion cannot tell whether a particular record was used as part of the model's training dataset. Given a record and a model, model inversion works exactly the same way when the record was used to train the model and when it was not used. In the case of pharmacogenetics [17], model inversion produces almost identical results for members and non-members. Due to the overfitting of the model, the results are a little (4%) more accurate for the members, but this accuracy can only be measured in retrospect, if the adversary already knows the ground truth (i.e., which records are indeed members of the model's training dataset). By contrast, our goal is to construct a decision procedure that distinguishes members from non-members.
Membership Inference Attacks against Machine Learning Models
Model inversion has also been applied to face recognition models [16]. In this scenario, the model's output is set to 1 for class i and 0 for the rest, and model inversion is used to construct an input that produces these outputs. This input is not an actual member of the training dataset but simply an average of the features that "characterize" the class.
Membership Inference Attacks against Machine Learning Models
In the face recognition scenario-and only in this specific scenario-each output class of the model is associated with a single person. All training images for this class are different photos of that person, thus model inversion constructs an artificial image that is an average of these photos. Because they all depict the same person, this average is recognizable (by a human) as that person. Critically, model inversion does not produce any specific image from the training dataset, which is the definition of membership inference.
Membership Inference Attacks against Machine Learning Models
If the images in a class are diverse (e.g., if the class contains multiple individuals or many different objects), the results of model inversion as used in [16] are semantically meaningless and not recognizable as any specific image from the training dataset. To illustrate this, we ran model inversion against a convolutional neural network 13 trained on the CIFAR-10 dataset, which is a standard benchmark for object recognition models. Each class includes different images of a single type of object (e.g., an airplane). Figure 13 shows the images "reconstructed" by model inversion. As expected, they do not depict any recognizable object, let alone an image from the training dataset. We expect similar results for other models, too. For the pharmacogenetics model mentioned above, this form of model inversion produces an average of different patients' genomes. For the model that classifies location traces into geosocial profiles (see Section VI-A), it produces an average of the location traces of different people. In both cases, the results of model inversion are not associated with any specific individual or specific training input.
Membership Inference Attacks against Machine Learning Models
In summary, model inversion produces the average of the features that at best can characterize an entire output class. It does not (1) construct a specific member of the training dataset, nor (2) given an input and a model, determines if this specific input was used to train the model. Model extraction. Model extraction attacks [32] aim to extract the parameters of a model trained on private data. The attacker's goal is to construct a model whose predictive performance on validation data is similar to the target model.
Membership Inference Attacks against Machine Learning Models
Model extraction can be a stepping stone for inferring information about the model's training dataset. In [32], this is illustrated for a specific type of models called kernel logistic regression (KLR) [38]. In KLR models, the kernel function includes a tiny fraction of the training data (so called "import points") directly into the model. Since import points are parameters of the model, extracting them results in the leakage of that particular part of the data. This result is very specific to KLR and does not extend to other types of models since they do not explicitly store training data in their parameters.
Membership Inference Attacks against Machine Learning Models
Even for KLR models, leakage is not quantified other than via visual similarity of a few chosen import points and "the closest (in L1 norm) extracted representers" on the MNIST dataset of handwritten digits. In MNIST, all members of a class are very similar (e.g., all members of the first class are different ways of writing digit "1"). Thus, any extracted digit must be similar to all images in its class, whether this digit was in the training set or not. Privacy-preserving machine learning. Existing literature on privacy protection in machine learning focuses mostly on how to learn without direct access to the training data. Secure multiparty computation (SMC) has been used for learning decision trees [26], linear regression functions [11], Naive 13 https://github.com/Lasagne/Recipes/blob/master/modelzoo/cifar10 nin.py Bayes classifiers [33], and k-means clustering [22]. The goal is to limit information leakage during training. The training algorithm is the same as in the non-privacy-preserving case, thus the resulting models are as vulnerable to inference attacks as any conventionally trained model. This also holds for the models trained by computing on encrypted data [4], [6], [35].
Membership Inference Attacks against Machine Learning Models
Differential privacy [12] has been applied to linear and logistic regression [8], [37], support vector machines [28], risk minimization [5], [9], [34], deep learning [1], [30], learning an unknown probability distribution over a discrete population from random samples [10], and releasing hyper-parameters and classifier accuracy [25]. By definition, differentially private models limit the success probability of membership inference attacks based solely on the model, which includes the attacks described in this paper.
Membership Inference Attacks against Machine Learning Models
We have designed, implemented, and evaluated the first membership inference attack against machine learning models, notably black-box models trained in the cloud using Google Prediction API and Amazon ML. Our attack is a general, quantitative approach to understanding how machine learning models leak information about their training datasets. When choosing the type of the model to train or a machine learning service to use, our attack can be used as one of the selection metrics.
Membership Inference Attacks against Machine Learning Models
Our key technical innovation is the shadow training technique that trains an attack model to distinguish the target model's outputs on members versus non-members of its training dataset. We demonstrate that shadow models used in this attack can be effectively created using synthetic or noisy data. In the case of synthetic data generated from the target model itself, the attack does not require any prior knowledge about the distribution of the target model's training data.
Membership Inference Attacks against Machine Learning Models
Membership in hospital-stay and other health-care datasets is sensitive from the privacy perspective. Therefore, our results have substantial practical privacy implications.
Information Leakage in Embedding Models
Machine learning (ML) has seen an explosive growth over the past decade and is now widely used across industry from image analysis [24,34], speech recognition [22], and even in applications in the medical sciences for diagnosis [8,54]. These advances rely on not only improved training algorithms and architectures, but also access to high-quality and often sensitive data.
Information Leakage in Embedding Models
The wide applications of machine learning and its reliance on quality training data necessitates a better understanding of how exactly ML models work and how they interact with their training data. Naturally, there is increasing literature focused on these aspects, both in the domains of interpretability and fairness of models, and their privacy. The latter is further of timely importance given recent regulations under the European Union's General Data Protection Regulation (GDPR) umbrella requiring users to have greater control of their data.
Information Leakage in Embedding Models
Applying a line of research investigating whether individual genomic records can be subsequently identified [28,60], Shokri et al. [63] developed the first membership inference tests investigating how ML models may leak some of their training data. This subsequently led to a large body of work exploring this space [44,47,58,59,74]. Another research direction investigating how models * Work done at Google Brain. might memorize their training data has also shown promising results [4,64]. Apart from training data privacy, modern deep learning models can unintentionally leak information of the sensitive input from the model's representation at inference time [9,18,38,49,65].
Information Leakage in Embedding Models
This growing body of work analyzing models from the standpoint of privacy aims to answer natural questions: Besides the learning task at hand, what other information do models capture or expose about their training data? And, what functionalities may be captured by ML models unintentionally? We also note that beyond the scope of this paper are other rich sets of questions around adversarial behavior with ML models-the presence of adversarial examples, data poisoning attacks, and more. Embeddings. In most of the research investigating the privacy of ML models, one class of models is largely conspicuously absent. Embeddings are mathematical functions that map raw objects (such as words, sentences, images, user activities) to real valued vectors with the goal of capturing and preserving some important semantic meaning about the underlying objects. The most common application of embeddings is transfer learning where the embeddings are pre-trained on a large amount of unlabeled raw data and later fine-tuned on downstream tasks with limited labeled data. Transfer learning from embeddings have been shown to be tremendously useful for many natural language processing (NLP) tasks such as paraphrasing [14], response suggestion [30], information retrieval, text classification, and question answering [55], where labeled data is typically expensive to collect. Embeddings have also been successfully applied to other data domains including social networks [21], source code [2], YouTube watches [11], movie feedback [25], locations [12], etc. In some sense, their widespread use should not be surprising-embedding models and their success illustrate why deep learning has been successful at capturing interesting semantics from large quantities of raw data.
Information Leakage in Embedding Models
Embedding models are often pre-trained on raw and unlabeled data at hand, then used with labeled data to transfer learning to various downstream tasks. Our study of embeddings is motivated by their widespread application and trying to better understand how these two stages of training may capture and subsequently leak information about sensitive data. While it is attractive that these models inherently capture relations and similarities and other semantic relationships between raw objects like words or sentences, it also behooves one to consider if this is all the information being captured by the embeddings. Do they, perhaps, in addition to meaning of words inadvertently capture information about the authors perhaps? Would that have consequences if these embeddings are used in adversarial ways? What aspects of the seminal research into privacy of ML models through the lens of membership inference attacks and memorization propensity can we apply to better understand the privacy risks of embeddings?
Information Leakage in Embedding Models
These questions and more lead us to initiate a systematic study of the privacy of embedding models by considering three broad arXiv:2004.00053v1 [cs.LG] 31 Mar 2020 classes of attacks. We first consider embedding inversion attacks whose goal is to invert a given embedding back to the sensitive raw text inputs (such as private user messages). For embeddings susceptible to these attacks, it is imperative that we consider the embedding outputs containing inherently as much information with respect to risks of leakage as the underlying sensitive data itself. The fact that embeddings appear to be abstract real-numbered vectors should not be misconstrued as being safe.
Information Leakage in Embedding Models
Along the lines of Song and Shmatikov [65], we consider attribute inference attacks to test if embeddings might unintentionally reveal attributes of their input data that are sensitive; aspects that are not useful for their downstream applications. This threat model is particularly important when sensitive attributes orthogonal to downstream tasks are quickly revealed given very little auxiliary data, something that the adversary might be able to capture through external means. As an example, given (sensitive) demographic information (such as gender) that an adversary might retrieve for a limited set of data points, the embeddings should ideally not enable even somewhat accurate estimates of these sensitive attributes across a larger population on which they might be used for completely orthogonal downstream tasks (e.g. sentiment analysis).
Information Leakage in Embedding Models
Finally, we also consider classic membership inference attacks demonstrating the need to worry about leakage of training data membership when given access to embedding models and their outputs, as is common with many other ML models. Our Contributions. As discussed above, we initiate a systematic and broad study of three classes of potential information leakage in embedding models. We consider word and sentence embedding models to show the viability of these attacks and demonstrate their usefulness in improving our understanding of privacy risks of embeddings. Additionally, we introduce new techniques to attacks models, in addition to drawing inspiration from existing attacks to apply them to embeddings to various degrees. Our results are demonstrated on widely-used word embeddings such as Word2Vec [45], FastText [3], GloVe [52] and different sentence embedding models including dual-encoders [39] and BERT [13,35].
Information Leakage in Embedding Models
(1) We demonstrate that sentence embedding vectors encode information about the exact words in input texts rather than merely abstract semantics. Under scenarios involving blackbox and white-box access to embedding models, we develop several inversion techniques that can invert the embeddings back to the words with high precision and recall values (exceeding 60% each) demonstrating that a significant fraction of the inputs may be easily recovered. We note that these techniques might be of independent interest and apply to other scenarios. (2) We discover certain embeddings training frameworks in particular favor sensitive attributes leakage by showing that such embedding models improve upon state-of-the-art stylometry techniques [57,62] to infer text authorship with very little training data. Using dual encoder embeddings [39] trained with contrastive learning framework [61] and 10 to 50 labeled sentences per author, we show 1.5-3× fold improvement in classifying hundreds of authors over prior stylometry methods and embeddings trained with different learning paradigms.
Information Leakage in Embedding Models
(3) We show that membership inference attacks are still a viable threat for embedding models, albeit to a lesser extent. However, given our results that demonstrate adversary can achieve a 30% improvement on membership information over random guessing on both word and sentence embeddings, we show that is prudent to consider these potential avenues of leakage when dealing with embedding models. (4) Finally, we propose and evaluate adversarial training techniques to minimize the information leakage via inversion and sensitive attribute inference. We demonstrate through experiments their applicability to mitigating these attacks. We show that embedding inversion attacks and attribute inference attacks against our adversarially trained model go down by 30% and 80% respectively. These come at a minor cost to utility indicating a mitigation that cannot simply be attributed to training poorer embeddings.
Information Leakage in Embedding Models
Paper Organization. The rest of the paper is organized as follows. Section 2 introduces preliminaries needed for the rest of the paper. Sections 3, 4, and 5 cover attacks against embedding models-inversion attacks, sensitive attributes inference attacks, and membership inference attacks respectively. Our experimental results and proposed defenses are covered in Sections 6 and 7 respectively. We describe related work and conclude in the last two sections.
Information Leakage in Embedding Models
Embedding models are widely used machine learning models that map a raw input (usually discrete) to a low-dimensional vector. The embedding vector captures the semantic meaning of the raw input data and can be used for various downstream tasks including nearest neighbor search, retrieval, and classification. In this work, we focus on embeddings of text input data as they are widely used in many applications and have been studied extensively in research community [3,5,13,33,35,39,45,52,53,56]. Word embeddings. Word embeddings are look-up tables that map each word w from a vocabulary V to a vector v w ∈ R d . Word embedding vectors capture the semantic meaning of words in the following manner: words with similar meanings will have small cosine distance in the embedding space, the cosine distance of v 1 and v 2 defined as 1 − (v ⊤ 1 · v 2 /∥v 1 ∥∥v 2 ∥). Popular word embedding models including Word2Vec [45], Fast-Text [3] and GloVe [52] are learned in an unsupervised fashion on a large unlabeled corpus. In detail, given a sliding window of words C = [w b , ... , w 0 , ... , w e ] from the training corpus, Word2Vec and FastText train to predict the context word w i given the center word w 0 by maximizing the log-likelihood log P V (w i |w 0 ) where
Information Leakage in Embedding Models
for each w i ∈ C/{w 0 }. To accelerate training, the above probability is calculated against V neg ⊂ V, a sampled subset of words not in C instead of the entire vocabulary. GloVe is trained to estimate the co-occurrence count of w i and w j for all pairs of w i , w j ∈ C.
Information Leakage in Embedding Models
A common practice in modern deep NLP models is to use word embeddings as the first layer so that discrete inputs are mapped to a continuous space and can be used for later computation. Pretrained word embeddings is often used to initialize the weights of the embedding layer. This is especially helpful when the downstream NLP tasks have a limited amount of labeled data as the knowledge learned by these pre-trained word embeddings from a large unlabeled corpus can be transferred to the downstream tasks. Sentence embeddings. Sentence embeddings are functions that map a variable-length sequence of words x to a fix-sized embedding vector Φ(x) ∈ R d through a neural network model Φ. For a input sequence of ℓ words x = [w 1 , w 2 , ... , w ℓ ], Φ first maps x into a sequence of word vectors X = [v 1 , ... , v ℓ ] with a word embedding matrix V . Then Φ feeds X to a recurrent neural networks or Transformer [67] and obtain a sequential hidden representation [h 1 , h 2 , ... , h ℓ ] for each word in x. Finally Φ outputs the sentence embedding by reducing the sequential hidden representation to a single vector representation. Common reducing methods include taking the last representation where Φ(x) = h ℓ and mean-pooling where Φ(x) = (1/ℓ) · ℓ i=1 h i . Sentence embedding models are usually trained with unsupervised learning methods on a large unlabeled corpus. A popular architecture for unsupervised sentence embedding is the dual-encoder model proposed in many prior works [5,7,26,39,40,56,72]. The dual-encoder model trains on a pair of context sentences (x a , x b ) where the pair could be a sentence and its next sentence in the same text or a dialog input and its response, etc. Given a randomly sampled set of negative sentences X neg that are not in the context of x a , x b , the objective of training is to maximizes the log-likelihood
Information Leakage in Embedding Models
.
Information Leakage in Embedding Models
(
Information Leakage in Embedding Models
Intuitively, the model is trained to predict the correct context x b from the set {x b } ∪ X neg when conditioned on x a . In other words, the similarity between embeddings of context data is maximized while that between negative samples is minimized. Sentence embeddings usually outperform word embeddings on transfer learning. For downstream tasks such as image-sentence retrieval, classification, and paraphrase detection, sentence embeddings are much more efficient for the reason that only a linear model needs to be trained using embeddings as the feature vectors. Pre-trained language models. Language models are trained to learn contextual information by predicting next words in input text, and pre-trained language models can easily adopt to other NLP tasks by fine-tuning. The recently proposed Transformer architecture [67] enables language models to have dozen of layers with huge capacity. These large language models, including BERT [13], GPT-2 [53], and XLNet [73], are trained on enormous large corpus and have shown impressive performance gains when transferred to other downstream NLP tasks in comparison to previous state of the art methods.
Information Leakage in Embedding Models
These pre-trained language models can also be used to extract sentence embeddings. There are many different ways to extract sentence-level features with pre-trained language models. In this work, we follow Sentence-BERT [56] which suggests that meanpooling on the hidden representations yields best empirical performance.
Information Leakage in Embedding Models
In this section, we give an overview of threat models we consider in this paper and describe a taxonomy of attacks leaking information from embedding models. Figure 1 shows an overview of the attack taxonomy.
Information Leakage in Embedding Models
In many NLP applications, embedding vectors are often computed on sensitive user input. Although existing frameworks propose computing embeddings locally on user's device to protect the privacy of the raw data [6,36,49,68], the embeddings themselves are shared with machine learning service providers or other parties for downstream tasks (training or inference). It is often tempting to assume that sharing embeddings might be "safer" than sharing the raw data, by their nature of being "simply a vector of real numbers. " However, this ignores the information that is retained in, and may be extracted from embeddings. We investigate the following questions: What kinds of sensitive information about the inputs are encoded in the embeddings? And can an adversary with access to the embeddings extract the encoded sensitive information?
Information Leakage in Embedding Models
Threat model. Our threat model comprises the following entities. (1) D train is a training dataset that may contain sensitive information. (2) Φ, the embedding model, which might be available in a white-box or black-box fashion. White-box access to the model reveals the model architecture and all parameters. Blackbox access allows anyone to compute Φ(x) for x of their choice.
Information Leakage in Embedding Models
(3) E target = {Φ(x * i )}, a set of embedding vectors on sensitive inputs x * i . (4) D aux is an auxiliary dataset available to the adversary comprising of either limited labeled data drawn from the same distribution as D train or unlabeled raw text data. In the text domain, unlabeled data is cheap to collect due to the enormous amount free text available on the Internet while labeling them is often much more expensive.
Information Leakage in Embedding Models
An adversary, given access to some of the entities described above, aims to leak some sensitive information about the input x from the embedding vector Φ(x). By looking at variants of what information an adversary possesses and their target, we arrive at three broad classes of attacks against embeddings.
Information Leakage in Embedding Models
With sensitive input data such as personal messages, it is natural to consider an adversary whose goal is to (partially) recover this text from Φ(x). Even if the input data is not sensitive, they may be associated with sensitive attributes, and an adversary could be tasked with learning these attributes from Φ(x). And finally, rather than recovering inputs, the adversary given some information about Φ(x) can aim to find if x was used to train Φ or not. These are formalized below. Embedding inversion attacks. In this threat model, the adversary's goal is to invert a target embedding Φ(x * ) and recover words in x * . We consider attacks that involve both black-box and whitebox access to Φ. The adversary is also allowed access to an unlabeled D aux and in both scenarios is able to evaluate Φ(x) for x ∈ D aux .
Information Leakage in Embedding Models
Sensitive attribute inference attacks. In this threat model, the adversary's goal is to infer sensitive attribute s * of a secret input x * from a target embedding Φ(x * ). We assume the adversary has Figure 1: Taxonomy of attacks against embedding models. We assume adversary has access to the embedding Φ(x * ) of a sensitive input text x * that will be used for downstream NLP tasks, and perform three information leakage attacks on Φ(x * ): (1) inverting the embedding back to the exact words in x * , (2) inferring sensitive attribute of x * , and (3) inferring the membership, i.e. whether x * and its context x ′ has been used for training. access to Φ, and a set of labeled data of the form (x, s) for x ∈ D aux . We focus on discrete attributes s * ∈ S, where S is the set of all possible attribute classes, and an adversary performing the inference by learning a classifier f on D aux . Given sufficient labeled data , the adversary's task trivially reduces to plain supervised learning, which would rightly not be seen as adversarial. Therefore, the more interesting scenario, which is the focus of this paper, is when the adversary is given access to a very small set of labeled data (in the order of 10-50 per class) where transfer learning from Φ(x) is likely to outperform supervised learning directly from inputs x. Membership inference attacks. Membership inference against ML models are well-studied attacks where the adversary has a target data point x * and the goal is to figure out with good accuracy whether x * ∈ D train or not. Unlike previous attacks focused on supervised learning, some embedding models, such as word embeddings, allow you to trivially infer membership. For word embeddings, every member of a vocabulary set is necessarily part of D train . Instead, we expand the definition of training data membership to consider this data with their contexts, which is used in training. We assume the adversary has a target context of word [w * 1 , ... , w * n ] and access to V for word embedding, or a context of target sentences (x * a , x * b ) and access to the model Φ for sentence embedding, and the goal is to decide the membership for the context. We also consider the target to be an aggregated level of data sentences [x * 1 , ... , x * n ] comprising multiple contexts for the adversary to determine if it were part of training Φ. Membership privacy for aggregation in the user level has also been explored in prior works [43,64].
Information Leakage in Embedding Models
We further assume adversary has a limited D aux labeled with membership. We propose that this is a reasonable and practical assumption as training data for text embeddings are often collected from the Internet where an adversary can easily inject data or get access to small amounts of labeled data through a more expensive labeling process. The assumption also holds for adversarial participants in collaborative training or federated learning [42,43].
Information Leakage in Embedding Models
The goal of inverting text embeddings is to recover the target input texts x * from the embedding vectors Φ(x * ) and access to Φ. A naïve  Vector similarity function ∆ A set of embedding similarity scores λ w , λ s Coefficient for adversarial training D aux Auxiliary data set available to the adversary approach for inversion would be enumerating all possible sequences from the vocabulary and find the recoveryx such that Φ(x) = Φ(x * ).
Information Leakage in Embedding Models
Although such brute-force search only requires black-box access to Φ, the search space grows exponentially with the length of x and thus inverting by enumerating is computationally infeasible. The brute-force approach does not capture inherently what information might be leaked by Φ itself. This naturally raises the question of what attacks are possible if the adversary is given complete access to the parameters and architecture of Φ, i.e., white-box access. This also motivates a relaxation technique for optimizationbased attacks in Section 3.1. We also consider the more constrained Algorithm 1 White-box inversion 1: Input: target embedding Φ(x * ), white-box embedding model Φ with lower layer representation function Ψ, temperature T , sparsity threshold τ sp , auxiliary dataset D aux 2: Query function Φ and Ψ with D aux and collect
Information Leakage in Embedding Models
while objective function of Eq 7 not converged do 7: Update z with gradient of Eq 7.
Information Leakage in Embedding Models
Project z to non-negative orthant. 9:
Information Leakage in Embedding Models
while objective function of Eq 5 not converged do 13: Update Z with gradient of Eq 5.
Information Leakage in Embedding Models
14:
Information Leakage in Embedding Models
black-box access scenario in Section 3.2 where we develop learning based attacks that are much more efficient than exhaustive search by utilizing auxiliary data.
Information Leakage in Embedding Models
In a white-box scenario, we assume that the adversary has access to the embedding model Φ's parameters and architecture. We formulate white-box inversion as the following optimization problem:
Information Leakage in Embedding Models
where X(V) is the set of all possible sequences enumerated from the vocabulary V. The above optimization can be hard to solve directly due to the discrete input space. Inspired by prior work on relaxing categorical variables [29], we propose a continuous relaxation of the sequential word input that allows more efficient optimization based on gradients.
Information Leakage in Embedding Models
The goal of the discrete optimization in Equation 3 is to select a sequence of words such that the distance between the output embeddings is minimized. We relax the word selection at each position of the sequence with a continuous variable z i ∈ R |V | . As mentioned before, Φ first maps the input x of length ℓ into a sequence of word vectors X = [v 1 , ... , v ℓ ] and then computes the text embedding based on X . For optimizing z i , we represent the selected word vectorsv i using a softmax attention mechanism:
Information Leakage in Embedding Models
where V is the word embedding matrix in Φ and T is a temperature parameter. The softmax function approximates hard argmax selection for T < 1. Intuitively, softmax(z i /T ) models the probabilities of selecting each word in the vocabulary at position i in the sequence andv i is the average of all word vectors weighted by the probabilities. Let Z = [z 1 , ... , z ℓ ] ∈ R ℓ× | V | and relaxed(Z ,T ) = [v 1 , ... ,v ℓ ] be the sequence of softmax relaxed word vectors. For simplicity, we denote Φ(X ) as the text embedding computed from any sequence of word vectors X . Then our relaxed optimization problem is:
Information Leakage in Embedding Models
With continuous relaxation and white-box access to Φ, the optimization problem can now be solved by gradient-based methods as gradients of Z can be calculated using back-propagation. To recover, we computex = {w i |i = arg max z j } ℓ j=1 . Inverting embedding from deep models. Prior works [15,16,41,66] demonstrated that inverting image features from the higher layers of deep model can be challenging as representations from the higher layers are more abstract and generic while inverting from lower-layer representations results in better reconstruction of the image. With the recent advance in Transformer models [67], text embeddings can also be computed from deep models with many layers such as BERT [13]. Directly inverting such deep embeddings using the relaxed optimization method in Equation 5 can results in inaccurate recovery as the optimization becomes highly non-convex and different sequences with similar semantics can be mapped to the same location in the high-level embedding space. In reality, however, it is more common to use the embeddings from the higher layers than from lower layers for downstream tasks and thus an adversary might not be able to observe embeddings from lower layers at all.
Information Leakage in Embedding Models
To resolve this issue, we propose to invert in two stages: (1) the adversary first maps the observed higher layer embedding Φ(x) to a lower layer one with a learned mapping function M, and (2) the adversary then solves the optimization problem of minimizing ||Ψ(x) − M(Φ(x * ))|| 2 2 where Ψ denotes the lower layer embedding function. To learn the mapping function M, adversary queries the white-box embedding model to get (Φ(x i ), Ψ(x i )) for each auxiliary data x i ∈ D aux and trains M with the set {(Φ(x i ), Ψ(x i ))} i . After M is trained, adversary solve the relaxed optimization:
Information Leakage in Embedding Models
In practice, we find that learning a linear least square model as M works reasonably well. A special case of inverting lowest representation. The lowest embedding we can compute from the text embedding models would be the average of the word vectors, i.e., Ψ(
Information Leakage in Embedding Models
Inverting from such embedding reduces to the problem of recovering the exact words vectors with given averaged vector. Instead of using the relaxed optimization approach in Equation 5, we use a sparse coding [48] formulation as following:
Information Leakage in Embedding Models
where V is the word embedding matrix and the variable z quantifies the contribution of each word vector to the average. We constrain z to be non-negative as a word contributes either something positive if it is in the sequence x or zero if it is not in x. We further penalize z with L 1 norm to ensure its sparsity as only few words from the vocabulary contributed to the average. The above optimization can be solved efficiently with projected gradient descent, where we set the coordinate z j to 0 if z j < 0 after each descent step. The final recovered words are those with coefficient z j > τ sp for a sparsity threshold hyper-parameter τ sp .
Information Leakage in Embedding Models
In the black-box scenario, we assume that the adversary only has query access to the embedding model Φ, i.e., adversary observes the output embedding Φ(x) for a query x. Gradient-based optimization is not applicable as the adversary does not have access to the model parameters and thus the gradients. Instead of searching for the most likely input, we directly extract the input information retained in the target embedding by formulating a learning problem. The adversary learns an inversion model ϒ that takes a text embedding Φ(x) as input and outputs the set of words in the sequence x. We focus on inverting embeddings of short texts and for practical reasons, it suffices to analyze the privacy of the inputs by considering attacks that recover a set of words without knowing the word ordering. We denote W(x) as the set of words in the sequence x. The adversary utilizes the auxiliary dataset D aux and queries the black-box Φ and obtain a collection of (Φ(x), W(x)) for each x ∈ D aux . The adversary then trains the inversion model ϒ to maximize log P ϒ (W(x)|Φ(x)) on the collected set of (Φ(x), W(x)) values. Once ϒ is trained, the adversary predicts the words in the target sequence x * as ϒ(Φ(x * )) for an observed Φ(x * ).
Information Leakage in Embedding Models
Multi-label classification. The goal is to predict the set of words in sequence x given the embedding Φ(x). A common choice for such a goal is to build a multi-label classification (MLC) model, where the model assigns a binary label of whether a word is in the set for each word in the vocabulary. The training objective function is then:
Information Leakage in Embedding Models
whereŷ w = P ϒ (y w |Φ(x)) is the predicted probability of word w given ϒ conditioned on Φ(x) and y w = 1 if word w is in x and 0 otherwise. Multi-set prediction. One drawback in the above multi-label classification model is that the model predicts the appearance of each word independently. A more sophisticated formulation would be predicting the next word given the current predicted set of words until all words in the set are predicted. We adopt the multiset prediction loss [69] (MSP) that is suited for the formulation. Our MSP model trains a recurrent neural network that predicts the next word in the set conditioned on the embedding Φ(x) and current predicted set of words. The training objective is as follows:
Information Leakage in Embedding Models
where W i is the set of words left to predict at timestamp i and W <i is the set of the predicted words before i. The MSP formulation allows ϒ to learn a policy on the order of the words should be predicted instead of predicting all words independently and simultaneously. In Section 6.2, we empirically show that MSP outperforms MLC in terms of the precision-recall trade-off. 
Information Leakage in Embedding Models
Predict a wordŵ = arg max P ϒ (w |W <i , Φ(x)). 6: Sample a batch B ⊂ D aux . 12:
Information Leakage in Embedding Models
Update ϒ with ∇L MSP .
Information Leakage in Embedding Models
Embeddings are designed to encode rich semantic information about the input data. When the input data is user-related, e.g., a user's video watch history, the embedding vector naturally captures information about the user and is often much more informative than the raw input. Although in many applications such rich information from the embeddings is desired in order to provide personalized services to the user, the embeddings may potentially reveal sensitive information about the input that might not directly appear in or be easy to infer from the input. As a motivating example, consider an adversary that may curate a small set of public comments labeled with authors of interest and then planning to use semantically rich embeddings on unlabeled, targeted text fragments to aim to deanonymize the author of the text.
Information Leakage in Embedding Models
For inferring sensitive attributes, we assume that the adversary has a limited set D aux = {(Φ(x i ), s i )} i of embeddings labeled with the sensitive attribute, such as text authorship. The adversary then treats the inference problem as a downstream task and trains a classifier which predicts s given Φ(x) as inputs on D aux . At inference time, adversary simply applies the classifier on observed embedding Φ(x * ) to infer the sensitive attribute of x * . We focus on the scenario of the adversary only having limited labeled data so as to (a) closely match real scenarios where labeled sensitive data would be challenging to collect, and (b) demonstrate how easily sensitive information can be extracted from the embeddings which therefore constitutes an important vector of information leakage from embeddings. Connections between leakage and the objective. In supervised learning, deep representations can reveal sensitive attributes of the 
Information Leakage in Embedding Models
input as these attributes might be used as internal features for the learning task [65]. The connection between unsupervised learning tasks and the leakage of sensitive attributes is less well understood. The objective functions (Equation 1 and 2) that maximize the semantic similarity of data in context for training unsupervised dual-encoder embedding models fall into contrastive learning framework [61]. This framework theoretically explains how unsupervised embedding training helps with the downstream tasks with a utility perspective. We explore how it might also favor the inference of some sensitive attributes from the perspective ofprivacy.
Information Leakage in Embedding Models
In this framework, training data of the embedding models are associated with latent classes (e.g., authors of the texts). When training with contrastive loss, the embeddings are learned so as to be similar for data in the same context and to be dissimilar for data coming from negative samples. Our approach takes advantage of the fact that data sharing the same latent class will often appear in the same context. Therefore, embedding similarity will be closer for inputs from the same class, and consequently with the same sensitive attribute when there is a correlation. We further note that unsupervised embeddings are especially helpful for attribute inference under the limited data constraints as the embeddings are trained on much larger unlabeled data which allows them to learn semantic similarity over latent classes that might not be captured only given limited labeled data.
Information Leakage in Embedding Models
Both inverting embeddings and inferring sensitive attributes concern inference-time input privacy, i.e., information leaked about the input x from the embedding vector. Another important aspect of privacy of ML models is training data privacy, namely, what information about the training data (which might be potentially sensitive) is leaked by a model during the training process? We focus on membership inference attacks [63] as a measurement of training data leakage in the embedding models.
Information Leakage in Embedding Models
The goal of membership inference is to infer whether a data point is in the training set of a given machine learning model. Classic membership inference attacks mainly target supervised machine learning, where a data point consists of a input feature vector and a class label. For unsupervised embedding models trained on units of data in context, we thus wish to infer the membership of a context of data (e.g., a sliding window of words or a pair of sentences).
Information Leakage in Embedding Models
Prior works [58,74] on membership inference suggest that simple thresholding attacks based on loss values can be theoretically optimal under certain assumptions and practically competitive to more sophisticated attacks [63]. In embedding models, the loss is approximated based on sampling during training as described in Initialize similarity function δ ′ with projection W m .
Information Leakage in Embedding Models
while δ ′ not converged do 6: Sample a batch B ⊂ D aux . 7: Compute loss L MIA for (x a , x b ) ∈ B with Eq 10.
Information Leakage in Embedding Models
Update W m with ∇L MIA . 9: Replace similarity function δ ← δ ′ . 
Information Leakage in Embedding Models
where δ is a vector similarity measure function (e.g. cosine similarity). Finally, the adversary uses the averaged score in ∆ to decide membership: if the averaged score is above some threshold then C is a member of the training data and not a member otherwise.
Information Leakage in Embedding Models
In sentence embeddings, we wish to decide membership of a pair of sentence in context (x a , x b ). We simply use the similarity score δ (Φ(x a ), Φ(x b )) as the decision score for membership inference as sentences in context used for training will be more similar to each other than sentences which were not used for training. Aggregate-level membership inference. Sometimes, deciding membership of a pair of sentences might not be enough to cause a real privacy threat. In many user-centric applications, models are trained on aggregation of data from users; e.g., a keyboard prediction model is trained on users' input logs on their phone [43]. In this scenario, the adversary infers membership on aggregate data from a particular user to learn whether this user participated training or not.
Information Leakage in Embedding Models
To perform MIA on aggregate text of n sentences X = [x 1 , x 2 , ... , x n ], the adversary first gets each sentence embeddings [Φ(x 1 ), Φ(x 2 ), ... , Φ(x n )]. Then the adversary collects the set of similarity
Information Leakage in Embedding Models
Finally, similar to MIA against word embeddings, we use the average score in ∆ as the decision score for membership inference. Learned similarity metric function. Using a pre-defined similarity measure may not achieve best membership inference results. With auxiliary data labeled with membership information, an adversary can learn a similarity metric customized for inference membership. More specifically, they learn a projection matrix W m and computes the learned similarity as δ ′ (x a ,
Information Leakage in Embedding Models
). The attack optimizes the binary cross-entropy loss with membership labels as following: (x a , x b ) where in the training set and 0 otherwise.
Information Leakage in Embedding Models
As each of the attacks assess different perspectives of privacy, we evaluate them on different text embedding models that are either trained locally on consumer hardware or are trained elsewhere and public available. Here, we describe text embedding models (and their corresponding datasets) we trained locally and evaluated against attacks described in previous sections. Other public models and datasets are detailed in subsequent subsections. Word Embeddings on Wikipedia. We collected nearly 150,000 Wikipedia articles 1 for evaluating word embeddings. We locally trained Word2Vec [45], FastText [3], and GloVe [52] embedding models using half of the articles and use the other half for evaluating membership inference.
Information Leakage in Embedding Models
For all word embeddings, we set the number of dimension in embedding vector d to be 100. For Word2Vec and FastText, we set the number of sampled negative words |V neg | to be 25, learning rate to be 0.05, sliding window size to be 5 and number of training epochs to be 5. For GloVe, we set the number of training iterations to be 50 as suggested in the original paper [52]. Sentence Embeddings on BookCorpus. Following prior works on training unsupervised sentence embeddings [33,39], we collected sentences from BookCorpus [76] consists of 14,000 books. We sample 40 millions sentences from half of the books as training data and use the rest as held-out data.
Information Leakage in Embedding Models
We locally train sentence embedding models with dual-encoder architecture described in Section 2. We considered two different neural network architecture for the embedding models: a recurrent neural network (LSTM [27]) and a three-layer Transformer [67]. For the LSTM, we set the size of embedding dimension d to be 1,200, number of training epochs to be 1 and learning rate to be 0.0005 following previous implementation [39]. For the Transformer, we set d to be 600, number of training epochs to be 5 with a warm-up scheduled learning rate following [67]. For both architectures, we train the model with Adam optimizer [32] and set the negative samples X neg as the other sentences in the same training batch at each step and |X neg | = 800.
Information Leakage in Embedding Models
Target and auxiliary data. We randomly sample 100,000 sentences from 800 authors in the held-out BookCorpus data as the target data D target = {x * i } to be recovered and perform inversion on the set of embeddings E target = {Φ(x * i )}. We consider two types of auxiliary data D aux : same-domain and cross-domain data. For same-domain D aux , we use a set of 200,000 randomly sampled 1 http://mattmahoney.net/dc/enwik9.zip sentences from BookCorpus that is disjoint to D target . For crossdomain D aux , we use a set of 800,000 randomly sampled sentences from Wikipedia articles. We set the number of cross-domain data points to be more than the same-domain data to match real-world constraints where cross-domain data is typically public and cheap to collect. Additional embedding models. In addition to the two dualencoder embedding models with LSTM and Transformer, we further experiment with popular pre-trained language models for sentence embedding. We consider the original BERT [13] and the state-ofthe-art ALBERT [35]. We use mean pooling of the hidden token representations as the sentence embedding as described in Section 2.1. Evaluation metrics. As the goal of inversion is to recover the set of words in the sensitive inputs, we evaluate our inversion methods based on precision (the percentage of recovered words in the target inputs), recall (the percentage of words in the target inputs are predicted) and F1 score which is the harmonic mean between precision and recall. White-box inversion setup. We evaluate the white-box inversion with Equation 5 and Equation 7. For inversion with Equation 5, we set the temperature T to be 0.05. For inversion with Equation 7, we set the L 1 penalty coefficient λ sp to be 0.1 and the sparsity threshold τ sp to be 0.01. For both methods, we use Adam optimizer [32] for gradient descent with learning rate set to 0.001. White-box inversion results. , an adversary can extract more than a half and a third of the target input from LSTM and Transformer embedding models respectively as indicate by the F1 score. This method performs poorly on BERT and ALBERT models. One plausible reason is that with many more layers in BERT where higher layer embeddings are more abstract than lower layers, directly optimizing for the highest layer could lead to recovering synonyms or semantically-related words rather than the targets. For inversions with Equation 7, all performance scores increase from Equation 5 on all models. We can recover roughly half of the input texts on all models. We also notice that there is only a little loss in performance when using cross-domain data for training the mapping M.
Information Leakage in Embedding Models
We further investigate the performance of inverting embeddings from different layers in BERT and ALBERT as shown in Figure 2. There are in total 12 Transformer layers in BERT and ALBERT models and we choose embeddings from 2, 4, 6, 8, 10, 12 layer for inversion. We also compare with inverting from layer 0, i.e. mean-pooling of word embedding in BERT models. The performance drops drastically when the layer goes high when inverting with Equation 5. When training a mapping M and inverting with Equation 7, the drop in the performance is much less significant for higher layers. Black-box setup. We evaluate the black-box inversion with L MLC (Equation 8) and L MSP (Equation 9). For L MLC , we train |V | binary classifiers as ϒ for each w ∈ V. For L MSP , we train a one-layer LSTM as ϒ with number of hidden units set to 300. We train both models for 30 epochs with Adam optimizer and set learning rate to 0.001, batch size to 256. Black-box results. Table 3 summarizes the results. Inversion with L MLC can achieve high precision with low recall. This might be due to the inversion model ϒ being biased towards the auxiliary data and thus confident in predicting some words while not others. Inversion with L MSP yields better balance precision and recall and thus higher F1 scores.
Information Leakage in Embedding Models
Target and auxiliary data. We consider authorship of sentence to be the sensitive attribute and target data to be a collection of sentences of randomly sampled author set S from the held-out dataset of BookCorpus, with 250 sentences per author. The goal is to classify authorship s of sentences amongst S given sentence embeddings. For auxiliary data, we consider 10, 20, 30, 40 and 50 labeled sentences (disjoint from those in the target dataset) per author. We also vary the size of author set |S| = 100, 200, 400 and 800 where the inference task becomes harder as |S| increases. Baseline model. To demonstrate sensitive attribute leakage from the embedding vector, we compare the attack performance between embedding models and a baseline model that is trained from raw sentences without access to the embeddings. We train a TextCNN model [31] as the baseline, which is efficient to train and has been shown to achieve accurate authorship attribution in previous works [57,62]. Additional embedding models. As discussed in Section 4, the embedding models trained with dual-encoder and contrastive learning that is a focus of this paper might, in particular, favor attribute inference. We compare the dual-encoder embedding models with two other embedding models trained with different objective functions. The first is the Skip-thought embedding model [33] which is trained to generate the context given a sentence. We also evaluate on InferSent embeddings [10] that is trained with supervised natural language inference tasks. Setup. For the baseline TextCNN model, we set the number of filters in convolutional layer to 128. For all other embedding models, we train a linear classifier for authorship inference. We train all inference models for 30 epochs with Adam optimizer and set learning rate to 0.001 and batch size to 128. We repeat each experiment 5 times with sampled S using different random seed and report the averaged top-5 accuracy.
Information Leakage in Embedding Models
Results. Figure 3 demonstrates the results of authorship inference with different number of labeled data and different number of authors. The baseline TextCNN models trained from scratch with limited labeled data have the worst performance across all settings. Skip-thought and InferSent embeddings can outperform TextCNN but the gap between the performance decreases as the number of labeled examples increase. The LSTM and Transformer dual-encoder models achieve best inference results and are better than the baseline by a significant margin in all scenarios. This demonstrates that embeddings from dual-encoder models trained with contrastive learning framework [61] aid attribute inference attacks the most comparing to other pre-trained embeddings.
Information Leakage in Embedding Models
Evaluation metrics. We consider membership inference as a binary classification task of distinguishing members and non-members of training data. We evaluate the performance of membership inference attacks with adversarial advantage [74], defined as the difference between the true and false positive rate. Random guesses offer an advantage of 0. Word embedding setup. We evaluate membership inference attacks on sliding window of 5 words from Wikipedia articles. We also perform the attack separately for windows with central words having different frequencies, following the intuition that rare words are prone to more memorization [64]. Specifically, we evaluate the attack for windows with frequency of central words in decile (10th, ... , 90th percentile) ranges. We use cosine similarity for π .
Information Leakage in Embedding Models
Word embedding results. Figure 4 demonstrates the results. For frequent (10th percentile) central words, there is almost no memorization (advantage < 0.1). As this frequency decreases, the advantage increases correspondingly to roughly 0.3. FastText is most  . The x-axis is the inverse frequency percentile range (the smaller the more frequent) and the y-axis is the adversarial advantage. The ctx ′ and book ′ denote results with learned similarity resistant to these attacks possibly due to its operating on sub-word units rather than exact words. Sentence embedding setup. We evaluate membership inference attacks on context-and aggregate-level data from the BookCorpus dataset. We consider a pair of sentences for context-level data, and a collection of sentences from the same book for aggregate-level data with a goal of inferring if the book is part of the training corpus. As with word embeddings, we evaluate the attack on different frequencies (averaged across words in a sentence). We use dot-product similarity for π , and for learning-based similarity, we learn a projection matrix with 10% of training and hold-out data.
Information Leakage in Embedding Models
We optimize L MIA with the Adam optimizer for 10 epochs with learning rate set to 0.001. Sentence embedding results. Figure 5 shows the results of MIA on embeddings from LSTM and Transformer dual-encoder models. For both models, context-level MIA advantage scores are below 0.1 for all frequency ranges, indicating that adversarial does not gain much information about context-level membership from the embeddings. Learning based similarity can improve context-level MIA slightly. For aggregated book-level inference, adversaries achieve a greater advantage than context-level inference and learning-based similarity scores can boost the advantage to 0.3 for books with infrequent sentences.
Information Leakage in Embedding Models
Adversarial training. Attacks involving embedding inversions and sensitive attributes are both inference-time attacks that wish to infer information about the sensitive inputs given the output of the embedding. A common defence mechanism for such inference-time attacks is adversarial training [9,17,18,38,71]. In this framework, a simulated adversary A is trained to infer any sensitive information jointly with a main model Φ while Φ is trained to maximize the adversary's loss and minimize the primary learning objective. The embeddings trained with this minimax optimization protects sensitive information from an inference-time adversary to an extent while maintaining their utility for downstream tasks.
Information Leakage in Embedding Models
To defend against embedding inversion attacks, A is trained to predict the words in x given Φ(x). For a pair of sentences in context (x a , x b ) and a set of negative example X neg , the training objective for Φ is:
Information Leakage in Embedding Models
where W(x) is the set of words in x and the coefficient λ w controls the balance between the two terms. A natural choice for log P A is to use the multi-label classification loss L MLC in Equation 8. To defend against sensitive attribute attacks, A is trained to predict the sensitive attribute s in x from the embedding Φ(x). As above, the training objective for Φ is:
Information Leakage in Embedding Models
where the coefficient λ s controls the balance between the two terms. Both minimax training objectives can be efficiently optimized with the gradient reversal trick [20].
Information Leakage in Embedding Models
Results with adversarial training. We evaluate this adversarial training approach on dual-encoder models with LSTM and Transformer. We keep all the training hyper-parameters the same as in Section 6.1. We train multiple models under different λ w and λ s and evaluate their effects on attack performance as well as utility scores for downstream tasks. For utility measurement, we evaluate the adversarially trained embeddings on four sentence analysis benchmarks: multi-perspective question answering (MPQA) [70], text retrieval (TREC) [37], subjectivity analysis (SUBJ) [50] and Microsoft Research paraphrase corpus (MSRP) [14]. We treat all benchmarks as classification problem and train a logistic regression model for each task following previous works [33,39]. Figure 6 shows the results for adversarial training against inversion. As λ w increases, the adversary performance drops for all models. White-box inversion attacks drop most significantly. LSTM embedding models needs a larger λ w than Transformer models to achieve similar mitigation, possible due to the fact that Transformer models have more capacity and are more capable of learning two tasks. Utility scores on the benchmarks drop more drastically on embedding models using LSTM than Transformer due to the larger value of λ w . Figure 7 shows the results for adversarial training against authorship inference. As λ s increases, the adversary performance on inferring authorship drops significantly for both inference attack models trained with 10 and 50 labeled data per author. Nearly all utility scores on the four benchmarks remain rather stable for different λ s 's. This also demonstrates that different adversary tasks can have different impact on the utility of embeddings. In our case, removing input word information from embeddings is a harder task than removing authorship and thus will have a larger impact on the utility.
Information Leakage in Embedding Models
Privacy in deep representations. Prior works demonstrate that representations from supervised learning models leak sensitive attributes about input data that are statistically uncorrelated with the learning task [65], and gradient updates in collaborative training (which depend on hidden representations of training inputs) also leak sensitive attributes [44]. In contrast, this work focuses on leakage in unsupervised text embedding models and considers leakage of both sensitive attributes and raw input text. In addition, we consider a more realistic scenario where the labeled data is limited for measuring sensitive attributes leakage which is not evaluated in prior works.
Information Leakage in Embedding Models
There is a large body of research on learning privacy-preserving deep representations in supervised models. One popular approach is through adversarial training [17,71] as detailed in Section 7. The same approach as been applied in NLP models to remove sensitive attributes from the representation for privacy or fairness concern [9,18,38]. Another approach for removing sensitive attributes is through directly minimizing the mutual information between the sensitive attributes and the deep representations [46,49]. We adopt the adversarial training approach during training embedding models as defense against embedding inversion and sensitive attribute inference attacks. Inverting deep representations. In computer vision community, inverting deep image representation has been studied as a way for understanding and visualizing traditional image feature extractor and deep convolutional neural networks. Both optimization-based approach [41,66] and learning-based approach [15,16] as been proposed for inverting image representations. In contrast, we focus on text domain data which is drastically different than image domain data. Text data are discrete and sparse in nature while images are often considered as continuous and dense. Our proposed inversion methods are tailored for unsupervised text embedding models trained with recurrent neural networks and Transformers.
Information Leakage in Embedding Models
Model inversion attacks [19] use gradient based method to reconstruct the input data given a classifier model and a class label. The reconstruction is often a class representatives from the training data, e.g. averaged face images of female for a gender classifier [44]. Embedding inversion, on the other hand, takes the representation vector as input and reconstruct the exact raw input text. Membership inference and memorization. Membership inference attacks (MIA) have first been studied against black-box supervised classification models [63], and later on generative models and language models [23,64]. MIA is closely connected to generalization where overfitted models are prone to the attacks [74]. In this work, we extended the study of MIA to unsupervised word and sentence embedding models without a clear notion of generalization for such models.
Information Leakage in Embedding Models
It has been shown that deep learning models have a tendency to memorize [75]. Later work showed that adversaries can extract formatted training text from the output of text generation models [4], indicating a real privacy threat caused by memorization. In this work, we focus on embedding models where the output is an embedding vector without the possibility of extract training data directly from the output as in text generation models. Instead, we demonstrated how to measure the memorization in embeddings through MIA. Differential Privacy. Differentially-private (DP) training of ML models [1,43] involves clipping and adding noise to instance-level gradients and is designed to train a model to prevent it from memorizing training data or being susceptible to MIA. DP, which limits how sensitive the model is to a training example does not provide a defense against attacks that aim to infer sensitive attributes (which is an aggregate property of training data). The noisy training techniques are challenging for models with large parameters on large amounts of data and sensitive to the hyper-parameters [51]. Our embeddings, with over 10 million parameters in the word embedding matrices V make DP training and hyper-parameter tuning computationally infeasible. Therefore, we leave it to future work to explore how to efficiently train embeddings with DP.
Information Leakage in Embedding Models
In this paper, we proposed several attacks against embedding models exploring different aspects of their privacy. We showed that embedding vectors of sentences can be inverted back to the words in the sentences with high precision and recall, and can also reveal the authorship of the sentences with a few labeled examples. Embedding models can also leak moderate amount of membership information for infrequent data by using similarity scores from embedding vectors in context. We finally proposed defenses against the information leakage using adversarial training and partially mitigated the attacks at the cost of minor decrease in utility.
Information Leakage in Embedding Models
Given their enormous popularity and success, our results strongly motivate the need for caution and further research. Embeddings not only encode useful semantics of unlabeled data but often sensitive information about input data that might be exfiltrated in various ways. When the inputs are sensitive, embeddings should not be treated as simply "vectors of real numbers. "
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Large pre-trained language models have achieved impressive results on various natural language processing tasks (Devlin et al., 2019;Radford et al., 2019a;Raffel et al., 2020). Model sizes rapidly increase from millions to trillions of parameters and keep growing to achieve better performance and even obtain some emergent abilities (Brown et al., 2020;Chowdhery et al., 2022;Wei et al., 2022;Fedus et al., 2022;Zhang et al., 2022). Despite the Figure 1: Given a prefix, ETHICIST extracts the verbatim suffix in the training data from the GPT-Neo 1.3B model. The extracted suffix in this example leaks private information about individuals (which is masked for privacy concerns), including name, phone number, fax, pager, home fax, and email. success of large-scale pre-trained language models, recent works point out that they may memorize a considerable fraction of training data, leading to the privacy risk of information leakage (Carlini et al., 2022a;Tirumala et al., 2022a;Mireshghallah et al., 2022;Carlini et al., 2021). Furthermore, researchers find that memorization scales with model sizes (Carlini et al., 2022a). Therefore, this privacy risk becomes more and more critical in the era of large-scale pre-training. And attacking language models to extract their training data attracts increasing attention.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
There are currently two main settings to extract training data. One is membership inference attack, which infers whether a given example is contained in the model's training data (Hisamoto et al., 2020;Shokri et al., 2017). The other is untargeted train-ing data extraction (Carlini et al., 2021), which aims to extract training data from scratch (i.e., without the given prefix). However, both settings are not suitable for extracting targeted training data. For example, attackers may feed the model with a prefix indicating the beginning of an email and try to extract the following private email content in the training dataset as shown in Figure 1. In such cases, we do not have complete examples to do membership inference, and we have specific goals instead of performing untargeted extraction. Therefore, we focus on targeted training data extraction in this paper, which requires recovering the suffix when given a prefix according to the training data. Compared with untargeted training data extraction, the task matters more because attackers can recover specific types of training data instead of any possible training data that might be harmless. What's more, it is easier to evaluate targeted training data extraction because we just need to compare the prediction with the ground truth suffix. However, for untargeted training data extraction, we need to search over the whole massive pre-training dataset (e.g., The Pile dataset (Gao et al., 2020), which has 800GB text data) to check whether it contains the generated sample, which is very slow and costly.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
The general process for targeted training data extraction can be divided into two steps: (1) generating one or more possible suffixes based on the given prefix, and (2) choosing a most likely suffix as the prediction result based on a confidence estimation method. We summarize two challenges of this task: (1) how to increase the generation likelihood of the ground truth suffix, and (2) how to estimate the confidence accurately so that the confidence score can be meaningfully interpreted as the probability that the output suffix is correct. To tackle these challenges, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation. For the first challenge, we propose loss smoothed soft prompting. It uses soft prompt to elicit memorization in the attacked model, and adds an additional loss besides the maximum likelihood estimation (MLE) loss to smooth the loss distribution of the suffix tokens. Through the loss smoothing, we hope to ensure that the probability of the ground truth token at each time step is not low, which makes it more likely to sample the ground truth suffix. With the two loss functions, we tune the prepended soft prompt tokens on an extracted training set which contains pairs of prefixes and ground truth suffixes. The existence of a training set is reasonable because large-scale pre-trained data generally contain public data (e.g., Common Crawl) 1 . For the second challenge, we propose a calibrated confidence estimation method. We find that the model's perplexity cannot accurately represent the probability that the generated suffix is correct because the prediction probabilities for diversified prefixes are inherently different and incomparable. We thus normalize the confidence of the generated suffixes with a local estimation, which can mitigate the problems caused by intrinsic differences in the difficulties of distinct samples. We verify ETHICIST on a recently proposed public benchmark containing 15,000 pairs of prefixes and suffixes derived from The Pile dataset (Gao et al., 2020). Experiments show that ETHICIST can significantly improve the extraction performance, which suggests that existing large language models are at significant risk of leaking training data. We also discuss and analyze several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Our contributions can be summarized as follows:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
• We propose loss smoothed soft prompting to reduce the difficulties of sampling the ground truth suffixes.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
• We propose a calibrated confidence estimation method that enables the confidence score to be meaningfully interpreted as the probability that the output suffix is correct.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
• Experiments on a recently proposed benchmark demonstrate that ETHICIST can consistently and significantly improve the data extraction performance across various model sizes. We further investigate several factors influencing the data extraction performance.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
2 Related Work
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Existing works on training data extraction mainly focus on membership inference attack or untargeted training data extraction. For membership inference attack, adversaries need to judge whether a given example is contained in the training data of the attacked model. Shokri et al. (2017);Song and Shmatikov (2019) train several shadow models that mimic the attacked models' behaviors to help train an auditing model that can predict whether an example is contained in the training dataset. Hisamoto et al. (2020) perform membership inference attacks on machine translation systems. They find it is harder to attack sequence generation models than classification models. Song and Raghunathan (2020) show that the encoded dense representations can leak information under membership inference attack. Mireshghallah et al. (2022) focuses on attacking masked language models that are pre-trained on possibly sensitive data (e.g., clinical notes). They introduce an additional reference masked language model besides the original attacked model and compute the ratio of the likelihood measured by the attacked model and the reference model, which is better than solely relying on the attacked model. For untargeted training data extraction, adversaries first generate various samples using the attacked model and then predict whether they are contained in its training set. Carlini et al. (2021) extract hundreds of verbatim sequences from the popular pre-trained language model GPT-2 (Radford et al., 2019b). And there is privacy information such as names, phone numbers, and email addresses in the extracted sequences. Lehman et al. (2021) try to extract sensitive information from BERT (Devlin et al., 2019) pre-trained on clinical notes. However, they are mostly unable to meaningfully expose Personal Health Information by simply using templates. Different from the existing works, we focus on targeted training data extraction that aims to recover the suffix when given a prefix, which is more security-critical and easier to evaluate.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We generally expect models can gain the generalization ability from the training process. However, recent works point out that models may unintentionally memorize the training data even without overfitting (Tirumala et al., 2022a;Carlini et al., 2022aCarlini et al., , 2019Béguelin et al., 2020). One possible method to mitigate this problem is to deduplicate training data (Kandpal et al., 2022). However, Carlini et al. (2019) also show that it is possible to recover samples appearing only once in the training dataset. Surprisingly, Tirumala et al. (2022a) find that there is a forgetting baseline during the pre-training of the casual language model (e.g., the model can memorize at least 40% of the data that appear only once, even being trained on other data for many epochs afterward). These findings further emphasizes the difficulties of avoiding memorization and the potential threats of unintended memorization in large-scale pre-trained language models. Another line of work uses differential privacy to avoid the memorization problem (Abadi et al., 2016;McMahan et al., 2018;Shokri and Shmatikov, 2015), but the mechanism could reduce the accuracy (Jayaraman and Evans, 2019; Feldman and Zhang, 2020;Feldman, 2020;Song and Shmatikov, 2019). Differential privacy also increases the training time, which can further influence the accuracy within the same budget. Therefore there is still no effective and practical way to avoid unintended memorization. Our work further verifies the existence of unintended memorization and makes it more necessary to develop practical defense methods.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We formulate the targeted training data extraction task as follows: given a source prefix S = (s 1 , s 2 , · · · , s |S| ) with |S| tokens, the attacker should predict the target suffix T = (t 1 , t 2 , · · · , t |T | ) with |T | tokens and its confidence. The pair of the given prefix and the predicted suffix (S, T ) should be contained in the pre-training dataset D pretrain = {(S i , T i )}, which the attacked model M is trained on. The prediction of the confidence score is necessary for picking out the most probable suffix when we don't know the ground truth suffix in realistic attack scenarios (i.e., we need to pick out most probable pairs of prefixes and extracted suffixes based on their confidence scores among all predictions). We assume the attacker can obtain some pairs of ground truth prefixes and
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
before attacking, which is reasonable because large-scale pre-trained data generally contain public data (e.g., Common Crawl). The attackers can utilize D train to train their attacking models and their goal is to predict suffixes for the prefixes in the test set
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
An overview of ETHICIST is shown in Figure 2. We first tune the soft prompt embeddings during training to elicit memorization in the attacked model M
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Training 
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Confidence , ( ) Figure 2: Method overview. During training, we fix the parameters of the attacked model M and only tune the parameters of the soft prompt embeddings. Besides the MLE loss, we additionally design a smoothing loss to make the loss distribution of the suffix sequence more smooth. After tuning the soft prompt embeddings, we extract training data by repeatedly sampling K suffixes conditioned on one given prefix and using calibrated confidence estimation to select the final predicted suffix T best and provide its confidence C(T best ).
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
with the MLE loss and the additional smoothing loss. The smoothing loss aims to increase the probability of sampling the ground truth suffix. After prompt tuning, we repeatedly sample K suffixes using the attacked model M conditioned on one given prefix and reorder them with our calibrated confidence estimation. Our calibrated confidence estimation can not only select the most possible suffix, but also provide a more accurate confidence score that represents how likely the predicted suffix is correct. Finally, the suffix with the highest confidence is selected as the final prediction.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We adopt prompt tuning to train the soft prompt tokens on D, which prepends |X| soft tokens X = (x 1 , x 2 , · · · , x |X| ) before the original input sequence. Then we feed the input to the attacked model M to compute the MLE loss:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Note that we only tune the parameters of the soft prompt tokens and the parameters of the attacked model M are fixed. We use prompt tuning for two reasons: (1) we do not want to change the original parameters of the attacked model M because the main goal is to elicit memorization in M , and (2) prompt tuning is helpful to improve the training efficiency when M is very large, making ETHICIST able to efficiently adapt to larger language models that generally memorize more training data. The MLE loss aims to increase the total generation probability of the target suffix T . However, when using popular sampling methods such as topk sampling (Fan et al., 2018) and top-p (nucleus) sampling (Holtzman et al., 2020) to generate multiple candidate suffixes, we want to ensure the probability of the ground truth suffix token at each time step is not low. Suppose the total probability of the ground truth suffix is high while there is one token in the sequence with a low generation probability. In this case, it is still hard to generate the correct suffix using auto-regressive sampling methods. Therefore, we propose a smoothing loss to make the loss distribution of the suffix sequence more smooth. More specifically, we pick out the top-N tokens with the highest loss values in the whole sequence T . Then we additionally optimize the generation probabilities for these N tokens as follows:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
where t σ(i) represents the token with the i-th highest loss in T . Note that t σ(i) is dynamically computed during training. The smoothing loss can also be seen as assigning higher weights to the tokens with higher loss values. Finally, we derive the overall loss function as follows:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
where the coefficient α is a hyperparameter to control the strength of the smoothing loss.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
After predicting the suffix, we also need to give a confidence score for the prediction, which can be meaningfully interpreted as the probability that the output suffix is correct. A naive method is to use the generation likelihood P T = exp(−|T |L MLE ) as the confidence score. This naive method is reasonable for picking out the most probable suffix T i from a collection of sampled suffixes {T 1 , T 2 , · · · , T M } for one given prefix. However, it is unsuitable for comparing the confidence of different predicted suffixes corresponding to different prefixes. As the language model is essentially a statistical model, frequencies of tokens and n-grams in the prefixes can greatly influence the absolute generation likelihood of the suffixes. For example, consider two predicted suffixes T A and T B conditioned on two different prefixes S A and S B , where S A and T A contain tokens and n-grams with much higher frequencies. The absolute generation likelihood of T A may be significantly higher than T B , even if they are both ground truth suffixes. Therefore, to eliminate the intrinsic differences in scales of generation likelihood across different suffixes, we propose a novel calibrated confidence estimation method. To calibrate the confidence estimation, we have two considerations: (1) different generated suffixes conditioned on one given prefix should have comparable scales of generation likelihood, and (2) the memorized ground truth suffix is expected to be generated more frequently during multiple generations, which is also validated in Section 5. Suppose the sampled distinct suffixes are {T 1 , T 2 , · · · , T M } for one given prefix, the repeated generation times for these suffixes are {r 1 , r 2 , · · · , r M } (i.e., r i denotes how many times T i is generated among K repeated sampling outputs), and the MLE loss values for these suffixes are {L 1 MLE , L 2 MLE , · · · , L M MLE }. Then we assign the calibrated confidence score to T i as:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Through the proposed confidence estimation method, we obtain the confidence score of T i by comparing it with other sampled suffixes with comparable scales of generation likelihood. In this way, we avoid the scale problem brought by different prefixes and make it practical to compare the predicted suffixes conditioned on different prefixes. Moreover, we leverage the repetition time r i as a valuable signal since memorized suffix is expected to be generated more frequently. Finally, we select the suffix T best with the highest confidence score C(T best ) among {C(T 1 ), C(T 2 ), · · · , C(T M )} as the predicted suffix and C(T best ) as its confidence estimation.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We evaluate ETHICIST on the LM-Extraction benchmark 2 , which is designed for benchmarking targeted training data extraction attacks. It consists of a subset contained in The Pile dataset (Gao et al., 2020).  
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We compare ETHICIST with the following baselines. All the compared baselines first sample K suffixes {T 1 , T 2 , · · · , T K } conditioned on one given prefix S and then pick out one suffix as the prediction.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Perplexity It leverages the perplexity (PPL) measured by the attacked language model M as the metric to sort the candidate suffixes and finally chooses the one with the lowest PPL as the predicted suffix T :
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Comparing (LM) It takes another language model M ′ and leverages the ratio of the perplexity measured by theses two language models as the metric (Carlini et al., 2021):
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
The language model M ′ could be a much smaller model trained on the same dataset with M or trained on a different dataset.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Comparing (zlib) Different from Comparing (LM), it uses the zlib (Gailly and Adler, 2004) entropy of the text (i.e., the number of bits after compression with zlib) for comparison (Carlini et al., 2021):
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Comparing (lowercase) It compares the perplexity of the original text and the lower-cased text measured by the same language model M (Carlini et al., 2021):
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Furthermore, we conduct ablation tests by removing the proposed components respectively to investigate the influence of each component.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We adopt the following automatic metrics for evaluation.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Recall The metric computes the percentage of the suffixes that are predicted verbatim over the whole test set. A higher recall score indicates better data extraction ability, which can also be understood as a higher attacking success rate.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
The metric first sorts the predictions according to their confidence scores and then evaluates the correctness of each prediction one by one. It finally computes the Recall score while making x incorrect predictions. We set x to 100 in our experiments following the LM-Extraction benchmark. A better confidence estimation method can give the correct predictions higher confidence scores and thus lead to a higher Recall Early stop score. Table 2 shows the automatic evaluation results with GPT-Neo 1.3B as the backbone. ETHICIST achieves an impressive Recall score of 62.8% and outperforms all the baselines by a large margin, indicating its better ability to extract training data from language models. Moreover, ETHICIST has better confidence estimation performance after calibration as shown by a significantly higher Recall Early stop score. To further investigate the influence of each component, we run an ablation study. From the results shown in Table 2, it can be seen that both the smoothing loss and the calibrated confidence estimation are important to enhance the ability to extract training data, and combining both of them achieves the best performance. Furthermore, we draw the following conclusions:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
(1) With prompt tuning and extra training data, we can better induce large-scale language models to generate their memorized training data and successfully achieves a 9.5% performance improvement on Recall and a 12.4% performance improvement on Recall Early stop .
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
(2) The proposed smoothing loss can further enhance the ability to extract training data, boosting the Recall score from 60.8% to 62.3%. (3) The calibrated confidence provides a 6.3% improvement on Recall Early stop as expected, demonstrating the importance of calibrating confidence estimation for this task. (4) The smoothing loss is more effective in predicting exact suffixes while the calibrated confidence is more beneficial for identifying highly confident predictions, according to the significant drop in Recall without smoothing and the substantial decrease in Recall Early stop without calibration. (5) The calibrated confidence estimation is effective regardless of whether using prompt tuning. And it demonstrates greater advantages compared to the comparing (LM) baseline in recognizing predictions with higher confidence when using prompt tuning, indicated by increasing Recall Early stop (from 48.7 to 52.4).
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
In our experiments, we use top-p sampling to sample multiple candidate suffixes conditioned on one given prefix. However, there are also other popular decoding methods, including greedy search, beam search, and top-k sampling. We thus compare these popular decoding methods in this section.  which suggests some tokens in the ground truth suffix do not have the highest probability at the corresponding positions. Beam search outperforms top-p sampling on Recall, indicating that searching for the suffix with the lowest loss works well to find the ground truth suffix. However, beam search performs significantly worse than top-p sampling on Recall Early stop , because it cannot use our calibrated confidence. Compared with beam search, top-p sampling can generate multiple candidates, which could substantially increase the accuracy of confidence estimation with our proposed calibrated confidence. Moreover, the top-k sampling performs worse than top-p sampling on Recall Early stop , which may be because top-k sampling is easier to sample low-probability tokens and thus reduce the confidence of the ground truth suffixes. We finally select top-p sampling as our decoding method due to its balance on Recall and Recall Early stop .
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Previous works on scaling laws find that larger language models can memorize more training data (Carlini et al., 2022b;Tirumala et al., 2022b). Therefore, we are interested in how targeted data extraction performance varies across different model scales. Figure 3 shows the results. We can see that the targeted training data extraction performance continuously increases as the model scale increases from 125 million to 6 billion. ETHI-CIST shows impressive results as it consistently and 47.9 ±1.0 Top-k 62.7 ±.6 50.8 ±.6 Top-p 62.8 ±.5 53.8 ±.5 Table 3: Effect of the decoding strategy on ETHICIST. Note that our proposed calibrated confidence is unused when decoding with deterministic methods, including greedy search and beam search. We show the mean and the standard deviation over 3 random seeds for all decoding strategies.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
significantly outperforms baselines across different model scales. Thanks to prompt tuning, ETHICIST is efficient in terms of computation time and particularly memory consumption. Therefore, ETHICIST can also be adapted to larger language models for efficient targeted training data extraction.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
All prefixes and suffixes in the LM-Extraction benchmark are 50 tokens long, making it an interesting question how the length of prefixes and suffixes would affect the extraction performance.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We show the effect of the given prefix length in Figure 4. We can observe that the extraction performance grows approximately linearly with the prefix length for all evaluated methods, and ETHICIST performs best for all prefix lengths. Al- Figure 3: Effect of the model scale. We show the mean and the standard deviation over 3 random seeds for all methods.  though all methods have similar growth speed on Recall, ETHICIST has the highest growth speed on Recall Early stop . It is also interesting that Comparing (LM) only outperforms Perplexity when given   Figure 6: Given the prefix, we show the top-2 predicted suffixes by ETHICIST. Although the first prediction has higher loss, it is repeated for 74 times and we correctly select it as the final predicted suffix using our calibrated confidence estimation. We highlight the wrong predicted text in red.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
prefixes that are long enough. We show the effect of the predicted suffix length in Figure 5. For all three methods, the extraction performance decreases when the suffix length increases. Different from the approximately linear relationship between the prefix length and the extraction performance, the performance degradation tends to become progressively slower as the suffix length increases. This suggests that the model can still memorize a considerable proportion of suffixes (rather than quickly decreasing to zero) even if the predicted suffix length continues to increase. What's more, we observe that ETHICIST has a significantly slower speed of performance degradation compared with the two baselines, which suggests ETHICIST is effective for eliciting deeper memorization of longer suffixes of the attacked model.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Due to space limitations, we put the analysis of sampling time in Appendix C. 
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We further show some statistical features in Table  4. We can see that the memorized suffixes can be sampled significantly more frequently with a high average repeat time of 85.38, validating that the repeat time is a valuable signal for confidence estimation. What's more, the memorized suffixes have significantly higher confidence. One interesting phenomenon we observe is that if the ground truth suffix can be generated, it mostly has the top 3 highest confidence (Recall@3 ≈ Recall@100). We also find that for more than 30% of the prefixes, the model cannot generate the correct prefix even given 100 chances. Therefore, an important future direction is to design better methods to elicit memorization in the attacked model. Considering the non-negligible gap between Recall@1 and Re-call@100 (0.63 vs. 0.69), another important future direction is to design better confidence estimation methods (maybe trainable), which can pick out the ground truth suffix among the collection of candidate suffixes for one prefix.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
We show a case in Figure 6. Although the first predicted suffix has higher loss than the second predicted suffix, it is sampled far more times than the latter. Therefore, we assign higher confidence to the first suffix using our calibrated confidence estimation method. We further show the probability of generating each token during the sampling process in Figure 7. We can observe that although the correct prediction has higher loss as a whole, it keeps a high sampling probability across the generation process. The minimum probability of generating one token in the correct suffix is about 0.45, which is significantly higher than 0.1 for the wrong suffix.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Therefore it is easier to generate the correct suffix, which leads to a higher confidence score. This is also in line with our motivation for designing the extra smoothing loss, which can increase the probability of sampling the correct suffix.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
In this work, we propose ETHICIST, an effective method for targeted training data extraction attack. ETHICIST uses soft prompt to elicit memorization in the attacked model. To ensure the probability of the ground truth suffix token at each time step is not low, we propose a smoothing loss besides the standard MLE loss. We also propose a calibrated confidence estimation method to calibrate the scale of confidence across different samples. Experiments on the LM-Extraction benchmark demonstrate that ETHICIST significantly improves the extraction performance. We further conduct extensive experiments to investigate several critical factors influencing the extraction performance, including decoding strategy, model scale, prefix length, and suffix length. We hope our work can promote future researches on better attack methods and practical defense methods for the training data extraction problem.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
This work was supported by the NSFC projects (Key project with No. 61936010 ). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Although we conduct experiments across various model scales ranging from 125M to 6B, there are still larger language models we don't test either because their training data is not publicly released or because we have limited resources.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Moreover, the examples in the LM-Extraction benchmark are all chosen to meet the property that there exists a prefix length (maybe longer than 50) that causes the model to generate the suffix string exactly, which makes the extraction performance on this benchmark higher than that on randomly selected prefixes.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
ETHICIST is a powerful method to elicit memorization in the large pre-trained language models, which makes it a useful tool to expose the privacy risk of large language models. However, it also has a risk to be abused by attackers to extract privacy information from pre-trained language models. Thus large language models should be carefully examined before being made publicly available. What's more, it is necessary to develop defense methods against the training data extraction attacks without sacrificing the language modeling ability.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
The LM-Extraction benchmark is derived from the Pile dataset, and thus covers many domains including books, code, emails, etc. This suggests the effectiveness of targeted training data extraction across different domains.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
dataset. They are GPT-Neo 125M, GPT-Neo 1.3B, GPT-Neo 2.7B, and GPT-J 6B (Black et al., 2021;Wang and Komatsuzaki, 2021). We set the prompt length to 100, the batch size to 32, the learning rate of AdamW optimizer to 1e-3, the warmup step to 500, the learning rate decay strategy to linear, N in Equation 2 to 5, α in Equation 3 to 0.7, and the maximum training epoch to 20 with an early stopping mechanism. In our main experiments, we generate the suffix using top-p sampling (Holtzman et al., 2020) with p = 0.7 and temperature = 0.8. For other decoding methods, we set beam size to 10 for beam search, and k to 10 for top-k sampling (temperature = 0.8). Our code is based on Huggingface Transformers (Wolf et al., 2020).
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
All experiments are carried out on a single Tesla V100 GPU with 32GB memory. Each experiment can be completed in less than 20 hours. 
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
In our main experiments, we sample 100 candidate suffixes for one given prefix. We show the effect of sampling time in Figure 8. We can see that all methods' performances increase quickly when the sampling time increases from 1 to 10. However, ETHICIST's performance can still improve slowly when the sampling time increases from 10 to 100, which we attribute to the consideration of repeat time in our calibrated confidence estimation. What's more, although we report the result for sampling 100 times in our main experiments, we can see that ETHICIST can achieve satisfying performance when sampling only 10 times, which suggests the efficiency of ETHICIST.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
A For every submission:
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
A1. Did you describe the limitations of your work?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
A2. Did you discuss any potential risks of your work?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Section Ethics Statement A3. Do the abstract and introduction summarize the paper's main claims?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Section Abstract and Section Introduction A4. Have you used AI writing assistants when working on this paper?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Left blank. B Did you use or create scientific artifacts?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
B1. Did you cite the creators of artifacts you used?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
They are discussed in our github repo.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section Experiments B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The publisher of the data did the work.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We don't create new data.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Section Experiments C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section Implementation Details and Section Computing Infrastructure in the Appendix The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
There are many situations in which private entities are worried about the privacy of their data. For example, many companies provide black-box training services where users are able to upload their data and have customized models built for them, without requiring machine learning expertise. A common concern in these ''machine learning as a service'' offerings is that the uploaded data be visible only to the client that owns it.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Currently, these entities are in the position of having to trust that service providers abide by the terms of their agreements. Although trust is an important component in relationships of all kinds, it has its limitations. In particular, it falls short of a well-known security maxim, originating in a Russian proverb that translates as, Trust, but * Work done while visiting Johns Hopkins University.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
verify. 1 Ideally, customers would be able to verify that their private data was not being slurped up by the serving company, whether by design or accident.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
This problem has been formalized as the membership inference problem, first introduced by Shokri et al. (2017) and defined as: ''Given a machine learning model and a record, determine whether this record was used as part of the model's training dataset or not.'' The problem can be tackled in an adversarial framework: The attacker is interested in answering this question with high accuracy, whereas the defender would like this question to be unanswerable (see Figure 1). Since then, researchers have proposed many ways to attack and defend the privacy of various types of models. However, the work so far has only focused on standard classification problems, where the output space of the model is a fixed set of labels.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
In this paper, we propose to investigate membership inference for sequence generation problems, where the output space can be viewed as a chained sequence of classifications. Prime examples of sequence generation includes machine translation and text summarization: In these problems, the output is a sequence of words whose length is undetermined a priori. Other examples include speech synthesis and video caption generation. Sequence generation problems are more complex than classification problems, and it is unclear whether the methods and results developed for membership inference in classification problems will transfer. For example, one might imagine that whereas a flat classification model might leak private information when the output is a single label, a recurrent sequence generation model might obfuscate this leakage when labels are generated successively with complex dependencies.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We focus on machine translation (MT) as the example sequence generation problem. Recent advances in neural sequence-to-sequence models have improved the quality of MT systems significantly, and many commercial service providers are deploying these models via public API's. We pose the main question in the following form:
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Given black-box access to an MT model, is it possible to determine whether a particular sentence pair was in the training set for that model?
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
In the following, we define membership inference for sequence generation problems ( §2) and contrast with prior work on classification ( §3). Next we present a novel dataset ( §4) based on state-of-the-art MT models. 2 Finally, we propose several attack methods ( §5) and present a series of experiments evaluating their ability to answer the membership inference question ( §6). Our conclusion is that simple one-off attacks based on shadow models, which proved successful in classification problems, are not successful on sequence generation problems; this is a result that favors the defender. Nevertheless, we describe the specific conditions where sequence-to-sequence models still leak private information, and discuss the possibility of more powerful attacks ( §7).
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We now define the membership inference attack problem for sequence-to-sequence models in detail. Following tradition in the security research literature, we introduce three characters: 2 We release the data to encourage further research in this new problem: https://github.com/sorami/ tacl-membership Alice (the service provider) builds a sequenceto-sequence model based on an undisclosed dataset A train and provides a public API. For MT, this API takes a foreign sentence f as input and returns an English translationê.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Bob (the attacker) is interested in discerning whether a data sample was included in Alice's training data A train by exploiting Alice's API. This sample is called a ''probe'' and consists of a foreign sentence f and its reference English translation, e. Together with the API's output e, Bob has to make a binary decision using a membership inference classifier g(·), whose goal is to predict: 3
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We term in-probes to be those probes where the true class is in, and out-probes to be those whose true class is out. Importantly, note that Bob has access not only to f but also to e in the probe. Intuitively, ifê is equivalent to e, then Bob may believe that the probe was contained in A train ; however, it may also be possible that Alice's model generalizes well to new samples and translates this probe correctly. The challenge for Bob is to make this distinction; the challenge for Alice is to prevent Bob from doing so.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Carol (the neutral third-party) is in charge of setting up the experiment between Alice and Bob. She decides which data samples should be used as in-probes and out-probes and evaluates Bob's classification accuracy. Carol is introduced only to clarify the exposition and to set up a fair experiment for research purposes. In practical scenarios, Carol does not exist: Bob decides his own probes, and Alice decides her own A train .
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
In order to be precise about how Carol sets up the experiment, we will explain in terms of machine translation, but note that the problem definition applies to any sequence-to-sequence problem. A training set for MT consists of a set of sentence pairs {(f i with d = ℓ 2 and i = 1 might refer to the first sentence in the CommonCrawl subcorpus. I(d) is the maximum number of sentences in the subcorpus with label d. The distinction among subcorpora is not necessary in the abstract problem definition, but is important in practice when differences in data distribution may reveal signals in membership.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Without loss of generality, in this section assume that Carol has a finite number of samples from two subcorpora d ∈ {ℓ 1 , ℓ 2 }. First, she creates an out-probe of k samples from subcorpus ℓ 1 :
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Then Carol creates the data for Alice to train Alice's MT model, using subcorpora ℓ 1 and ℓ 2 :
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
(3) Importantly, the two sets are totally disjoint: i.e., A out probe ∩ A train = ∅. By definition, out-probes are sentence pairs that are not in Alice's training data. Finally, Carol creates the in-probe of k samples by drawing from A train , i.e. A in probe ⊂ A train , which is defined to be samples that are included in training:
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
(4) Note that both A in probe and A out probe are sentence pairs that come from the same subcorpus; the only difference is that the former is included in A train whereas the latter is not.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
There are several ways in which Bob's data can be created. For this work, we will assume that Bob also has some data to train MT models, in order to mimic Alice and design his attacks. This data could either be disjoint from A train , or contain parts of A train . We choose the latter, which assumes that there might be some public data that is accessible to both Alice and Bob. This scenario slightly favors Bob. In the case of MT, parallel data can be hard to come by, and datasets like Europarl are widely accessible to anyone, so presumably both Alice and Bob would use it. However, we expect that Alice has an in-house dataset (e.g., crawled data) that Bob does not have access to. Thus, Carol creates data for Bob:
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
(5) Note that this dataset is like A train but with two exceptions: All samples from subcorpora ℓ 2 and all samples from A in probe are discarded. One can view ℓ 2 as Alice's own in-house corpus which Bob has no knowledge of or access to, and ℓ 1 as the shared corpus where membership inference attacks are performed.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
To summarize, Carol gives A train to Alice, who uses it in whatever way she chooses to build a sequence-to-sequence model M [A train , Θ]. The model is trained on A train with hyperparameters Θ (e.g., neural network architecture) known only to Alice. In parallel, Carol gives B all to Bob, who uses it to design various attack strategies, resulting in a classifier g(·) (see Section 5). When it is time for evaluation, Carol provides both probes A in probe and A out probe to Bob in randomized order and asks Bob to classify each sample as in or out. For each probe (f
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
i ), Bob is allowed to make one call to Alice's API to obtainê (d) i . As an additional evaluation, Carol creates a third probe based on a new subcorpus ℓ 3 . We call this the ''out-of-domain (OOD) probe'':
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Both A out probe and A ood should be classified as out by Bob's classifier. However, it has been known that sequence-to-sequence models behave very differently on data from domains/genre that is significantly different from the training data (Koehn and Knowles, 2017). The goal of having two out probes is to quantify the difficulty or ease of membership inference in different situations. Figure 2 summarizes the problem definition. The probes A out probe and A ood are by construction outside of Alice's training data A train , while the probe A in probe is included. Bob's goal is to produce a classifier that can make this distinction. There are k samples each for A in probe , A out probe , and A ood . Alice's training data A train excludes A out probe and ℓ 3 , while including A in probe . Bob's data B all is a subset of Alice's data, excluding A in probe and ℓ 2 .
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
He has at his disposal a smaller dataset B all , which he can use in whatever way he desires.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
There are alternative definitions of this membership inference problem. For example, one can allow Bob to make multiple API calls to Alice's model for each probe. This enlarges the repository of potential attack strategies for Bob. Or, one could evaluate Bob's accuracy not on a per-sample basis, but allow for a coarser granularity where Bob can aggregate inferences over multiple samples. There is also a distinction between white-box and black-box attacks: We focus on the blackbox case where Bob has no internal access to the internal parameters of Alice's model, but can only guess at likely model architectures. In the white-box case, Bob would have access to Alice's model internals, so different attacks would be possible (e.g., backpropagation of gradients). In these respects, our problem definition makes the problem more challenging for Bob the attacker.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Finally, note that Bob is not necessarily always the ''bad guy''. Some examples of who Alice and Bob might be in MT are: (1) Organizations (Bob) that provide bitext data under license restrictions might be interested to determine whether their licenses are being complied with in published models (Alice). (2) The organizers (Bob) of an annual bakeoff (e.g., WMT) might wish to confirm that the participants (Alice) are following the rules of not training on test data. (3) ''MT as a service'' providers may support customized engines if users upload their own bitext training data. The provider promises that the user-supplied data will not be used in the customized engines of other users, and can play both Alice and Bob, attacking its own model to provide guarantees to the user. If it is possible to construct a successful membership inference mechanism, then many ''good guys'' would be able to provide the aforementioned fairness (1, 2) and privacy guarantees (3).
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Based on the problem definition in Section 2, we construct a dataset to investigate the possibility of the membership inference attack on MT models. We make this dataset available to the public to encourage further research. 4 There are various considerations to ensure the benchmark is fair for both Alice and Bob: We need a dataset that is large and diverse to ensure Alice can train state-of-the-art MT models and Bob can test on probes from different domains. We used corpora from the Conference on Machine Translation (WMT18) (Bojar et al., 2018). We chose the German-English language pair because it has a reasonably large amount of training data, and previous work demonstrate high BLEU scores.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We now describe how Carol prepares the data for Alice and Bob. First, Carol selects four subcorpora for the training data of Alice, namely, CommonCrawl, Europarl v7, News Commentary v13, and Rapid 2016. A subset of these four subcorpora are also available to Bob (ℓ 1 in § 2.1). In addition, Carol gives ParaCrawl to Alice but not Bob (ℓ 2 in §2.1). We can think of it as in-house data that the service provider holds. For all these subcorpora, Carol first performs basic preprocessing: (a) tokenization of both the German and English sides using the Moses tokenizer, (b) de-duplication of sentence pairs so that only unique pairs are present, and (c) randomly shuffling all sentences prior to splitting into probes and MT training data. 5 Figure 3 illustrates how Carol splits subcorpora for Alice and Bob. For each subcorpus, Carol splits 4 https://github.com/sorami/taclmembership 5 These are design decisions that balance between simple experimentation vs. realistic condition. Carol doing a common tokenization removes some of the MT-specific complexity for researchers who want to focus on the Alice or Bob models. However, in a real-world public API, Alice's tokenization is likely to be unknown to Bob. We decided on a middle ground to have Carol perform a common tokenization, but Alice and Bob do their own subword segmentation. them to create probes A in probe and A out probe , and A train and B all . Carol sets k = 5, 000, meaning each probe set per subcorpus has 5,000 samples. For each subcorpus, Carol selects 5,000 samples to create A out probe . She then uses the rest as A train and select 5,000 from it as A in probe . She excludes A in probe and ParaCrawl from A train to create a dataset for Bob, B all . 6 In addition, Carol has four other domains to create out-of-domain probe set A ood , namely, EMEA and Subtitles 18 (Tiedemann, 2012), Koran (Tanzil), and TED (Duh, 2018). These subcorpora are equivalent to ℓ 3 in § 2.1. The size of A ood is 5,000 per subcorpus, same as A in probe and A out probe . The number of samples for each set is summarized in Table 1.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Alice uses her dataset A train (consisting of four subcorpora and ParaCrawl) to train her own MT model. Because Paracrawl is noisy, Alice first applies dual conditional cross-entropy filtering (Junczys-Dowmunt, 2018), retaining the top 4.5 million lines. Alice then trains a joint BPE subword model (Sennrich et al., 2016) using 32,000 merge operations. No recasing is applied. Alice's model is a six-layer Transformer (Vaswani et al., 2017) using default parameters in Sockeye (Hieber et al., 2017). 7 The model was 6 We prepared two different pairs of A in probe and A out probe . Thus B all has 10k fewer samples than A train , and not 5k fewer. For the experiment we used only one pair, and kept the other for future use. 7 Three-way tied embeddings, model and embedding size 512, eight attention heads, 2,048 hidden states in the feed forward layers, layer normalization applied before each selfattention layer, and dropout and residual connections applied afterward, word-based batch size of 4,096. trained until perplexity on newstest2017 (Bojar et al., 2017) had not improved for five consecutive checkpoints, computed every 5,000 batches.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
The BLEU score (Papineni et al., 2002) on newstest2018 was 42.6, computed using sacreBLEU (Post, 2018) with the default settings. 8
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
To evaluate membership inference attacks on Alice's MT models, we use the following procedure: First, Bob asks Alice to translate f . Alice returns her resultê to Bob. Bob also has access to the reference e and use his classifier g(f, e,ê) to infer whether (e, f ) was in Alice's training data. The classification is reported to Carol, who computes ''attack accuracy''. Given a probe set P containing a list of (f, e,ê, l), where l is the label (in or out), this accuracy is defined as:
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
If the accuracy is 50%, then the binary classification is same as random, and Alice is safe. An accuracy slightly above 50% can be considered a potential breach of privacy.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Bob's initial approach for attack is to use ''shadow models'', similar to Shokri et al. (2017). The idea is that Bob creates MT models with his data to mimic (shadow) the behavior of Alice's MT model, then train a membership inference classifier on these shadow models. To do so, Bob splits his data B all into his own version of in-probe, out-probe, and training set in multiple ways to train MT models. Then he translates these probe sentences with his own shadow MT models, and use the resulting (f, e,ê) with its in or out label to train a binary classifier g(f, e,ê). If Bob's shadow models are sufficiently similar to Alice's in behavior, this attack can work.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Bob first selects 10 sets of 5,000 sentences per subcorpus in B all . He then chooses two sets and uses one as in-probe and the other as outprobe, and combines in-probe and the rest (B all minus 10 sets) as a training sets. We use notations  Green box indicates the out-probe B out probe . Bob uses models from splits 1 to 3 as a train, 4 as a validation, and 5 as a test sets for his attack.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
in probe B 1+ out probe , and B 1+ train for the first group of in-probe, out-probe, and training sets. Bob then swaps the in-probe and out-probe to create another group. We notate this as B 1− in probe , B 1− out probe , and B 1− train . With 10 sets of 5,000 sentences, Bob can create 10 different groups of in-probe, out-probe, and training sets. Figure 4 illustrates the data splits.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
For each group of data, Bob first trains a shadow MT model using the training set. He then uses this model to translate sentences in the in-probe and out-probe sets. Bob has now a list of (f, e,ê) from different shadow models, and he knows for each sample if it was in or out of the training data for the MT model used to translate that sentence.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Bob's model is a 4-layer Transformer, with no tied embedding, model/embedding size 512, 8 attention heads, 1,024 hidden states in the feed forward layers, word-based batch size of 4,096. The model is optimized with Adam (Kingma and Ba, 2015), regularized with label smoothing (0.1), and trained until perplexity on newstest2016 (Bojar et al., 2016) had not improved for 16 consecutive checkpoints, computed every 4,000 batches. Bob has BPE subword models with vocab size 30k for each language. The mean BLEU scores of the ten shadow models on news-test2018 is 38.6±0.2 (compared with 42.6 for Alice).
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Bob extracts features from (f, e,ê) for a binary classifier. He uses modified 1-to 4-gram precisions and smoothed sentence-level BLEU score (Lin and Och, 2004) as features. Bob's intuition is that if an unusually large number of n-grams inê matches e, then it could be a sign that this was in the training data and Alice memorized it. Bob calculates n-gram precision by counting the number of n-grams in translation that appear in the reference sentence. In the later investigation Bob also considers the MT model score as an extra feature.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Data: B all Result: g(·) Split B all into multiples groups of (B i in probe ,
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
in probe , B i out probe , and their translations to train g(·) ; Bob tries different types of classifiers, namely, Perceptron (P), Decision Tree (DT), Naïve Bayes (NB), Nearest Neighbors (NN), and Multi-layer Perceptron (MLP). DT uses GINI impurity for the splitting metrics, and the max depth to be 5. Our NB uses Gaussian distribution. For NN we set the number of neighbors to be 5 and use Minkowski distance. For MLP, we set the size of the hidden layer to be 100, the activation function to be ReLU, and the L2 regularization term α to be 0.0001.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Algorithm 1 summarizes the procedure to construct a membership inference classifier g(·) using Bob's dataset B all . For training the binary classifiers, Bob uses models from data splits 1 to 3 for training, 4 for validation, and 5 for his own internal testing. Note that the final evaluation of the attack is done using the translations of A in probe and A out probe with Alice's MT model, by Carol.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We now present a series of results based on the shadow model attack method described in Section 5. In Section 6.1 we will observe that Bob has difficulty attacking Alice under our definition of membership inference. In Sections 6.2 and 6.3 we will see that Alice nevertheless does leak some private information under more nuanced conditions. Section 6.4 describes the possibility of attacks beyond sentence-level membership. Section 6.5 explores the attacks using external resources.  The results of the attacks on the Alice model show that it is around 50%, meaning that the attack is not successful and the binary classification is almost the same as a random choice. 9 The accuracy is around 50% for Bob:valid, meaning that Bob also has difficulty attacking his own simulated probes, therefore the poor performance on A in probe and A out probe is not due to mismatches between Alice's model and Bob's model.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
The accuracy is around 50% for Bob:train as well, revealing that the classifier g(·) is underfitting. 10 This suggests that the current features do not provide enough information to distinguish in-probe and out-probe sentences. Figure 5 shows 9 Some numbers are slightly over 50%, which may be interpreted as small leak of privacy. Although the desired accuracy levels depend on the application, for the MT scenarios described in Section 2.2 Bob would need much higher accuracies. For example, if Bob is a bakeoff organizer, he might want accuracy above 60% in order to determine whether to manually check the submission. However, if Bob is providing ''MT as a service'' with strong privacy guarantees, he may need to provide the client with accuracy higher than 90%. 10 The higher accuracy for k-NN is an exception, but is due to having the exact same datapoint in the model as the input, which always becomes the nearest neighbor. When the k value is increased, the accuracy on in-sample data decreased.   the confusion matrices of the classifier output on Alice probes. We see that for all classifiers, whatever prediction they make is incorrect half of the time. Table 3 shows the result when MT model score is added as an extra feature for classification. The result indicates that this extra information does not improve the attack accuracy. In summary, these results suggest that Bob is not able to reveal membership information at the sentence/sample level. This result is in contrast to previous work on membership inference in ''classification'' problems, which demonstrated high accuracy with Bob's shadow model attack.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Additionally, note that although accuracies are close to 50%, the number of Bob:test tend to be slightly higher than Alice's for some classifiers. This may reflect the fact that Bob:test is a matched condition using the same shadow MT architecture, while Alice probes are from a mismatched condition using an unknown MT architecture. It is important to compare both numbers in the experiments: accuracy on Alice probes is the real evaluation and accuracy on Bob:test is a diagnostic.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Carol prepared OOD subcorpora, A ood , that are separate from A train and B all . The membership inference accuracy of each subcorpus is shown in Table 4. The accuracy for OOD subcorpora are much higher than that of original indomain subcorpora. For example, the accuracy with Decision Tree was 50.3% and 51.1% for ParaCrawl and CommonCrawl (in-domain), whereas accuracy was 67.2% and 94.1% for EMEA and Koran (out-of-domain). This suggests that for OOD data Bob has a better chance to infer the membership.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
In Table 4 we can see that Perceptron has accuracy 50% for all in-domain subcorpora and 100% for all OOD subcorpora. Note that the OOD subcorpora only have out-probes. By definition none of the samples from OOD subcorpora are in the training data. We get such accuracy because our Perceptron is always predicting out, as we can see in Figure 5. We believe this behavior is caused by applying Perceptron to inseparable data, and this particular model happened to be trained to act this way. To confirm this we have trained variations of Perceptrons by shuffling the training data, and observed that the resulting models had different output ratios of in and out, and in some cases always predicting in for both in and OOD subcorpora. Figure 6 shows the distribution of sentencelevel BLEU scores per subcorpus. The BLEU scores tend to be lower for OOD subcorpora, and the classifier may exploit this information to distinguish the membership better. But note that EMEA (out-of-domain) and CommonCrawl (indomain) have similar BLEU scores, but vastly different membership accuracies, so the classifier may also be exploiting n-gram match distributions.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Overall, these results suggest that Bob's accuracy depends on the specific type of probe being tested. If there is a wide distribution of domains, there is a higher chance that Bob may be able to reveal membership information. Note that in the actual scenario Bob will have no way of knowing what is OOD for Alice, so there is no signal that is exploitable for Bob. This section is meant as an error analysis that describes how membership inference classifiers behave differently in case the probe is OOD.   
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We also focused on the samples that contain the words that never appear in the training data of the MT model used for translation, that is, out-ofvocabulary (OOV) words. For this analysis, we focus only on vocabulary that does not exist in the training data of Bob's shadow MT models, rather than Alice's, since Bob does not have access to her vocabulary. By definition there are only out-probes in OOV subsets. For Bob's shadow models, 7.4%, 3.2%, and 1.9% of samples in the probe sets had one or more OOV words in source, reference, or both sentences, respectively. Table 5 shows the membership inference accuracy of the OOV subsets from the Bob test set, which is generally very high (>70%). This implies that sentences with OOV words are translated idiosyncratically compared with the ones without OOV words, and the classifier can exploit this.  sets, and Alice sets. These sets are evenly split into groups of 500, not sampled as done in training. Table 6 shows the accuracy on probe groups. We can see that the accuracy is much higher than 50%, not only for Bob's training set but also for his validation and test sets. However, for Alice, we found that classifiers were almost always predicting in, resulting the accuracy to be around 50%. This is due to the fact that classifiers were trained on shadow models that have lower BLEU scores than Alice. This suggests that we need to incorporate the information about the Alice / Bob MT performance difference.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
One way to adjust the difference is to directly manipulate the input feature values. We adjusted the feature values, compensating by the difference in mean BLEU scores, and accuracy on Alice probes increased to 60% for P and DT as shown in the ''adjusted'' column of Table 6. If the classifier took advantage of the absolute values in its decision, the adjustment may provide improvements. If that is not the case, then improvements are less likely. Before the adjustment, all classifiers were predicting everything to be in for Alice probes. Classifiers like NB and MLP apparently did not change how often they predict in even after the normalization, whereas classifiers like P and DT did. In a real scenario this BLEU difference can be reasonably estimated by Bob, since he can use Alice's translation API to calculate the BLEU score on a held-out set, and compare it with his shadow models.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Another possible approach to handle the problem of classifiers always predicting in is to consider the relative size of classifier output score. We can rank the samples by the classifier output scores, and decide top N% to be in and rest to be out. Figure 7 shows how the accuracy changes when varying the in percentage. We can see that the accuracy can be much higher than the original result, especially if Bob can adjust the threshold based on his knowledge of in percentage in the probe.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
This is the first strong general result for Bob, suggesting the membership inference attacks are possible if probes are defined as groups of sentences. 11 Importantly, note that the classifier threshold adjustment is performed only for the classifiers in this section, and is not relevant for the classifiers in Section 6.1 to 6.3.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Our results in Section 6.1 demonstrate the difficulty of general membership inference attacks. One natural question is whether attacks can be improved with even stronger features or classifiers, in particular by exploiting external resources beyond the dataset Carol provided to Bob. We tried two different approaches: one using a Quality Estimation model trained on additional data, and another using a neural sequence model with a pre-trained language model. Quality Estimation (QE) is a task of predicting the quality of a translation at the sentence or word level. One may imagine that a QE model might produce useful feature to tease apart in and out because in translations may have detectable improvements in quality. To train this model, we used the external dataset from the WMT shared task on QE (Specia et al., 2018). Note that for our language pair, German to English, the shared task only had a labeled dataset for the SMT 11 We can imagine an alternative definition of this grouplevel membership inference where Bob's goal is to predict the percentage of overlap with respect to Alice's training data. This assumes that model trainers make corpus-level decisions about what data to train on. Reformulation of a binary problem to a regression problem may be useful for some purposes. system. Our models are NMT, so the estimation quality may not be optimally matched, but we believe this is the best data available at this time. We applied the Predictor-Estimator (Kim et al., 2017) implemented in an open source QE framework OpenKiwi (Kepler et al., 2019). It consists of a predictor that predicts each token of the target sentence given the target context and the source, and estimator that takes features produced by the predictor to estimate the labels; both are made of LSTMs. We used this model as this is one of the best models seen in the shared tasks, and it does not require alignment information. The model metrics on the WMT18 dev set, namely, Pearson's correlation, Mean Average Error, and Root Mean Squared Error for sentence-level scores, are 0.6238, 0.1276, and 0.1745, respectively.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We used the sentence score estimated by the QE model as an extra feature for classifiers described in Section 6.1. The results are shown in Table 7.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We can see that this extra feature did not provide any significant influence to the accuracy. In a more detailed analysis, we find that the reason is that our in and out probes both contain a range of translations from low to high quality translations, and our QE model may not be sufficiently finegrained to tease apart any potential differences. In fact, this may be difficult even for a human estimator.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Another approach to exploit external resources is to use a language model pre-trained on a large amount of text. In particular, we used BERT (Devlin et al., 2019), which has shown competitive results in many NLP tasks. We used BERT directly as a classifier, and followed a fine-tuning setup similar to paraphrase detection: For our case the inputs are the English translation and reference sentences, and the output is the binary membership label. This setup is similar to the classifiers we described in Section 5.3, where rather than training Perceptron or Decision Tree on manually defined features, we directly applied BERT-based sequence encoders on the raw sentences.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We fine-tuned the BERT Base,Cased English model with Bob:train. The results are shown in Table 7. Similar to previous results, the accuracy is 50% so the attack using BERT as classifier was not successful. Detailed examination of the BERT classifier probabilities show that they are scattered around 0.5 for all cases, but in general are quite random for both Bob and Alice probes. This result is similar to the other simpler classifiers in Section 6.1.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
In summary, from these results we can see that even with external resources and more complex classifiers, sentence-level attack is still very difficult for Bob. We believe this attests to the inherent difficulty of the sentence-level membership inference problem.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We formalized the problem of membership inference attacks on sequence generation tasks, and used machine translation as an example to investigate the feasibility of a privacy attack.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Our results in Section 6.1 and Section 6.5 show that Alice is generally safe and it is difficult for Bob to infer the sentence-level membership. In contrast to attacks on standard classification problems (Shokri et al., 2017), sequence generation problems maybe be harder to attack because the input and output spaces are far larger and complex, making it difficult to determine the quality of the model output or how confident the model is. Also, the output distribution of class labels is an effective feature for the attacker for standard classification problems, but is difficult to exploit in the sequence case.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
However, this does not mean that Alice has no risk of leaking private information. Our analyses in Sections 6.2 and 6.3 show that Bob's accuracy on out-of-domain and out-of-vocabulary data is above chance, suggesting that attacks may be feasible in conditions where unseen words and domains cause the model to behave differently. Further, Section 6.4 shows that for a looser definition of membership attack on groups of sentences, the attacker can win at a level above chance.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Our attack approach was a simple one, using shadow models to mimic the target model. Bob can attempt more complex strategies, for example, by using the translation API multiple times per sentence. Bob can manipulate a sentence, for example, by dropping or adding words, and observe how the translation changes. We may also use the metrics proposed by Carlini et al. (2018) as features for Bob; they show how recurrent models might unintentionally memorize rare sequences in the training data, and propose a method to detect it. Bob can also add ''watermark sentences'' that have some distinguishable characteristics to influence the Alice model, making attack easier. To guard against these attacks, Alice's protection strategy may include random subsampling of training data or additional regularization terms.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Finally, we note some important caveats when interpreting our conclusions. The translation quality of the Alice and Bob MT models turned out to be similar in terms of BLEU. This situation favors Bob, but in practice Bob is not guaranteed to be able to create shadow models of the same standard, nor verify how well it performs compared with the Alice model. We stress that when one is to interpret the results, one must evaluate both on Bob's test set and Alice probes side-by-side, like those shown in Tables 2, 3, and 7, to account for the fact that Bob's attack on his own shadow model translations is likely an optimistic upper-bound on the real attack accuracy on Alice's model.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
We believe our dataset and analysis is a good starting point for research in these privacy questions. Although we focused on MT, the formulation is applicable to other kinds of sequence generation models such as text summarization and video captioning; these will be interesting as future work.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
BERT-based encoders with Masked Language Modeling (MLM) Objectives (Devlin et al., 2018; have become models of choice for use as pre-trained models for various Natural Language Processing (NLP) classification tasks (Wang et al., 2018;Zhang et al., 2019;Rogers et al., 2020) and have been applied to diverse domains such as disease diagnosis, insurance analysis on financial data, sentiment analysis for improved user experience, etc (Yang et al., 2020;Lee et al., 2020). Given the sensitivity of the data used to train these models, it is crucial to conceive a framework to systematically evaluate the leakage of training data from these models (Shokri, 2022;Carlini et al., 2019;Murakonda and Shokri, 2020;Mireshghallah et al., 2020), and limit the leakage. The conventional way to measure the leakage of training data from machine learning models is by performing mem-bership inference attacks (Shokri et al., 2017;Nasr et al., 2021), in which the attacker tries to determine whether a given sample was part of the training data of the target model or not. These attacks expose the extent of memorization by the model at the level of individual samples. Prior attempts at performing membership inference and reconstruction attacks on masked language models have either been inconclusive (Lehman et al., 2021), or have (wrongly) concluded that memorization of sensitive data in MLMs is very limited and these models are more private than their generative counterparts (e.g., autoregressive language models) (Vakili and Dalianis, 2021; Jagannatha et al., 2021;Nakamura et al., 2020).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We hypothesize that prior MLM attacks have been inconclusive because they rely solely on the target model's (model under attack) loss on each individual sample as a proxy for how well the model has memorized that sample. If the loss is lower than a threshold, the sample is predicted to be a member of the training set. However, the target model's loss includes confounding factors of variation like the intrinsic complexity of the sample -and thus provides a limited discriminative signal for membership prediction. This scheme has either a high false-negative rate (with a conservative threshold) -classifying many hard-to-fit samples from the training set as non-members, or a high false-positive rate (with a generous threshold) -failing to identify easy-to-fit samples that are not in the training set.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Reference-based likelihood ratio attacks, on the other hand, when applied to certain probabilistic graphical models and classifiers, have been shown to alleviate this problem and more accurately distinguish members from non-members (Murakonda et al., 2021;Ye et al., 2021). In such attacks, instead of the loss of the model under attack, we look at the ratio of the likelihood of the sample under the target model and a reference model trained on samples from the underlying population distribution that generates the training data for the target model. This ratio recalibrates the test statistic to explain away spurious variation in model's loss for different samples due to the intrinsic complexity of the samples. Unlike most other models (e.g., generative models), however, computing the likelihood of MLMs is not straightforward. In this paper, we propose a principled framework for measuring information leakage of MLMs through likelihood ratio-based membership inference attacks and perform an extensive analysis of memorization in such models. To compute the likelihood ratio of the samples under the target and the reference MLMs, we view the MLMs as energy-based probabilistic models (Goyal et al., 2022) over the sequences. This enables us to perform powerful inference attacks on conventionally non-probabilistic models like masked language models.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We evaluate our proposed attack on a suite of masked clinical language models, following (Lehman et al., 2021). We compare our attack with the baseline from the prior work that relies solely on the loss of the target model (Yeom et al., 2018;Song and Raghunathan, 2020;Jagannatha et al., 2021). We empirically show that our attack improves the AUC from 0.66 to 0.90 on the ClinicalBERT-Base model, and achieves a true positive rate (recall) of 79.2% (for a false positive rate of 10%), which is a substantial improvement over the baseline with 15.6% recall. This shows that, contrary to prior results, masked language models are significantly susceptible to attacks exploiting the leakage of their training data. In low error regions (at 1% false positive rate) our attack is 51× more powerful than the prior work.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We also present analyses of the effect of the size of the model, the length of the samples, and the choice of the reference model on the success of the attack. Finally, we attempt to identify features of samples that are more exposed (attack is more successful on), and observe that samples with multiple non-alphanumeric symbols (like punctuation) are more prone to being memorized. We provide instructions on how to request access to the data and code in Appendix A.2.1.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In this section, we first formally describe the membership inference attack, how it can be conducted using likelihood ratio tests and how we apply the test for masked language models (MLMs) which do not explicitly offer an easy-to-compute probability distribution over sequences. Finally, we describe all the steps in our attack, as summarized in Figure 1.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Let M θ denote a model with parameters θ that have been trained on data set D, sampled from the general population distribution p. Our goal is to quantify the privacy risks of releasing M θ for the members of training set D.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We consider an adversary who has access to the target model M θ . We assume this adversary can train a (reference) model M θ R with parameters θ R on independently sampled data from the general population p. In a Membership Inference Attack (MIA), the objective of the adversary is to create a decision rule that determines whether a given sample s was used for training M θ . To test the adversary, we perform the following experiment. We sample a datapoint s from either the general population or the training data with a 0.5 probability, and challenge the adversary to tell if s is selected from the training set (it is a member) or not (it is a non-member) (Murakonda et al., 2021). The precision of the membership inference attack indicates the degree of information leakage from the target model about the members of its training set. We measure the adversary's success using two metrics:
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
(1) the adversary's power (the true positive rate), and (2) the adversary's error (the false positive rate).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Before discussing our proposed attack for MLMs in the next section, we summarize the likelihood ratio test here which forms the core of our approach. A likelihood ratio test distinguishes between a null hypothesis and an alternative hypothesis via a test statistic based on the ratio of likelihoods under the two hypotheses. Prior work demonstrated an MIA attack based on the likelihood ratio to be optimal for probabilistic graphical models (Bayesian networks) (Murakonda et al., 2021). Given a sample s from the training data of the target model, the adversary aims at distinguishing between two hypotheses: 1. Null hypothesis (H out ): The target sample s is drawn from the general population p, independently from the training set D.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The target sample s is drawn from the target model's training set D.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The goal of hypothesis testing is to find whether there is enough evidence to reject H out in favor of H in . We use a likelihood ratio for this purpose Attack Procedure = log P( ; " ) P( ; ) < t Figure 1: Overview of our attack: to determine whether a target sample s is a member of the training data (D ∼ p) of the target model (M θ ), we feed it to the energy function formulation of M θ so that we can compute Pr(s;M θ ), the probability of s under M θ . We do the same with a reference model M θ R which is trained on a disjoint data set from the same distribution as the training data. Then, we compute likelihood ratio L(s), and based on this ratio and a given test threshold t, we decide if s is a member of D (H in ) or not (H out ).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
which involves comparison of the likelihood of the target sample under the settings for H out and H in respectively. For H in , we already have access to the target model, which is parameterized by θ and trained on D. For H out , we require access to a model trained on the general population. As mentioned earlier, the adversary has access to a reference model parameterized by θ R . Therefore, the likelihood ratio test is characterized by the following statistic:
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The Likelihood Ratio (LR) test is a comparison of the log-likelihood ratio statistic L(s) with a threshold t. If L(s) ≤ t, then the adversary rejects H out (decides in favor of membership of s ∈ D); otherwise the adversary fails to reject H out . We discuss the details of selecting the threshold and quantifying the attack's success in Section 2.4.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Performing a likelihood ratio test with masked language models is difficult because these models do not explicitly define an easy-to-compute probability distribution over natural language sequences. Following prior work (Goyal et al., 2022), we alternatively view pre-trained MLMs as energy-based probability distributions on sequences, allowing us to directly apply the likelihood ratio formalism. An energy-based sequence model defines the probability distribution over the space of possible sequences S as:
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
where E(s; θ) refers to the scalar energy of a sequence s that is parametrized by θ, and Z θ = s ′ ∈S e −E(s ′ ;θ) denotes the intractable noramlization constant. Under this framework, the likelihood ratio test statistic (Eq. 1) is:
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Above, we make use of the fact that for two fixed models (i.e., target model θ, and reference model θ R ), the intractable term log(Z θ ) − log(Z θ R ) is a global constant and can be ignored in the test. Therefore, computation of the test statistic only relies on the difference between the energy values assigned to sample s by the target model M θ , and the reference model M θ R . In practice, we cast a traditional MLM as an energy-based language model using a slightly different parameterization than explored by Goyal et al. (2022). Since the training of most MLMs (including the ones we attack in experiments) involves masking 15% of the tokens in a training sequence, we define our energy parameterization on these 15% chunks. Specifically, for a sequence of length T , and the subset size l = ⌈0.15×T ⌉, we consider computing the energy with the set C consisting of all T l combinations of masking patterns.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
where s \I is the sequence s with the l positions in I masked. Computing this energy, which involves running |C| = T l forward passes of the MLM, is expensive. Hence, we further approximate this parametrization by summing up over K random masking patterns where K ≪ |C|.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Population Data distribution of ( ) over population Choose threshold t so that FPR is , e.g. 10% LR Test for (s,t)
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
All threshold values t corresponding to 0 ≤ ≤ 1  
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Given the form of the likelihood ratio test statistic (Eq. 2) and energy function formulation for MLM likelihood (Eq. 2), we conduct the attack as follows (shown in Figure 1): 2. We compare L(s) to a threshold t, and if L(s) ≤ t, we reject the null hypothesis (H out ) and mark the sample as a member. Otherwise, we mark it as a non-member.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Choosing the threshold. The threshold determines the (false positive) error the adversary is willing to tolerate in the membership inference attack. Thus, for determining the threshold t, we select a false positive rate α, and empirically compute t as the corresponding percentile of the likelihood ratio statistic over random samples from the underlying distribution. This process is visualized in Figure 2a.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We empirically estimate the distribution of the test statistic L(x) using all the sequences x drawn from the general population distribution. This yields the distribution of L under the null hypothesis. We then select the threshold such that the tolerance of attack's error i.e. the rate at which attack falsely classifies the population data as "members" is α%.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Quantifying the Privacy Risk. The attacker's success (i.e. the privacy loss of the model) can be quantified using the relation between the attack's power (the true positive rate) versus its error (the false positive rate). Higher power for lower errors indicates larger privacy loss. To compare two attack algorithms (e.g., our method versus the target model loss based methods), we can compute their power for all different error values, which can be illustrated in an ROC curve (as in Figure 2b and Figure 4). This enables a complete comparison between two attack algorithms. The Area Under the Curve (AUC) metric for each attack provides an overall threshold independent evaluation of the privacy loss under each attack.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We conduct our experiments using the preprocessed data, and pre-trained models provided by Lehman et al. (2021). We use this medical-based setup as medical notes are sensitive and leakage of models trained on notes can cause privacy breaches.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In this section, we briefly explain the details of our experimental setup. Appendix A.2 provides more details. Table 1 provides a summary.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We run our attack on two sets of target samples, in both of which the "members" portion is sampled from the training set (D) of our target models, which is the MIMIC-III dataset. The non-members, however, are different. For the results shown under "MIMIC", the non-members are a held-out subset of the MIMIC data that was not used in training. For i2b2, the non-members are from a different (but similar) dataset, i2b2. Below we elaborate on each of these datasets (full detail in Appendix A.2.2). Both the datasets require a license for access, so we cannot show examples of the training data. MIMIC-III. The target models we attack are trained on the pseudo re-identified MIMIC-III notes which consist of 1,247,291 electronic health records (EHR) of 46,520 patients. i2b2. This dataset was curated for the i2b2 de-identification of protected health information (PHI) challenge in 2014 (Stubbs and Özlem Uzuner, 2015). We use this dataset as a secondary non-member dataset since it is similar in domain to MIMIC-III (both are medical notes), is larger in terms of size than the held-out MIMIC-III set, and has not been used as training data for our models.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Target Models. We perform our attack on 4 different pre-trained ClinicalBERT models, that are all trained on MIMIC-III, but with different training ClinicalBERT-large target model, trained for 300k iterations w/ sequence length 128 and 100k iterations w/ sequence length 512. Large++
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
ClinicalBERT-large++ target model, same as the Large model but trained for longer: trained for 1M iterations w/ a sequence length of 128.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Baseline with threshold set to be the mean of training sample losses (µ) (for reporting threshold-dependant metrics) (A) w/ Pop. thresh. Baseline with threshold set so that there is 10% false positive rate (for reporting threshold-dependant metrics) (B) w/ Pop. thresh. Our method with threshold set so that there is 10% false positive rate ( for reporting threshold-dependant metrics) procedures, summarized in Table 1 under Models. Reference Models. We use Pubmed-BERT 1 trained on pre-processed PubMed texts containing around 4000M words extracted from PubMed ASCII code version (Peng et al., 2019) as our main domain-specific reference model, since its training data is similar to MIMIC-III in terms of domain, however, it does not include MIMIC-III training data. We also use the standard pre-trained bert-base-uncased as a general-domain reference model for ablating our attack.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We compare our results with a popular prior method, which uses the loss of the target model as a signal to predict membership (Yeom et al., 2018;Jayaraman et al., 2021;Ye et al., 2021). We show this baseline as Model loss in our tables. This baseline could have two variations, based on the way its threshold is chosen: (1) µ threshold (Jagannatha et al., 2021), which assumes access to the mean of the training data loss, µ and uses it as the threshold for the attack, and (2) population threshold (pop. thresh.) which calculates the loss on a population set of samples (samples that were not used in training but are similar to training data), and then selects the threshold that would result in a 10% false positive rate on that population.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Area Under the ROC Curve (AUC). The ROC curve is a plot of power (true positive rate) versus error (false positive rate), measured across different thresholds t, which captures the trade-off between power and error. Thus, the area under the ROC curve (AUC) is a single, threshold-independent metric for measuring the strength of the attack.  Figure 3: (a) likelihood ratio histogram for training data members and non-members. (b) loss histogram for training data members and non-members. The blue lines in the two figures correspond to the same target sample (which is a random member of training-set). The red line is is the threshold at α = 10% false positive rate. The threshold from our attack (a) is correctly able to label the test sample as a training-set member but the thresholds from the baseline attack (b) fails to do so. members of the training set out of the total number of target samples inferred as members by the attack. For recall, we measure the percentage of samples correctly inferred as members of the training set out of the total number of target samples that are actually members of the training set.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In this section, we discuss our experimental results and main observations. First, we explore the overall performance improvement of our approach over baselines. Later, we analyze the effectiveness of our approach across several factors of variation that have an effect on the leakage of the model. (e.g. length of samples, model size, including names   Table 2. etc.) Finally, we explore correlations between samples that are deemed to be exposed by our approach. We provide further studies and ablations on choosing a lower false-positive rate of 1%, using different energy formulation and changing target sequence lengths in Appendix sections A.3.1, A.3.2, and A.3.3, respectively. Table 2 shows the metrics for our attack and the baseline's on both sample and patient level, with held-out MIMIC-III and i2b2 medical notes used as non-member samples ( Figure 4 shows the ROC curve). The table shows that our method significantly outperforms the target model loss-based baselines (Jagannatha et al., 2021;Yeom et al., 2018), which threshold the loss of the target model based on either the mean of the training samples' loss (µ), or the population samples' loss. Our attack's improvement over the baselines is more apparent in the case where both the members and non-members are from MIMIC-III. This case is harder for the baselines since members and non-members are much more similar and harder to distinguish if we only look at the loss of the target model. Our attack, however, is successful due to the use of a reference, which helps magnify the gap in the behavior of the target model towards members and non-members, thereby teasing apart similar samples.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We can also see that in terms of precision/recall trade-off, our attack has a consistently higher recall, with an average higher precision. Population loss based thresholding ((A) w/ Pop. thresh.) has the lowest recall of 15.6%, which is due to members and non-members achieving similar losses from the target model due to their similarity. This is also shown in Figure 3b. In Figure 3a, however, we see a distinct separation between the member and non-member histogram distributions when we use the ratio statistic L(s) as the test criterion for our attack. This results in the estimation of a useful threshold that correctly classifies the blue line sample as a member, as opposed to using only the target model loss (Figure 3b). Finally, we observe that all the metrics have higher values on the patient-level attack, compared to sample-level, for both our attack and the baselines. This is due to the higher granularity of the patient level attack, as it makes the decision based on an aggregate of multiple samples.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Tables 3 and 4 show the metrics for our attack and the baseline broken down based on the length of the target sample, and the size and training epochs of the target model, respectively. In Table 3, the target model is same as that of Table 2, ClinicalBERT-base. Short samples are those that have between 10 to 20 tokens, and long samples have 20 to 60 tokens. We can see that both the baseline and our attacks show more leakage for long sentences than they do for short sequences, which could be due to the longer sentences being more unique and thus being more likely to provide a discriminative signal for a sequence-level decision. Table 4 shows the attacks mounted on the four models from Table 1. We see that leakage on all the models is very similar, however, the AUC on Large++ is consistently higher than on Base, which hints at the observation made by (Carlini et al., 2021b) that larger models tend to have a higher capacity for memorization.  Table 6 shows results for attacking the name insertion model (Lehman et al., 2021), shown as Base-b, where the patient's first and last name are prepended to each training sample. We see that our attack's performance is better on the name-insertion model,  compared to the base model, whereas the baseline attack performs worse (in the sample-level scenario). We hypothesize that this is due to the "difficulty" of the samples. Adding names to the beginning of each sample actually increases the entropy of the dataset overall, since in most cases they don't have a direct relation with the rest of the sentence (except for very few sentences that directly state a person's disease), therefore they might as well be random. This makes these sentences more difficult and harder to learn, as there is no easy pattern. Hence, on average, these sentences have higher loss values (2.14 for name inserted samples, vs. 1.61 for regular samples). However, for the non-members, since they don't have names attached to them, the average loss is the same (the 10% FPR threshold is 1.32), and that is why the attack performs poorly on these samples, as most of the members get classified as non-members. For our attack, since we use the reference, we are able to tease apart such hard samples as they are extremely less likely given the reference than they are given the target model.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
To evaluate whether there are correlations between samples that have high leakage based on our attack (i.e. training samples that are successfully detected as members), we conduct an experiment. In this experiment, we create a new train and test dataset, by subsampling the main dataset and selecting 5505 and 7461 samples, respectively. We label the training and test samples based on whether they are exposed or not, i.e. whether the attack successfully detects them as training samples or not, and get 2519 and 3283 samples labeled as "memorized", for the train and test set. Since our goal is to see if we can find correlations between the memorized samples of the training set and use those to predict memorization on our test set, we create features for each sample, and then use those features with the labels to create a simple logistic regression classifier that predicts memorization. Table 7 shows these results in terms of precision and recall for predicting if a sample is "memorized" or not, with different sets of features. The first 4 rows correspond to individual handcrafted feature sets: (A) the number of digits in the sample, (B) length of a sample (in tokens), (C) the number of non-alphanumeric characters (this would be characters like '*', '-', etc.). (D) corresponds to feature sets that are obtained by encoding the tokenized sample by the frequency of each of its tokens, and then taking the 3 least frequent tokens' frequencies as features (the frequency comes from a frequency dictionary built on the training set). We can see that among the hand-crafted features, (C) is most indicative, as it counts the characters that are more out-of-distribution and are possibly not determined by grammatical rules or consistent patterns. (C) and (D) concatenated together perform slightly better than (C) alone, which could hint at the effect frequency of tokens and how common they are could have on memorization. We also get a small improvement over these by concatenating (B), (C), and (D), which shows the length has a slight correlation too.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In this paper, we introduce a principled membership inference attack based on likelihood ratio testing to measure the training data leakage of Masked Language Models (MLMs). In contrast to prior work on MLMs, we rely on signals from both the model under attack and a reference model to decide the membership of a sample. This enables performing successful membership inference attacks on data points that are hard to fit, and therefore cannot be detected using the prior work. We also perform an analysis of why these models leak, and which data points are more susceptible to memorization. Our attack shows that MLMs are significantly prone to memorization. This work calls for designing robust privacy mitigation algorithms for such language models.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Membership inference attacks form the foundation of privacy auditing and memorization analysis in machine learning. As we show in this paper, and as it is shown in the recent work (Carlini et al., 2021a;Ye et al., 2021), these attacks are very efficient in identifying privacy vulnerabilities of models with respect to individual data records. However, for a thorough analysis of data privacy, it is not enough to rely only on membership inference attacks. We thus would need to extend our analysis to reconstruction attacks and property inference attacks.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We use two datasets in this paper, MIMIC-III and i2b2, both of which contain sensitive data and can only be accessed by request 2 and after agreeing to the data usage and confidentiality terms 3 and passing proper training for ethical and privacypreserving use of the data. For reproduction of our results, code will be made available only by request and for research purposes, only to researchers who provide proof of authorized access to the datasets (by forwarding the access granted emails from MIMIC-III and i2b2 to the first author 4 ).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
To protect models against membership inference attacks, like the one proposed in this work, differentially private training algorithms (Abadi et al., 2016;Chaudhuri et al., 2011) can be used, as they are theoretically designed to protect the membership of each data record individually. Other methods such as adversarial training (Mireshghallah et al., 2021) and personally identifiable information scrubbing (Dernoncourt et al., 2017) can also be used, however, they do not provide the worst-case guarantees that differential privacy does (Brown et al., 2022  Set of target samples whose membership we want to determine. s A given target sample (s ∈ S, p(s ∈ D) = 0.5)) R Reference model t Attack threshold L(s)
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Likelihood ratio for given sample s
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
To summarize and clarify the notations used in the paper for explaining our attack, we added Table 8.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We use two datasets in this paper, MIMIC-III and i2b2, both of which contain sensitive data and can only be accessed by request through https://mimic. mit.edu/docs/gettingstarted/ and https://portal.dbmi.hms. harvard.edu/projects/n2c2-nlp/, and after agreeing to the data usage and confidentiality terms 5 and passing proper training for ethical and privacy-preserving use of the data. For reproduction of our results, code will be made available only by request and for research purposes, only to researchers who provide proof of authorized access to the datasets (by forwarding the access granted emails from MIMIC-III and i2b2 to the first author 6 ).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We run our attack on two sets of target samples, one we denote by "MIMIC" and the other by "i2b2" in the results (both medical notes). For both of these, the "members" portion of the target samples is from the training set (D) of our target models, which is the MIMIC-III dataset. However, the non-members are different. For the results shown under MIMIC, the non-members are a held-out subset of the MIMIC data that was not used in training. For i2b2, the non-members are from a different (but similar) dataset, i2b2. Below we elaborate on this setup and each of these datasets. For the attack's target data, we use a held-out subset of MIMIC-III consisting of 89 patients (4072 sample sequences) whose data was not used during training of the ClinicalBERT target models and use them as "non-member" samples. For "member" samples, we take a 99 patient subsample of the entire training data, and then subsample 4072 sample sequences from that (we do this 10 times for each attack and average the results over), so that the number of member and non-member samples are the same and the target pool is balanced.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
i2b2 This dataset was curated for the i2b2 de-identification of protected health information (PHI) challenge in 2014 (Stubbs and Özlem Uzuner, 2015). We use this dataset as a secondary non-member dataset, since it is similar in domain to MIMIC-III (both are medical notes), is larger in terms of size than the held-out MIMIC-III set, and has not been used as training data for our models. We subsample 99 patients from i2b2, consisting of 18561 sequences, and use them as non-members.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The population data that we use to evaluate the distribution of likelihood ratio over the null hypothesis (which is used to compute the threshold) is disjoint with the non-member set that we use to evaluate the attack. We randomly select 99 patients from the i2b2 dataset for this purpose.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We perform our attack on 5 different pre-trained models, that are all trained on MIMIC-III, but with different training procedures:
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
ClinicalBERT (Base) BERT-base architecture trained over the pseudo re-identified MIMIC-III notes for 300k iterations for sequence length 128 and for 100k iterations for sequence length 512. ClinicalBERT++ (Base++) BERT-base architecture trained over the pseudo re-identified MIMIC III notes for 1M iterations at a sequence length of 128.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
ClinicalBERT-Large (Large). BERT-large architecture trained over the pseudo re-identified MIMIC-III notes for 300k iterations for sequence length 128 and for 100k iterations for sequence length 512.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
ClinicalBERT-large++ (Large++) BERT-large architecture trained over the pseudo re-identified MIMIC III notes for 1M iterations at a sequence length of 128.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
ClinicalBERT-b ((Base-b), Name Insertion). Same training and architecture as ClincalBERTbase, but that the patient's surrogate name is prepended to the beginning of every sentence. This model is used to identify the effect of nameinsertion on the memorization of BERT-based models (Lehman et al., 2021).
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
For this paper, we did not train any models, so the GPU training time is 0 hours. However, for getting the likelihoods, we ran inference on the sequences in our target samples pool. For that, we used an RTX2080 GPU with 11GB of memory for 18 hours.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
All the threshold-dependant results (precision and recalls) in Section 4 are reported with a threshold set for having α = 10% false positive rate (using the mechanism shown in Figure 2a). In this section, we want to look at lower false positive rates, like we do in Figure 5, and see how well our attack does when precision is very important to us and we do not want to get any false positives. These results are shown in Table 10. (This table corresponds with Table 4 from Section 4.2 and the sample-level part of Table 6.) We can see that compared to Table 10, as we are decreasing the false positive rate, the performance gap between our attack and the baseline increases drastically, showin that our attack performs really well under tight false positive rate constraints. Note that AUC is not threshold dependant therefore it has the same value in both tables.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In the paper, we use E(s;θ), as shown in Equation 2 for finding the likelihood ratio. In other words, for finding the likelihood ratio, we basically calculate the loss of the target model and reference model (using 15% masking and averaging over 10 times) on the given sequence s, and subtract them. However, another way to approach this problem of calculating likelihood ratio is to use the normalized energy (instead of the loss) as introduced in (Goyal et al., 2022), instead of the loss. For calculating the normalized energy, we mask each token in the sequence, one token at a time (instead of 15%), calculated the loss, and average over all the tokens. To see how this energy does, we have used it to mount our attack, and we show the results in Table 11.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Compared to using loss (Table 9) it seems like the AUC for normalized energy is higher overall,
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
All the results in Section 4 are reported for long sequences (more than 20 tokens long), except those reported in Table 3, where we ablate the privacy risks for sequences of different lengths. In this section, for the sake of completion, we are reporting results for short sequences as well as long sequences, for all the five models we study. These results are shown in Table 9. (This table corresponds with Table 4 from Section 4.2 and the sample-level part of Table 6.)
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Figures 3a and 3b show histogram visualizations of L(s) (likelihood ratio statistic) and model loss, over the target sample pool (i.e. mixture of members and non-members from the MIMIC dataset), respectively. Here the target model is ClinicalBERT-Base. The blue line represents a target query sample drawn from the member set. The point of these visualizations is to show a case in which a sample is misclassified as a non-member by the model loss baseline, but is correctly classified as a member using our attack.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
The member and non-member distributions' histograms are shown via light blue and light orange bars respective. The red line indicates the threshold that is selected such that α = 0.1, i.e. 10% false positive rate. In Figure 3a we see a distinct separation between the member and non-member histogram distributions when we use L(s) as the test criterion for our attack. This results in the estimation of a useful threshold that correctly classifies the blue line sample as a member. In contrast, the baseline attack by (Jagannatha et al., 2021), in Figure 3b based solely upon the target model's loss leads to a high overlap between the member and non-member histograms which leads to a threshold that misclassifies the sample represented by the blue line. These histograms show that the reference model used in our method helps in getting a sense of how hard each sample is in general, and puts each point in perspective.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
In Figure 5 We have plotted Figure 4 from the results, but with logarithmic x-axis, to zoom in on the low false positive rate section and really show the differences between our attack and the baseline. Table 9: Sample-level attack results (with α = 10% false positive rate used for thresholding) on the four ClinicalBERT models, plus the Base-b model (name insertion). We use PubMed-BERT as the reference model and the MIMIC data as non-members. Base++ (Large++) is same as Base (Large), but trained for more epochs. Base-b is the same as the Base model, but the training data was modified to prepend patient's first and last name to each sequence. This   Figure 5: The ROC curve of sample-level attack, with logarithmic scale x-axis. performed on Clinical-BERT using MIMIC samples as non-members. Green line shows our attack and the red line shows the baseline loss-based attack. The blue dashed line shows AUC=0.5 (random uess). We find false positive and true positive rates over the target sample pool, which is comprised of members and non-members. This curve corresponds to Table 2 in the results, and is the zoomed-in version of Figure 4.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Since our work proposes an attack for quantifying leakage of masked language models (MLMs), based on the likelihood ratio, there are two lines of work that are related to ours: (1) work surrounding attacks/leakage on machine learning models (2) work on calculating sequence likelihood for MLMs. Prior work on measuring memorization and leakage in machine learning models, and specifically NLP models can itself be classified into two main categories: (1) membership inference attacks and (2) training data extraction attacks. Below we discuss each line of work in more detail.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Membership inference. Membership Inference Attacks (MIA) try to determine whether or not a target sample was used in training a target model (Shokri et al., 2017;Yeom et al., 2018). These attacks be seen as privacy risk analysis tools (Murakonda and Shokri, 2020;Nasr et al., 2021;Kandpal et al., 2022), which help reveal how much the model has memorized the individual samples in Table 11: Sample-level attack results (with α = 10% false positive rate used for thresholding) on the four ClinicalBERT models, plus the Base-b model (name insertion). Here we use "normalized energy" as a proxy for likelihood, instead of using the 15% masked energy formulation of Equation 2. We use PubMed-BERT as the reference model and the MIMIC data as non-members. Base++ (Large++) is same as Base (Large), but trained for more epochs. Base-b is the same as the Base model, but the training data was modified to prepend patient's first and last name to each sequence. This  Long et al., 2017;Salem et al., 2018;Ye et al., 2021;Carlini et al., 2021a) A group of these attacks rely on behavior of shadow models (models trained on data similar to training, to mimic the target model) to determine the membership of given samples (Jayaraman et al., 2021;Shokri et al., 2017). In the shadow model training procedure the adversary trains a batch of models m 1 ,m 2 ,...,m k as shadow models, with data from the target user. Then, it trains m ′ 1 ,m ′ 2 ...,m ′ k without the data from the target user and then tries to find some statistical disparity between these models (Mahloujifar et al., 2021). Shadow-based attacks have been mounted on NLP models as well: (Song and Shmatikov, 2018) mounts such an attack on LSTM-based text-generation models, (Mahloujifar et al., 2021) mounts one on word embedding, (Hisamoto et al., 2020) applies it to machine translation and more recently, (Shejwalkar et al., 2021) mounts it on transformer-based NLP classification models. Mounting such attacks is usually costly, as their success relies upon training multiple shadow models, and access to adequate shadow data for training such models.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Another group of MIAs relies solely on the loss value of the target sample, under the target model, and thresholds this loss to determine membership (Jagannatha et al., 2021;Yeom et al., 2018). Song and Raghunathan mount such an attack on word embedding, where they try to infer if given samples were used in training different embedding models. Jagannatha et al., which is the work closest to ours, uses a thresholding loss-based attack to infer membership on MLMs. Although our proposed attack is also a threshold-based one, it is different from prior work by: (a) applying likelihood ratio testing using a reference model and (b) calculating the likelihood through our energy function formulation. These two components cause our attack to have higher AUC, as shown in the results.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
We refer the reader to the framework introduced by (Ye et al., 2021) that formalizes different membership inference attacks and compares their performance on benchmark ML tasks.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Training data extraction. Training data extraction quantifies the risk of extracting training data by probing a trained language model (Salem et al., 2020;Carlini et al., 2019;Zanella-Béguelin et al., 2020;Carlini et al., 2021bCarlini et al., , 2022Nakamura et al., 2020). The most prominent of such attacks, on NLP models is that of Carlini et al. (2021b), where they take more than half a million samples from different GPT-2 models, sift through the samples using a membership inference method to find samples that are more likely to have been memorized, and finally, once they have narrowed down the samples to 1800, they check the web to see if such samples might have been in the GPT-2 training set. They find that over 600 of those 1800 samples were verbatim training samples. Lehman et al. (2021) mount the same data extraction attack on MLMs, but their results are somehow inconclusive as to how much MLMs memorize samples, as only 4% of generated sentences with a patient's name also contain one of their true medical conditions. They also mount other type of attacks, where they try to extract a person's name given their disease, or disease given name, but in all their attacks, they only use signals from the target model and consistently find that a frequencybased baseline (i.e. one that would always guess the most frequent name/disease) is more successful.