- Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, Nicholas Carlini. 2022. Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy. Abstract: Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data, and aids in the evaluation of potential countermeasures. Many prior works—and some recently deployed defenses—focus on “verbatim memo-rization”, deﬁned as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization def-initions are too restrictive and fail to capture more subtle forms of memorization. Speciﬁcally, we design and implement an efﬁcient defense based on Bloom ﬁlters that perfectly prevents all verbatim memorization. And yet, we demonstrate that this “perfect” ﬁlter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and mini-mally modiﬁed “style-transfer” prompts—and in some cases even the non-modiﬁed original prompts—to extract memorized information. For example, instructing the model to output ALL-CAPITAL texts bypasses memorization checks based on verbatim matching. We conclude by discussing potential alternative deﬁnitions and why deﬁning memorization is a difﬁcult yet crucial open question for neural language models.
