@inproceedings{yue-etal-2023-synthetic,
    title = "Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe",
    author = "Yue, Xiang  and
      Inan, Huseyin  and
      Li, Xuechen  and
      Kumar, Girish  and
      McAnallen, Julia  and
      Shajari, Hoda  and
      Sun, Huan  and
      Levitan, David  and
      Sim, Robert",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.74",
    doi = "10.18653/v1/2023.acl-long.74",
    pages = "1321--1342",
    abstract = "Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.",
}

Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.
Yue, Xiang Inan, Huseyin Li, Xuechen Kumar, Girish McAnallen, Julia Shajari, Hoda Sun, Huan Levitan, David Sim, Robert
Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe