@inproceedings{tang-etal-2023-diffusion,
    title = "Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference !",
    author = "Tang, Zecheng  and
      Wang, Pinzheng  and
      Zhou, Keyan  and
      Li, Juntao  and
      Cao, Ziqiang  and
      Zhang, Min",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.721",
    doi = "10.18653/v1/2023.findings-acl.721",
    pages = "11359--11386",
    abstract = "Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space. However, there exist nonnegligible gaps between training and inference, owing to the absence of the forward process during inference. Thus, the model only predicts based on the previously generated reverse noise rather than the noise computed by the forward process. Besides, the widely-used downsampling strategy in speeding up the inference will cause the mismatch of diffusion trajectories between training and inference. To understand and mitigate the above two types of training-inference discrepancies, we launch a thorough preliminary study. Based on our observations, we propose two simple yet effective methods to bridge the gaps mentioned above, named Distance Penalty and Adaptive Decay Sampling. Extensive experiments on \textbf{6} generation tasks confirm the superiority of our methods, which can achieve $\mathbf{100}\times \rightarrow \mathbf{200}\times$ speedup with better performance. Our code will be released at \url{https://github.com/CODINNLG/Bridge_Gap_Diffusion}.",
}

Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space. However, there exist nonnegligible gaps between training and inference, owing to the absence of the forward process during inference. Thus, the model only predicts based on the previously generated reverse noise rather than the noise computed by the forward process. Besides, the widely-used downsampling strategy in speeding up the inference will cause the mismatch of diffusion trajectories between training and inference. To understand and mitigate the above two types of training-inference discrepancies, we launch a thorough preliminary study. Based on our observations, we propose two simple yet effective methods to bridge the gaps mentioned above, named Distance Penalty and Adaptive Decay Sampling. Extensive experiments on \textbf{6} generation tasks confirm the superiority of our methods, which can achieve $\mathbf{100}\times \rightarrow \mathbf{200}\times$ speedup with better performance. Our code will be released at \url{https://github.com/CODINNLG/Bridge_Gap_Diffusion}.
Tang, Zecheng Wang, Pinzheng Zhou, Keyan Li, Juntao Cao, Ziqiang Zhang, Min
Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference !