@inproceedings{tian-etal-2023-unsupervised,
    title = "Unsupervised Melody-to-Lyrics Generation",
    author = "Tian, Yufei  and
      Narayan-Chen, Anjali  and
      Oraby, Shereen  and
      Cervone, Alessandra  and
      Sigurdsson, Gunnar  and
      Tao, Chenyang  and
      Zhao, Wenbo  and
      Chen, Yiwen  and
      Chung, Tagyoung  and
      Huang, Jing  and
      Peng, Nanyun",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.513",
    doi = "10.18653/v1/2023.acl-long.513",
    pages = "9235--9254",
    abstract = "Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical interest and more challenging than unconstrained lyric generation as the music imposes additional constraints onto the lyrics. The training data is limited as most songs are copyrighted, resulting in models that underfit the complicated cross-modal relationship between melody and lyrics. In this work, we propose a method for generating high-quality lyrics without training on any aligned melody-lyric data. Specifically, we design a hierarchical lyric generation framework that first generates a song outline and second the complete lyrics. The framework enables disentanglement of training (based purely on text) from inference (melody-guided text generation) to circumvent the shortage of parallel data. We leverage the segmentation and rhythm alignment between melody and lyrics to compile the given melody into decoding constraints as guidance during inference. The two-step hierarchical design also enables content control via the lyric outline, a much-desired feature for democratizing collaborative song creation. Experimental results show that our model can generate high-quality lyrics that are more on-topic, singable, intelligible, and coherent than strong baselines, for example SongMASS, a SOTA model trained on a parallel dataset, with a 24{\%} relative overall quality improvement based on human ratings. Our code is available at \url{https://github.com/amazon-science/unsupervised-melody-to-lyrics-generation}.",
}

Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical interest and more challenging than unconstrained lyric generation as the music imposes additional constraints onto the lyrics. The training data is limited as most songs are copyrighted, resulting in models that underfit the complicated cross-modal relationship between melody and lyrics. In this work, we propose a method for generating high-quality lyrics without training on any aligned melody-lyric data. Specifically, we design a hierarchical lyric generation framework that first generates a song outline and second the complete lyrics. The framework enables disentanglement of training (based purely on text) from inference (melody-guided text generation) to circumvent the shortage of parallel data. We leverage the segmentation and rhythm alignment between melody and lyrics to compile the given melody into decoding constraints as guidance during inference. The two-step hierarchical design also enables content control via the lyric outline, a much-desired feature for democratizing collaborative song creation. Experimental results show that our model can generate high-quality lyrics that are more on-topic, singable, intelligible, and coherent than strong baselines, for example SongMASS, a SOTA model trained on a parallel dataset, with a 24{\%} relative overall quality improvement based on human ratings. Our code is available at \url{https://github.com/amazon-science/unsupervised-melody-to-lyrics-generation}.
Tian, Yufei Narayan-Chen, Anjali Oraby, Shereen Cervone, Aless ra Sigurdsson, Gunnar Tao, Chenyang Zhao, Wenbo Chen, Yiwen Chung, Tagyoung Huang, Jing Peng, Nanyun
Unsupervised Melody-to-Lyrics Generation