@inproceedings{zan-etal-2023-large,
    title = "Large Language Models Meet {NL}2{C}ode: A Survey",
    author = "Zan, Daoguang  and
      Chen, Bei  and
      Zhang, Fengji  and
      Lu, Dianjie  and
      Wu, Bingchao  and
      Guan, Bei  and
      Yongji, Wang  and
      Lou, Jian-Guang",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.411",
    doi = "10.18653/v1/2023.acl-long.411",
    pages = "7443--7464",
    abstract = "The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are {``}Large Size, Premium Data, Expert Tuning{''}. In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website \url{https://nl2code.github.io} to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.",
}

The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are {``}Large Size, Premium Data, Expert Tuning{''}. In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website \url{https://nl2code.github.io} to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.
Zan, Daoguang Chen, Bei Zhang, Fengji Lu, Dianjie Wu, Bingchao Guan, Bei Yongji, Wang Lou, Jian-Guang
Large Language Models Meet NL2Code: A Survey