@inproceedings{wang-etal-2023-recode,
    title = "{R}e{C}ode: Robustness Evaluation of Code Generation Models",
    author = "Wang, Shiqi  and
      Li, Zheng  and
      Qian, Haifeng  and
      Yang, Chenghao  and
      Wang, Zijian  and
      Shang, Mingyue  and
      Kumar, Varun  and
      Tan, Samson  and
      Ray, Baishakhi  and
      Bhatia, Parminder  and
      Nallapati, Ramesh  and
      Ramanathan, Murali Krishna  and
      Roth, Dan  and
      Xiang, Bing",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.773",
    doi = "10.18653/v1/2023.acl-long.773",
    pages = "13818--13843",
    abstract = "Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model{'}s robustness performance. With human annotators, we verified that over 90{\%} of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.",
}

Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model{'}s robustness performance. With human annotators, we verified that over 90{\%} of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.
Wang, Shiqi Li, Zheng Qian, Haifeng Yang, Chenghao Wang, Zijian Shang, Mingyue Kumar, Varun Tan, Samson Ray, Baishakhi Bhatia, Parminder Nallapati, Ramesh Ramanathan, Murali Krishna Roth, Dan Xiang, Bing
ReCode: Robustness Evaluation of Code Generation Models