@inproceedings{ding-etal-2023-static,
    title = "A Static Evaluation of Code Completion by Large Language Models",
    author = "Ding, Hantian  and
      Kumar, Varun  and
      Tian, Yuchen  and
      Wang, Zijian  and
      Kwiatkowski, Rob  and
      Li, Xiaopeng  and
      Ramanathan, Murali Krishna  and
      Ray, Baishakhi  and
      Bhatia, Parminder  and
      Sengupta, Sudipta",
    editor = "Sitaram, Sunayana  and
      Beigman Klebanov, Beata  and
      Williams, Jason D",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-industry.34",
    doi = "10.18653/v1/2023.acl-industry.34",
    pages = "347--360",
    abstract = "Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the other hand, static analysis tools such as linters, which can detect errors without running the program, haven{'}t been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.",
}

Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the other hand, static analysis tools such as linters, which can detect errors without running the program, haven{'}t been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.
Ding, Hantian Kumar, Varun Tian, Yuchen Wang, Zijian Kwiatkowski, Rob Li, Xiaopeng Ramanathan, Murali Krishna Ray, Baishakhi Bhatia, Parminder Sengupta, Sudipta
A Static Evaluation of Code Completion by Large Language Models