@inproceedings{yin-etal-2023-natural,
    title = "Natural Language to Code Generation in Interactive Data Science Notebooks",
    author = "Yin, Pengcheng  and
      Li, Wen-Ding  and
      Xiao, Kefan  and
      Rao, Abhishek  and
      Wen, Yeming  and
      Shi, Kensen  and
      Howland, Joshua  and
      Bailey, Paige  and
      Catasta, Michele  and
      Michalewski, Henryk  and
      Polozov, Oleksandr  and
      Sutton, Charles",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.9",
    doi = "10.18653/v1/2023.acl-long.9",
    pages = "126--173",
    abstract = "Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1078 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions. Arcade is publicly available at \url{https://github.com/google-research/arcade-nl2code/}.",
}

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1078 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions. Arcade is publicly available at \url{https://github.com/google-research/arcade-nl2code/}.
Yin, Pengcheng Li, Wen-Ding Xiao, Kefan Rao, Abhishek Wen, Yeming Shi, Kensen Howl , Joshua Bailey, Paige Catasta, Michele Michalewski, Henryk Polozov, Oleks r Sutton, Charles
Natural Language to Code Generation in Interactive Data Science Notebooks