“Looks Okay to Me”: A Study of Best Practice in Data Analysis Code Review

Education in statistical computing requires that we train students not only in programming skills and principles, but also in good data science habits. In this study, we investigate the question of how certain habits, routines, or intuitions contribute to quality analysis, in the context of identifying errors in an existing work. Volunteers from two populations—professional data scientists and mid-degree college students—were supplied with pre-populated R Markdown notebooks and asked to comb through the reports' code and discussion in search of errors. We then conducted a qualitative analysis of subject behavior during the study, based on video recordings of these sessions. Ultimately, we identified many common themes in how the subjects interacted with the code, text, and integrated development environment during their error-checking process.