
# Raw Data from MultiPL-E

**Uploads are a work in progress. If you are interested in a split that is not yet available, please contact a.guha@northeastern.edu.**

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the 
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use [the following script](https://github.com/nuprl/MultiPL-E/blob/main/upload_completions.py-)
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named `base_dataset`.`language`.`model`.`temperature`.`variation`

- `base_dataset` is either `humaneval` or `mbpp`.
- `language` is the file extension of the programming language. E.g., `py` for Python or `sh` for Bash.
- `model` is the name of the model. Some model names used by MultiPL-E:
   - `bigcode_15b_1000m`:  StarCoderBase
   - `bigcode_15b_200m`, `bigcode_15b_400m`, `bigcode_15b_600m`, `bigcode_15b_800m`: the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
  - `pystarcoder2`: StarCoder
   - there are others, just have a look :)
- `temperature`: this is either 0.2 or 0.8
- `variation`: the variation of the MultiPL-E prompt to use, which should be `reworded`. MultiPL-E has several prompt ablations and the `reworded` prompts are the canonical variation.
code region:us
{"pretty_name": "MultiPL-E generated programs and execution results", "dataset_info": {"features": [{"name": "experiment", "dtype": "string"}, {"name": "problem", "dtype": "string"}, {"name": "language", "dtype": "string"}, {"name": "top_p", "dtype": "float64"}, {"name": "max_tokens", "dtype": "int64"}, {"name": "prompt", "dtype": "string"}, {"name": "tests", "dtype": "string"}, {"name": "stop_tokens", "sequence": "string"}, {"name": "completions", "sequence": "string"}, {"name": "programs", "sequence": "string"}, {"name": "stdouts", "sequence": "string"}, {"name": "stderrs", "sequence": "string"}, {"name": "exit_codes", "sequence": "int64"}, {"name": "statuses", "sequence": "string"}, {"name": "timestamps", "sequence": "int64"}], "splits": [{"name": "humaneval.py.bigcode_15b_800m.0.2.reworded", "num_bytes": 50941974, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_200m.0.2.reworded", "num_bytes": 57850786, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_400m.0.2.reworded", "num_bytes": 52404545, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_600m.0.2.reworded", "num_bytes": 55071293, "num_examples": 161}, {"name": "humaneval.rkt.bigcode_15b_800m.0.2.reworded", "num_bytes": 77194321, "num_examples": 161}, {"name": "humaneval.py.chatgpt.0.2.reworded", "num_bytes": 5473126, "num_examples": 161}, {"name": "humaneval.r.bigcode_15b_800m.0.2.reworded", "num_bytes": 73160389, "num_examples": 161}, {"name": "humaneval.r.bigcode_15b_1000m.0.2.reworded", "num_bytes": 63088872, "num_examples": 161}, {"name": "humaneval.r.bigcode_15b_200m.0.2.reworded", "num_bytes": 77532102, "num_examples": 161}, {"name": "humaneval.r.bigcode_15b_400m.0.2.reworded", "num_bytes": 80103073, "num_examples": 161}, {"name": "humaneval.r.bigcode_15b_600m.0.2.reworded", "num_bytes": 76123407, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_1000m.0.2.reworded", "num_bytes": 47282373, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_1000m.0.1.reworded", "num_bytes": 11724023, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_1000m.0.4.reworded", "num_bytes": 12167610, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_1000m.0.6.reworded", "num_bytes": 12344190, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_1000m.0.8.reworded", "num_bytes": 12825651, "num_examples": 161}, {"name": "humaneval.py.codegeex.0.2.reworded", "num_bytes": 49547494, "num_examples": 161}, {"name": "humaneval.py.codegen.0.2.reworded", "num_bytes": 55391634, "num_examples": 161}, {"name": "humaneval.py.Salesforce_codegen_16B_mono.0.2.reworded", "num_bytes": 54757013, "num_examples": 161}, {"name": "humaneval.py.cushman001.0.2.reworded", "num_bytes": 5081696, "num_examples": 161}, {"name": "humaneval.js.pystarcoder2.0.2.reworded", "num_bytes": 6784781, "num_examples": 161}, {"name": "humaneval.py.santacoder.0.2.reworded", "num_bytes": 57098223, "num_examples": 161}, {"name": "humaneval.py.santacoder_fim_prompt.0.2.reworded", "num_bytes": 5473782, "num_examples": 161}, {"name": "humaneval.lua.bigcode_15b_1000m.0.2.reworded", "num_bytes": 53170918, "num_examples": 161}, {"name": "mbpp.py.bigcode_15b_1000m.0.2.reworded", "num_bytes": 76438518, "num_examples": 397}, {"name": "mbpp.py.pystarcoder2.0.2.reworded", "num_bytes": 78424728, "num_examples": 397}, {"name": "mbpp.lua.davinci.0.2.reworded", "num_bytes": 82574073, "num_examples": 401}, {"name": "humaneval.js.davinci.0.2.transform", "num_bytes": 61113074, "num_examples": 161}, {"name": "humaneval.py.davinci.0.2.transform", "num_bytes": 46653237, "num_examples": 161}, {"name": "humaneval.ts.davinci.0.2.transform", "num_bytes": 60332695, "num_examples": 159}, {"name": "humaneval.ts.davinci.0.2.reworded", "num_bytes": 60256365, "num_examples": 159}, {"name": "humaneval.cpp.davinci.0.2.reworded", "num_bytes": 67044215, "num_examples": 159}, {"name": "humaneval.cs.davinci.0.2.reworded", "num_bytes": 103369582, "num_examples": 156}, {"name": "humaneval.d.davinci.0.2.reworded", "num_bytes": 66641398, "num_examples": 156}, {"name": "humaneval.go.davinci.0.2.reworded", "num_bytes": 71050586, "num_examples": 154}, {"name": "humaneval.java.davinci.0.2.reworded", "num_bytes": 71969268, "num_examples": 156}, {"name": "humaneval.jl.davinci.0.2.reworded", "num_bytes": 76515733, "num_examples": 159}, {"name": "humaneval.js.davinci.0.2.reworded", "num_bytes": 61674621, "num_examples": 161}, {"name": "humaneval.lua.davinci.0.2.reworded", "num_bytes": 54016568, "num_examples": 161}, {"name": "humaneval.php.davinci.0.2.reworded", "num_bytes": 61403511, "num_examples": 161}, {"name": "humaneval.pl.davinci.0.2.reworded", "num_bytes": 60281076, "num_examples": 161}, {"name": "humaneval.py.davinci.0.2.reworded", "num_bytes": 48223052, "num_examples": 161}, {"name": "humaneval.rb.davinci.0.2.reworded", "num_bytes": 65236003, "num_examples": 161}, {"name": "humaneval.r.davinci.0.2.reworded", "num_bytes": 48031115, "num_examples": 161}, {"name": "humaneval.rkt.davinci.0.2.reworded", "num_bytes": 66273931, "num_examples": 161}, {"name": "humaneval.rs.davinci.0.2.reworded", "num_bytes": 56522566, "num_examples": 156}, {"name": "humaneval.scala.davinci.0.2.reworded", "num_bytes": 84153007, "num_examples": 158}, {"name": "humaneval.sh.davinci.0.2.reworded", "num_bytes": 58089427, "num_examples": 158}, {"name": "humaneval.swift.davinci.0.2.reworded", "num_bytes": 61663115, "num_examples": 158}, {"name": "humaneval.java.bigcode_15b_1000m.0.2.reworded", "num_bytes": 75580922, "num_examples": 158}, {"name": "humaneval.java.bigcode_15b_200m.0.2.reworded", "num_bytes": 84307515, "num_examples": 158}, {"name": "humaneval.java.bigcode_15b_400m.0.2.reworded", "num_bytes": 85145220, "num_examples": 158}, {"name": "humaneval.java.bigcode_15b_600m.0.2.reworded", "num_bytes": 80223655, "num_examples": 158}, {"name": "humaneval.java.bigcode_15b_800m.0.2.reworded", "num_bytes": 78345454, "num_examples": 158}, {"name": "humaneval.java.codegeex.0.2.reworded", "num_bytes": 74859734, "num_examples": 158}, {"name": "humaneval.java.codegen.0.2.reworded", "num_bytes": 76705002, "num_examples": 158}, {"name": "humaneval.java.cushman001.0.2.reworded", "num_bytes": 7860926, "num_examples": 158}, {"name": "humaneval.java.replit_code.0.2.reworded", "num_bytes": 20396812, "num_examples": 158}, {"name": "humaneval.java.pystarcoder2.0.2.reworded", "num_bytes": 73528078, "num_examples": 158}, {"name": "humaneval.java.davinci.0.2.keep", "num_bytes": 71389774, "num_examples": 161}, {"name": "humaneval.java.davinci.0.2.remove", "num_bytes": 64493441, "num_examples": 157}, {"name": "humaneval.java.davinci.0.2.transform", "num_bytes": 72660740, "num_examples": 156}, {"name": "humaneval.java.davinci.0.8.keep", "num_bytes": 72358729, "num_examples": 161}, {"name": "humaneval.java.davinci.0.8.reworded", "num_bytes": 75043862, "num_examples": 156}, {"name": "humaneval.java.incoder.0.2.keep", "num_bytes": 110176353, "num_examples": 161}, {"name": "humaneval.java.incoder.0.2.remove", "num_bytes": 88903768, "num_examples": 157}, {"name": "humaneval.java.incoder.0.2.reworded", "num_bytes": 109021885, "num_examples": 156}, {"name": "humaneval.java.incoder.0.2.transform", "num_bytes": 107183302, "num_examples": 156}, {"name": "humaneval.java.incoder.0.8.keep", "num_bytes": 75299144, "num_examples": 139}, {"name": "humaneval.java.incoder.0.8.reworded", "num_bytes": 100533855, "num_examples": 158}, {"name": "mbpp.java.codegen.0.2.reworded", "num_bytes": 144592215, "num_examples": 373}, {"name": "mbpp.java.codegen.0.8.reworded", "num_bytes": 47521423, "num_examples": 120}, {"name": "mbpp.java.davinci.0.2.keep", "num_bytes": 135567713, "num_examples": 373}, {"name": "mbpp.java.davinci.0.2.reworded", "num_bytes": 136848151, "num_examples": 373}, {"name": "mbpp.java.incoder.0.2.reworded", "num_bytes": 225046095, "num_examples": 373}, {"name": "mbpp.java.incoder.0.8.reworded", "num_bytes": 167923488, "num_examples": 373}, {"name": "humaneval.cpp.davinci.0.2.keep", "num_bytes": 63794632, "num_examples": 161}, {"name": "humaneval.cpp.davinci.0.2.remove", "num_bytes": 58355394, "num_examples": 158}, {"name": "humaneval.cpp.davinci.0.2.transform", "num_bytes": 66852210, "num_examples": 159}, {"name": "humaneval.cpp.davinci.0.8.keep", "num_bytes": 61668425, "num_examples": 161}, {"name": "humaneval.cpp.bigcode_15b_1000m.0.2.reworded", "num_bytes": 67353068, "num_examples": 161}, {"name": "humaneval.cpp.bigcode_15b_200m.0.2.reworded", "num_bytes": 73914809, "num_examples": 161}, {"name": "humaneval.cpp.bigcode_15b_400m.0.2.reworded", "num_bytes": 68514672, "num_examples": 161}, {"name": "humaneval.cpp.bigcode_15b_600m.0.2.reworded", "num_bytes": 70059227, "num_examples": 161}, {"name": "humaneval.cpp.bigcode_15b_800m.0.2.reworded", "num_bytes": 69289473, "num_examples": 161}, {"name": "humaneval.cpp.codegeex.0.2.reworded", "num_bytes": 70250543, "num_examples": 161}, {"name": "humaneval.cpp.codegen.0.2.reworded", "num_bytes": 65355449, "num_examples": 161}, {"name": "humaneval.cpp.cushman001.0.2.reworded", "num_bytes": 6878097, "num_examples": 161}, {"name": "humaneval.cpp.replit_code.0.2.reworded", "num_bytes": 18647873, "num_examples": 161}, {"name": "humaneval.cs.bigcode_15b_1000m.0.2.reworded", "num_bytes": 115265463, "num_examples": 158}, {"name": "humaneval.cs.bigcode_15b_200m.0.2.reworded", "num_bytes": 128116325, "num_examples": 158}, {"name": "humaneval.cs.bigcode_15b_400m.0.2.reworded", "num_bytes": 116443233, "num_examples": 158}, {"name": "humaneval.cs.bigcode_15b_600m.0.2.reworded", "num_bytes": 110736924, "num_examples": 158}, {"name": "humaneval.cs.bigcode_15b_800m.0.2.reworded", "num_bytes": 116921504, "num_examples": 158}, {"name": "humaneval.cs.codegeex.0.2.reworded", "num_bytes": 108831398, "num_examples": 158}, {"name": "humaneval.cs.codegen.0.2.reworded", "num_bytes": 115085420, "num_examples": 158}, {"name": "humaneval.cs.cushman001.0.2.reworded", "num_bytes": 11455476, "num_examples": 158}, {"name": "humaneval.cs.replit_code.0.2.reworded", "num_bytes": 29978496, "num_examples": 158}, {"name": "humaneval.d.bigcode_15b_1000m.0.2.reworded", "num_bytes": 69856838, "num_examples": 156}, {"name": "humaneval.d.bigcode_15b_200m.0.2.reworded", "num_bytes": 69168908, "num_examples": 156}, {"name": "humaneval.d.bigcode_15b_400m.0.2.reworded", "num_bytes": 66130665, "num_examples": 156}, {"name": "humaneval.d.bigcode_15b_600m.0.2.reworded", "num_bytes": 60081870, "num_examples": 156}, {"name": "humaneval.d.bigcode_15b_800m.0.2.reworded", "num_bytes": 68285500, "num_examples": 156}, {"name": "humaneval.d.codegeex.0.2.reworded", "num_bytes": 67554723, "num_examples": 156}, {"name": "humaneval.d.codegen.0.2.reworded", "num_bytes": 69538065, "num_examples": 156}, {"name": "humaneval.d.cushman001.0.2.reworded", "num_bytes": 6543145, "num_examples": 156}, {"name": "humaneval.d.replit_code.0.2.reworded", "num_bytes": 19332975, "num_examples": 156}, {"name": "humaneval.go.bigcode_15b_1000m.0.2.reworded", "num_bytes": 75966586, "num_examples": 154}, {"name": "humaneval.go.bigcode_15b_200m.0.2.reworded", "num_bytes": 90496893, "num_examples": 154}, {"name": "humaneval.go.bigcode_15b_400m.0.2.reworded", "num_bytes": 80263304, "num_examples": 154}, {"name": "humaneval.go.bigcode_15b_600m.0.2.reworded", "num_bytes": 80653936, "num_examples": 154}, {"name": "humaneval.go.bigcode_15b_800m.0.2.reworded", "num_bytes": 79636433, "num_examples": 154}, {"name": "humaneval.go.codegeex.0.2.reworded", "num_bytes": 74466402, "num_examples": 154}, {"name": "humaneval.go.codegen.0.2.reworded", "num_bytes": 82565036, "num_examples": 154}, {"name": "humaneval.go.cushman001.0.2.reworded", "num_bytes": 7919252, "num_examples": 154}, {"name": "humaneval.go.replit_code.0.2.reworded", "num_bytes": 21740421, "num_examples": 154}, {"name": "humaneval.jl.bigcode_15b_1000m.0.2.reworded", "num_bytes": 64541752, "num_examples": 159}, {"name": "humaneval.jl.bigcode_15b_200m.0.2.reworded", "num_bytes": 64272523, "num_examples": 159}, {"name": "humaneval.jl.bigcode_15b_400m.0.2.reworded", "num_bytes": 84674386, "num_examples": 159}, {"name": "humaneval.jl.bigcode_15b_600m.0.2.reworded", "num_bytes": 83951098, "num_examples": 159}, {"name": "humaneval.jl.bigcode_15b_800m.0.2.reworded", "num_bytes": 71891875, "num_examples": 159}, {"name": "humaneval.jl.codegeex.0.2.reworded", "num_bytes": 49376484, "num_examples": 159}, {"name": "humaneval.jl.codegen.0.2.reworded", "num_bytes": 49686685, "num_examples": 159}, {"name": "humaneval.jl.cushman001.0.2.reworded", "num_bytes": 5594623, "num_examples": 159}, {"name": "humaneval.jl.replit_code.0.2.reworded", "num_bytes": 12432167, "num_examples": 159}, {"name": "humaneval.js.bigcode_15b_1000m.0.2.reworded", "num_bytes": 63930510, "num_examples": 161}, {"name": "humaneval.js.bigcode_15b_200m.0.2.reworded", "num_bytes": 71006276, "num_examples": 161}, {"name": "humaneval.js.bigcode_15b_400m.0.2.reworded", "num_bytes": 71141641, "num_examples": 161}, {"name": "humaneval.js.bigcode_15b_600m.0.2.reworded", "num_bytes": 66406645, "num_examples": 161}, {"name": "humaneval.js.bigcode_15b_800m.0.2.reworded", "num_bytes": 65906688, "num_examples": 161}, {"name": "humaneval.js.codegeex.0.2.reworded", "num_bytes": 68965171, "num_examples": 161}, {"name": "humaneval.js.codegen.0.2.reworded", "num_bytes": 71850674, "num_examples": 161}, {"name": "humaneval.js.cushman001.0.2.reworded", "num_bytes": 6756809, "num_examples": 161}, {"name": "humaneval.js.replit_code.0.2.reworded", "num_bytes": 20658701, "num_examples": 161}, {"name": "humaneval.lua.bigcode_15b_200m.0.2.reworded", "num_bytes": 56733662, "num_examples": 161}, {"name": "humaneval.lua.bigcode_15b_400m.0.2.reworded", "num_bytes": 57525953, "num_examples": 161}, {"name": "humaneval.lua.bigcode_15b_600m.0.2.reworded", "num_bytes": 53575875, "num_examples": 161}, {"name": "humaneval.lua.bigcode_15b_800m.0.2.reworded", "num_bytes": 54309789, "num_examples": 161}, {"name": "humaneval.lua.codegeex.0.2.reworded", "num_bytes": 53766400, "num_examples": 161}, {"name": "humaneval.lua.codegen.0.2.reworded", "num_bytes": 63642889, "num_examples": 161}, {"name": "humaneval.lua.cushman001.0.2.reworded", "num_bytes": 5726991, "num_examples": 161}, {"name": "humaneval.lua.replit_code.0.2.reworded", "num_bytes": 14458988, "num_examples": 161}, {"name": "humaneval.php.bigcode_15b_1000m.0.2.reworded", "num_bytes": 62087493, "num_examples": 161}, {"name": "humaneval.php.bigcode_15b_200m.0.2.reworded", "num_bytes": 67992787, "num_examples": 161}, {"name": "humaneval.php.bigcode_15b_400m.0.2.reworded", "num_bytes": 65415347, "num_examples": 161}, {"name": "humaneval.php.bigcode_15b_600m.0.2.reworded", "num_bytes": 64025429, "num_examples": 161}, {"name": "humaneval.php.bigcode_15b_800m.0.2.reworded", "num_bytes": 67914229, "num_examples": 161}, {"name": "humaneval.php.codegeex.0.2.reworded", "num_bytes": 63599818, "num_examples": 161}, {"name": "humaneval.php.codegen.0.2.reworded", "num_bytes": 71759630, "num_examples": 161}, {"name": "humaneval.php.cushman001.0.2.reworded", "num_bytes": 6680669, "num_examples": 161}, {"name": "humaneval.php.replit_code.0.2.reworded", "num_bytes": 18347062, "num_examples": 161}, {"name": "humaneval.pl.bigcode_15b_1000m.0.2.reworded", "num_bytes": 69839042, "num_examples": 161}, {"name": "humaneval.pl.bigcode_15b_200m.0.2.reworded", "num_bytes": 79671308, "num_examples": 161}, {"name": "humaneval.pl.bigcode_15b_400m.0.2.reworded", "num_bytes": 78788842, "num_examples": 161}, {"name": "humaneval.pl.bigcode_15b_600m.0.2.reworded", "num_bytes": 69916889, "num_examples": 161}, {"name": "humaneval.pl.bigcode_15b_800m.0.2.reworded", "num_bytes": 73552220, "num_examples": 161}, {"name": "humaneval.pl.codegeex.0.2.reworded", "num_bytes": 72617126, "num_examples": 161}, {"name": "humaneval.pl.codegen.0.2.reworded", "num_bytes": 74351768, "num_examples": 161}, {"name": "humaneval.pl.cushman001.0.2.reworded", "num_bytes": 7317844, "num_examples": 161}, {"name": "humaneval.pl.replit_code.0.2.reworded", "num_bytes": 23014112, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_200m.0.8.reworded", "num_bytes": 55679581, "num_examples": 161}, {"name": "humaneval.py.bigcode_15b_400m.0.8.reworded", "num_bytes": 49813429, "num_examples": 161}, {"name": "humaneval.py.replit_code.0.2.reworded", "num_bytes": 16222771, "num_examples": 161}, {"name": "humaneval.py.starcoder.0.8.reworded", "num_bytes": 50428866, "num_examples": 161}, {"name": "humaneval.py.starcoderprompted.0.1.reworded", "num_bytes": 53971758, "num_examples": 161}, {"name": "humaneval.rb.bigcode_15b_1000m.0.2.reworded", "num_bytes": 67446763, "num_examples": 161}, {"name": "humaneval.rb.bigcode_15b_200m.0.2.reworded", "num_bytes": 70571683, "num_examples": 161}, {"name": "humaneval.rb.bigcode_15b_400m.0.2.reworded", "num_bytes": 67565830, "num_examples": 161}, {"name": "humaneval.rb.bigcode_15b_600m.0.2.reworded", "num_bytes": 71419194, "num_examples": 161}, {"name": "humaneval.rb.bigcode_15b_800m.0.2.reworded", "num_bytes": 69995749, "num_examples": 161}, {"name": "humaneval.rb.codegeex.0.2.reworded", "num_bytes": 63388920, "num_examples": 161}, {"name": "humaneval.rb.codegen.0.2.reworded", "num_bytes": 68918022, "num_examples": 161}, {"name": "humaneval.rb.cushman001.0.2.reworded", "num_bytes": 7084615, "num_examples": 161}, {"name": "humaneval.rb.replit_code.0.2.reworded", "num_bytes": 17797810, "num_examples": 161}, {"name": "humaneval.r.codegeex.0.2.reworded", "num_bytes": 64172735, "num_examples": 161}, {"name": "humaneval.r.codegen.0.2.reworded", "num_bytes": 75777642, "num_examples": 161}, {"name": "humaneval.r.cushman001.0.2.reworded", "num_bytes": 6509329, "num_examples": 161}, {"name": "humaneval.rkt.bigcode_15b_1000m.0.2.reworded", "num_bytes": 71049799, "num_examples": 161}, {"name": "humaneval.rkt.bigcode_15b_200m.0.2.reworded", "num_bytes": 72642020, "num_examples": 161}, {"name": "humaneval.rkt.bigcode_15b_400m.0.2.reworded", "num_bytes": 73564249, "num_examples": 161}, {"name": "humaneval.rkt.bigcode_15b_600m.0.2.reworded", "num_bytes": 73730273, "num_examples": 161}, {"name": "humaneval.rkt.codegeex.0.2.reworded", "num_bytes": 70940774, "num_examples": 161}, {"name": "humaneval.rkt.codegen.0.2.reworded", "num_bytes": 90161741, "num_examples": 161}, {"name": "humaneval.rkt.cushman001.0.2.reworded", "num_bytes": 6030454, "num_examples": 161}, {"name": "humaneval.rkt.replit_code.0.2.reworded", "num_bytes": 18423402, "num_examples": 161}, {"name": "humaneval.r.replit_code.0.2.reworded", "num_bytes": 19677779, "num_examples": 161}, {"name": "humaneval.rs.bigcode_15b_1000m.0.2.reworded", "num_bytes": 59702550, "num_examples": 156}, {"name": "humaneval.rs.bigcode_15b_200m.0.2.reworded", "num_bytes": 67759741, "num_examples": 156}, {"name": "humaneval.rs.bigcode_15b_400m.0.2.reworded", "num_bytes": 68044357, "num_examples": 156}, {"name": "humaneval.rs.bigcode_15b_600m.0.2.reworded", "num_bytes": 54658037, "num_examples": 156}, {"name": "humaneval.rs.bigcode_15b_800m.0.2.reworded", "num_bytes": 62854949, "num_examples": 156}, {"name": "humaneval.rs.codegeex.0.2.reworded", "num_bytes": 67976569, "num_examples": 156}, {"name": "humaneval.rs.codegen.0.2.reworded", "num_bytes": 82843583, "num_examples": 156}, {"name": "humaneval.rs.cushman001.0.2.reworded", "num_bytes": 6310774, "num_examples": 156}, {"name": "humaneval.rs.replit_code.0.2.reworded", "num_bytes": 17624999, "num_examples": 156}, {"name": "humaneval.scala.bigcode_15b_1000m.0.2.reworded", "num_bytes": 84533532, "num_examples": 160}, {"name": "humaneval.scala.bigcode_15b_800m.0.2.reworded", "num_bytes": 87023720, "num_examples": 160}, {"name": "humaneval.scala.codegeex.0.2.reworded", "num_bytes": 79619828, "num_examples": 160}, {"name": "humaneval.scala.codegen.0.2.reworded", "num_bytes": 128885303, "num_examples": 160}, {"name": "humaneval.scala.cushman001.0.2.reworded", "num_bytes": 8500865, "num_examples": 160}, {"name": "humaneval.scala.replit_code.0.2.reworded", "num_bytes": 22458222, "num_examples": 160}, {"name": "humaneval.sh.bigcode_15b_1000m.0.2.reworded", "num_bytes": 62768941, "num_examples": 158}, {"name": "humaneval.sh.bigcode_15b_200m.0.2.reworded", "num_bytes": 75630478, "num_examples": 158}, {"name": "humaneval.sh.bigcode_15b_400m.0.2.reworded", "num_bytes": 77050658, "num_examples": 158}, {"name": "humaneval.sh.bigcode_15b_600m.0.2.reworded", "num_bytes": 65325746, "num_examples": 158}, {"name": "humaneval.swift.bigcode_15b_600m.0.2.reworded", "num_bytes": 70424335, "num_examples": 158}, {"name": "humaneval.scala.bigcode_15b_200m.0.2.reworded", "num_bytes": 89054581, "num_examples": 160}, {"name": "humaneval.scala.bigcode_15b_400m.0.2.reworded", "num_bytes": 83343360, "num_examples": 160}, {"name": "humaneval.scala.bigcode_15b_600m.0.2.reworded", "num_bytes": 89752223, "num_examples": 160}, {"name": "humaneval.sh.bigcode_15b_800m.0.2.reworded", "num_bytes": 66811937, "num_examples": 158}, {"name": "humaneval.sh.codegeex.0.2.reworded", "num_bytes": 65196768, "num_examples": 158}, {"name": "humaneval.sh.codegen.0.2.reworded", "num_bytes": 99280481, "num_examples": 158}, {"name": "humaneval.sh.cushman001.0.2.reworded", "num_bytes": 6237965, "num_examples": 158}, {"name": "humaneval.sh.replit_code.0.2.reworded", "num_bytes": 18134838, "num_examples": 158}, {"name": "humaneval.swift.bigcode_15b_1000m.0.2.reworded", "num_bytes": 68129948, "num_examples": 158}, {"name": "humaneval.swift.bigcode_15b_200m.0.2.reworded", "num_bytes": 76924134, "num_examples": 158}, {"name": "humaneval.swift.bigcode_15b_400m.0.2.reworded", "num_bytes": 72042977, "num_examples": 158}, {"name": "humaneval.swift.bigcode_15b_800m.0.2.reworded", "num_bytes": 70027106, "num_examples": 158}, {"name": "humaneval.swift.codegeex.0.2.reworded", "num_bytes": 73605273, "num_examples": 158}, {"name": "humaneval.swift.codegen.0.2.reworded", "num_bytes": 76081675, "num_examples": 158}, {"name": "humaneval.swift.cushman001.0.2.reworded", "num_bytes": 6766506, "num_examples": 158}, {"name": "humaneval.swift.replit_code.0.2.reworded", "num_bytes": 21605861, "num_examples": 158}, {"name": "humaneval.ts.bigcode_15b_1000m.0.2.reworded", "num_bytes": 61005831, "num_examples": 159}, {"name": "humaneval.ts.bigcode_15b_200m.0.2.reworded", "num_bytes": 68875546, "num_examples": 159}, {"name": "humaneval.ts.bigcode_15b_400m.0.2.reworded", "num_bytes": 62805583, "num_examples": 159}, {"name": "humaneval.ts.bigcode_15b_600m.0.2.reworded", "num_bytes": 53733690, "num_examples": 159}, {"name": "humaneval.ts.bigcode_15b_800m.0.2.reworded", "num_bytes": 64371975, "num_examples": 159}, {"name": "humaneval.ts.codegeex.0.2.reworded", "num_bytes": 58487751, "num_examples": 159}, {"name": "humaneval.ts.codegen.0.2.reworded", "num_bytes": 69981611, "num_examples": 159}, {"name": "humaneval.ts.cushman001.0.2.reworded", "num_bytes": 6768589, "num_examples": 159}, {"name": "humaneval.ts.replit_code.0.2.reworded", "num_bytes": 18208741, "num_examples": 159}, {"name": "mbpp.py.codegeex.0.2.reworded", "num_bytes": 86906502, "num_examples": 397}, {"name": "mbpp.py.codegen.0.2.reworded", "num_bytes": 92562493, "num_examples": 397}, {"name": "mbpp.py.cushman001.0.2.reworded", "num_bytes": 7629346, "num_examples": 397}, {"name": "mbpp.py.pystarcoder2.0.1.reworded", "num_bytes": 142442817, "num_examples": 397}, {"name": "mbpp.py.Salesforce_codegen_16B_mono.0.2.reworded", "num_bytes": 86067040, "num_examples": 397}, {"name": "humaneval.matlab.keep.gpt_35_turbo_0301.0.2.reworded", "num_bytes": 4837906, "num_examples": 161}], "download_size": 1286123094, "dataset_size": 14746371358}, "tags": ["code"]}
TAGS
#code #region-us 


# Raw Data from MultiPL-E

Uploads are a work in progress. If you are interested in a split that is not yet available, please contact a.guha@URL.

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the 
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use the following script
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named 'base_dataset'.'language'.'model'.'temperature'.'variation'

- 'base_dataset' is either 'humaneval' or 'mbpp'.
- 'language' is the file extension of the programming language. E.g., 'py' for Python or 'sh' for Bash.
- 'model' is the name of the model. Some model names used by MultiPL-E:
   - 'bigcode_15b_1000m':  StarCoderBase
   - 'bigcode_15b_200m', 'bigcode_15b_400m', 'bigcode_15b_600m', 'bigcode_15b_800m': the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
  - 'pystarcoder2': StarCoder
   - there are others, just have a look :)
- 'temperature': this is either 0.2 or 0.8
- 'variation': the variation of the MultiPL-E prompt to use, which should be 'reworded'. MultiPL-E has several prompt ablations and the 'reworded' prompts are the canonical variation.
# Raw Data from MultiPL-E

Uploads are a work in progress. If you are interested in a split that is not yet available, please contact a.guha@URL.

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the 
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use the following script
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named 'base_dataset'.'language'.'model'.'temperature'.'variation'

- 'base_dataset' is either 'humaneval' or 'mbpp'.
- 'language' is the file extension of the programming language. E.g., 'py' for Python or 'sh' for Bash.
- 'model' is the name of the model. Some model names used by MultiPL-E:
   - 'bigcode_15b_1000m':  StarCoderBase
   - 'bigcode_15b_200m', 'bigcode_15b_400m', 'bigcode_15b_600m', 'bigcode_15b_800m': the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
  - 'pystarcoder2': StarCoder
   - there are others, just have a look :)
- 'temperature': this is either 0.2 or 0.8
- 'variation': the variation of the MultiPL-E prompt to use, which should be 'reworded'. MultiPL-E has several prompt ablations and the 'reworded' prompts are the canonical variation.
TAGS
#code #region-us 
 # Raw Data from MultiPL-E

Uploads are a work in progress. If you are interested in a split that is not yet available, please contact a.guha@URL.

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the 
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use the following script
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named 'base_dataset'.'language'.'model'.'temperature'.'variation'

- 'base_dataset' is either 'humaneval' or 'mbpp'.
- 'language' is the file extension of the programming language. E.g., 'py' for Python or 'sh' for Bash.
- 'model' is the name of the model. Some model names used by MultiPL-E:
   - 'bigcode_15b_1000m':  StarCoderBase
   - 'bigcode_15b_200m', 'bigcode_15b_400m', 'bigcode_15b_600m', 'bigcode_15b_800m': the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
  - 'pystarcoder2': StarCoder
   - there are others, just have a look :)
- 'temperature': this is either 0.2 or 0.8
- 'variation': the variation of the MultiPL-E prompt to use, which should be 'reworded'. MultiPL-E has several prompt ablations and the 'reworded' prompts are the canonical variation.