|
--- |
|
datasets: |
|
- Marcus2112/minipile_density-proportioned |
|
language: |
|
- en |
|
base_model: |
|
- EleutherAI/pythia-160m-deduped |
|
--- |
|
|
|
| Benchmark | Measure | | 160M MiniPile | 160M Density | |
|
| ---------------- | ---------- | --- | --------------------------- | -------------------------------- | |
|
| ARC-Challenge | acc | ↑ | **0.2125 ± 0.0120** | 0.1920 ± 0.0115 | |
|
| MMLU | acc | ↑ | **0.2699 ± 0.0037** | 0.2295 ± 0.0035 | |
|
| HellaSwag | acc | ↑ | 0.2560 ± 0.0044 | **0.2604 ± 0.0044** | |
|
| WinoGrande | acc | ↑ | 0.4720 ± 0.0140 | **0.5201 ± 0.0140** | |
|
| Lambada (OpenAI) | acc | ↑ | 0.0000 ± 0.0000 | 0.0000 ± 0.0000 | |
|
| Lambada (OpenAI) | perplexity | ↓ | 3033175.2693 ± 288926.5827 | **2099002.0912 ± 170652.6222** | |
|
| Lambada (Std) | acc | ↑ | 0.0000 ± 0.0000 | 0.0000 ± 0.0000 | |
|
| Lambada (Std) | perplexity | ↓ | 27067951.3460 ± 2710040.191 | **13347273.6076 ± 1997894.6360** | |
|
| BLiMP | acc | ↑ | 0.5194 ± 0.0018 | **0.5501 ± 0.0017** | |
