| ### Single Pass | |
| ``` | |
| hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 32 | |
| | Task |Version| Metric |Value | |Stderr| | |
| |-------------|------:|--------|-----:|---|-----:| | |
| |arc_challenge| 0|acc |0.2355|_ |0.0124| | |
| | | |acc_norm|0.2671|_ |0.0129| | |
| |arc_easy | 0|acc |0.4444|_ |0.0102| | |
| | | |acc_norm|0.4276|_ |0.0102| | |
| |boolq | 1|acc |0.5358|_ |0.0087| | |
| |hellaswag | 0|acc |0.3784|_ |0.0048| | |
| | | |acc_norm|0.5034|_ |0.0050| | |
| |openbookqa | 0|acc |0.1580|_ |0.0163| | |
| | | |acc_norm|0.2840|_ |0.0202| | |
| |piqa | 0|acc |0.6518|_ |0.0111| | |
| | | |acc_norm|0.6464|_ |0.0112| | |
| |winogrande | 0|acc |0.5422|_ |0.0140| | |
| ``` | |
| ### 16x Passees | |
| ``` | |
| hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 64 | |
| | Task |Version| Metric |Value | |Stderr| | |
| |-------------|------:|--------|-----:|---|-----:| | |
| |arc_challenge| 0|acc |0.2466|_ |0.0126| | |
| | | |acc_norm|0.2824|_ |0.0132| | |
| |arc_easy | 0|acc |0.3649|_ |0.0099| | |
| | | |acc_norm|0.3582|_ |0.0098| | |
| |boolq | 1|acc |0.6214|_ |0.0085| | |
| |hellaswag | 0|acc |0.3085|_ |0.0046| | |
| | | |acc_norm|0.3614|_ |0.0048| | |
| |openbookqa | 0|acc |0.1900|_ |0.0176| | |
| | | |acc_norm|0.2800|_ |0.0201| | |
| |piqa | 0|acc |0.5702|_ |0.0116| | |
| | | |acc_norm|0.5729|_ |0.0115| | |
| |winogrande | 0|acc |0.5399|_ |0.0140| | |
| ``` | |