File size: 1,895 Bytes
cc30631 8a8aa43 11f90ce cc30631 32fd5c7 cc30631 32fd5c7 cc30631 32fd5c7 cc30631 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
### Single Pass
```
hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 32
| Task |Version| Metric |Value | |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge| 0|acc |0.2355|_ |0.0124|
| | |acc_norm|0.2671|_ |0.0129|
|arc_easy | 0|acc |0.4444|_ |0.0102|
| | |acc_norm|0.4276|_ |0.0102|
|boolq | 1|acc |0.5358|_ |0.0087|
|hellaswag | 0|acc |0.3784|_ |0.0048|
| | |acc_norm|0.5034|_ |0.0050|
|openbookqa | 0|acc |0.1580|_ |0.0163|
| | |acc_norm|0.2840|_ |0.0202|
|piqa | 0|acc |0.6518|_ |0.0111|
| | |acc_norm|0.6464|_ |0.0112|
|winogrande | 0|acc |0.5422|_ |0.0140|
```
### 16x Passees
```
hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 64
| Task |Version| Metric |Value | |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge| 0|acc |0.2466|_ |0.0126|
| | |acc_norm|0.2824|_ |0.0132|
|arc_easy | 0|acc |0.3649|_ |0.0099|
| | |acc_norm|0.3582|_ |0.0098|
|boolq | 1|acc |0.6214|_ |0.0085|
|hellaswag | 0|acc |0.3085|_ |0.0046|
| | |acc_norm|0.3614|_ |0.0048|
|openbookqa | 0|acc |0.1900|_ |0.0176|
| | |acc_norm|0.2800|_ |0.0201|
|piqa | 0|acc |0.5702|_ |0.0116|
| | |acc_norm|0.5729|_ |0.0115|
|winogrande | 0|acc |0.5399|_ |0.0140|
```
|