checkpoint @ 31,404,851,200 tokens trained, settings used:

```
batch_size: 512
context_length: 1024
learning_rate: 2e-4
schedule: cosine with 10% warmup from 0 to 2e-4, cooldown to 0
```

requirements:
```
pytorch
transformers
einops
```
{"language": ["en"], "license": "apache-2.0", "datasets": ["cerebras/SlimPajama-627B"]}
transformers pytorch gpt2a text-generation custom_code en dataset:cerebras/SlimPajama-627B license:apache-2.0 autotrain_compatible region:us