Doge-CheckPoint
Collection
A series of checkPoint weights that can continue training on new datasets without spikes of the training.
•
3 items
•
Updated
•
1
Doge 60M checkpoint
Doge uses wsd_scheduler as the training scheduler, which divides the learning rate into three stages: warmup, stable, and decay. It allows us to continue training on any new dataset from any checkpoint in the stable stage without spikes of the training.
Here are the initial learning rates required to continue training at each checkpoint:
| Model | Learning Rate | Schedule | Warmup Steps | Stable Steps |
|---|---|---|---|---|
| Doge-20M | 8e-3 | wsd_scheduler | 800 | 6400 |
| Doge-60M | 6e-3 | wsd_scheduler | 1600 | 12800 |
| Doge-160M | 4e-3 | wsd_scheduler | 2400 | 19200 |
| Doge-320M | 2e-3 | wsd_scheduler | 3200 | 25600 |
- Downloads last month
- 20
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API does not yet support model repos that contain custom code.