O0506TEST2

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -6.1946

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.5234	0.09	10	-1.5732
-1.6148	0.18	20	-1.6455
-1.6632	0.27	30	-1.6791
-1.7073	0.36	40	-1.7412
-1.771	0.45	50	-1.8170
-1.8527	0.54	60	-1.9152
-1.9647	0.63	70	-2.0400
-2.1058	0.73	80	-2.1959
-2.2879	0.82	90	-2.4163
-2.5127	0.91	100	-2.6733
-2.7623	1.0	110	-2.9418
1.9772	1.09	120	3.2375
-0.6558	1.18	130	-2.4587
-2.9863	1.27	140	-3.3977
-3.5664	1.36	150	-3.8086
-3.9393	1.45	160	-4.1914
-4.292	1.54	170	-4.5058
-4.5875	1.63	180	-4.8033
-4.8602	1.72	190	-5.0741
-5.1734	1.81	200	-5.2974
-5.3674	1.9	210	-5.4358
-5.4772	1.99	220	-5.6335
-5.7073	2.08	230	-5.7721
-5.7794	2.18	240	-5.8909
-5.7909	2.27	250	-5.9810
-5.9785	2.36	260	-6.0502
-6.0635	2.45	270	-6.0989
-5.9563	2.54	280	-6.1409
-6.1511	2.63	290	-6.1660
-6.1327	2.72	300	-6.1826
-6.1021	2.81	310	-6.1911
-6.0999	2.9	320	-6.1941
-6.1908	2.99	330	-6.1946

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0