llama_8b_lima_43

This model is a fine-tuned version of meta-llama/Llama-3.1-8B on the open_webui_dataset dataset. It achieves the following results on the evaluation set:

• Loss: 0.9357

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.5e-06
• train_batch_size: 3
• eval_batch_size: 2
• seed: 66
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 5
• total_train_batch_size: 30
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: polynomial
• lr_scheduler_warmup_steps: 50
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.0422	0.0750	80	1.0044
0.9127	0.1500	160	0.9607
0.8605	0.2251	240	0.9440
0.9068	0.3001	320	0.9343
0.9147	0.3751	400	0.9293
1.0192	0.4501	480	0.9250
0.8303	0.5251	560	0.9192
1.0284	0.6002	640	0.9292
0.9183	0.6752	720	0.9389
0.9897	0.7502	800	0.9337
1.0209	0.8252	880	0.9390
0.9118	0.9002	960	0.9374
0.9077	0.9752	1040	0.9356

Framework versions

• Transformers 4.46.1
• Pytorch 2.4.0
• Datasets 3.1.0
• Tokenizers 0.20.3