collapse_gemma-2-27b_hs2_replace_iter5_sftsd0

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3588
• Num Input Tokens Seen: 2363424

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.1282	0
4.0483	0.1269	5	1.1138	306092
3.8882	0.2538	10	1.2045	612464
3.7347	0.3807	15	1.2446	916160
3.5411	0.5075	20	1.3019	1219264
3.4466	0.6344	25	1.3385	1521040
3.351	0.7613	30	1.3513	1822060
3.3188	0.8882	35	1.3558	2123260

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1