See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: unsloth/SmolLM-135M
batch_size: 8
bf16: true
chat_template: tokenizer_default_fallback_alpaca
datasets:
- data_files:
  - 2144ebae5d455e44_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/2144ebae5d455e44_train_data.json
  type:
    field_instruction: prompt
    field_output: original_response
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
early_stopping_patience: 3
eval_steps: 50
flash_attention: true
gpu_memory_limit: 80GiB
gradient_checkpointing: true
group_by_length: true
hub_model_id: willtensora/ecf2f7d5-ff6e-46d0-baf3-23d54fa38ba2
hub_strategy: checkpoint
learning_rate: 0.0002
logging_steps: 10
lora_alpha: 256
lora_dropout: 0.1
lora_r: 128
lora_target_linear: true
lr_scheduler: cosine
micro_batch_size: 1
model_type: AutoModelForCausalLM
num_epochs: 100
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resize_token_embeddings_to_32x: false
sample_packing: false
save_steps: 50
sequence_len: 2048
tokenizer_type: GPT2TokenizerFast
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.1
wandb_entity: ''
wandb_mode: online
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: default
warmup_ratio: 0.05
xformers_attention: true

ecf2f7d5-ff6e-46d0-baf3-23d54fa38ba2

This model is a fine-tuned version of unsloth/SmolLM-135M on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.4037

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 517
• num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0012	1	1.6175
1.6014	0.0603	50	1.5669
1.417	0.1206	100	1.5081
1.4265	0.1809	150	1.4801
1.4553	0.2413	200	1.4600
1.3311	0.3016	250	1.4486
1.3884	0.3619	300	1.4392
1.3114	0.4222	350	1.4347
1.4563	0.4825	400	1.4244
1.3847	0.5428	450	1.4214
1.4362	0.6031	500	1.4189
1.4121	0.6634	550	1.4157
1.286	0.7238	600	1.4085
1.3552	0.7841	650	1.4072
1.3352	0.8444	700	1.4045
1.3053	0.9047	750	1.4026
1.3281	0.9650	800	1.3991
1.2612	1.0253	850	1.3985
1.3015	1.0856	900	1.3953
1.2802	1.1460	950	1.3945
1.3213	1.2063	1000	1.3938
1.2842	1.2666	1050	1.3919
1.2355	1.3269	1100	1.3920
1.3006	1.3872	1150	1.3937
1.2391	1.4475	1200	1.4037

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1