
# Background

Textbook-Yawp-Openstax-merged datasets, is now applied the the Intel/Neural-Chat-7b-v3.1. The training peramters are below.

| Key | Value |
|-|-|
| output_dir | "./results" |
| num_train_epochs | 15 |
| per_device_train_batch_size | 32 |
| gradient_accumulation_steps | 4 |  
| optim | "paged_adamw_8bit" |
| save_steps | 500 |
| logging_steps | 5 |
| learning_rate | 1e-4 |
| weight_decay | 0.001 |
| fp16 | False |
| bf16 | False |
| max_grad_norm | 0.3 |
| max_steps | -1 |
| warmup_ratio | 0.3 |
| group_by_length | True |
| lr_scheduler_type | "linear" |
| report_to | "wandb" |


| Key | Value |
|-|-|
| train/epoch | ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███ |
| train/global_step | ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███ |  
| train/learning_rate | ▁▂▂▃▄▄▅▅▆▇▇███▇▇▇▇▆▆▆▆▆▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▁▁ |
| train/loss | █▇▆▆▆▅▄▄▃▂▃▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ |
| train/total_flos | ▁ |
| train/train_loss | ▁ |
| train/train_runtime | ▁ |  
| train/train_samples_per_second | ▁ |
| train/train_steps_per_second | ▁ |
| train/epoch | 15.0 |
| train/global_step | 270 |
| train/learning_rate | 0.0 |
| train/loss | 0.4845 |
| train/total_flos | 3.550242517150925e+16 |  
| train/train_loss | 0.89085 |
| train/train_runtime | 2501.3387 | 
| train/train_samples_per_second | 13.805 |
| train/train_steps_per_second | 0.108 |



## Dataset
https://huggingface.co/datasets/ambrosfitz/textbook-openstax-yawp-merge

## Models
https://huggingface.co/ambrosfitz/tinyllama-history-chat-v1.1
{"license": "apache-2.0"}
transformers safetensors mistral text-generation license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us