|
--- |
|
library_name: transformers |
|
license: llama3.1 |
|
base_model: meta-llama/Llama-3.1-8B-Instruct |
|
tags: |
|
- axolotl |
|
- generated_from_trainer |
|
model-index: |
|
- name: L3.1-Pneuma-8B |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl) |
|
<details><summary>See axolotl config</summary> |
|
|
|
axolotl version: `0.5.0` |
|
```yaml |
|
base_model: meta-llama/Llama-3.1-8B-Instruct |
|
|
|
load_in_8bit: false |
|
load_in_4bit: false |
|
strict: false |
|
|
|
load_in_8bit: false |
|
load_in_4bit: false |
|
strict: false |
|
|
|
datasets: |
|
- path: Sandevistan_cleaned.jsonl |
|
type: customllama3_stan |
|
dataset_prepared_path: last_run_prepared |
|
val_set_size: 0.05 |
|
output_dir: ./outputs/out |
|
|
|
fix_untrained_tokens: true |
|
|
|
sequence_len: 4096 |
|
sample_packing: true |
|
pad_to_sequence_len: true |
|
|
|
wandb_project: Pneuma |
|
wandb_entity: |
|
wandb_watch: |
|
wandb_name: |
|
wandb_log_model: |
|
|
|
gradient_accumulation_steps: 16 |
|
micro_batch_size: 8 |
|
num_epochs: 2 |
|
optimizer: paged_adamw_8bit |
|
lr_scheduler: cosine |
|
learning_rate: 0.0000078 |
|
max_grad_norm: 1 |
|
|
|
train_on_inputs: false |
|
group_by_length: false |
|
bf16: auto |
|
fp16: |
|
tf32: false |
|
|
|
gradient_checkpointing: unsloth |
|
early_stopping_patience: |
|
resume_from_checkpoint: |
|
logging_steps: 1 |
|
xformers_attention: |
|
flash_attention: true |
|
eval_sample_packing: false |
|
|
|
plugins: |
|
- axolotl.integrations.liger.LigerPlugin |
|
liger_rope: true |
|
liger_rms_norm: true |
|
liger_swiglu: true |
|
liger_fused_linear_cross_entropy: true |
|
|
|
hub_model_id: Replete-AI/L3.1-Pneuma-8B |
|
hub_strategy: every_save |
|
|
|
warmup_steps: 0 |
|
evals_per_epoch: 3 |
|
eval_table_size: |
|
saves_per_epoch: 3 |
|
debug: |
|
deepspeed: |
|
weight_decay: 0.1 |
|
fsdp: |
|
fsdp_config: |
|
special_tokens: |
|
bos_token: "<|begin_of_text|>" |
|
eos_token: "<|end_of_text|>" |
|
pad_token: "<|end_of_text|>" |
|
tokens: |
|
|
|
``` |
|
|
|
</details><br> |
|
|
|
# L3.1-Pneuma-8B |
|
|
|
This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on the [Sandevistan](https://huggingface.co/datasets/Replete-AI/Sandevistan) dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 2.4357 |
|
|
|
## Model description |
|
|
|
This model is designed to challenge common paradigms in training Large Language Models, giving them a focus on user experience over profitability. These are highly experimental, and need preference training in order to increase their effectiveness. It seems to have retained a large amount of the biases that we were trying to eliminate from the corporate instruct models. |
|
|
|
## Intended uses & limitations |
|
|
|
Chatting, conversation, and assistance in small downstream tasks. |
|
|
|
Large Language Models work incredibly differently from humans, so while we are capable of training and rewarding them to act just like us in many ways, you should treat it as a simulation and use the Socratic method when engaging with them. You, as an end-user should always remain in control of your own thoughts and decisions, and use AI as a way to improve yourself rather than becoming dependent on it. |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 7.8e-06 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 16 |
|
- total_train_batch_size: 128 |
|
- optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
|
- lr_scheduler_type: cosine |
|
- num_epochs: 2 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|:-------------:|:------:|:----:|:---------------:| |
|
| 1.0731 | 0.0023 | 1 | 2.7679 | |
|
| 0.6458 | 0.3338 | 143 | 2.4576 | |
|
| 0.6504 | 0.6675 | 286 | 2.4407 | |
|
| 1.112 | 1.0019 | 429 | 2.4358 | |
|
| 0.6014 | 1.3357 | 572 | 2.4358 | |
|
| 0.6194 | 1.6694 | 715 | 2.4357 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.46.1 |
|
- Pytorch 2.3.1+cu121 |
|
- Datasets 3.0.1 |
|
- Tokenizers 0.20.3 |
|
|
