peft pytorch safetensors llama generated_from_trainer en dataset:Open-Orca/SlimOrca-Dedup dataset:gardner/glaive-function-calling-v2-sharegpt base_model:TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T license:apache-2.0 model-index region:us
---
license: apache-2.0
library_name: peft
tags:
- generated_from_trainer
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model-index:
- name: TinyLlama-1.1B-SlimOrca-Function-Calling-3T
  results:
    - task:
        type: text-generation
      metrics:
        - name: Average
          type: Average
          value: 37.38
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: ARC
          type: ARC
          value: 36.09
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: HellaSwag
          type: HellaSwag
          value: 59.66
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: MMLU
          type: MMLU
          value: 28.21
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: TruthfulQA
          type: TruthfulQA
          value: 36.74
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: Winograde
          type: Winograde
          value: 59.12
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

    - task:
        type: text-generation
      metrics:
        - name: GSM8K
          type: GSM8K
          value: 4.47
      source:
        name: Open LLM Leaderboard
        url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
datasets:
  - Open-Orca/SlimOrca-Dedup
  - gardner/glaive-function-calling-v2-sharegpt
language: en
---

# TinyLlama-1.1B-SlimOrca-Function-Calling-3T

![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/638581711769b7c4b10f0523/KMYjgnAE5D41YJWx_mPT8.jpeg)

This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) on the [SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca-Dedup) and [glaive-function-calling-v2](https://huggingface.co/datasets/gardner/glaive-function-calling-v2-sharegpt) datasets.

# Evaluation

It achieves the following results on the evaluation set:
- Loss: 0.7403

Please see the `scripts/llm-eval.py` to recreate the evaluation results from the test split as published here: [gardner/tinyllama-function-calling-eval](https://huggingface.co/datasets/gardner/tinyllama-function-calling-eval). The model responds with function calling when expected and refuses when it doesn't have access to tools. In the linked dataset, `result1` is generated by this model and `result2` is from the test dataset.

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.3.0`
```yaml
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: Open-Orca/SlimOrca-Dedup
    type: sharegpt
    conversation: chatml

  - path: gardner/glaive-function-calling-v2-sharegpt
    type: sharegpt
    conversation: chatml

dataset_prepared_path: ./.prepared-datasets/glaive-function-calling-v2-sharegpt
val_set_size: 0.05
output_dir: ./tinyllama/function-calling/chatml

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

```

</details><br>



## Training procedure

The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: True
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 4

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.2492        | 0.0   | 1     | 1.2363          |
| 0.7621        | 0.25  | 1896  | 0.8096          |
| 0.757         | 0.5   | 3792  | 0.7852          |
| 0.6424        | 0.75  | 5688  | 0.7717          |
| 0.5944        | 1.04  | 7584  | 0.7625          |
| 0.73          | 1.29  | 9480  | 0.7585          |
| 0.6781        | 1.54  | 11376 | 0.7521          |
| 0.829         | 1.79  | 13272 | 0.7471          |
| 0.6964        | 2.08  | 15168 | 0.7467          |
| 0.6652        | 2.33  | 17064 | 0.7453          |
| 0.7645        | 2.58  | 18960 | 0.7420          |
| 0.5702        | 2.83  | 20856 | 0.7392          |
| 0.7049        | 3.12  | 22752 | 0.7418          |
| 0.6087        | 3.37  | 24648 | 0.7412          |
| 0.6064        | 3.62  | 26544 | 0.7405          |
| 0.7125        | 3.87  | 28440 | 0.7403          |


### Framework versions

- PEFT 0.7.0
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0