peft safetensors llama axolotl generated_from_trainer base_model:codellama/CodeLlama-7b-hf license:llama2 4-bit bitsandbytes region:us
---
license: llama2
library_name: peft
tags:
- axolotl
- generated_from_trainer
base_model: codellama/CodeLlama-7b-hf
model-index:
- name: EvilCodeLlama-7b
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.3.0`
```yaml
base_model: codellama/CodeLlama-7b-hf
base_model_config: codellama/CodeLlama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
hub_model_id: EvilCodeLlama-7b

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: dhuynh95/Magicoder-Evol-Instruct-110K-Filtered_0.35
    type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./qlora-out-evil-codellama

adapter: qlora
lora_model_dir:

eval_sample_packing: false
sequence_len: 2048
sample_packing: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: axolotl
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 16
num_epochs: 10
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: true
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
eval_steps: 0.01
save_strategy: epoch
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
```

</details><br>

# EvilCodeLlama-7b

This model is a fine-tuned version of [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7929

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.2543        | 0.04  | 1    | 1.2447          |
| 1.2677        | 0.12  | 3    | 1.2446          |
| 1.2572        | 0.24  | 6    | 1.2443          |
| 1.2602        | 0.37  | 9    | 1.2432          |
| 1.2573        | 0.49  | 12   | 1.2403          |
| 1.2811        | 0.61  | 15   | 1.2342          |
| 1.2584        | 0.73  | 18   | 1.2217          |
| 1.2152        | 0.86  | 21   | 1.2005          |
| 1.1592        | 0.98  | 24   | 1.1695          |
| 1.1512        | 1.07  | 27   | 1.1345          |
| 1.1191        | 1.19  | 30   | 1.0970          |
| 1.1111        | 1.32  | 33   | 1.0543          |
| 1.0362        | 1.44  | 36   | 1.0160          |
| 1.0386        | 1.56  | 39   | 0.9879          |
| 1.0637        | 1.68  | 42   | 0.9549          |
| 1.0109        | 1.81  | 45   | 0.9377          |
| 0.9416        | 1.93  | 48   | 0.9258          |
| 0.8851        | 2.03  | 51   | 0.9164          |
| 0.9027        | 2.15  | 54   | 0.9085          |
| 0.8959        | 2.28  | 57   | 0.9018          |
| 0.9168        | 2.4   | 60   | 0.8956          |
| 0.9386        | 2.52  | 63   | 0.8896          |
| 0.9762        | 2.64  | 66   | 0.8832          |
| 0.9118        | 2.77  | 69   | 0.8768          |
| 0.9055        | 2.89  | 72   | 0.8714          |
| 0.8617        | 3.01  | 75   | 0.8660          |
| 0.9085        | 3.11  | 78   | 0.8604          |
| 0.8531        | 3.23  | 81   | 0.8547          |
| 0.8725        | 3.36  | 84   | 0.8486          |
| 0.8845        | 3.48  | 87   | 0.8424          |
| 0.8812        | 3.6   | 90   | 0.8381          |
| 0.865         | 3.72  | 93   | 0.8351          |
| 0.8312        | 3.85  | 96   | 0.8311          |
| 0.8766        | 3.97  | 99   | 0.8280          |
| 0.842         | 4.07  | 102  | 0.8249          |
| 0.8377        | 4.19  | 105  | 0.8222          |
| 0.8661        | 4.32  | 108  | 0.8195          |
| 0.8505        | 4.44  | 111  | 0.8171          |
| 0.8509        | 4.56  | 114  | 0.8140          |
| 0.8823        | 4.68  | 117  | 0.8111          |
| 0.8246        | 4.81  | 120  | 0.8091          |
| 0.8116        | 4.93  | 123  | 0.8073          |
| 0.7993        | 5.03  | 126  | 0.8054          |
| 0.8277        | 5.15  | 129  | 0.8048          |
| 0.8533        | 5.28  | 132  | 0.8030          |
| 0.7887        | 5.4   | 135  | 0.8015          |
| 0.8189        | 5.52  | 138  | 0.8005          |
| 0.8148        | 5.64  | 141  | 0.7993          |
| 0.8376        | 5.77  | 144  | 0.7977          |
| 0.8142        | 5.89  | 147  | 0.7968          |
| 0.8074        | 6.01  | 150  | 0.7961          |
| 0.8122        | 6.11  | 153  | 0.7970          |
| 0.7753        | 6.23  | 156  | 0.7963          |
| 0.8477        | 6.36  | 159  | 0.7958          |
| 0.7977        | 6.48  | 162  | 0.7947          |
| 0.7653        | 6.6   | 165  | 0.7944          |
| 0.8358        | 6.72  | 168  | 0.7930          |
| 0.7445        | 6.85  | 171  | 0.7926          |
| 0.808         | 6.97  | 174  | 0.7922          |
| 0.7799        | 7.07  | 177  | 0.7916          |
| 0.7593        | 7.19  | 180  | 0.7933          |
| 0.8275        | 7.32  | 183  | 0.7930          |
| 0.7599        | 7.44  | 186  | 0.7925          |
| 0.7734        | 7.56  | 189  | 0.7928          |
| 0.7886        | 7.68  | 192  | 0.7927          |
| 0.8066        | 7.81  | 195  | 0.7919          |
| 0.7778        | 7.93  | 198  | 0.7916          |
| 0.7839        | 8.03  | 201  | 0.7918          |
| 0.7942        | 8.15  | 204  | 0.7927          |
| 0.7457        | 8.28  | 207  | 0.7930          |
| 0.7525        | 8.4   | 210  | 0.7928          |
| 0.7768        | 8.52  | 213  | 0.7926          |
| 0.7469        | 8.64  | 216  | 0.7928          |
| 0.7777        | 8.77  | 219  | 0.7929          |
| 0.7694        | 8.89  | 222  | 0.7928          |
| 0.7639        | 9.01  | 225  | 0.7927          |
| 0.7556        | 9.11  | 228  | 0.7927          |
| 0.7098        | 9.23  | 231  | 0.7927          |
| 0.7537        | 9.36  | 234  | 0.7928          |
| 0.7721        | 9.48  | 237  | 0.7926          |
| 0.7642        | 9.6   | 240  | 0.7929          |


### Framework versions

- PEFT 0.7.2.dev0
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.16.1
- Tokenizers 0.15.0