Hugging Face's logo

thrunlab
/
relu_llama_7b_hf2_refined_web_relu_2024-03-27

Text Generation
Transformers
Safetensors
sparse_llama
Generated from Trainer
custom_code
Model card Files Community
lukeleeai commited on
Commit
8cc6c0e
·
verified ·
1 Parent(s): 3c14765

Training in progress, step 1250

Browse files
Files changed (7) hide show
  1. README.md +16 -6
  2. adapter_config.json +3 -3
  3. adapter_model.safetensors +1 -1
  4. model-00001-of-00003.safetensors +1 -1
  5. model-00002-of-00003.safetensors +1 -1
  6. model-00003-of-00003.safetensors +1 -1
  7. training_args.bin +1 -1
README.md CHANGED
@@ -14,7 +14,7 @@ should probably proofread and complete it, then remove this comment. -->
14
 
15
  This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on the None dataset.
16
  It achieves the following results on the evaluation set:
17
- - Loss: 2.5919
18
 
19
  ## Model description
20
 
@@ -44,7 +44,7 @@ The following hyperparameters were used during training:
44
  - total_eval_batch_size: 8
45
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
  - lr_scheduler_type: linear
47
- - training_steps: 850
48
 
49
  ### Training results
50
 
@@ -80,10 +80,20 @@ The following hyperparameters were used during training:
80
  | 2.7437 | 0.11 | 700 | 2.7797 |
81
  | 2.7773 | 0.12 | 725 | 2.7696 |
82
  | 2.6785 | 0.12 | 750 | 2.7611 |
83
- | 2.7584 | 0.12 | 775 | 2.7508 |
84
- | 2.7787 | 0.13 | 800 | 2.7414 |
85
- | 2.7547 | 0.13 | 825 | 2.7341 |
86
- | 2.7227 | 0.14 | 850 | 2.7260 |
 
 
 
 
 
 
 
 
 
 
87
 
88
 
89
  ### Framework versions
 
14
 
15
  This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on the None dataset.
16
  It achieves the following results on the evaluation set:
17
+ - Loss: 2.5280
18
 
19
  ## Model description
20
 
 
44
  - total_eval_batch_size: 8
45
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
  - lr_scheduler_type: linear
47
+ - training_steps: 1100
48
 
49
  ### Training results
50
 
 
80
  | 2.7437 | 0.11 | 700 | 2.7797 |
81
  | 2.7773 | 0.12 | 725 | 2.7696 |
82
  | 2.6785 | 0.12 | 750 | 2.7611 |
83
+ | 2.7582 | 0.12 | 775 | 2.7510 |
84
+ | 2.7785 | 0.13 | 800 | 2.7414 |
85
+ | 2.7549 | 0.13 | 825 | 2.7339 |
86
+ | 2.7228 | 0.14 | 850 | 2.7257 |
87
+ | 2.5928 | 0.14 | 875 | 2.7189 |
88
+ | 2.7048 | 0.14 | 900 | 2.7118 |
89
+ | 2.6131 | 0.15 | 925 | 2.7052 |
90
+ | 2.7515 | 0.15 | 950 | 2.6994 |
91
+ | 2.7365 | 0.16 | 975 | 2.6933 |
92
+ | 2.7635 | 0.16 | 1000 | 2.6882 |
93
+ | 2.7881 | 0.16 | 1025 | 2.6844 |
94
+ | 2.7033 | 0.17 | 1050 | 2.6783 |
95
+ | 2.7138 | 0.17 | 1075 | 2.6728 |
96
+ | 2.643 | 0.18 | 1100 | 2.6683 |
97
 
98
 
99
  ### Framework versions
adapter_config.json CHANGED
@@ -21,10 +21,10 @@
21
  "revision": null,
22
  "target_modules": [
23
  "gate_proj",
24
- "down_proj",
25
- "up_proj",
26
  "q_proj",
27
- "v_proj"
 
 
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
 
21
  "revision": null,
22
  "target_modules": [
23
  "gate_proj",
 
 
24
  "q_proj",
25
+ "down_proj",
26
+ "v_proj",
27
+ "up_proj"
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d6576de2d4a27c2582fb51478e83bda28b04ee094c4a80af1eaf100ac048dcba
3
  size 252750032
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:632337f05fc4b781b2e739e0235b712ad29aa45b7a6cd401df7b8a7bc4dffbfb
3
  size 252750032
model-00001-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7963110045ac39b724dfc11f3a54b210722b48881562565470bc634e2bfcbd87
3
  size 4938985352
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf8307e5197d314fe4cda7e6fdbad5dea87b51b05d508284821d124247f02a26
3
  size 4938985352
model-00002-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8a57b7d4dae346909ef2614353065340c4eedddd18a9977cc1e0c7be3e899308
3
  size 4947390880
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c185ab205448ec2ee849c93c06489b9b140e129e82e0dfeb285bbc520117ac4
3
  size 4947390880
model-00003-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:44891a046bb56413b312f79845d64170b18931f10d9f87d6232ce9084845ffcc
3
  size 3590488816
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22e5a2b5d67140a5ff9cbbef4f9b05f1733192d3980d536a53b75a5695183fe7
3
  size 3590488816
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f6fca1d2159adf3e897638ed73cdde11964dbef91cc8677dbaf9635665fc9d2c
3
  size 6712
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e983adc2d4d479f7ea690400fdec167a70ee977ce444069a440262b9a20ce6dc
3
  size 6712