End of training

README.md

@@ -0,0 +1,111 @@
+---
+library_name: peft
+license: bsd-3-clause
+base_model: hugohrban/progen2-base
+tags:
+- generated_from_trainer
+model-index:
+- name: Progen2_Kinase_PhosphositeGen_dkz_traindata
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# Progen2_Kinase_PhosphositeGen_dkz_traindata
+
+This model is a fine-tuned version of [hugohrban/progen2-base](https://huggingface.co/hugohrban/progen2-base) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 2.0955
+- Perplexity: 8.1296
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 16
+- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- training_steps: 5000
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Perplexity |
+|:-------------:|:------:|:----:|:---------------:|:----------:|
+| 4.7229        | 0.1455 | 100  | 2.1862          | 8.9015     |
+| 4.2858        | 0.2909 | 200  | 2.1091          | 8.2405     |
+| 4.2112        | 0.4364 | 300  | 2.0519          | 7.7824     |
+| 4.1146        | 0.5818 | 400  | 2.0049          | 7.4252     |
+| 4.0772        | 0.7273 | 500  | 1.9859          | 7.2855     |
+| 3.9871        | 0.8727 | 600  | 1.9478          | 7.0130     |
+| 3.891         | 1.0175 | 700  | 1.9204          | 6.8236     |
+| 3.4841        | 1.1629 | 800  | 1.8889          | 6.6122     |
+| 3.4596        | 1.3084 | 900  | 1.8696          | 6.4854     |
+| 3.4659        | 1.4538 | 1000 | 1.8430          | 6.3152     |
+| 3.433         | 1.5993 | 1100 | 1.8105          | 6.1137     |
+| 3.3728        | 1.7447 | 1200 | 1.7991          | 6.0441     |
+| 3.3853        | 1.8902 | 1300 | 1.7924          | 6.0040     |
+| 3.1832        | 2.0349 | 1400 | 1.7975          | 6.0348     |
+| 2.8198        | 2.1804 | 1500 | 1.7924          | 6.0041     |
+| 2.7867        | 2.3258 | 1600 | 1.7604          | 5.8149     |
+| 2.8669        | 2.4713 | 1700 | 1.7437          | 5.7183     |
+| 2.795         | 2.6167 | 1800 | 1.7307          | 5.6445     |
+| 2.8152        | 2.7622 | 1900 | 1.7188          | 5.5779     |
+| 2.7734        | 2.9076 | 2000 | 1.6911          | 5.4256     |
+| 2.5299        | 3.0524 | 2100 | 1.7682          | 5.8605     |
+| 2.2126        | 3.1978 | 2200 | 1.7346          | 5.6669     |
+| 2.2435        | 3.3433 | 2300 | 1.7104          | 5.5310     |
+| 2.2663        | 3.4887 | 2400 | 1.7144          | 5.5536     |
+| 2.2463        | 3.6342 | 2500 | 1.7338          | 5.6620     |
+| 2.3117        | 3.7796 | 2600 | 1.6879          | 5.4079     |
+| 2.2655        | 3.9251 | 2700 | 1.6946          | 5.4447     |
+| 1.9936        | 4.0698 | 2800 | 1.8444          | 6.3244     |
+| 1.7929        | 4.2153 | 2900 | 1.8653          | 6.4577     |
+| 1.8214        | 4.3607 | 3000 | 1.7600          | 5.8123     |
+| 1.8505        | 4.5062 | 3100 | 1.7855          | 5.9628     |
+| 1.8382        | 4.6516 | 3200 | 1.7955          | 6.0225     |
+| 1.7945        | 4.7971 | 3300 | 1.7754          | 5.9028     |
+| 1.8238        | 4.9425 | 3400 | 1.7820          | 5.9418     |
+| 1.573         | 5.0873 | 3500 | 1.8691          | 6.4823     |
+| 1.4562        | 5.2327 | 3600 | 1.8905          | 6.6225     |
+| 1.47          | 5.3782 | 3700 | 2.0037          | 7.4163     |
+| 1.4649        | 5.5236 | 3800 | 1.8911          | 6.6268     |
+| 1.4778        | 5.6691 | 3900 | 1.9307          | 6.8940     |
+| 1.4985        | 5.8145 | 4000 | 1.9265          | 6.8655     |
+| 1.4587        | 5.96   | 4100 | 1.9128          | 6.7720     |
+| 1.258         | 6.1047 | 4200 | 2.0383          | 7.6773     |
+| 1.2239        | 6.2502 | 4300 | 2.0444          | 7.7244     |
+| 1.2186        | 6.3956 | 4400 | 2.0497          | 7.7658     |
+| 1.2174        | 6.5411 | 4500 | 2.0454          | 7.7323     |
+| 1.2051        | 6.6865 | 4600 | 2.0195          | 7.5345     |
+| 1.2189        | 6.832  | 4700 | 2.0461          | 7.7376     |
+| 1.2061        | 6.9775 | 4800 | 2.0435          | 7.7176     |
+| 1.0575        | 7.1222 | 4900 | 2.0885          | 8.0727     |
+| 1.048         | 7.2676 | 5000 | 2.0955          | 8.1296     |
+
+
+### Framework versions
+
+- PEFT 0.13.2
+- Transformers 4.47.1
+- Pytorch 2.1.0.post301
+- Datasets 3.0.2
+- Tokenizers 0.21.0

adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "hugohrban/progen2-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "qkv_proj",
+    "out_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67a68ef6562a5590c925e8b4103f77d75af532ae245e58dfe71d3ba464586b61
+size 127416480

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<|endoftext|>": 30
+}

merges.txt

@@ -0,0 +1 @@
+#version: 0.2

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer.json

@@ -0,0 +1,105 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|pad|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|bos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|eos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 30,
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "post_processor": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<|pad|>": 0,
+      "<|bos|>": 1,
+      "<|eos|>": 2,
+      "1": 3,
+      "2": 4,
+      "A": 5,
+      "B": 6,
+      "C": 7,
+      "D": 8,
+      "E": 9,
+      "F": 10,
+      "G": 11,
+      "H": 12,
+      "I": 13,
+      "K": 14,
+      "L": 15,
+      "M": 16,
+      "N": 17,
+      "O": 18,
+      "P": 19,
+      "Q": 20,
+      "R": 21,
+      "S": 22,
+      "T": 23,
+      "U": 24,
+      "V": 25,
+      "W": 26,
+      "X": 27,
+      "Y": 28,
+      "Z": 29
+    },
+    "merges": []
+  }
+}

tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e3cf2103e69321d14b1705f1fe73f150b66c3fa5129bf4d01a1531ca0fb3e6b
+size 5432

vocab.json

@@ -0,0 +1 @@
+{"<|pad|>":0,"<|bos|>":1,"<|eos|>":2,"1":3,"2":4,"A":5,"B":6,"C":7,"D":8,"E":9,"F":10,"G":11,"H":12,"I":13,"K":14,"L":15,"M":16,"N":17,"O":18,"P":19,"Q":20,"R":21,"S":22,"T":23,"U":24,"V":25,"W":26,"X":27,"Y":28,"Z":29}