File size: 3,001 Bytes
2fa8088 f025ad6 2fa8088 f025ad6 2fa8088 b3ef1f5 2fa8088 b3ef1f5 2fa8088 6578d2d 2fa8088 f025ad6 2fa8088 1439616 2fa8088 f025ad6 2fa8088 1439616 2fa8088 f025ad6 2fa8088 f025ad6 751fbe8 5e2c7aa f025ad6 751fbe8 f025ad6 751fbe8 f025ad6 2fa8088 b3ef1f5 2fa8088 f025ad6 2fa8088 d2b72a2 fc101d6 ec26f69 1439616 6578d2d fc101d6 ec26f69 6578d2d fc101d6 6578d2d 2fa8088 f025ad6 2fa8088 f025ad6 2fa8088 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 |
---
library_name: transformers
tags:
- code
- gemma-2b
- finetune
- qlora
license: apache-2.0
datasets:
- SaikatM/Code-Platypus
language:
- en
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
This model is a fine-tuned version of google/gemma-2b on an SaikatM/Code-Platypus dataset.
## Model Details
### Model Description
- **Finetuned from model:** google/gemma-2b
### Model Sources
Training code can be found here: https://github.com/Saikat-M/LLM-Finetuning
### Direct Use
* Code generation tasks
### Training Data
* Dataset: https://huggingface.co/datasets/SaikatM/Code-Platypus
* Source Dataset: https://huggingface.co/datasets/garage-bAInd/Open-Platypus
### Training Procedure
Used QLoRA from PEFT and used SFTTrainer.
#### Preprocessing
From the Open-Platypus dataset filtering-out rows which has leetcode_ne in it's data_source column.
#### Training Hyperparameters
LoraConfig(
r=4,
lora_alpha=2,
target_modules=modules,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
TrainingArguments(
output_dir="gemma-2b-code-platypus",
num_train_epochs=1,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
gradient_checkpointing=True,
optim="paged_adamw_8bit",
logging_steps=1,
save_strategy="epoch",
bf16=False,
tf32=False,
learning_rate=2e-4,
max_steps= 100,
max_grad_norm=0.3,
warmup_ratio=0.03,
lr_scheduler_type="constant",
push_to_hub=False,
report_to="tensorboard",
)
SFTTrainer(
model=model,
train_dataset=train_data,
eval_dataset=test_data,
dataset_text_field="text",
peft_config=lora_config,
max_seq_length=512,
tokenizer=tokenizer,
args=training_arguments,
)
#### Speeds, Sizes, Times
Took around 1 hour to train.
### Results
* Test Result 1:
```
Write a fucntion to sort a list in python
Answer:
def sort_list(list):
return sorted(list)<eos>
Response: None
```
* Test Result 2:
```
Write a function to count Consonants in a Given Word in Python
Response: None
```
* Test Result 3:
```
Write a function to count the number of vowels in a given string in Python.
Example 1:
Input: s = "leetcodeisgreat"
Output: 5
Explanation: The vowels are 'e', 'i', 'a', 'o', and 'u'.
Example 2:
Input: s = "leetcodeisgreat"
Output: 0
Explanation: The vowels are 'e', 'i', 'a', 'o', and 'u'.
Constraints:
* 1 <= s.length <= 100
* s consists of lowercase English letters.
def countVowels(s):
count = 0
for c in s:
if c in 'aeiou':
count += 1
return count
<eos>
Response: None
```
### Compute Infrastructure
Trained in Google Colab
#### Hardware
T4 GPU Hardware accelerator.
|