Model Card for Model ID

This model is a fine-tuned version of google/gemma-2b on an SaikatM/Code-Platypus dataset.

Model Details

Model Description

Finetuned from model: google/gemma-2b

Model Sources

Training code can be found here: https://github.com/Saikat-M/LLM-Finetuning

Direct Use

Code generation tasks

Training Data

Dataset: https://huggingface.co/datasets/SaikatM/Code-Platypus
Source Dataset: https://huggingface.co/datasets/garage-bAInd/Open-Platypus

Training Procedure

Used QLoRA from PEFT and used SFTTrainer.

Preprocessing

From the Open-Platypus dataset filtering-out rows which has leetcode_ne in it's data_source column.

Training Hyperparameters

LoraConfig(

r=4,
lora_alpha=2,
target_modules=modules,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"

)

TrainingArguments(

output_dir="gemma-2b-code-platypus",      
num_train_epochs=1,                    
per_device_train_batch_size=4,            
gradient_accumulation_steps=4,            
gradient_checkpointing=True,              
optim="paged_adamw_8bit",                 
logging_steps=1,                          
save_strategy="epoch",                    
bf16=False,
tf32=False,
learning_rate=2e-4,                       
max_steps= 100,
max_grad_norm=0.3,                        
warmup_ratio=0.03,                        
lr_scheduler_type="constant",             
push_to_hub=False,                        
report_to="tensorboard",

)

SFTTrainer(

model=model,
train_dataset=train_data,
eval_dataset=test_data,
dataset_text_field="text",
peft_config=lora_config,
max_seq_length=512,
tokenizer=tokenizer,
args=training_arguments,

)

Speeds, Sizes, Times

Took around 1 hour to train.

Results

Test Result 1:

Write a fucntion to sort a list in python

Answer:

def sort_list(list):
    return sorted(list)<eos>
Response:  None

Test Result 2:

Write a function to count Consonants in a Given Word in Python

Response:  None

Test Result 3:

Write a function to count the number of vowels in a given string in Python.

Example 1:

Input:  s =  "leetcodeisgreat"
Output:  5
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Example 2:

Input:  s =  "leetcodeisgreat"
Output:  0
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Constraints:

*   1 <= s.length <= 100
*   s consists of lowercase English letters.


def countVowels(s):
    count = 0
    for c in s:
        if c in 'aeiou':
            count += 1
    return count
<eos>
Response:  None

Compute Infrastructure

Trained in Google Colab

Hardware

T4 GPU Hardware accelerator.

SaikatM
/

Code-Gemma-v1