peft safetensors gemma base_model:Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_2700_steps_beta region:us
---
library_name: peft
base_model: Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_2700_steps_beta
languages: en
---

# Model Card for Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_2700_steps_beta

### This Repo contains the Fine-Tuned JOSIE model from the pretrained 'Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_2700_steps_beta' model
This is JOSIE, my own private AI assistant that can call funtinos, and greet me based on the time and day.

### Quantised Models are here: [official quantized models](https://huggingface.co/Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_3600_steps_beta-GGUF)

- **Developed by:** Gökdeniz Gülmez
- **Funded by:** Gökdeniz Gülmez
- **Shared by:** Gökdeniz Gülmez
- **Model type:** Gemma
- **Language(s) (NLP):** English
- **License:** MIT

- **Pre-Trained model** = Isaak-Carter/JOSIExgemma_2b_chat_checkpoint_2700_steps_beta
- **Fine-Tuned model** = JOSIExgemma_2b_chat_checkpoint_3600_steps_beta
- **Dataset for Fine-Tuning** = Isaak-Carter/MAIN_JOSIE_wizard_vicuna_70k_unfiltered

## Dataset Informations
Here is the Dataset I used to Fine-Tune the AI.

full_dataset_size = 34598
dataset_train_size = 31139
dataset_val_size = 3459

# QLoRA parameters
- lora_r (LoRA attention dimension) = 64
- lora_alpha (Alpha parameter for LoRA scaling) = 32
- lora_dropout (Dropout probability for LoRA layers) = 0.05

# Bitsandbytes parameters
- use_4bit = True
- bnb_4bit_compute_dtype = float16
- bnb_4bit_quant_type = nf4
- use_nested_quant = False

# Training Arguments
- num_train_epochs = 1
- max_steps = 900
- fp16 = False
- bf16 = True
- per_device_train_batch_size = 1
- per_device_eval_batch_size = 1
- gradient_accumulation_steps = 4
- gradient_checkpointing = True
- max_grad_norm = 0.3
- learning_rate = 2e-05
- weight_decay = 0.001
- optim = paged_adamw_8bit
- lr_scheduler_type = cosine
- warmup_ratio = 0.02
- warmup_steps = 100
- max_seq_length = 8192
- packing = False

# Hardware settings
- GPU name = A40
- GPU amount = 2

### Framework versions

- PEFT 0.7.1