nicksnlp/shrimp · Hugging Face

Model Fine-Tuning

This model was fine-tuned using supervised fine-tuning (SFT) with the following key configuration details:

Pre-trained Model

Model: mistralai/Mistral-7B-v0.3
Tokenizer: Corresponding tokenizer for the Mistral-7B model.

Dataset

Dataset: vicgalle/alpaca-gpt4
Subset Size: First 1,000 examples.

Training Parameters

Epochs: 1
Batch Size: 8 (with gradient accumulation of 2)
Learning Rate: 2e-4
Optimizer: paged_adamw_8bit
Weight Decay: 0.001
Max Grad Norm: 0.3
Warm-up Ratio: 0.3
Gradient Accumulation: 2 steps
FP16/BF16: Disabled
Max Steps: -1 (training will stop when dataset is exhausted)
Scheduler: Linear learning rate scheduler
Monitoring: Weights & Biases (wandb)

PEFT Configuration

The model uses LoRA (Low-Rank Adaptation) for efficient fine-tuning with the following configuration:

lora_alpha: 8
lora_dropout: 0.1
r: 16
Bias: "none"
Task Type: CAUSAL_LM
Target Modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj"]

Quantization Configuration

The model is quantized to 4-bit using BitsAndBytes to reduce memory usage for training and inference:

Load in 4-bit: Yes
Quantization Type: nf4
Compute Data Type: float16

Training Environment

Platform: Google Colab (GPU enabled)
Monitoring: Weights & Biases (W&B)

Model Saving and Upload

Fine-tuned Model: Saved to Hugging Face repository: nicksnlp/shrimp
Tokenizer: Pushed to Hugging Face repository.

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load the fine-tuned model
model = PeftModel.from_pretrained("nicksnlp/shrimp")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.3")

# Example inference
prompt = "What is Newton's 3rd Law and its formula?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

nicksnlp
/

shrimp