fine-tuned-smolLM2-135M-with-LoRA-on-camel-ai-physics
This model is a fine-tuned version of HuggingFaceTB/SmolLM2-135M on the dataset akhilfau/physics_decontaminated_2. This dataset was created by decontaminating the camel-ai/physics dataset from mmlu:college_physics.
Model Performance
This model was evaluated on MMLU: college_physics using LightEval. The evaluation compared the base model (HuggingFaceTB/SmolLM2-135M) and the fine-tuned model (akhilfau/fine-tuned-smolLM2-135M-with-LoRA-on-camel-ai-physics). Results are as follows:
Model Description
The fine-tuned model leverages LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning. The base model is SmolLM2-135M, which uses the LlamaForCausalLM architecture, and it was fine-tuned to enhance its understanding of physics-related questions and answers using the akhilfau/physics_decontaminated_2 dataset.
Training and Evaluation Data
Dataset Details:
- Training Dataset: akhilfau/physics_decontaminated_2
- Evaluation Dataset: mmlu:college_physics
The training dataset was decontaminated to ensure no overlap with the evaluation dataset for fair performance testing.
Training Procedure
Training Hyperparameters
| Hyperparameter | Value |
|---|---|
| Learning Rate | 0.0005 |
| Train Batch Size | 4 |
| Eval Batch Size | 4 |
| Seed | 42 |
| Optimizer | AdamW with betas=(0.9, 0.999), epsilon=1e-8 |
| LR Scheduler Type | Cosine |
| Number of Epochs | 8 |
Training Results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.0151 | 1.0 | 4000 | 1.0407 |
| 1.0234 | 2.0 | 8000 | 1.0087 |
| 0.9995 | 3.0 | 12000 | 0.9921 |
| 0.9528 | 4.0 | 16000 | 0.9824 |
| 0.9353 | 5.0 | 20000 | 0.9755 |
| 0.9121 | 6.0 | 24000 | 0.9720 |
| 0.9175 | 7.0 | 28000 | 0.9707 |
| 0.9197 | 8.0 | 32000 | 0.9706 |
Intended Use
This model is specifically fine-tuned for physics-related reasoning tasks and QA tasks. It may perform well on datasets that require understanding physics-related problems and concepts. Evaluation results show a measurable improvement compared to the base model on MMLU college physics tasks.
Framework Versions
- PEFT: 0.13.2
- Transformers: 4.46.2
- Pytorch: 2.4.1+cu121
- Datasets: 3.1.0
- Tokenizers: 0.20.3
- Downloads last month
- 9