Create README.md

README.md

@@ -0,0 +1,65 @@
+---
+license: apache-2.0
+datasets:
+- randomani/MedicalQnA-llama2
+language:
+- en
+library_name: adapter-transformers
+pipeline_tag: question-answering
+tags:
+- medical
+---
+
+# Fine-Tuning LLaMA-2 Chat Model with Medical QnA Dataset using QLoRA
+
+This repository contains the code and configuration for fine-tuning the LLaMA-2 chat model using the Medical QnA dataset with the QLoRA technique.Used only 2k data elements for training due to constrained gpu resources.
+
+## Model and Dataset
+
+- **Pre-trained Model**: `NousResearch/Llama-2-7b-chat-hf`
+- **Dataset for Fine-Tuning**: `randomani/MedicalQnA-llama2`
+- **Fine-Tuned Model Name**: `Llama-2-7b-Medchat-finetune`
+
+## QLoRA Parameters
+
+- **LoRA Attention Dimension** (`lora_r`): 64
+- **LoRA Scaling Alpha** (`lora_alpha`): 16
+- **LoRA Dropout Probability** (`lora_dropout`): 0.1
+
+## bitsandbytes Parameters
+
+- **Use 4-bit Precision** (`use_4bit`): True
+- **4-bit Compute Dtype** (`bnb_4bit_compute_dtype`): float16
+- **4-bit Quantization Type** (`bnb_4bit_quant_type`): nf4
+- **Use Nested Quantization** (`use_nested_quant`): False
+
+## Training Arguments
+
+- **Number of Training Epochs** (`num_train_epochs`): 1
+- **Use fp16** (`fp16`): False
+- **Use bf16** (`bf16`): False
+- **Training Batch Size per GPU** (`per_device_train_batch_size`): 4
+- **Evaluation Batch Size per GPU** (`per_device_eval_batch_size`): 4
+- **Gradient Accumulation Steps** (`gradient_accumulation_steps`): 1
+- **Enable Gradient Checkpointing** (`gradient_checkpointing`): True
+- **Maximum Gradient Norm** (`max_grad_norm`): 0.3
+- **Initial Learning Rate** (`learning_rate`): 2e-4
+- **Weight Decay** (`weight_decay`): 0.001
+- **Optimizer** (`optim`): paged_adamw_32bit
+- **Learning Rate Scheduler Type** (`lr_scheduler_type`): cosine
+- **Maximum Training Steps** (`max_steps`): -1
+- **Warmup Ratio** (`warmup_ratio`): 0.03
+- **Group Sequences by Length** (`group_by_length`): True
+- **Save Checkpoints Every X Steps** (`save_steps`): 0
+- **Logging Steps** (`logging_steps`): 25
+
+## Supervised Fine-Tuning (SFT) Parameters
+
+- **Maximum Sequence Length** (`max_seq_length`): None
+- **Packing Multiple Short Examples** (`packing`): False
+
+## References
+
+For more details and access to the dataset, visit the [Hugging Face Dataset Page](https://huggingface.co/datasets/randomani/MedicalQnA-llama2).
+
+