Update README.md

README.md

@@ -14,7 +14,7 @@ model-index:
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-# **Llama-3.2-3B-Code-Knowledge-Value-Eval **
+# **Llama-3.2-3B-Code-Knowledge-Value-Eval**
 
 This model is a fine-tuned version of [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) on the [kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval) dataset.
 It achieves the following results on the evaluation set:
@@ -38,6 +38,135 @@ The model focuses on understanding the structure, syntax, and logic of various p
 2. **Language and Domain Limitations**: Since the dataset is sourced from `bigcode/the-stack`, it may not cover all programming languages or specialized domains. The model may perform less effectively in underrepresented languages or niche coding styles not well-represented in the dataset.
 3. **Not Suitable for All Educational Levels**: While the model is designed to evaluate code for educational purposes, its outputs may be better suited for certain levels (e.g., beginner or intermediate coding), and its recommendations might not fully cater to advanced or highly specialized learners.
 
+
+## How to use this model?
+
+```python
+import torch
+import numpy as np
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+# Define the model name or path for loading the tokenizer and model
+model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
+
+# Load the tokenizer from the pre-trained model
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+
+# Load the pre-trained model for sequence classification and map it to the first CUDA device
+model = AutoModelForSequenceClassification.from_pretrained(
+    model_name_or_path, 
+    device_map="cuda:0",
+)
+
+# Example code snippet to be evaluated
+code = [
+"""
+import torch
+import numpy as np
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+# Define the model name or path for loading the tokenizer and model
+model_name_or_path = "kimsan0622/Llama-3.2-1B-Code-Knowledge-Value-Eval"
+
+# Load the tokenizer from the pre-trained model
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+
+# Load the pre-trained model for sequence classification and map it to the first CUDA device
+model = AutoModelForSequenceClassification.from_pretrained(
+    model_name_or_path, 
+    device_map="cuda:0",
+)
+
+# Example code snippet to be evaluated
+code = ["code 1"]
+
+# Tokenize the input code, setting max length, padding, and truncation
+batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")
+
+# Perform inference with the model, without computing gradients (for faster inference)
+with torch.no_grad():
+    # Pass the input IDs and attention mask to the model, using the CUDA device
+    res = model(
+            input_ids=batch["input_ids"].to("cuda:0"),
+            attention_mask=batch["attention_mask"].to("cuda:0"),
+        )
+    
+    # Move the logits to the CPU, convert them to a numpy array
+    preds = res.logits.cpu().numpy()
+    
+    # Get the predicted class by taking the argmax of the logits across the classification axis
+    preds = np.argmax(preds, axis=1).tolist()
+"""
+]
+
+# Tokenize the input code, setting max length, padding, and truncation
+batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")
+
+# Perform inference with the model, without computing gradients (for faster inference)
+with torch.no_grad():
+    # Pass the input IDs and attention mask to the model, using the CUDA device
+    res = model(
+            input_ids=batch["input_ids"].to("cuda:0"),
+            attention_mask=batch["attention_mask"].to("cuda:0"),
+        )
+    
+    # Move the logits to the CPU, convert them to a numpy array
+    preds = res.logits.cpu().numpy()
+    
+    # Get the predicted class by taking the argmax of the logits across the classification axis
+    preds = np.argmax(preds, axis=1).tolist()
+    print(preds)
+```
+
+### 8 Bit quantization
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
+
+# Define the model name or path for loading the model
+model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
+
+# Configure the model to load in 8-bit precision for memory efficiency
+bnb_config = BitsAndBytesConfig(load_in_8bit=True)
+
+# Load the pre-trained model for sequence classification with quantization for 8-bit precision
+# This helps reduce memory usage, particularly for large models, and map it to the first CUDA device
+model = AutoModelForSequenceClassification.from_pretrained(
+    model_name_or_path, 
+    quantization_config=bnb_config,  # Apply 8-bit quantization
+    device_map="cuda:0",  # Map the model to the first CUDA device
+)
+```
+
+### 4 Bit quntization
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
+import torch
+
+# Define the model name or path for loading the model
+model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"
+
+# Define configuration parameters for 4-bit quantization
+bnb_config_params = {
+    "bnb_4bit_quant_type": "fp4",  # Use FP4 for 4-bit quantization type
+    "bnb_4bit_compute_dtype": torch.bfloat16,  # Use bfloat16 for computation to balance performance and precision
+    "bnb_4bit_use_double_quant": False,  # Disable double quantization, which is typically used to further reduce precision
+    "bnb_4bit_quant_storage": torch.bfloat16,  # Store quantized values in bfloat16 format
+}
+
+# Configure the model to load in 4-bit precision for memory and performance optimization
+bnb_config = BitsAndBytesConfig(load_in_4bit=True, **bnb_config_params)
+
+# Load the pre-trained model for sequence classification with 4-bit quantization settings
+# This reduces memory usage while still maintaining reasonable accuracy, mapping the model to the first CUDA device
+model = AutoModelForSequenceClassification.from_pretrained(
+    model_name_or_path, 
+    quantization_config=bnb_config,  # Apply 4-bit quantization configuration
+    device_map="cuda:0",  # Map the model to the first CUDA device
+)
+```
+
+
 ## Training and evaluation data
 
 [kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval)
@@ -110,3 +239,64 @@ The following hyperparameters were used during training:
 | **weighted avg** |  0.55      |    0.56    |     0.55     |   18232     |
 
 
+
+## 8 Bit quantization model
+
+### Confusion matrix
+
+| **y_true** |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
+|-------|-------|-------|-------|-------|-------|-------|
+|   0   |  933  |  272  |  111  |   56  |   5   |   0   |
+|   1   |  320  |  333  |  340  |  224  |  25   |   2   |
+|   2   |  129  |  244  |  502  |  752  |  139  |   8   |
+|   3   |   43  |  119  |  463  | 2627  | 1604  |  69   |
+|   4   |    7  |   15  |   67  | 1227  | 4191  |  817  |
+|   5   |    0  |    2  |    0  |   30  | 1034  | 1522  |
+
+
+### Classification report
+
+|  **y_true**   | **precision** | **recall** | **f1-score** | **support** |
+|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
+|       0       |     0.65      |    0.68    |     0.66     |    1377     |
+|       1       |     0.34      |    0.27    |     0.30     |    1244     |
+|       2       |     0.34      |    0.28    |     0.31     |    1774     |
+|       3       |     0.53      |    0.53    |     0.53     |    4925     |
+|       4       |     0.60      |    0.66    |     0.63     |    6324     |
+|       5       |     0.63      |    0.59    |     0.61     |    2588     |
+|  **accuracy** |               |            |     0.55     |   18232     |
+|  **macro avg**|     0.52      |    0.50    |     0.51     |   18232     |
+| **weighted avg** |  0.55      |    0.55    |     0.55     |   18232     |
+
+
+
+## 4 Bit quantization model
+
+### Confusion matrix
+
+| **y_true** |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
+|-------|-------|-------|-------|-------|-------|-------|
+|   0   |  695  |  581  |   42  |   55  |    4  |   0   |
+|   1   |  151  |  662  |  190  |  215  |   24  |   2   |
+|   2   |   53  |  485  |  353  |  716  |  159  |   8   |
+|   3   |   20  |  277  |  335  | 2446  | 1765  |  82   |
+|   4   |    4  |   31  |   60  | 1104  | 4211  |  914  |
+|   5   |    0  |    2  |    0  |   24  |  934  | 1628  |
+
+
+
+
+### Classification report
+
+|               | **precision** | **recall** | **f1-score** | **support** |
+|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
+|       0       |     0.75      |    0.50    |     0.60     |    1377     |
+|       1       |     0.32      |    0.53    |     0.40     |    1244     |
+|       2       |     0.36      |    0.20    |     0.26     |    1774     |
+|       3       |     0.54      |    0.50    |     0.52     |    4925     |
+|       4       |     0.59      |    0.67    |     0.63     |    6324     |
+|       5       |     0.62      |    0.63    |     0.62     |    2588     |
+|  **accuracy** |               |            |     0.55     |   18232     |
+|  **macro avg**|     0.53      |    0.50    |     0.51     |   18232     |
+| **weighted avg** |  0.55      |    0.55    |     0.54     |   18232     |
+