
# Model Card for SantaFixer

<!-- Provide a quick summary of what the model is/does. -->

This is a LLM for code that is focussed on generating bug fixes using infilling. 

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** [codelion](https://huggingface.co/codelion)
- **Model type:** GPT-2
- **Finetuned from model:** [bigcode/santacoder](https://huggingface.co/bigcode/santacoder)


## How to Get Started with the Model

Use the code below to get started with the model.

```python
# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "lambdasec/santafixer"
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint,
              trust_remote_code=True).to(device)

input_text = "<fim-prefix>def print_hello_world():\n
              <fim-suffix>\n print('Hello world!')
              <fim-middle>"
inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```

## Training Details

- **GPU:** Tesla P100
- **Time:** ~5 hrs

### Training Data

The model was fine-tuned on the [CVE single line fixes dataset](https://huggingface.co/datasets/lambdasec/cve-single-line-fixes)

### Training Procedure 

Supervised Fine Tuning (SFT)

#### Training Hyperparameters

- **optim:** adafactor
- **gradient_accumulation_steps:** 4
- **gradient_checkpointing:** true
- **fp16:** false

## Evaluation

The model was tested with the [GitHub top 1000 projects vulnerabilities dataset](https://huggingface.co/datasets/lambdasec/gh-top-1000-projects-vulns)
{"language": ["code"], "license": "apache-2.0", "tags": ["code"], "datasets": ["lambdasec/cve-single-line-fixes", "lambdasec/gh-top-1000-projects-vulns"], "programming_language": ["Java", "JavaScript", "Python"], "inference": false, "model-index": [{"name": "SantaFixer", "results": [{"task": {"type": "text-generation"}, "dataset": {"name": "HumanEval", "type": "openai/human-eval-infilling"}, "metrics": [{"type": "pass@1", "value": 0.47, "name": "single-line infilling pass@1", "verified": false}, {"type": "pass@10", "value": 0.74, "name": "single-line infilling pass@10", "verified": false}]}, {"task": {"type": "text-generation"}, "dataset": {"name": "GH Top 1000 Projects Vulnerabilities", "type": "lambdasec/gh-top-1000-projects-vulns"}, "metrics": [{"type": "pass@1", "value": 0.26, "name": "pass@1 (Java)", "verified": false}, {"type": "pass@10", "value": 0.48, "name": "pass@10 (Java)", "verified": false}, {"type": "pass@1", "value": 0.31, "name": "pass@1 (Python)", "verified": false}, {"type": "pass@10", "value": 0.56, "name": "pass@10 (Python)", "verified": false}, {"type": "pass@1", "value": 0.36, "name": "pass@1 (JavaScript)", "verified": false}, {"type": "pass@10", "value": 0.62, "name": "pass@10 (JavaScript)", "verified": false}]}]}]}
transformers pytorch gpt2 text-generation code custom_code dataset:lambdasec/cve-single-line-fixes dataset:lambdasec/gh-top-1000-projects-vulns license:apache-2.0 model-index autotrain_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #code #custom_code #dataset-lambdasec/cve-single-line-fixes #dataset-lambdasec/gh-top-1000-projects-vulns #license-apache-2.0 #model-index #autotrain_compatible #has_space #text-generation-inference #region-us 


# Model Card for SantaFixer



This is a LLM for code that is focussed on generating bug fixes using infilling. 

## Model Details

### Model Description





- Developed by: codelion
- Model type: GPT-2
- Finetuned from model: bigcode/santacoder


## How to Get Started with the Model

Use the code below to get started with the model.



## Training Details

- GPU: Tesla P100
- Time: ~5 hrs

### Training Data

The model was fine-tuned on the CVE single line fixes dataset

### Training Procedure 

Supervised Fine Tuning (SFT)

#### Training Hyperparameters

- optim: adafactor
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- fp16: false

## Evaluation

The model was tested with the GitHub top 1000 projects vulnerabilities dataset
# Model Card for SantaFixer



This is a LLM for code that is focussed on generating bug fixes using infilling. ## Model Details ### Model Description





- Developed by: codelion
- Model type: GPT-2
- Finetuned from model: bigcode/santacoder ## How to Get Started with the Model

Use the code below to get started with the model. ## Training Details

- GPU: Tesla P100
- Time: ~5 hrs ### Training Data

The model was fine-tuned on the CVE single line fixes dataset ### Training Procedure 

Supervised Fine Tuning (SFT) #### Training Hyperparameters

- optim: adafactor
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- fp16: false ## Evaluation

The model was tested with the GitHub top 1000 projects vulnerabilities dataset
TAGS
#transformers #pytorch #gpt2 #text-generation #code #custom_code #dataset-lambdasec/cve-single-line-fixes #dataset-lambdasec/gh-top-1000-projects-vulns #license-apache-2.0 #model-index #autotrain_compatible #has_space #text-generation-inference #region-us 
 # Model Card for SantaFixer



This is a LLM for code that is focussed on generating bug fixes using infilling. ## Model Details ### Model Description





- Developed by: codelion
- Model type: GPT-2
- Finetuned from model: bigcode/santacoder ## How to Get Started with the Model

Use the code below to get started with the model. ## Training Details

- GPU: Tesla P100
- Time: ~5 hrs ### Training Data

The model was fine-tuned on the CVE single line fixes dataset ### Training Procedure 

Supervised Fine Tuning (SFT) #### Training Hyperparameters

- optim: adafactor
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- fp16: false ## Evaluation

The model was tested with the GitHub top 1000 projects vulnerabilities dataset
TAGS
#transformers #pytorch #gpt2 #text-generation #code #custom_code #dataset-lambdasec/cve-single-line-fixes #dataset-lambdasec/gh-top-1000-projects-vulns #license-apache-2.0 #model-index #autotrain_compatible #has_space #text-generation-inference #region-us 
# Model Card for SantaFixer



This is a LLM for code that is focussed on generating bug fixes using infilling.## Model Details### Model Description





- Developed by: codelion
- Model type: GPT-2
- Finetuned from model: bigcode/santacoder## How to Get Started with the Model

Use the code below to get started with the model.## Training Details

- GPU: Tesla P100
- Time: ~5 hrs### Training Data

The model was fine-tuned on the CVE single line fixes dataset### Training Procedure 

Supervised Fine Tuning (SFT)#### Training Hyperparameters

- optim: adafactor
- gradient_accumulation_steps: 4
- gradient_checkpointing: true
- fp16: false## Evaluation

The model was tested with the GitHub top 1000 projects vulnerabilities dataset