|
--- |
|
license: llama2 |
|
datasets: |
|
- AlfredPros/smart-contracts-instructions |
|
language: |
|
- en |
|
tags: |
|
- code |
|
- blockchain |
|
- solidity |
|
- smart contract |
|
--- |
|
# Code LLaMA 7B Instruct Solidity |
|
|
|
A finetuned 7 billion parameters Code LLaMA - Instruct model to generate Solidity smart contract using 4-bit QLoRA finetuning provided by PEFT library. |
|
|
|
# Training Dataset |
|
|
|
Dataset used to finetune the model is AlfredPros' Smart Contracts Instructions (https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions). |
|
A dataset containing 6,003 GPT-generated human instruction and Solidity source code data pairs. This dataset has been processed for training LLMs. |
|
|
|
# Training Parameters |
|
|
|
## Bitsandbytes quantization configurations |
|
- Load in 4-bit: true |
|
- 4-bit quantization type: NF4 |
|
- 4-bit compute dtype: float16 |
|
- 4-bit use double quantization: true |
|
|
|
## Supervised finetuning trainer parameters |
|
- Number of train epochs: 1 |
|
- FP16: true |
|
- FP16 option level: O1 |
|
- BF16: false |
|
- Per device train batch size: 1 |
|
- Gradient accumulation steps: 1 |
|
- Gradient checkpointing: true |
|
- Max gradient normal: 0.3 |
|
- Learning rate: 2e-4 |
|
- Weight decay: 0.001 |
|
- Optimizer: paged AdamW 32-bit |
|
- Learning rate scheduler type: cosine |
|
- Warmup ratio: 0.03 |
|
|
|
# Training Details |
|
- GPU used: 1x NVIDIA GeForce GTX 1080Ti |
|
- Training time: 21 hours, 4 minutes, and 57 seconds |
|
|
|
# Training Loss |
|
``` |
|
Step Training Loss |
|
100 0.330900 |
|
200 0.293000 |
|
300 0.276500 |
|
400 0.290900 |
|
500 0.306100 |
|
600 0.302600 |
|
700 0.337200 |
|
800 0.295000 |
|
900 0.297800 |
|
1000 0.299500 |
|
1100 0.268900 |
|
1200 0.257800 |
|
1300 0.264100 |
|
1400 0.294400 |
|
1500 0.293900 |
|
1600 0.287600 |
|
1700 0.281200 |
|
1800 0.273400 |
|
1900 0.266600 |
|
2000 0.227500 |
|
2100 0.261600 |
|
2200 0.275700 |
|
2300 0.290100 |
|
2400 0.290900 |
|
2500 0.316200 |
|
2600 0.296500 |
|
2700 0.291400 |
|
2800 0.253300 |
|
2900 0.321500 |
|
3000 0.269500 |
|
3100 0.295600 |
|
3200 0.265800 |
|
3300 0.262800 |
|
3400 0.274900 |
|
3500 0.259800 |
|
3600 0.226300 |
|
3700 0.325700 |
|
3800 0.249000 |
|
3900 0.237200 |
|
4000 0.251400 |
|
4100 0.247000 |
|
4200 0.278700 |
|
4300 0.264000 |
|
4400 0.245000 |
|
4500 0.235900 |
|
4600 0.240400 |
|
4700 0.235200 |
|
4800 0.220300 |
|
4900 0.202700 |
|
5000 0.240500 |
|
5100 0.258500 |
|
5200 0.236300 |
|
5300 0.267500 |
|
5400 0.236700 |
|
5500 0.265900 |
|
5600 0.244900 |
|
5700 0.297900 |
|
5800 0.281200 |
|
5900 0.313800 |
|
6000 0.249800 |
|
6003 0.271939 |
|
``` |
|
|
|
# Example Usage |
|
```py |
|
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM |
|
import torch |
|
import accelerate |
|
|
|
use_4bit = True |
|
bnb_4bit_compute_dtype = "float16" |
|
bnb_4bit_quant_type = "nf4" |
|
use_double_nested_quant = True |
|
compute_dtype = getattr(torch, bnb_4bit_compute_dtype) |
|
|
|
# BitsAndBytesConfig 4-bit config |
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=use_4bit, |
|
bnb_4bit_use_double_quant=use_double_nested_quant, |
|
bnb_4bit_quant_type=bnb_4bit_quant_type, |
|
bnb_4bit_compute_dtype=compute_dtype, |
|
load_in_8bit_fp32_cpu_offload=True |
|
) |
|
|
|
# Load model in 4-bit |
|
tokenizer = AutoTokenizer.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity") |
|
model = AutoModelForCausalLM.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity", quantization_config=bnb_config, device_map="balanced_low_0") |
|
|
|
# Make input |
|
input='Make a smart contract to create a whitelist of approved wallets. The purpose of this contract is to allow the DAO (Decentralized Autonomous Organization) to approve or revoke certain wallets, and also set a checker address for additional validation if needed. The current owner address can be changed by the current owner.' |
|
|
|
# Make prompt template |
|
prompt = f"""### Instruction: |
|
Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task: |
|
|
|
### Task: |
|
{input} |
|
|
|
### Solution: |
|
""" |
|
|
|
# Tokenize the input |
|
input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda() |
|
# Run the model to infere an output |
|
outputs = model.generate(input_ids=input_ids, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.001, pad_token_id=1) |
|
|
|
# Detokenize and display the generated output |
|
print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):]) |
|
``` |
