|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
base_model: |
|
- NovaSky-AI/Sky-T1-32B-Preview |
|
--- |
|
|
|
Bitsandbytes quantization of https://huggingface.co/NovaSky-AI/Sky-T1-32B-Preview. |
|
|
|
See https://huggingface.co/blog/4bit-transformers-bitsandbytes for instructions. |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from transformers import BitsAndBytesConfig |
|
import torch |
|
|
|
# Define the 4-bit configuration |
|
nf4_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_compute_dtype=torch.bfloat16 |
|
) |
|
|
|
# Load the pre-trained model with the 4-bit quantization configuration |
|
model = AutoModelForCausalLM.from_pretrained("NovaSky-AI/Sky-T1-32B-Preview", quantization_config=nf4_config) |
|
|
|
# Load the tokenizer associated with the model |
|
tokenizer = AutoTokenizer.from_pretrained("NovaSky-AI/Sky-T1-32B-Preview") |
|
|
|
# Push the model and tokenizer to the Hugging Face hub |
|
model.push_to_hub("onekq-ai/Sky-T1-32B-Preview-bnb-4bit", use_auth_token=True) |
|
tokenizer.push_to_hub("onekq-ai/Sky-T1-32B-Preview-bnb-4bit", use_auth_token=True) |
|
``` |
