Update README.md
Browse files
README.md
CHANGED
@@ -43,30 +43,15 @@ For more details about the training protocol of this model, please refer to the
|
|
43 |
|
44 |
# Usage
|
45 |
|
46 |
-
Currently to use this model you can
|
47 |
|
48 |
-
## 🤗 transformers
|
49 |
-
|
50 |
-
```python
|
51 |
-
import torch
|
52 |
-
from transformers import AutoModelForCausalLM, AutoTokenizer
|
53 |
-
|
54 |
-
model_id = "tiiuae/Falcon3-7B-Instruct-1.58bit"
|
55 |
-
|
56 |
-
model = AutoModelForCausalLM.from_pretrained(
|
57 |
-
model_id,
|
58 |
-
torch_dtype=torch.bfloat16,
|
59 |
-
).to("cuda")
|
60 |
-
|
61 |
-
# Perform text generation
|
62 |
-
```
|
63 |
|
64 |
## BitNet
|
65 |
|
66 |
```
|
67 |
git clone https://github.com/microsoft/BitNet && cd BitNet
|
68 |
pip install -r requirements.txt
|
69 |
-
|
70 |
python run_inference.py -m models/Falcon3-10B-1.58bit/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
|
71 |
```
|
72 |
|
|
|
43 |
|
44 |
# Usage
|
45 |
|
46 |
+
Currently to use this model you can rely on [BitNet](https://github.com/microsoft/BitNet) library. You can also play with the model using the [falcon-1.58bit playground](https://huggingface.co/spaces/tiiuae/falcon3-1.58bit-playground) (only for the 7B instruct version).
|
47 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
48 |
|
49 |
## BitNet
|
50 |
|
51 |
```
|
52 |
git clone https://github.com/microsoft/BitNet && cd BitNet
|
53 |
pip install -r requirements.txt
|
54 |
+
huggingface-cli download tiiuae/Falcon3-10B-Instruct-1.58bit-GGUF ggml-model-i2_s.gguf --local-dir models/Falcon3-10B-1.58bit/
|
55 |
python run_inference.py -m models/Falcon3-10B-1.58bit/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
|
56 |
```
|
57 |
|