|
--- |
|
library_name: transformers |
|
tags: |
|
- bitnet |
|
- falcon3 |
|
base_model: tiiuae/Falcon3-3B-Base |
|
--- |
|
|
|
%3C!-- HTML_TAG_END --> |
|
|
|
|
|
# Table of Contents |
|
|
|
0. [TL;DR](#TL;DR) |
|
1. [Model Details](#model-details) |
|
2. [Training Details](#training-details) |
|
3. [Usage](#usage) |
|
4. [Evaluation](#evaluation) |
|
5. [Citation](#citation) |
|
|
|
|
|
# TL;DR |
|
|
|
# Model Details |
|
|
|
## Model Description |
|
|
|
- **Developed by:** [https://www.tii.ae](https://www.tii.ae) |
|
- **Model type:** Causal decoder-only |
|
- **Architecture:** Pure-transformer - 1.58bit version |
|
- **Language(s) (NLP):** Mainly English |
|
- **License:** TII Falcon License 2.0 |
|
|
|
# Training details |
|
|
|
The model has been trained following the training strategies from the recent [1-bit LLM HF blogpost](https://huggingface.co/blog/1_58_llm_extreme_quantization) and [1-bit LLM paper](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf). |
|
For more details about the training protocol of this model, please refer to the Falcon-3 technical report, section *Compression*. |
|
|
|
|
|
# Usage |
|
|
|
Currently to use this model you can either rely on Hugging Face transformers library or [BitNet](https://github.com/microsoft/BitNet) library. You can also play with the model using the [falcon-1.58bit playground](https://huggingface.co/spaces/tiiuae/falcon3-1.58bit-playground) (only for the 7B instruct version). |
|
|
|
## 🤗 transformers |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model_id = "tiiuae/Falcon3-3B-Base-1.58bit" |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_id, |
|
torch_dtype=torch.bfloat16, |
|
).to("cuda") |
|
|
|
# Perform text generation |
|
``` |
|
|
|
## BitNet |
|
|
|
``` |
|
git clone https://github.com/microsoft/BitNet && cd BitNet |
|
pip install -r requirements.txt |
|
python setup_env.py --hf-repo tiiuae/Falcon3-3B-Base-1.58bit -q i2_s |
|
python run_inference.py -m models/Falcon3-3B-Base-1.58bit/ggml-model-i2_s.gguf -p "Hi how are you doing today?" -cnv |
|
``` |
|
|
|
# Evaluation |
|
We report in the following table our internal pipeline benchmarks: |
|
|
|
**Note evaluation results are normalized score from v2 leaderboard tasks - reported results of original models in the blogpost are raw scores** |
|
|
|
<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;"> |
|
<colgroup> |
|
<col style="width: 10%;"> |
|
<col style="width: 10%;"> |
|
<col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;"> |
|
</colgroup> |
|
<thead> |
|
<tr> |
|
<th>Benchmark</th> |
|
<th>Llama3-8B-1.58-100B-tokens</th> |
|
<th>Falcon3-7B-Instruct-1.58bit </th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<td>IFEval</td> |
|
<td>17.91</td> |
|
<td><b>27.49</b></td> |
|
</tr> |
|
<tr> |
|
<td>MUSR</td> |
|
<td><b>4.87</b></td> |
|
<td>4.64</td> |
|
</tr> |
|
<tr> |
|
<td>GPQA</td> |
|
<td><b>1.83<b></td> |
|
<td>0.00</td> |
|
</tr> |
|
<tr> |
|
<td>BBH</td> |
|
<td>5.36</td> |
|
<td><b>2.97</b></td> |
|
</tr> |
|
<tr> |
|
<td>MMLU-PRO</td> |
|
<td><b>2.78<b></td> |
|
<td><b>1.47</b></td> |
|
</tr> |
|
<tr> |
|
<td>MATH</td> |
|
<td>0.26</td> |
|
<td><b>0.43</b></td> |
|
</tr> |
|
<tr> |
|
<td>Average</td> |
|
<td>5.5</td> |
|
<td><b>6.17</b></td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
|
|
# Citation |
|
|
|
Coming soon .. |
