File size: 9,266 Bytes
cfa3f9d 48f06f2 cfa3f9d 48f06f2 847af4a 48f06f2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
base_model: utter-project/EuroLLM-9B-Instruct
---
This is a quantization of the [EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct).
The EuroLLM project has the goal of creating a suite of LLMs capable of understanding and generating text in all European Union languages as well as some additional relevant languages. EuroLLM-9B is a 9B parameter model trained on 4 trillion tokens divided across the considered languages and several data sources: Web data, parallel data (en-xx and xx-en), and high-quality datasets. EuroLLM-9B-Instruct was further instruction tuned on EuroBlocks, an instruction tuning dataset with focus on general instruction-following and machine translation.
## Evaluations
This model provides an accuracy recovery of 99.61%.
| __English__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
|:--------------|:--------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------|
| Avg. | 66.35 | 65.35 |
| Arc | 63.3 | 61.7 |
| Hellaswag | 69.4 | 69.0 |
| | | |
| __French__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg. | 61.67 | 61.3 |
| Arc | 58.1 | 57.3 |
| Hellaswag | 70.2 | 70.3 |
| MMLU | 56.7 | 56.3 |
| | | |
| __German__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg. | 60.0 | 60.37 |
| Arc | 57.2 | 56.7 |
| Hellaswag | 66.3 | 67.1 |
| MMLU | 56.5 | 57.3 |
| | | |
| __Italian__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg. | 61.8 | 61.7 |
| Arc | 58.3 | 58.2 |
| Hellaswag | 69.9 | 69.4 |
| MMLU | 57.2 | 57.5 |
| | | |
| __Portuguese__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg. | 61.47 | 61.37 |
| Arc | 59.1 | 59.3 |
| Hellaswag | 70.3 | 70.2 |
| MMLU | 55.0 | 54.6 |
| | | |
| __Spanish__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg. | 62.03 | 61.53 |
| Arc | 59.7 | 59.3 |
| Hellaswag | 71.4 | 71 |
| MMLU | 55 | 54.3 |
We did not check for data contamination.
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`.
## Usage
Install **vLLM** and
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
```
python -m vllm.entrypoints.openai.api_server --model cortecs/EuroLLM-9B-Instruct-FP8-Dynamic --gpu-memory-util 0.95
```
Access the model:
```
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' {
"model": "cortecs/EuroLLM-9B-Instruct-FP8-Dynamic",
"prompt": "San Francisco is a"
} '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**1976 tokens per second** using one NVIDIA L4 ⚡ |