
# ToRoLaMa-7B-GGUF
- Author: [All By AI](https://huggingface.co/allbyai)
- Original model: [ToRoLaMa 7b v1.0](https://huggingface.co/allbyai/ToRoLaMa-7b-v1.0)

## Description

GGUF format model files for [ToRoLaMa 7b v1.0](https://huggingface.co/allbyai/ToRoLaMa-7b-v1.0).

## Prompt template: (Chat Format)

```markdown
Cuộc hội thoại giữa người dùng và một trí thông minh nhân tạo. Đưa ra câu trả lời chính xác, giúp ích cho người dùng.

USER: chào
ASSISTANT: Xin chào! Tôi có thể giúp gì cho bạn hôm nay?</s>
USER:...
```

## Quantisation methods (using llama.cpp b1796)

- GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
- GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
- GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
- GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
- GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw

## How to use GGUF files

**Note:** Don't clone the entire repo! Only pick and download a single file.

Automatically download models using:

* LM Studio
* LoLLMS Web UI
* Faraday.dev

{"language": ["vi", "en"], "license": "llama2", "model_name": "ToRoLaMa-7b-v1.0", "base_model": "allbyai/ToRoLaMa-7b-v1.0", "inference": false, "model_creator": "All By AI", "model_type": "llama", "quantized_by": "ndthai"}
transformers gguf llama vi en base_model:allbyai/ToRoLaMa-7b-v1.0 license:llama2 text-generation-inference region:us