
# Quantized Qwen models for llama.cpp

Download a model:
```bash
huggingface-cli download Lemmih/Qwen-GGUF qwen-1_8b-chat-q4_0.gguf --local-dir .
```

Run the model (from the folder where you built llama.cpp):
```bash
./main --model qwen-1_8b-chat-q4_0.gguf -i -cml -f prompts/chat-with-qwen.txt
```

If in doubt, use the `chat` models rather than the non-chat models.

{"license": "other", "license_name": "tongyi-qianwen", "license_link": "https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT"}
gguf license:other region:us