# Yi-6B-Infinity-Chat-HQQ
This is [wbag's Yi-6B-Infinity-Chat model](https://huggingface.co/Walmart-the-bag/Yi-6B-Infinity-Chat) quantized to 4bit HQQ

# Usage
To run this quantization, you can use the following code.

```bash
pip install git+https://github.com/mobiusml/hqq/ transformers -U
```
```python
model_id = 'HQQman/Yi-6B-Infinity-Chat-HQQ'

from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = HQQModelForCausalLM.from_quantized(model_id)

prompt = """
### Instruction: What is AI?
### Response:"""


input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda')

output = model.generate(input_ids=input_ids, max_length=50, num_return_sequences=1, do_sample=True, top_k=50)

generated_sequence = tokenizer.decode(output[0].cuda(), skip_special_tokens=True)
print(generated_sequence)
```

# Read About HQQ
https://mobiusml.github.io/hqq_blog/

{"license": "cc", "inference": false}
transformers llama text-generation conversational license:cc autotrain_compatible text-generation-inference region:us