Can i create endpoint using quantized model?
hello everyone, I’m trying to create endpoint using quantied model ''TheBloke/llama-2-7B-chat-GPTQ" but facing errors.So I’m wondering if i could create endpoint using it!
2023-09-26T11:53:16Z
What errors are you facing? Are you selection “gptq” in the advanced section?
2023-10-31T16:12:47Z
Our organization is also testing out HF inference endpoints.It’s not totally clear from the documentation how to run quantized models.For example if I select the modelmistral-7b-instruct-v0-1and select GPTQ in the container configuration, will it work? I just see errors in the logs and failings to update.And what if I choose a quantized model to begin with such asTheBloke/Mistral-7B-Instruct-v0.1-GPTQorTheBloke/Mistral-7B-OpenOrca-AWQ, what do I select in the containter config for quantization? I assume None.In either case I have not been able to successfully get an endpoint to work yet with a quantized model.Like it’s not clear whether “GPTQ in the container configuration” is saying quantize this model or if its saying I chose a quantized model so make sure to check this box.
2024-01-16T17:59:18Z
Were you able to get an answer for this?