Serving AWQ models without a custom container
Is it possible to serve AWQ models using huggingface’s inference endpoints without using a custom container?
2023-11-13T13:18:52Z
Hi@p-christ! Good news–AWQ has been added as a quantization option so you can now use it with Inference Endpoints
2023-11-13T13:55:05Z
oh great thanks a LOT!Do you also know if there’s an easy way of using vLLM with inference endpoints?