
You can run it on 4G mem GPU,quantize base QuIP# method, a weights-only quantization method that is able to achieve near fp16 performance using only 2 bits per weight.

url:https://github.com/Cornell-RelaxML/quip-sharp/tree/release20231203
{"license": "apache-2.0", "tags": ["2bit", "llama", "XVERSE"]}
transformers safetensors llama text-generation 2bit XVERSE license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us