amd
/

Llama-3.2-11B-Vision-Instruct-FP8-KV

bowenbaoamd commited on 22 days ago

Commit

d544aec

verified ·

1 Parent(s): 4fba5f5

Upload README.md with huggingface_hub

Files changed (1) hide show

README.md CHANGED Viewed

@@ -24,7 +24,8 @@ python3 quantize_quark.py \
         --kv_cache_dtype fp8 \
         --num_calib_data 128 \
         --model_export quark_safetensors \
-        --no_weight_matrix_merge
 # If model size is too large for single GPU, please use multi GPU instead.
 python3 quantize_quark.py \
         --model_dir $MODEL_DIR \
@@ -34,7 +35,8 @@ python3 quantize_quark.py \
         --num_calib_data 128 \
         --model_export quark_safetensors \
         --no_weight_matrix_merge \
-        --multi_gpu
 ```
 ## Deployment
 Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).

         --kv_cache_dtype fp8 \
         --num_calib_data 128 \
         --model_export quark_safetensors \
+        --no_weight_matrix_merge \
+        --custom_mode fp8
 # If model size is too large for single GPU, please use multi GPU instead.
 python3 quantize_quark.py \
         --model_dir $MODEL_DIR \
         --num_calib_data 128 \
         --model_export quark_safetensors \
         --no_weight_matrix_merge \
+        --multi_gpu \
+        --custom_mode fp8
 ```
 ## Deployment
 Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).