Update README.md

README.md

@@ -3,4 +3,112 @@ license: llama2
 pipeline_tag: text-generation
 tags:
 - text-generation-inference
----
+---
+
+<div align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/64ccdc322e592905f922a06e/VhwQtaklohkUXFWkjA-3M.png" width="450"/>
+
+English | [简体中文](README_zh-CN.md)
+
+</div>
+
+<p align="center">
+    👋 join us on <a href="https://twitter.com/intern_lm" target="_blank">Twitter</a>, <a href="https://discord.gg/xa29JuW87d" target="_blank">Discord</a> and <a href="https://r.vansin.top/?r=internwx" target="_blank">WeChat</a>
+</p>
+
+
+# W4A16 LLM Model Deployment
+
+LMDeploy supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80.
+
+Before proceeding with the inference, please ensure that lmdeploy(>=v0.0.4) is installed.
+
+```shell
+pip install lmdeploy
+```
+
+## 4-bit LLM model Inference
+
+You can download the pre-quantized 4-bit weight models from LMDeploy's [model zoo](https://huggingface.co/lmdeploy) and conduct inference using the following command.
+
+Alternatively, you can quantize 16-bit weights to 4-bit weights following the ["4-bit Weight Quantization"](#4-bit-weight-quantization) section, and then perform inference as per the below instructions.
+
+Take the 4-bit Llama-2-7B model from the model zoo as an example:
+
+```shell
+git-lfs install
+git clone https://huggingface.co/lmdeploy/llama2-chat-7b-w4
+```
+
+As demonstrated in the command below, first convert the model's layout using `turbomind.deploy`, and then you can interact with the AI assistant in the terminal
+
+```shell
+
+## Convert the model's layout and store it in the default path, ./workspace.
+python3 -m lmdeploy.serve.turbomind.deploy \
+    --model-name llama2 \
+    --model-path ./llama2-chat-7b-w4 \
+    --model-format awq \
+    --group-size 128
+
+## inference
+python3 -m lmdeploy.turbomind.chat ./workspace
+```
+
+## Serve with gradio
+
+If you wish to interact with the model via web ui, please initiate the gradio server as indicated below:
+
+```shell
+python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} ----server_port {port}
+```
+
+Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model
+
+## Inference Performance
+
+We benchmarked the Llama 2 7B and 13B with 4-bit quantization on NVIDIA GeForce RTX 4090 using [profile_generation.py](https://github.com/InternLM/lmdeploy/blob/main/benchmark/profile_generation.py). And we measure the token generation throughput (tokens/s) by setting a single prompt token and generating 512 tokens. All the results are measured for single batch inference.
+
+| model       | llm-awq | mlc-llm | turbomind |
+| ----------- | ------- | ------- | --------- |
+| Llama 2 7B  | 112.9   | 159.4   | 206.4     |
+| Llama 2 13B | N/A     | 90.7    | 115.8     |
+
+```shell
+python benchmark/profile_generation.py \
+  ./workspace \
+  --concurrency 1 --input_seqlen 1 --output_seqlen 512
+```
+
+## 4-bit Weight Quantization
+
+It includes two steps:
+
+- generate quantization parameter
+- quantize model according to the parameter
+
+### Step 1: Generate Quantization Parameter
+
+```shell
+python3 -m lmdeploy.lite.apis.calibrate \
+  --model $HF_MODEL \
+  --calib_dataset 'c4' \             # Calibration dataset, supports c4, ptb, wikitext2, pileval
+  --calib_samples 128 \              # Number of samples in the calibration set, if memory is insufficient, you can appropriately reduce this
+  --calib_seqlen 2048 \              # Length of a single piece of text, if memory is insufficient, you can appropriately reduce this
+  --work_dir $WORK_DIR \             # Folder storing Pytorch format quantization statistics parameters and post-quantization weight
+```
+
+### Step2: Quantize Weights
+
+LMDeploy employs AWQ algorithm for model weight quantization.
+
+```shell
+python3 -m lmdeploy.lite.apis.auto_awq \
+  --model $HF_MODEL \
+  --w_bits 4 \                       # Bit number for weight quantization
+  --w_sym False \                    # Whether to use symmetric quantization for weights
+  --w_group_size 128 \               # Group size for weight quantization statistics
+  --work_dir $WORK_DIR \             # Directory saving quantization parameters from Step 1
+```
+
+After the quantization is complete, the quantized model is saved to `$WORK_DIR`. Then you can proceed with model inference according to the instructions in the ["4-Bit Weight Model Inference"](#4-bit-llm-model-inference) section.