Llama-2-70b-chat-hf get worse result than Llama-2-70B-Chat-GPTQ
System Infotransformersversion: 4.36.0Platform: Linux-4.15.0-213-generic-x86_64-with-glibc2.27Python version: 3.9.18Huggingface_hub version: 0.19.4Safetensors version: 0.4.1Accelerate version: 0.25.0Accelerate config: not foundPyTorch version (GPU?): 2.1.1+cu121 (True)Tensorflow version (GPU?): not installed (NA)Flax version (CPU?/GPU?/TPU?): not installed (NA)Jax version: not installedJaxLib version: not installedUsing GPU in script?:Using distributed or parallel set-up in script?:Who can help?No responseInformationThe official example scripts My own modified scriptsTasksAn officially supported task in theexamplesfolder (such as GLUE/SQuAD, …) My own task or dataset (give details below)ReproductionI am trying to use Llama-2-70b-chat-hf as zero-shot text classifier for my datasets. Here is my setups.vLLM + Llama-2-70b-chat-hfI used vLLM as my inference engine as run it with:python api_server.py     --model /nas/lili/models_hf/70B-chat     --tensor-parallel-size 8api_server.py is theexample fileand I do not modify anything.client code:data = {
            "prompt": prompt,
            "use_beam_search": False,
            "n": 1,
            "temperature": 0.1,
            "max_tokens": 128,
        }
        res = _post(data)
        return eval(res.content)['text'][0].strip()And my prompt is:You will be provided with a product name. The product name will be delimited by 3 backticks, i.e.```. 
Classify the product into a primary category.

Primary categories: 
Clothing, Shoes & Jewelry
Automotive
Home & Kitchen
Beauty & Personal Care
Electronics
Sports & Outdoors
Patio, Lawn & Garden
Handmade Products
Grocery & Gourmet Food
Health & Household
Musical Instruments
Toys & Games
Baby Products
Pet Supplies
Tools & Home Improvement
Appliances
Office Products
Cell Phones & Accessories

Product name:```Cambkatl Men's Funny 3D Fake Abs T-Shirts Casual Short Sleeve Chest Graphic Printed Crewneck Novelty Pullover Tee Tops```.

Only answer the category name, no other words.The classification accuracy is 0.352. And I also tried to use the same prompt and parameter(temperature and max_token) to call chatgpt and gpt-4, the got 0.68 and 0.72 respectively.Llama 2 shouldn’t be significantly worse than ChatGPT. There must be something wrong with it. So I suspect it may be related to vLLM. So I tried the following method.Transformer + flaskIt’s not a good serving method, maybe I should use tgi. But I think it’s easy for locating problem.from transformers import LlamaTokenizer, LlamaForCausalLM

tokenizer_path = "/nas/lili/models_hf/70B-chat-hf/"
model_path = "/nas/lili/models_hf/70B-chat-hf/"

tokenizer = LlamaTokenizer.from_pretrained(tokenizer_path)

model = LlamaForCausalLM.from_pretrained(
        model_path,
        #load_in_8bit=True,
        #torch_dtype=torch.float16,
        device_map="auto",
    )


from flask import Flask, request, jsonify
from flask_cors import CORS
from transformers.generation import GenerationConfig

app = Flask(__name__)
CORS(app)

@app.route('/generate', methods=['POST'])
def generate():
    json = request.get_json(force=True)
    prompt = json['prompt']
    num_beams = json.get('num_beams')
    temperature = json.get('temperature')
    max_tokens = json.get('max_tokens')
    do_sample = json.get('do_sample')
    top_k = json.get('top_k') or 10
    model_inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
    cfg = GenerationConfig(
        num_beams = num_beams,
        max_new_tokens = max_tokens,
        temperature = temperature,
        do_sample = do_sample,
        top_k = top_k
    )
    output = model.generate(**model_inputs, generation_config=cfg, pad_token_id=tokenizer.eos_token_id)
    input_length = model_inputs["input_ids"].shape[1]
 
    output = tokenizer.decode(output[0][input_length:], skip_special_tokens=True)
    output = output.strip()

    return jsonify({'text': [output]})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)And the client code:data = {
            "prompt": prompt,
            "do_sample": True, 
            "temperature": 0.1,
            "max_tokens": 128,
            "num_beams":5
        }
        res = _post(data, url=self.url)
        return eval(res.content)['text'][0].strip()This time I used a large num_beams=5(I should use 1 but I made a mistake)I used the same prompt as before. And the accuracy is 0.368. It’s not much better than using vLLM(the gain may from large num_beams).Now it seems there is not the problem of vLLM. What’s wrong with it? Is Llama 2 70b a very bad model? I don’t think so. So I tried the 3rd method.Transformer(using Llama-2-70B-Chat-GPTQ ) + flaskThe setup is the same as method 2, I only change model:tokenizer_path = "/nas/lili/models_hf/7B-chat/"
model_path = "/nas/lili/models_hf/Llama-2-70B-chat-GPTQ/"I saved Llama-2-70B-chat-GPTQ by saved_pretrained and forget saved the tokenizer, So I use the tokenizer of Llama2 7B-chat(I think all Llama 2 tokenizer is the same for different mode size). This time I got a better result of 0.56. It’s not good as chatgpt but is significant better than uncompressed Llama-2-70B-chat.So I am confused that original Llama-2-70B-chat is 20% worse than Llama-2-70B-chat-GPTQ. Method 2 and Method 3 are exactly the same except for different model.Expected behaviorLlama 2 70b got a similar or better result than Llama-2-70B-chat-GPTQ.