Error while Trying to run inference using gaudi on a finetuned llama2 model using habana repo
image1946×330 54.4 KBUsing this repo heregithub.com//github.com/intel/intel-extension-for-transformers/tree/main/workflows/chatbot/inferencecreated a container image using the dockerfile mentioned in the instructionsran it on 2 computes : 1 HPU, 16 CPU and 32 GB Memory and 1 HPU, 50 CPU and 200 GB memory
2023-08-16T07:50:16Z
Hi@gildesh, could you share the command you used to run inference please?
2023-08-16T08:44:29Z
thanks for replying!running it as an endpoint with these params: -#model_config={“model_name”: “/root/.cache/intel/neural-chat-7b-v2”, “tokenizer_name”: “/root/.cache/intel/llama/neural-chat-7b-v2”, “device”: “hpu”, “use_hpu_graphs”: true, “peft_path”:“/input/finetune/output/peft_model”}MicrosoftTeams-image1440×767 46.1 KB
2023-08-16T10:45:48Z
Could you share thegenerate.pyfile that is used in this endpoint?
2023-08-16T11:10:15Z
github.comintel/intel-extension-for-transformers/blob/main/workflows/chatbot/inference/generate.pyimport argparse
import copy, time
from datetime import datetime
import torch
import re, os, logging
from threading import Thread
import contextlib
from typing import List
from transformers import (
    GenerationConfig,
    AutoModelForCausalLM,
    AutoModelForSeq2SeqLM,
    AutoTokenizer,
    AutoConfig,
    TextIteratorStreamer,
    StoppingCriteriaList,
    StoppingCriteria,
)

# Set necessary env variablesThis file has been truncated.show original
2023-08-16T12:35:17Z
Thanks!I don’t have access tointel/neural-chat-7b-v2, it’s not on the Hugging Face Hub it seems. Do you have aconfig.jsonfile somewhere? If yes, could you tell me the value of the fieldmodel_typeplease?For instance, forIntel/neural-chat-7b-v1-1, I see that the model is based on MPT:config.json · Intel/neural-chat-7b-v1-1 at main
2023-08-18T10:19:58Z
_name_or_path": “/models/llama-v2-latest-20230719/models_hf/Llama-2-7b”,“architectures”: [“LlamaForCausalLM”],“bos_token_id”: 1,“eos_token_id”: 2,“hidden_act”: “silu”,“hidden_size”: 4096,“initializer_range”: 0.02,“intermediate_size”: 11008,“max_position_embeddings”: 2048,“model_type”: “llama”,
2023-08-18T12:57:23Z
And which version of Optimum Habana do you use?
2023-08-21T04:26:55Z
actually, we probably just use latest as we source it from this repogithub.com//github.com/intel/intel-extension-for-transformers/tree/main/workflows/chatbot/fine_tuningjust give this in requirements.txtoptimum
2023-08-21T06:54:51Z
Could show me the output ofpip show optimum-habanaplease?