Issue - ValueError: Unsupported model type mixtral
Trying to deploy the PreTrained version of the below model to SageMaker Endpoint.Model Name: mistralai/Mixtral-8x7B-Instruct-v0.1Using the g5.48xlarge instance for the same.But getting the below error:File “/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py”, line 124, in serve_innermodel = get_model(model_id, revision, sharded, quantize, trust_remote_code)File “/opt/conda/lib/python3.9/site-packages/text_generation_server/models/init.py”, line 314, in get_modelraise ValueError(f"Unsupported model type {model_type}")ValueError: Unsupported model type mixtral.Please find the below deployed commands for the same:from sagemaker.huggingface import get_huggingface_llm_image_uriretrieve the llm image urillm_image = get_huggingface_llm_image_uri(“huggingface”,version=“1.1.0”)print(f"llm image uri: {llm_image}")import jsonfrom sagemaker.huggingface import HuggingFaceModelsagemaker configinstance_type = “ml.g5.48xlarge”number_of_gpu = 1health_check_timeout = 1000Define Model and Endpoint configuration parameterconfig = {‘HF_MODEL_ID’: “mistralai/Mixtral-8x7B-v0.1”, # model_id fromModels - Hugging Face‘SM_NUM_GPUS’: json.dumps(number_of_gpu), # Number of GPU used per replica‘MAX_INPUT_LENGTH’: json.dumps(1024),  # Max length of input text‘MAX_TOTAL_TOKENS’: json.dumps(2048),  # Max length of the generation (including input text)}create HuggingFaceModel with the image urillm_model = HuggingFaceModel(role=role,env=config,transformers_version = ‘4.28’,            # the transformers version used in the training jobpytorch_version      = ‘2.0’,             # the pytorch_version version used in the training jobpy_version           = ‘py310’,)llm = llm_model.deploy(initial_instance_count=1,instance_type=instance_type,container_startup_health_check_timeout=health_check_timeout,endpoint_name = ‘mixtral-inference-testing1’)Please let me know if I am using the right versions or if I am doing anything wrong.Let me know if any other information is required.Thanks.
2023-12-28T16:11:07Z
On troubleshooting, found that the initial or above mentioned error was due to the transformer version. Earlier was using 4.34. Now changed it to 4.36.2After the change, getting the below error while the logic is trying to download the Model.**AlgorithmError: ExecuteUserScriptError: ExitCode 1 ErrorMessage "ImportError: /opt/conda/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c106SymIntltEl **training_function model = AutoModelForCausalLM.from_pretrained(script_args.model_id, File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 565, in from_pretrained model_class = _get_model_class(config, cls._model_mapping) File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 387, in _get_model_class supported_models = model_mapping[type(config)] File "/opt/conda/lib/python3.10/site-packagBut the same logic is working (the Model is getting downloaded and also I am able to do some training in the same) when I am running it outside the AWS Sagemaker.