Error while trying to host finetuned model on inference endpoint
Hi,I have model that was finetuned using unsloth, I’m trying to host the model on an inference endpoint, but i end up getting the following error when trying to initialise the model on a t4 GPU with 64 gb ram8:50.284772Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286334Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286333Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286883Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286915Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.688669Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.879857Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.886106Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:51.088917Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} Error: WebserverFailedhere is a link to the repositoryhuggingface.coSaran12/Blusapphire_AI · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Is it due to erroneous settings in the inference endpoint?, or was the repository uploaded missing something? Any help would be appreciated.
2024-05-21T15:59:04Z
Hi@Saran12Can you please try using this model with an A10G or A100 instance instead of the T4? Let us know if you continue experiencing issues!
2024-05-22T05:42:11Z
Yup it is working on A10G, thank you!