Accelerated Inference API can't load a model on GPU
Hi, I am trying to run models via accelerated inference API using GPU.I already subscribed Community Pro plan in order to use GPU.The inference API works fine if I use CPU.However, the API cannot load models on GPU.I used the following code (where MY_API_TOKEN is replaced by my actual API token string):import requestsimport jsonAPI_TOKEN = ‘MY_API_TOKEN’API_URL = “https://api-inference.huggingface.co/models/Salesforce/codegen-350M-multi”headers = {“Authorization”: f"Bearer {API_TOKEN}"}payload = {‘inputs’: ‘def hello_world():’, ‘options’: {‘use_gpu’: True}}data = json.dumps(payload)response = requests.request(“POST”, API_URL, headers=headers, data=data)post_result = json.loads(response.content.decode(“utf-8”))print(post_result)Then I get the message below no matter how long I wait:{‘error’: ‘Model Salesforce/codegen-350M-multi is currently loading’, ‘estimated_time’: 31.894704818725586}What’s wrong with the accelerated inference API?How can I load models on GPU?
2022-08-29T16:23:17Z
cc@Narsil
2022-09-07T08:41:02Z
@nielsrThanks for your attention to this topic.I still can’t load a model on GPU.Meanwhile, I tried to use pinned models using CPU (since it looks like using CPU is fast enough once the model is loaded),but I think my pinned models are not preloaded becase I get the error message (‘error’: ‘…model is currently loading’) for the first inference.Could you look into these issues? (GPU and pinned models)
2022-09-07T15:27:39Z
Hi@kernelpanic! Thanks for reaching out and letting us know about the issue and error message. We’ll take a look to see what’s going on and follow-up soon.Thanks again!Michelle
2022-09-22T07:21:49Z
Hi@michellehbn. Today I received a bill including $120 for Pinned Models on CPU and $5 for Pinned Models on GPU, which I could never use as mentioned aboveCan I get a refund on these pinned models?
2022-09-22T07:29:21Z
Also, I get charged $0.02 for Accelerated Inference API on GPU.I don’t understand why this is included in the bill because(1) The description of accelerated inference API looks like there is no extra charge for GPU if I subscribe Community Pro plan, andsee “Accelerated inference for a number of supported models on CPU and GPU (GPU requires a Community Pro or Organization Lab plan)”  inOverview(2) I could not use the inference API on GPU as I mentioned in the first post.Could you explain why $0.02 for Accelerated Inference API on GPU is included in the bill?
2022-09-22T09:07:27Z
Hi@kernelpanic, Pricing for the Inference API is based on the number of characters going through the endpoint: you’ll have up to 30k input characters /month for free (on CPU). Pro plan users will have up to 1M input characters /month for text tasks and up to 2h of audio for audio tasks, then pay-as-you-go with $10/M characters on CPU, $50/M characters on GPU. When you pin a model on CPU it’s $1/day or on GPU it will be $5/day with one day started being one day due.Would you still like to us to continue working on loading ‘Salesforce/codegen-350M-multi’ on GPU?Sorry to hear about the pinned models you were unable to use, would you mind sending an email toapi-enterprise@huggingface.cowith the invoice or invoice number please? Thanks so much!
2022-09-22T10:30:52Z
Thank you so much for the kind reply@michellehbn.The current pricing page has no information about what you explained.I think it should be clearly described in the pricing page.(Please let me know if there is an official pricing page about this.)I would still like you to continue working on loading the model on GPU.The reason why I asked for a refund is because I couldn’t use it (not because I don’t need it…).I sent an email toapi-enterprise@huggingface.co.I will post an update as soon as I receive a response.Thanks again.
2022-09-22T12:35:26Z
Sure thing@kernelpanic! I haven’t seen the email come through just yet, do you mind sending to me atmichelle@huggingface.coas well please? Sorry for that, I’ll help follow this through.
2022-09-23T01:20:38Z
Hi@michellehbn, I sent another email tomichelle@huggingface.cowith the invoice number.Thanks for your support and help.
2022-10-25T13:25:20Z
Hi@kernelpanic! Thanks for bearing with us and sorry for the delay in follow-up!We’ve made some changes to the Inference API - this is now free for use, as a solution to easily explore and evaluate models, and Inference Endpoints is our new paid inference solution for production use cases! The free Inference API is subject to rate limiting for heavy use cases.For larger volumes of requests, or if you need guaranteed latency/performance, you can use our new solutionInference Endpointsto easilydeploy your modelson dedicated, fully-managed infrastructure. Inference Endpoints will give you the flexibility to quickly create endpoints on CPU or GPU resources, and is billed by compute uptime vs character usage. Further pricing information can be foundhere.We’re all ears for feedback so please let us know what you think! Thanks so much!
2022-10-26T01:40:38Z
Hi@michellehbn. Thank you very much for letting me know the update.It is amazing that you made the Inference API free for use.I will take a look at the new inference solution as well.Thank you and have a nice day!
2023-01-05T12:46:07Z
Hello@michellehbn,Is this still valid for Inference API: With Pro account for 1M character per month and if we hit this limit in thesame monthwe will pay $10 for each M for our usage?Or do we have to migrate Inference Endpoints?
2023-01-16T11:06:55Z
Hi@metkor! Happy new year! Sorry for my delay in response. The PRO subscription will give you higher Inference API rate limits than the free Inference API plan. The pricing mentioned earlier ($10/M characters, etc) is no longer valid. If you need guaranteed latency/performance, you should use our new solutionInference Endpointsto easilydeploy your modelson dedicated, fully-managed infrastructure, and is billed by compute uptime vs character usage. Further pricing information can be foundhere. Please let us know if there are any other questions! Thanks again!