hey folks, is there a plan to add support for embedding (retrieval) models in the future? I'm not sure I follow how to run them using NIM Serverless
Borys Zibrov
borowis
AI & ML interests
None yet
Recent Activity
replied to
MoritzLaurer's
post
21 days ago
The new NIM Serverless API by HF and Nvidia is a great option if you want a reliable API for open-weight LLMs like Llama-3.1-405B that are too expensive to run on your own hardware.
- It's pay-as-you-go, so it doesn't have rate limits like the standard HF Serverless API and you don't need to commit to hardware like for a dedicated endpoint.
- It works out-of-the box with the new v0.25 release of our huggingface_hub.InferenceClient
- It's specifically tailored to a small collection of popular open-weight models. For a broader selection of open models, we recommend using the standard HF Serverless API.
- Note that you need a token from an Enterprise Hub organization to use it.
Details in this blog post: https://huggingface.co/blog/inference-dgx-cloud
Compatible models in this HF collection: https://huggingface.co/collections/nvidia/nim-serverless-inference-api-66a3c6fcdcb5bbc6e975b508
Release notes with many more features of huggingface_hub==0.25.0: https://github.com/huggingface/huggingface_hub/releases/tag/v0.25.0
Copy-pasteable code in the first comment:
Organizations
None yet