How many user requests can spaces process in parrallel?
Sorry if I missed this in the documentation.Are user requests processed in parallel or sequential?Many thanks in advance for your time.
2023-04-18T17:57:15Z
Hi@harrycoppockHere are some key points to consider:Firstly, the request capacity will depend on the hardware resources behind the server.Secondly, the processing of requests can be influenced by the chosen SDK.If you’re using Gradio, it utilizes uvicorn and FastAPI as the web server, which can queue requests for your prediction function. Gradio also provides a way to batch requests for better performance:Setting Up A Demo For Maximum PerformanceWhen using Streamlit, you can learn more about how it handles requests here:Does streamlit is running on a single-threaded development server by default or not? - #2 by tim - 🚀 Deployment - StreamlitIf you’re employing the Docker SDK, it’s up to you to set up a web server and manage multiple requests.In summary, the processing of user requests will depend on the hardware resources and web server implementation.image1702×1290 89.2 KB
2023-04-19T15:08:57Z
Hi@radamesthank you for the speedy response - this is super useful. Wow an amazing service. Out of interest, why do you call them demos as it appears you can scale to many thousands of users?
2023-04-20T00:12:36Z
hi@harrycoppockfor production inference we offer a more robust, autoscaling and dedicated infrastructure manage by us.huggingface.coInference Endpoints - Hugging FaceTransformers in production: solvedhuggingface.co🤗 Inference EndpointsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2023-09-03T20:06:10Z
Hi@radames– Has spaces replicas been discontinued? I’m a pro user and I don’t have access to this. Did you mean the enterprise clients? Also, how does this work with the persistent storages? Do replicas share a common volume? All the best!
2023-09-05T04:27:10Z
hi@akgunomerfaruk, please contactapi-enterprise@huggingface.cofor individual scaling requests, and the persistent storage is shared across replicas. Thanks for your interest .
2023-09-06T04:05:50Z
Thank you@radamesfor explanations. One more dumb question from me, the price will increase linearly with the number of GPUs that are added? For example if I need one more GPU because I have (predefined) too many request it will just add the price of that GPU and no extra cost?
2023-09-08T00:21:12Z
sorry,@coralexbadeacould you also please reach out toapi-enterprise@huggingface.co? Thanks