Unable to Process Concurrent User Request
I am using thehttps://huggingface.co/ktrapeznikov/biobert_v1.1_pubmed_squad_v2/tree/main#for the question answering model. I am sending around 50 abstracts from Pubmed and then asking the question. It is working fine for 1 user , but when scale to 10 concurrent users the model is taking too long. Can anybody help
2020-12-01T09:19:50Z
There’s a number of avenues that you could use to reduce inference time:Scale your deployment vertically or horizontally.Move to a smaller model.Improve your preprocessing/postprocessing efficiency.