Run vllm docker on space
I just try to run vLLM docker with space,but there is something weird.I can callhttps://xxx.hf.space/v1/modelsBut I got error when callhttps://xxx.hf.space/v1/chat/completionsthe error was“Server unavailable, error code: 349453”I try to find the error on google but found nothing.Thanks
2024-01-23T18:24:17Z
hi@sofianhw,Could please share a Space with Dockerfile example?
2024-01-24T05:21:51Z
I just add prefix,and now it works
2024-01-24T09:10:10Z
I have the same error.What do you mean by ‘I just add prefix’?Can you give an example?Thanx for your help
2024-01-24T17:29:11Z
I just copy all files in these folder:github.com//github.com/vllm-project/vllm/tree/main/vllm/entrypoints/openaiand change everything route in api_server.py that start (“/v1/xxx”) to (“/api/v1/xxx”).and just run the python api_server.py with arguments.
2024-01-24T18:21:12Z
Tx for your help !I’ve found another way:Add just ‘‘extra_query={‘id’: time.time_ns()}’’ on call params for client.chat.completions.create (with openai client lib 1.9.0)
2024-01-25T02:39:30Z
Cool,I will try also
2024-01-25T14:39:41Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.