Hugging Face's logo

Spaces:
yusufs
/
vllm-inference
Paused

App Files
vllm-inference

Commit History

feat(runner.sh): using MODEL_ID only
490e6a3

yusufs commited on

feat(runner.sh): using runner.sh to select llm in the run time
69c6372

yusufs commited on

feat(seed): Random seed for reproducibility.
d4b0956

yusufs commited on

feat(/app/run-llama.sh): /app/run-llama.sh
cab183f

yusufs commited on

feat(/app/run-sailor.sh): /app/run-sailor.sh
6d92442

yusufs commited on

feat(llama3.2): using llama model first for cost saving, until we want test sailor
92a4a4a

yusufs commited on

docs(sailor): add not about minimum resources of sailor
6dac0d0

yusufs commited on

feat(sailorchat): using sailor chat model
0f3cd25

yusufs commited on

feat(quantization): T4 not support bfloat16
0345d26

yusufs commited on

feat(llama3.2): run llama3.2 using bfloat16 with cache dtype fp8 with same model len
38d356a

yusufs commited on

feat(sail/Sailor-4B-Chat): try increase gpu-memory-utilization to 0.9 before changing the token length
4a9e328

yusufs commited on

feat(sailor-8B): using sailor-8b
811d851

yusufs commited on

feat(llama3.2): using Llama-3.2-3B-Instruct 0cb88a4f764b7a12671c53f0838cd831a0843b95
8b37c20

yusufs commited on

feat(llama3.2): change model to llama3.2
b826155

yusufs commited on

feat(dep_sizes.txt): removes dep_sizes.txt during build, it not needed
8e49b3b

yusufs commited on

feat(download_model.py): remove download_model.py during build, it causing big image size
c360fd3

yusufs commited on

docs(Dockerfile): add comment about estimated image size after compile
8dc2050

yusufs commited on

feat(add-model): always download model during build, it will be cached in the consecutive builds
8679a35

yusufs commited on

feat(reduce-max-num-batched-tokens): Reducing max-num-batched-tokens even the error state it want to reduce max_model_len
13a5c22

yusufs commited on

feat(hf_token): set hf token during build
493a5f1

yusufs commited on

fix(hf_token): export HF_TOKEN during build
c6efe6a

yusufs commited on

feat(download-model): add download model at runtime
fc30f26

yusufs commited on

feat(change-model): change to sail/Sailor-4B-Chat 89a866a7041e6ec023dd462adeca8e28dd53c83e
d90e4d6

yusufs commited on

feat(endpoint): add prefix /api on each endpoint
5f3bf21

yusufs commited on

feat(refactor): move the files to root
7935381

yusufs commited on

fix(run.sh): move files
6479dc6

yusufs commited on

feat(run.sh): add script for running openai server
ded2af7

yusufs commited on

fix(python): fix absolute path of python script
d2e0be1

yusufs commited on

fix(cmd): fix 'error: failed to solve: dockerfile parse error on line 19: unknown instruction: "python3",'
de6b236

yusufs commited on

feat(openai): VLLM OpenAI compatible server
147b3a2

yusufs commited on

feat(parse): parse output
b44271e

yusufs commited on

feat(response): should dict only
b41be20

yusufs commited on

feat(one-model): one model at a time
35decf8

yusufs commited on

fix(remove): use_cached_output is not an option
6b1968a

yusufs commited on

feat(max_model_len): reducing max_model_len for T4 support
c41cdb4

yusufs commited on

fix(half-precision): use half precision for T4
d51e450

yusufs commited on

fix(tensor_parallel_size): set to 1
84c6c4a

yusufs commited on

feat(cuda): add cuda information
2457cd7

yusufs commited on

fix(remove-params): Removing max_model_len
0ef012d

yusufs commited on

feat(sailor-chat): add sail/Sailor-4B-Chat with the same context length
586265c

yusufs commited on

feat(reduce-max-length): reduce maximum length
2425953

yusufs commited on

feat(t4-gpu): add t4 gpu capability
4998ce7

yusufs commited on

fix(expose-port): add EXPOSE in Dockerfile
6d19ece

yusufs commited on

fix(module): fix 'error module app' it should be 'main'
6a914f2

yusufs commited on

feat(first-commit): follow examples and tutorials
ae7cfbb

yusufs commited on

initial commit
1a7087e
verified

yusufs commited on