text
Just tried LitServe from the good folks at 
Just tried LitServe from the good folks at 
mention
@LightningAI
LightningAI
text
!
!
new_line
new_line
text
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
new_line
new_line
text
That's where LitServe comes in!
That's where LitServe comes in!
new_line
new_line
text
LitServe is a high-throughput serving engine for AI models built on FastAPI.
LitServe is a high-throughput serving engine for AI models built on FastAPI.
new_line
new_line
text
Yes, built on FastAPI. That's where the advantage and the issue lie.
Yes, built on FastAPI. That's where the advantage and the issue lie.
new_line
new_line
text
It's extremely flexible and supports multi-modality and a variety of models out of the box.
It's extremely flexible and supports multi-modality and a variety of models out of the box.
new_line
new_line
text
But in my testing, it lags far behind in speed compared to vLLM.
But in my testing, it lags far behind in speed compared to vLLM.
new_line
new_line
text
Also, no OpenAI API-compatible endpoint is available as of now.
Also, no OpenAI API-compatible endpoint is available as of now.
new_line
new_line
text
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
new_line
new_line
text
GitHub: 
GitHub: 
link
https://github.com/Lightning-AI/LitServe
https://github.com/Lightning-AI/LitServe
Just tried LitServe from the good folks at @LightningAI!

Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!

That's where LitServe comes in!

LitServe is a high-throughput serving engine for AI models built on FastAPI.

Yes, built on FastAPI. That's where the advantage and the issue lie.

It's extremely flexible and supports multi-modality and a variety of models out of the box.

But in my testing, it lags far behind in speed compared to vLLM.

Also, no OpenAI API-compatible endpoint is available as of now.

But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...

GitHub: https://github.com/Lightning-AI/LitServe
👀
John6666 victor theQuert
🚀
Norod78