text
On-device AI framework ecosystem is blooming these days:
On-device AI framework ecosystem is blooming these days:
new_line
new_line
text
1. llama.cpp - All things Whisper, LLMs & VLMs - run across Metal, CUDA and other backends (AMD/ NPU etc)
1. llama.cpp - All things Whisper, LLMs & VLMs - run across Metal, CUDA and other backends (AMD/ NPU etc)
new_line
link
https://github.com/ggerganov/llama.cpp
https://github.com/ggerganov/llama.cpp
new_line
new_line
text
2. MLC - Deploy LLMs across platforms especially WebGPU (fastest WebGPU LLM implementation out there)
2. MLC - Deploy LLMs across platforms especially WebGPU (fastest WebGPU LLM implementation out there)
new_line
link
https://github.com/mlc-ai/web-llm
https://github.com/mlc-ai/web-llm
new_line
new_line
text
3. MLX - Arguably the fastest general purpose framework (Mac only) - Supports all major Image Generation (Flux, SDXL, etc), Transcription (Whisper), LLMs
3. MLX - Arguably the fastest general purpose framework (Mac only) - Supports all major Image Generation (Flux, SDXL, etc), Transcription (Whisper), LLMs
new_line
link
https://github.com/ml-explore/mlx-examples
https://github.com/ml-explore/mlx-examples
new_line
new_line
text
4. Candle - Cross-platform general purpose framework written in Rust - wide coverage across model categories
4. Candle - Cross-platform general purpose framework written in Rust - wide coverage across model categories
new_line
link
https://github.com/huggingface/candle
https://github.com/huggingface/candle
new_line
new_line
text
Honorable mentions:
Honorable mentions:
new_line
new_line
text
1. Transformers.js - Javascript (WebGPU) implementation built on top of ONNXruntimeweb
1. Transformers.js - Javascript (WebGPU) implementation built on top of ONNXruntimeweb
new_line
link
https://github.com/xenova/transformers.js
https://github.com/xenova/transformers.js
new_line
new_line
text
2. Mistral rs - Rust implementation for LLMs & VLMs, built on top of Candle
2. Mistral rs - Rust implementation for LLMs & VLMs, built on top of Candle
new_line
link
https://github.com/EricLBuehler/mistral.rs
https://github.com/EricLBuehler/mistral.rs
new_line
new_line
text
3. Ratchet - Cross platform, rust based WebGPU framework built for battle-tested deployments
3. Ratchet - Cross platform, rust based WebGPU framework built for battle-tested deployments
new_line
link
https://github.com/huggingface/ratchet
https://github.com/huggingface/ratchet
new_line
new_line
text
4. Zml - Cross platform, Zig based ML framework
4. Zml - Cross platform, Zig based ML framework
new_line
link
https://github.com/zml/zml
https://github.com/zml/zml
new_line
new_line
text
Looking forward to how the ecosystem would look 1 year from now - Quite bullish on the top 4 atm - but open source ecosystem changes quite a bit! 🤗
Looking forward to how the ecosystem would look 1 year from now - Quite bullish on the top 4 atm - but open source ecosystem changes quite a bit! 🤗
new_line
new_line
text
Also, which frameworks did I miss?
Also, which frameworks did I miss?
On-device AI framework ecosystem is blooming these days:

1. llama.cpp - All things Whisper, LLMs & VLMs - run across Metal, CUDA and other backends (AMD/ NPU etc)
https://github.com/ggerganov/llama.cpp

2. MLC - Deploy LLMs across platforms especially WebGPU (fastest WebGPU LLM implementation out there)
https://github.com/mlc-ai/web-llm

3. MLX - Arguably the fastest general purpose framework (Mac only) - Supports all major Image Generation (Flux, SDXL, etc), Transcription (Whisper), LLMs
https://github.com/ml-explore/mlx-examples

4. Candle - Cross-platform general purpose framework written in Rust - wide coverage across model categories
https://github.com/huggingface/candle

Honorable mentions:

1. Transformers.js - Javascript (WebGPU) implementation built on top of ONNXruntimeweb
https://github.com/xenova/transformers.js

2. Mistral rs - Rust implementation for LLMs & VLMs, built on top of Candle
https://github.com/EricLBuehler/mistral.rs

3. Ratchet - Cross platform, rust based WebGPU framework built for battle-tested deployments
https://github.com/huggingface/ratchet

4. Zml - Cross platform, Zig based ML framework
https://github.com/zml/zml

Looking forward to how the ecosystem would look 1 year from now - Quite bullish on the top 4 atm - but open source ecosystem changes quite a bit! 🤗

Also, which frameworks did I miss?
👍
nicolay-r Nymbo victor John6666 AtAndDev osanseviero umair894 adamelliotfields cfahlgren1 atasoglu
🔥
osanseviero cfahlgren1 ZeroWw
https://cdn-avatars.huggingface.co/v1/production/uploads/64b695dcd3df8086e5ed7c89/06Toh65jDEz3WJbIM6ZmZ.jpeg
Adam Fields
adamelliotfields
user