text
MLX RAG with GGUF Models
MLX RAG with GGUF Models
new_line
text
Minimal, clean code implementation of RAG with mlx inferencing for GGUF models.
Minimal, clean code implementation of RAG with mlx inferencing for GGUF models.
new_line
new_line
text
Code: 
Code: 
link
https://github.com/Jaykef/mlx-rag-gguf
https://github.com/Jaykef/mlx-rag-gguf
new_line
new_line
text
The code here builds on vegaluisjose's example, it has been optimized to support RAG-based inferencing for .gguf models. I am using BAAI/bge-small-en for the embedding model, tinyllama-1.1b-chat-v1.0.Q4_0.gguf as base model and the custom vector database script for indexing texts in a pdf file. Inference speeds can go up to ~413 tokens/sec for prompts and ~36 tokens/sec for generation on my M2 Air.
The code here builds on vegaluisjose's example, it has been optimized to support RAG-based inferencing for .gguf models. I am using BAAI/bge-small-en for the embedding model, tinyllama-1.1b-chat-v1.0.Q4_0.gguf as base model and the custom vector database script for indexing texts in a pdf file. Inference speeds can go up to ~413 tokens/sec for prompts and ~36 tokens/sec for generation on my M2 Air.
new_line
new_line
text
Queries make use of both .gguf (base model) and .npz (retrieval model) simultaneouly resulting in much higher inferencing speeds.
Queries make use of both .gguf (base model) and .npz (retrieval model) simultaneouly resulting in much higher inferencing speeds.
MLX RAG with GGUF Models
Minimal, clean code implementation of RAG with mlx inferencing for GGUF models.

Code: https://github.com/Jaykef/mlx-rag-gguf

The code here builds on vegaluisjose's example, it has been optimized to support RAG-based inferencing for .gguf models. I am using BAAI/bge-small-en for the embedding model, tinyllama-1.1b-chat-v1.0.Q4_0.gguf as base model and the custom vector database script for indexing texts in a pdf file. Inference speeds can go up to ~413 tokens/sec for prompts and ~36 tokens/sec for generation on my M2 Air.

Queries make use of both .gguf (base model) and .npz (retrieval model) simultaneouly resulting in much higher inferencing speeds.
❤️
samusenps