 In a previous video, we looked at light rag, which is a simple and fast retrieval augmented generation system that combines knowledge graphs with embedding based retrieval and is a great alternative to graph rack from Microsoft. They just added support for local models using all Lama and also added some visualizations of the graph. In this video, I'm going to show you how to set this up using Olamour. So everything is going to be running locally. Now I'll highly recommend to watch my previous video, which they have kindly linked in their main GitHub repo for better understanding of how light drag works. So the first step is going to be to clone this repo. In my case, the repo is already cloned. So I'm going to just use the git pull command to ensure that I have all the latest changes. Next, you need to create a virtual environment. In my case, I have created a virtual environment, which I'm calling lightrag. So we'll activate that virtual environment. And since there are changes to the GitHub repo, so we need to install all the new packages that are added. So if you want to build it from source, you're going to use the pip install dash e dot command. This will recreate the project from the source. Now to run the local models using Olama, make sure that you have Olama downloaded and installed on your local machine. Okay. If you go to the repo under examples, there is an example file called lightrag Olama demo. So in here you can provide your model name and also the embedding model that you want to use. By default, they're using the NOMID Embedged Text, which is a text embedding model from NOMICAI. But you can replace this with any embedding model that you want. Similarly, you can replace this LLM with any LLM that can be hosted. If you go to Olamour, you can search for models in here. In this video, we're going to be using the QIN2 model because that seems to be a pretty good model. Now for the embedding models, I will recommend to search for something like embed. And then you can see that there are a whole bunch of different embeddings that are available. And you will need to run both the embedding as well as the LLM through OLAMA for this to work. Now, one more thing to consider OLAMA limits the context window to 2048 tokens in order to preserve the VRAM that is needed. So we need to expand that context window because during the chunking and vector creation process, we need a much longer context window. Assuming that you have OLAMA running, we need a much longer context window. Assuming that you have Olama running, we need to start two different servers, one for the LLM and the other one for the embedding model. So we're going to start the server for the embedding model, which is easiest of the two. In that case, we will use the default NOMIC embedding. So just go to the Onama website and copy this command. Now in here, I'm going to paste the command and just run it. So this will download the model and load it. Next, we need to set up our LLM. Now you'll need to do a couple of additional steps. So assuming that we want to use the Quinn model. So so first I'm going to go here and copy this command. This will download the Quinn 2 model for you. The same process is going to work with any LLM that is available on Olama. In my case, the model is already downloaded. That's why it just loads the model. Now, the next step that we need to do is to change the context size or context window, and you can do it in two different ways. One is you type in Olamar run, Quinn two, and then once the model loads, you can set there, and then we are going to be setting the context window. You can type number of context and provide the new context window. So for example, if I want to set it to 32,000 tokens, that's how I'm going to set it. Now, if you type this in, it says that the parameter is set to 32,000 tokens, but I want to follow the official instructions in the light rack repo. And I highly recommend to do it because this probably is the easiest solution. Now the way we can change this within the configuration file is that we're going to ask Olama to show the model file for Qint2. So if you run this command, you are going to see that it will create another file within this folder. And this is basically the main configurations of the Qintu model that are stored through Olama. So I'll just go to this parameter section. And in here, we're going to add this new parameter, which is set the context window to 32,768 tokens. So just save this. And instead of overwriting the configurations for Quintu, we will create another model file. So this command will create a new model file. So this created a new model file for us. And now we can run this new model file using this new model that we created. So now I can do olama, then run Quinn to M, which is the modified model. And if you run this, this will load the new model with the updated context window. So now we need to go back to our example file that we want to run. In here, we're going to provide this new model name, which is going to be Quinn to M m that is the model we just created and here we are running our text embedding model okay another thing that you can check out is if you go to any terminal and just type olama ps that's basically show all the processes that are currently running so you can see that is currently running the quint 2, which is the modified model file that we created. And right now it's using the 100% GPU. So now we are all set. Again, we will need some sort of example data set to run this on. So based on the example, the repo has provided, I downloaded the same file, which is this textbook Christmas Carol by Charles Deakins. And we stored this as a book.txt. Now in my case, I deleted quite a bit of the text from this file, just to make sure that we're able to run the demo pretty quickly. But in your case, you will need to parse your PDF for HTML files and put them in plain text. And you can run the indexing process on the plain text. So just a quick recap of how this works. I already created this. I'm going to actually delete the index that was previously created. So first you have to provide the working directory. I am calling it deacons underscore ulama. Then if that directory does not exist it will create that directory for us now in terms of the configuration that we are using we are saying that look at the working directory so if there is no indexed or knowledge graph within that working directory it will start creating that for us then the we want to use the olama model so that's why olama model complete that's the. Then the we want to use the old armor models. So that's why all our model complete, that's the function, provide the name of the LLM that you want to use, which is going to M that we just created. And then the embedding model that you want to use. Now, I think you can mix and match. So for example, we can use a local LLM, but with the API based embedding model, that probably is going to work out of the box, but the configurations are going to be a little different. I'll probably show that in another video. Now, once we create this basic light rag object, then we load the text and pass that text to the rag system to index it. So it will create the knowledge graph. Plus the embeddings are going to be computed. I highly recommend to watch my previous video. If you haven't also, if you want to understand how rag system works, I have a course on advanced rack systems and rack concepts. We'll highly recommend to check that out. I'd probably create a more detailed video on light drag and include that in the course at some point. So do check it out. But then once we create the index, we can just run queries on the index. And the way you do it is you call this query function on your index. You provide the query. In this case, I'm asking is who is Mr. Scrooge and what is his relationship with other characters? Now it supports four different types of query mechanisms or query modes. I have discussed these in my previous videos, but you can run naive rag, then using the local search on the knowledge graph or using the global mode, or you can combine both the local and global mode. And depending on the type of the query, in some cases, this hybrid approach can be extremely beneficial anyways. So we have set this up now in order to run this, we need to run the light drag or limit demo file, which is under the example folder. So we're going to type Python examples and then light rag and Olama. Okay. So after running this, you're going to see a few things happens. So the first one is it will show you all the different parameters. The embedding model that we are using has an output, a vector of 768 different dimensions. The max number of tokens are here so seems like everything is correctly loaded we are using the correct model name plus the max number of tokens are set correctly as well so this is good to know now there was a single document that has 20 different chunks by default it's using 1200 tokens chunk. So it created all 20 vectors for those chunks. And then right now it's running the entity extraction process. So this can take a while depending on the length of your documents and you can provide multiple documents if you want. So first it ran the embeddings for each of the chunks. And now it's using the LLM to create those relationships. One more thing that you can do is you can go back to your terminal and then again, run the olama ps command. So it right now shows that two different processes are running. One is the LLM and the second one is the embedded model. The knowledge graph creation process is complete along with the indexing and it created 99 nodes with 31 edges for those 20 chunks. It took close to 10 minutes and I have seen the indexing process can take a variable amount of time for the same documents. So keep that in mind. And then we ran the same query through these four different modes. Now for the naive rag approach, it is I think taking the 20 chunks, but they are only using three chunks in this approach. And the result is pretty good. Not bad at all. So it talks about Mr. Scrooge and it says that he's a central character in the novel. Then he is a portrayed as a greedy and miserly businessman. Then the different relationships with different characters, they are highlighted in here. If you look at the local approach, which is basically using the knowledge graph we see the data is presented in a little different way so in here the query uses 60 entities 24 relationships within those three text units and i think these text units are very different than the text units. And I think these texts units are very different than the text chunks. So again, it talks about who Mr. Scrooge is, and then the relationships that this specific character has with some of the other characters. Now you see similar patterns for the other modes as well, but the number of entities differ based on whether you are using a global, if you were to add more data to it, it's a non destructive operation. So if you look at something like a graph rag, in that case, it will recreate the whole index. But in this case, you just need to call the insert function provide the text, and it will update the knowledge graph plus the index rather than evaluating everything. That is a good thing for a graph rag. Now let's change one of these questions. So I'm going to just say who is Bob. And if you run the same query or the same file again, it should take a lot less time because it's going to be just reusing the index that it already created. So it persists on the on the disk, and it's going to reuse that index for us. So I'm going to run this and then I'm going to show you how the index itself looks like. And then we were going to look at some visualization. So this is a real time process of how I'm running it. So it loaded that. And it says that the local query uses 60 entities, 26 relationships, three text units, depending on your hardware, it could take some time for running the actual query. Okay. So I would say it took close to 10 or 15 seconds, but he has the response. Bob is a character and he is portrayed as Mr. Scrooge clerk, highlighting him as someone who struggles financially and works diligently for Scrooge in his office. Right? So this is a quick response. We can see that the number of entities that it used was close to 60 with 26 different relationships. Now, if you go to the actual folder that it created, there is this graph file, which preserve the actual relationships between different entities. And we're going to look at a visualization of this. There are other files. For example, this is the LLM response cache. Then I think there is even like different chunks that it created, plus the entities that it found in the document, and then order the different relationships. So within the example folder, there are two other files that are, I think, interesting one is the graph visualization with HTML. So you can run this and this will visualize the graph. Now, in this case, I had to change the folder. So by default, the folder name is De in this case, I had to change the folder. So by default, the folder name is Deakins, but I change it to Deakins Olama because we created the index in this updated folder. So let me show you how to run this part for visualization. So we're going to type Python examples and then graph visualization. So we want to use the simple HTML one. There is also a new 4J, but in that case, you will need to create an account on the new 4J platform. So I just want to see a representation of the graph on a HTML file. So if you run this, you're going to see that it created a new knowledge underscore graph dot HTML file. So if you run this, you're going to see that it created a new knowledge underscore graph dot HTML file. We can open this file. Actually, I'll have to go to the main folder and then run that HTML file. So here's the folder. If we run it, now here is how the visualization looks like. We can zoom in. So in the center here, we have Scrooge and then the different relationships with other entities. So for example, Christmas Eve, Liberty, ghost, clerk, right. Then there is another one, another character. I am assuming there are some of these other entities, which don't really have any type of relationships. I was using a subset of the data. So it's not the complete data. And that's why probably not all of the relationships are preserved in here, where it's a pretty neat visualization, pretty decent overview of different entities and what their relationships are. This was how to use Olama. Now there are other demos. So for example, in here, if you look look at this file which gives you a demo of using the models available on bedrock you will just need to provide your model name that is served through bad amazon bedrock and then the embedding model that you want to use similarly you can run models available on hugging face directly without using ol Lama. Now I ran this example. The main issue with this specific part was that it was loading the model for each request. So for example, if it was trying to figure out entities in a chunk for each chunk, it would reload the model that takes forever. I hope that the team is going to look into this specific problem. There is another example for using other models through the API, which have, which are using the open AI compatible API endpoint. So potentially you can use something like Grok or Mistral in here. I tried to use Grok for this specific example. i was running into rate limits because it is making a lot of api calls there is also a demo on vram management because some of them these models can actually take quite a lot of your vram depending on how much context it's using from the llm so this is a project which is under active development but i have seen some really good progress in the last few days. Now, this was a quick demo of how you can use Olama with light rag. I haven't done extensive testing on how good this approach is compared to some of the other approaches, but based on the performance tables that they have shown, plus the results in their technical report, it does seems to outperform most of the other RAG approaches. So I think it's definitely worth checking out, especially it has an MIT license. So it's not too restrictive either. Let me know if you want me to make more videos on light RAG or any other topics that you are interested in. If you want to support my channel, there are quite a few options. Details are in a few options. Details are in the video description. Also make sure to subscribe to the channel. If you are interested in technical content related to rag agents and anything in between, if you want to get a better understanding of how light drag works, I created a previous video link is going to be at the top. Or if you are interested in the graph rag project, link to that video is going to be here. Thanks for watching. And as always, see you in the next one.
Local LightRAG: A GraphRAG Alternative but Fully Local with Ollama
In this video, we explore how to set up and run LightRAG—a retrieval augmented generation (RAG) system that combines knowledge graphs with embedding-based retrieval—locally using OLLAMA. This video provides a step-by-step guide on cloning the repo, configuring local models like the Qwen2 LLM, adjusting context windows, and visualizing knowledge graphs generated from example data such as "A Christmas Carol" by Charles Dickens. 


LINKS:
https://github.com/HKUDS/LightRAG
https://lightrag.github.io/
https://arxiv.org/pdf/2410.05779
https://microsoft.github.io/graphrag/
https://youtu.be/vX3A96_F3FU


💻 RAG Beyond Basics Course:
https://prompt-s-site.thinkific.com/courses/rag

Let's Connect: 
🦾 Discord: https://discord.com/invite/t4eYQRUcXB
☕ Buy me a Coffee: https://ko-fi.com/promptengineering
|🔴 Patreon: https://www.patreon.com/PromptEngineering
💼Consulting: https://calendly.com/engineerprompt/consulting-call
📧 Business Contact: engineerprompt@gmail.com
Become Member: http://tinyurl.com/y5h28s6h

💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off).  

Signup for Newsletter, localgpt:
https://tally.so/r/3y9bb0

TIMESTAMP
00:00 LightRAG with local models
01:38 Setup with Ollama
02:53 Serving Embeddings with Ollama
03:40 Changing the context window of the LLM
07:00 Configuring the Ingestion process
08:12 Advanced RAG Course
09:14 Indexing and Knowledge Graph Creation
10:45 Testing it out with local models


All Interesting Videos:
Everything LangChain: https://www.youtube.com/playlist?list=PLVEEucA9MYhOu89CX8H3MBZqayTbcCTMr

Everything LLM: https://youtube.com/playlist?list=PLVEEucA9MYhNF5-zeb4Iw2Nl1OKTH-Txw

Everything Midjourney: https://youtube.com/playlist?list=PLVEEucA9MYhMdrdHZtFeEebl20LPkaSmw

AI Image Generation: https://youtube.com/playlist?list=PLVEEucA9MYhPVgYazU5hx6emMXtargd4z