 All right, so let's go and get started. My name is Jonathan Larson and today I'm excited to come here today to talk to you about GraphRAG. GraphRAG is an LLM-derived knowledge graph for RAG, and just to give you a small teaser, what you see over here on the right-hand side is actually a picture of Kevin Scott's Behind the Tech podcast as an LLM-derived knowledge graph, an LLM memory representation. We'll be covering this in about the middle of the presentation here today, so I just wanted to give you, though, a teaser of what's to come. So what is GraphRag? GraphRag is really a two-step process. It is first an indexing process that's run over top of private data to create LLM-derived knowledge graphs. These knowledge graphs serve as a form of an LLM memory representation, which can then be used by subsequent steps to then do better retrieval, which leads us to the second step. The second part of it is an LLM orchestration mechanism that utilizes those pre-built indices that I just talked about, and then those indices can be used to construct much, much better, much more empowered RAG operations. And that really leads us to the key differentiators of what GraphRAG allows us to do. The first is it allows us to help enhance search relevancy. This is because it has a holistic view of the semantics across the entire data set. Second, it helps us enable new scenarios that would today require a very large context. For example, doing holistic data set analysis for trends, summarization, aggregation, things like that. So if you haven't had a chance to read it yet in the pre-reads, I'd encourage you to take a look at the blog post, take a look at the archive. They have a lot more technical details and measurements and evaluations in this space. But the one thing I want to ask of you today as we go through this presentation is how should we best drive impact with this technology. Okay, so let's go ahead and explore how GraphRag actually works. The way that we explain GraphRag is first to explain, of course, how baseline RAG works. So in baseline RAG, what you do is you take a private data set, you chunk it up using embeddings, and you start into a vector database. Then you perform nearest neighbor search, and you can use those nearest neighbor searches to augment the context window. GraphRag is a parallel process to the way that baseline rag works. What we do with this is we actually take the same text chunks and then we take those sentences that are being extracted and we ask the LLM to perform reasoning operations over top of each sentence in a single pass through over all of the data. So let's take this example sentence here. We have the POK leader, Sylvia Marek, who took the stage with Lucio Jacob, founder of Sea of Our Wildlands. Now, the major differentiation here is you can see there's some named entity recognition done over top of the sentence. And that's pretty typical for this type of text analysis. However, the major differentiation here is we're not just looking for the named entities, we're looking for the relationships between those entities and the strength of those relationships. And this is where GPT-4 really comes in to play a very strong leading role in the capability of this technology. For example, we can actually see that Sylvia Marek is very strong and really related to the POK because she is the leader. We can see that she is perhaps weakly related to Save Our Wildlands, because she is apparently taking the stage with its leader, but she's not the leader of that. And that's a major differentiation here, is that GPT-4 can understand the semantics of these relationships, and that allows us to create weighted graphs from those relationships that are far richer than just like co-occurrence networks, which is where traditional NER would typically take this type of problem. So once we create these knowledge graphs, let's say, for example, we took all these sentences across this data set, we create a knowledge graph here, what you get are a series of nodes that are connected to each other via these relationships. But that's not all we can do. Once we have the graph, we can utilize graph machine learning to do semantic aggregations and hierarchical agglomerations over top of those structures. So if you take this graph right here that we just created, which has no labeling on it, no colors, if you will, we can then create a labeling on that at one level, and then we can hierarchically create subpartitions and subpartitions until you get down to individual nodes. This allows us effectively a granular filter that allows us to ask questions at any level of granularity across the dataset for a semantic topic. Once that is built, we can then take that into a variety of different end use cases. Just to list out a couple, we could do dataset question generation, we could summarize Q&A, and a variety of other methods that we can talk about here later today. So let's go ahead and jump into some demonstrations here and show you this technology in action. So as I bring up the screen here, what you're gonna see here are three different columns. In addition, I need to explain the dataset that is behind each of these different columns that we're going to be analyzing. So we have three different RAG systems that are implemented over top of one data set. This data set is about 3,000 articles in total. It's articles that originate from both the Russian and the Ukrainian side of the war, of the conflict. And we're going to ask it a question to all three different RAG systems. We're going to ask it a question that to all three different RAG systems, we're going to ask it, what is Novorossiya and what are its targets? So it's actually two questions. Now, if you're not familiar with what Novorossiya is, Novorossiya are the Russian occupied portions of Ukraine. It's also a political movement. And then the what are its targets are looking for what Novorossiya might be looking to destroy is basically. We know that there's some information inside this data set. Let's see how we can actually use RAG based retrieval to find those particular targets. So using traditional lang chain based semantic search, this is just using baseline RAG right here, we can see it actually failed to answer either of the two questions. Sometimes it can answer the first part of the question okay, but today it decided it couldn't come up with anything. The second column here is a much improved version of RAG. This is a, as people who've been using RAG know, it oftentimes requires a lot of tuning, prompt engineering, and other improvements to actually make the RAG work more effectively. And we can see that here. So the left-hand side here is looking at baseline RAG. The middle column here is a much improved version. It's a supercharged version of RAG. Now if you read through this text right here, I'm just going to scroll through here real quick, you will actually see that it actually does an okay job of actually talking about the first part of this question, what is Novarosia? But it actually fails to mention anything about the specific targets that Novorossiya are looking to destroy. In comparison, over here on the left-hand side, I'm going to highlight this first paragraph. This first paragraph here actually addresses the what is Novorossiya part of the question. That's good. But the second paragraph right here gives us a list of very highly specific targets that Novorossiya was looking to destroy. For example, the national television company Ukraine, the radio station, the cannery, looks like also some private bank and Roshan properties, and they're also planning on terror attacks in the city of Odessa. And so this is exactly what we're looking for. These are the specifics that we were seeing recall failures for on the baseline RA reg operations that we performed in the left two columns. Now, when we perform GraphRag, one of the nice features about it is it also allows us to actually look at the underlying provenance. So you'll see inside the text here, it refers to relationships. We can actually take a look at those relationships. I'm going to move to a second tool here, which is this is a purely GraphRag tool. So I have two columns here. One is doing a local search. One is doing a community based search. We'll get into the details of what each of these two are later. Both of these answer the question correctly, but I'm going to focus on that question about Novorossiya has targeted several entities for destruction. Let's go and click into those relationships. And if we open up the raw text text we can get the English translation because I can't read Russian or Ukrainian and actually get to the originating text chunk that was used to make the specific claim made above right here. So this can really help aid in understanding if there are hallucinations being made in the system and detecting them and then also providing grounding and evidence which is critical for analysts trying to use this data for their production purposes. The other thing we can do in here, of course, as well, too, is we can use a second agent to really help reduce hallucinations by providing a verification score. In this process, what we can actually do is we can also use the information that's provided in the context and the answer that was provided and use an independent agent to evaluate the two of those together to ask it if anything was hallucinated in those results. And so the score can really help provide an after-the-fact analysis as to whether the information was correctly grounded or not. Now let's go on to some of the capabilities where GraphRag can show new opportunity spaces where regular RAG struggles. I'm going to ask it the question, what are the top five themes in the data? Now, when you ask this question to a typical baseline RAG system, it's going to take this phrase. It's going to vectorize it using like an ADA embedding. Look for the nearest neighbors. The problem here is there's nothing in this query that would indicate any specific filtering over top of that data set unless someone inside of this data set already wrote what are the top five themes of the data, which is highly unlikely because each of these articles are independent of one another and they don't have any knowledge of one another. So what does baseline right give us in this case? In this case it gives us state of Russian economy. It actually did in this case also come up with one example here about acknowledging that there seems to be a war the national rating the investment climate improving the quality of life and the meaning of Vladimir Putin so I want to just take a step back for a second and kind of emphasize this is a data set that's primarily about a conflict like 80 of the articles about a conflict and you can see about 80 of the bullet points here don't have anything to do with the fact that there's a war going on. In contrast, because we can use the semantic and thematic and agglomerative approaches that we've built over top of the graph machine learning parts of GraphRAG, we actually have a holistic understanding of what's happening in the data set. So on the GraphRAG side of things, you can see front and center, the first thing it's talking about is the conflict and the military activity and how that plays through on each of the major themes that it then gives you in context. Now I do want to draw your attention to one other thing on this. With BaselineRag, it's using about 5,000 tokens of context and it returned in about eight seconds. GraphRag here is a lot more expensive. It took about 50,000 tokens of context and took about 71 seconds to respond. But the important piece here is that though GraphRag is using a lot more resources, it's providing a much, much richer and correct answer, which we've seen for our customers is the crucial piece and what they really care about. And they're willing to actually pay the extra cost to get these much better answers returned. Next, let's take a look at this data set from a network map perspective. So this is actually a visualization of the entire graph network map of this data set, which is called the Vena data set over the Russian-Ukrainian war. Now again, I said about 80% of it's among the war, so the main core of this is going to be about war topics. On the periphery, however, we'll find lots of normal topics, and one of the ones I'd like to look at in particular is this group over here on the left. It turns out a lot of these entities here are about soccer, and you can see that they're semantically grouped next to each other in the embedded space, which is also great because it starts showing us that there are a group of entities that are co-related with one another. And then these colors here serve as a way for us to actually look at the communities that define these semantic boundaries. So for example, we can actually choose one of these colors. I'm going to choose a color for community, it looks like number 450. I can extract the subgraph for it, and we can also extract out pre-generated reports that the LLM has already generated on each of these community structures. So in this case here we can see clearly this is the Nova Rocia community that was being used to answer some of those queries that I made earlier in the demonstration. All right let's go and switch context a little bit to a different data set. So this is a data set where we took in all of the transcripts from the Behind the Tech podcast, which is, of course, hosted by Kevin Scott. So we can actually use the same type of methods here with GraphRag and ask it holistic, thematic, and trend-type questions over top of this data set, where, again, regular RAG would tend to fail. So the first question I'm going to ask it are what are the top 10 technology trends in the podcast. And so you can see here it has some good breadth and diversity. We'll dive more into this here later too when we go to compare this with Gemini. So I'll ask you to maybe comment on some of these later because we'll actually be able to see it in side by side with Gemini. The next question, and this is an important one because we have a lot of other demonstrations tied to this one, or what are the most odd conversations discussed? Again, this is a question where if you imagine baseline RAG being run over the top of this, it would perform very poorly. It would effectively pull back random chunks, and if those chunks had randomly picked up something with some odd conversations, it would comment on them, but it's not going to have the comprehensiveness and diversity that a graph RAG approach will in this case. The last example I'm going to show you here is actually a side-by-side. As I mentioned before, we have orchestration that runs over top of the indices. We have two of those methods that we're highlighting here today. On this orchestration, we have one that does a local search of the knowledge graph. It looks for, in this case, nodes that discuss artificial intelligence and looks at the knowledge graph. So it looks for like in this case, nodes that discuss artificial intelligence and looks at the nearest neighbors. Or we have a global search, which is going to be a more expensive operation that looks at the community summaries. And you can see the comparison of the depth and breadth between these two. So in this case here, you can see that very clearly the global summary here is providing you a much more comprehensive and diverse view than you are in the local search. So let's take a look at Kevin's podcast, Knowledge Graph, an interactive graph visualization tool here. Now, each of the nodes here represents an entity that the LLM extracted. And again, I just want to take you back for a second. This Knowledge Graph did not exist before the LLM was exposed to all the transcripts. It read through all those transcripts and created this from nothing. And that's pretty cool. So if we take a look at the colors here, those colors are semantic partitions that represent a high level topic. Now, one of the things I should mention, because I can only show one level of that hierarchy at a time, is that we're looking at the root hierarchy of this graph right now. So if we zoom into this node right here, let me go ahead and just pull it up here. We can see that this node right here represents Kevin Scott. Zooming out and looking up a little bit, we can see that this node is Microsoft. That's another very highly connected one. And another one right here, which is a major landmark, is Christina Wharton, who helps on the podcast as well too. Now if we zoom into one of these colors in particular let's go ahead and take a look at this green section over here. One of the things we're going to notice is we can see all the entities that are being pulled out. So we can see things like RNA virus, spike glycoprotein, SARS coronavirus 2, some things like synthetic biology, computational biology. And what's interesting here is it's actually grouped two episodes together into the same semantic topic. We have one with Drew Endy in episode 22. We have one with, oh, yeah, one episode 33 of David Baker. I looked these two people up. It looks like they're both in the biology field, so it totally makes sense. And it actually goes to show that GraphRag is actually working in the sense that it pulled out all the entities and then actually grouped them semantically together, which is exactly what we want because that will help us answer those questions. So if we get biology type questions on the orchestration side of the house, we can then come to this portion of the graph to then help better answer and augment those questions.
GraphRAG: LLM-Derived Knowledge Graphs for RAG
Watch my colleague Jonathan Larson present on GraphRAG!

GraphRAG is a research project from Microsoft exploring the use of knowledge graphs and large language models for enhanced retrieval augmented generation. It is an end-to-end system for richly understanding text-heavy datasets by combining text extraction, network analysis, LLM prompting, and summarization.
 
For more details on GraphRAG check out aka.ms/graphrag

Read the blogpost: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Check out the arxiv paper: https://arxiv.org/abs/2404.16130

And follow me on other platforms so you’ll never miss out on my updates!

💌 Sign up for my free AI newsletter Chaos Theory: https://alexchao.substack.com/subscribe
🐦 Follow me on Twitter https://twitter.com/alexchaomander
📷 And Instagram! https://www.instagram.com/alexchaomander/
🎥 And TikTok! https://www.tiktok.com/@alexchaomander
👥 Connect with me on LinkedIn https://www.linkedin.com/in/alexchao56/