 Hello everyone. We work at Neo4j and the GenAI team and today we will be talking about measuring the performance of vector rag versus graph rag. So I assume that people in this session are already familiar with the concept of rag and the different rag strategies. But in case you don't, don't worry. I'll be briefly explaining those concepts in this session. But I highly recommend you to join the dedicated session. There are plenty of them diving deeply into these concepts. So in this presentation, I'm going to talk about the motivation. So why do we need an evaluation for the RAG strategies and the main objectives behind this work? I'm going also to talk about the different RAG strategies that could be used using Neo4j database. I'll be also talking about the framework that we use for evaluation, which is called RAGUS. And we will be also talking about experiments that we conducted and share with you some insights and of course, complete the future work. So what is RAG? Why do we need an evaluation for RAG pipelines? And what are the challenges? So RAG stands for Retrieval Augmented Generation. And it's basically when you don't only have a basic interaction with LLM where you only provide an input query and a prompt, and you just receive an answer that is generated by the LLM, but you provide it with a grounding context, which means an additional information that you acquire from external sources, such as documents, APIs, but most interestingly, knowledge graphs. So this has many advantages, but mainly it helps the LLM to overcome the knowledge cut off. So for instance, there's plenty of knowledge that wasn't present in the training data. And there are also like domain-specific knowledge and confidential data or enterprise-related data. So when you provide the grounding context, you provide this additional information and you help the LLM to answer better the questions and to reduce hallucination. It's also considered as a cost-effective implementation with respect to other implementations such as fine-tuning LLMs. So in general, there are two phases in the RAC pipeline. The first phase is the retrieval phase, where you acquire this additional information from external sources. Here in this presentation, we'll be focusing on the Neo4j database. There are different strategies that you can use in order to get this additional information, and you have different parameters or hyperparameters to configure. For instance, if you're relying on something that is called vector search, you have to configure the K for the top K nodes that can be retrieved from the graph, similar to the query. You can also configure the similarity threshold. We'll be talking about that in the next slides. And the generation phase, the LLM used this additional context or additional information and answer the question so here uh also we have different configurations and different parameters to choose for instance we have uh to choose which llm to rely on in order to generate the answer the maximum answer size etc so here in this work we added a third phase which is called evaluation in order to prove which strategy performs the best, because we have different configuration for retrieval, different configuration for generation. So we want to evaluate each strategy in order to identify weak points and to be able to improve them in the future and monitor, of course, the pipeline and production. So there are many challenges for the for doing the evaluation. So first there's currently no standard process for evaluation and if the evaluation is not automated it's really very hard to generate a ground truth answer. So it's time and effort consuming. And we also need a framework that can give us an overall score for the evaluation of the whole framework, but also to be able to give scores related to each component of the five times. So for instance, further retrieval apart, further generation apart. So here I'm going to talk about the strategies that we used using Neo4j databases. Again, there are plenty of other sessions on VectorRAG and GraphRAG. I also recommend to watch the recording of the workshop on the Graphrag Python package that was developed by Neo4j, which dive deeply into the concept of vectorrag and graphrag. But briefly, what is vectorrag and what is graphrag? So vectorrag is when you rely on vector similarity in order to acquire this additional information that we call grounding context. And this is done usually by doing a similarity function between vector of embeddings that represent the query and vector of embeddings that represent some textual properties on nodes and the graph. In the graph rag, you basically rely on the power of the graph and the relationship between the nodes to not only acquire information from text nodes, but also from the neighborhood, so so from related nodes thanks to the power of graph traversal so here in the uh in this work of evaluation we use two uh different strategies for graph frag the first one is augmented vector search so you basically rely on vector search uh to um search for the top case similar nodes with respect to a query. And then you rely on those nodes as starting nodes, and then you traverse the graph using a Cypher query, and you can acquire additional information, which are concatenated in the context and provided to the LLM. The other strategy that is also considered a GraphFact strategy is Text2Cypher. And here, you basically rely on Text2Cypher to automatically translate a natural language query into a Cypher query. And the Cypher query is executed on your graph and you get a subgraph and you can concatenate information from the subgraph and provide it to the ELLM to get better answers. So let's say you have a graph, a Neo4j graph that represents movies and actors who acted in those movies. So let's say the embeddings in your graph are generated only on nodes related to movies, and the query is on the movies, then vector search would do good for this, because it will retrieve a relevant context and provide it to the LLM. But if your query is on actors who played specific roles in those movies, then you get a limited context with the vector search. While with the GraphFrag, you can, based on the nodes movies, and then using a cipher query, you can traverse the graph and then acquire additional information. And in this case, information about actors will be concatenated in the context and it will be provided to the LLM. But this requires, as I said, a retrieval query to be configured, which is a cipher query. Whereas with text to cipher, you don't need to configure a cipher query because text to cipher will automatically translate the natural language query into a cipher query. It will be executed and then information related to actors will be concatenated in the context. Now, we used the RAGAS framework for the evaluation. I'll be talking about RAGAS, and I'll be explaining some metrics that we considered relevant for the evaluation. So we basically looked at different tools and frameworks that are based on an automatic evaluation of the RAC pipeline. I'm sure this is not an exhaustive list. There are plenty of tools and frameworks that could be used, but we relied on Raga specifically, not only because it's RAC- and it also um involves uh core met well-known metrics for for the evaluation and it's open source but most importantly because it requires no to very minimal human intervention and if you are also working with other well-known frameworks such as lang chain and llama index it's really very easy to work with RAGAS, too, for the evaluation part, because it offers a good integration with these framework. So RAGAS stands for Automated Assessment of RAC System. As I said, it's an open source project, and this is the link of the related GitHub project. It is based on the LLM to provide the evaluation or metrics for each component of the RAC pipeline. So basically, there are some tailored prompts, and they are given to the ELLM, so the ELLM can generate the scores and evaluation for each component of the RAC pipeline. The good thing with RAGA is that you can get an overall score evaluating the whole RAC pipeline, but also some scores relative to the retrieval and the generation separately. So this is an example of the Ragus metrics that we relied on in this work to compare vector rag and graph rag. So here, for instance, it's a list for the rag, for the ragas metrics that are relevant for retrieval. So some of these metrics do require the presence of ground truth answers generated by humans or by LLMs. But other, like the context relevancy, do not. So for instance, context relevancy, basically the LLM is asked to count the number of statements in the context that were relevant to answering the question. And here by statements, I mean, like, consider the following sentence, Nathalie is a software engineer and works at Neo4j. So here we have two statements the first one is that natalie works uh at neo4j and the second one is that natalie is a software engineer um the other two like context precision and context the context recall um do require ground truth uh answers so here basically the llm is asked to find ground truth elements either in the first case sentences of the context or in the whole context now there are other metrics in ragas for the generation part so so for instance um answer relevancy um you ask basically the llm to infer questions from answers and then to compare inferred questions with the original one in order to provide a score for answer relevancy faithfulness for instance um you you ask theLM to count the number of statements and the answer that were supported or can be inferred from the given context. And now I'll hand it over to Magbule, who will be talking about experiments. Thank you, Nathalie. In the rest of the presentation, I will be talking about the experimental setup and the results. We will be talking about how we prepared the dataset and which kind of databases we have used. We have used two databases from two domains. One of them is movies and the other one is products. We have used the available databases which are publicly shared on DemoLabs, DemoNeo4J Labs databases, so you can access them. We have expanded them by preparing embeddings for them. In these databases, we have some nodes, such as for movies we have users, movies, actor or director, information, and genre. Each of these nodes have some properties. For example, in movies we know the title or the release year and we have relationship between these nodes. For example, user has rated a movie and also we know the property of it. The rating is some value. And similarly for products, we have product, article, department and customer notes, their relationship and their properties. Next step would be the dataset preparation. As you know, for question answering, such as retrieval purposes, we need a question and ground-root answers. These questions are what users would ask and users would ask in their natural language question. For example, it can be the user could be asking about the top five movies by the IMDB rating. And the ground rules would contain the list of those movies. However, preparing this kind of data sets is not very easy. What we have done is we started with a core set which are created manually. Initially, we started to be prepared 15 to 20 questions for each domain and these questions have different complexity levels. We have identified four levels. The first one was the basic very simple questions which are asking about the properties of the nodes. Let's say, release the year of a movie. And the part is the very much more difficult questions, the hardest questions, which contain many constraints or aggregations and etc. Having this core set, then we try to expand it. For this purpose, we have used LLMs. In the first approach, we use this core set and ask LLM to rewrite these questions. We ask them, we and ask LLM to rewrite these questions. We ask the LLM to rewrite in a more formal way, rewriting as if it is a teenager, rewriting using synonyms and etc. The second version of expansion directly uses LLM. Since we already have the schema of the database, we know the relationships, nodes and properties. We provided this information to LLM and asked LLM to generate questions for us. have the schema of the database we know the relationships nodes and properties we provided this information to llm and asked llm to generate questions for us however in this step not all questions are meaningful or not all of them are relevant to the data we have as a result we need a step for human validation for this purpose we have used SexyCypher to generate cipher queries and we collected the output from cipher and compared them manually using the human annotators. And since this step requires human annotators we were able to collect around 100 question and answers for movies and around 250 questions for products. When we check the complexity levels, we have seen that movies dataset is a little bit more complex. It contains more questions about aggregations. However, products complexity was around 1.6. When we check the results of the experiments, as you know, we have used the RUGAS framework. In the next slide, you'll see it. We have used, there are two kinds of metrics, as Natali has already explained, retrieval relevant and generation relevant. When we check the retrieval relevant metrics, we see that graph RUG strategies, augmented vector search and text cipher, performs better than vector search. And when we check the best results usually obtained by text cipher, only exception is context precision, where vector search performs better. But it is expected because precision is relevant to how many items are returned and how many of them are right. Since vector search are returning a few number of information, having higher precision is easier for that kind of approach. In the next slide, we will see that the generated relevant metrics. Here we see the similar pattern. GraphRack strategies perform better, and TextCypher has better results, relatively slightly better results than AugVVS. However, there is an exception. As you see, Faithfulness value for AugVVS is very high compared to the others. And we were wondering why Text2Cypher couldn't achieve that result. And we think that it is relevant to how we provided the context to the generator. Since in the text cipher we have provided cipher queries together with the context we have collected and we believe that that creates noise for the LLM and that could be the reason why we have a different pathfulness result. When we check the product question answer set, we have similar observations. GraphRack strategies outperforms vector search results. Except for generation, for answer similarity and paidfulness are very high in vector search. But we believe it is relevant to the complexity of the data set as i have mentioned vector search is better at asking questions about properties of the nodes and for simpler questions less complex question set they perform better and products data set was relatively simpler less complex compared to the Moody's dataset. We believe this result is because of these complexity levels. But of course, these things require many more exploration. When we check the questions themselves, we explore the questions and their answers. As you see, in our database, we are looking for the genres of the assertion movie. In our database, the genres are different nodes. That means we need to execute one-hop neighbor execution, neighbor exploration. And when we look at the results, vector search is not able to answer the question. Augmented vector search also wasn't able to. However, TextoCypher was able to find the right answers. In the next slide, we'll see that a more complex question and answers. This time we are using some aggregations. We are looking for the first three genres. We are looking for most rated genres that have the most movies associated with them. And the right answers are seen on the right side of the screen as expected output. So the results are drama, comedy and thriller. TextCypher was good at it and V and vector search was not able to answer it at all. One surprising result is from augmented vector search. It is able to return two of the right answers, comedy and drama, but it returns some additional information, like this tall movie is relevant to those things, but the numbers are irrelevant because it is not able to access all the database. It is hallucinating some numbers itself based on knowledge it has. So in short, what we have seen is that a graph-break strategies are outperforming vector search results. Augmented vector search performance depends on the input of the Cypher retrieval queries. So if you have expert knowledge, if you are able to prepare right Cypher queries, it can perform really well. Take Cypher compared to the others is more consistent retrieval. And for vector search, we can say that it is able to answer questions which are directly relevant to the properties of the nodes. But for others, it has some challenges, struggles. So we will conclude our presentation right now. In this presentation, we have talked about the evaluation and comparison of different drug strategies. We have explored vector search and two graph strategies, which are augmented vector search and text cipher. We have used two domains, movies and products, and relevant databases. We have talked about how we prepared these questions and answers. And as a result, what we have seen is that, in general, graph RAC strategies outperform the vector search strategy. In the future, of course, we want to use larger data sets, which are well known in the domain instead of preparing them ourselves. And we want to explore the shortcomings of these strategies and cover them as much as we can. Also, we want to use different data sets from different domains to reveal their shortcomings again. This is all from us. If you have any questions, we can have a live question and answer. That's all from us. Thank you. So my name is Clavin Tulli and I'm the lead data scientist at Gallo. I have over 10 years of experience developing AI solutions for manufacturing execution systems, logistics, and supply chain domains. That's my LinkedIn QR code right there. Feel free to connect. So the outline of the presentation, first we are going to talk about the use case. And our use case is track and trace in manufacturing. I think track and trace, it's kind of like self-explanatory. We are just going to be tracking and tracing where our product is. Next I'm going to talk a little bit about generative AI, prompting methods, then I jump into knowledge graphs, describe what they are, then the graph retrieval augmented generation, which is the graph rag, then customizing it with tuning, and lastly, the results and findings. So in terms of the use case, trick and trace in manufacturing here is the background manufacturing does demand near perfect execution and over over time process management methods like six sigma have evolved and they call for a target of 3.4 defects per million opportunities. I think you can actually see that this is like a gold standard and, you know, we need to achieve that. But regardless, defects do happen and they tend to be very costly and time consuming when they happen. We also lose production. And if you think about product recalls, they are a major pinpoint for most manufacturing companies. Therefore, the ability to track and trace with agility, it can vastly improve problem identification in our supply chains and value chains. And it can also improve the time required to resolution. And we think that generative AI and knowledge graphs, they offer exciting capabilities to transform trick and trace. So, if you are familiar with the manufacturing value chain, we all know that it kind of starts with suppliers, raw materials are transported, processed, and there's further processing, QA and packaging, and there's further process transportation to warehouses and distribution centers and ultimately to the customer. The biggest challenge within manufacturing is that most of the systems are siloed and the data is siloed. And when we are doing track and trace and we are faced with questions like what products depend on this supplier? Where did this material come from what equipment was used where is the product going where did this product come from from a customer perspective it becomes very challenging so the ability to integrate the data across all these siloed systems it's a game changer in how we do track and trace. So we will see how we achieve that. But in the meantime, I wanted to introduce you to generative AI. So what is generative AI? Generative AI is all about generating content and here i've just shown a few techniques of prompting to generating content zero short prompting few short prompting and chain of thought prompting there are other prompting techniques like meta prompting and so forth but i just wanted to keep it simple so for example if we look at zero short prompting the question I'm going to ask my model is how would you describe the test of a smoothie with too many oranges and then it gave me that answer a smoothie made with too many oranges can be overwhelmingly citrusy and tangy and all that and when i looked at the answer i was like i liked it and i generated um more for different foods fruits like watermelons strawberries and so forth now with few short prompting I have to give my model an example one example or many examples so that it can learn what kind of content I'm expecting in this case I'm prompting my model and saying that you have knowledge about how machinery fails tell me more about a breakdown where the motor fails, limit output to 50 weights. Then I give it an example here, and then I later prompt it for bearings wearing out, and then it gives me an answer that can like consistent with the example that I had shown it. Now with the chain of thought prompting, it's used to break down the problem into sub problems so that we can increase the accuracy of how our model is generating content. For example, here I'm giving it all kinds of events that are happening and i wanted to tell me the total duration of the breakdown in the last one right so in in the example i give it i ask it to first identify the start time which is 11 a.m identify the end time which is 4 20 pm and then calculate the difference so the chain of thought is in the sequence of how the model is solving the problem and once it kind of like knows how to do that then i will prompt it and it will give me an answer so this is generative ai in a nutshell but here we are not using any external data sources we would want to use external data sources so that we can give more context and make our responses more meaningful based on the data that is within our enterprises. So that will be achieved by retrieval augmented generation. So retrieval as in retrieving from external data sources. Now in terms of knowledge graphs, there are kind of like two approaches to build knowledge graphs. One of the approaches, approaches it uses unstructured data and you have so many documents. If you think about Wikipedia, it's a good example. And then you can use AI to create a knowledge graph. There are large language models that you can use to create a knowledge graph. Within the manufacturing industry, I would say proceed with caution with this method. Remember, we mentioned that manufacturing demands near perfect execution. So you would not want to take any chances with how the chunking is being done, whether it's hierarchical chunking, whatever. You want to be careful in how your data is organized in the knowledge graph. That's why the best option is approach number two which uses semi-structured data and if you have your relational databases it's going to have all kinds of tables in there and some of these tables they can have columns that have um free text and using all these tables you can use um ontology mapping and ontology it's pretty much the study of things and how they are connected and with this ontology mapping you can create a knowledge graph and that's what's going to be a knowledge your knowledge base for how you are going to implement your graph rag so we went ahead and created our knowledge graph using ontology for our sub for our value chain and here's what can what it looks like the good thing about knowledge graphs is that they are intuitive um so it's very easy to read them or query them for example here we see that we have this supplier uh with our we like which is shown by the orange bubble it supplied materials and the materials were processed to a product it could it might be refined or not and then it was shipped to a customer so that's how you can like um read a knowledge graph this is like a trivial example i'm showing here and when you look at how you query it with cipher it's also kind of like very straightforward it's very intuitive and this is the good thing about knowledge graphs their intuitiveness it makes them work very well with generative ai solutions so on the nodes for example the supplier material product and customer nodes we can also have properties there for example we can have the customer address we can have a project description we can have a material description and so forth same Same as the relationships, we can also have data there. For example, when we say processed to, we can tell what equipment was used to process and if we encountered any downtime, we can have the reason codes there and also the downtime descriptions and so forth. So that's the nice thing about graphs they are so intuitive now once we have our graph we are kind of like going to overlay it on our value chain and supply chain and as you can see it starts to connect information or data sources that are siloed. This is the most powerful thing about these graphs. They help us connect all these siloed data systems. And once we connect the data, it's much easier to answer all those questions which we were concerned about earlier. For example example what products depend on this supplier where did this material come from it's going to be much easier to answer all those questions with a graph now the other interesting thing we can do about graph is we can add freeform text to our nodes or properties. So, for example, when we are processing our materials into a product, we had an issue with an equipment that broke down here so we can um add that free form text here like increase the noise vibration and so forth and that's gonna be a property on this relationship proper process process too and we can also include all the reason codes and the equipment that was used same as the product we can add maybe the product description in this case that this is the text that we were generating earlier how would a smoothie made with too many oranges test so we add that to our product node and we add this freeform text as a property so the reason we add this text as property is because we can create embeddings out of this freeform text. And once we create these embeddings, we can now perform a semantic search on our graph. So if you, so most of vector rags, right, they kind of like just store these chunks of text as embeddings and it's just like a vector store but in a graph we can see that we are able to achieve the vector store and also we are able to capture the connections so a graph rag pretty much is a superset of the vector like it can do everything that effect Iraq does and it also provide those connections that we really care about so what do embeddings of the free-form text look like it's going to be something like this so if you remember from your linear algebra class about vectors, a vector is going to have maybe three dimensions if it's X, Y, Z. But when we are dealing with these models, the vectors for text they are going to have a length of um 1024 or 1520 and 36 that's how big the dimensions are uh as you as you can see here the embedding is too many numbers uh for for the text we show here so these embeddings here that's what we store in our property graph and when we are doing a semantic search we can use cosine similarity to retrieve what matches what best matches our prompt and once we find what best matches our prompt we can also traverse the graph the neighboring nodes and retrieve more information and bring all that information to the large language model so that it can enrich and personalize the response that's the good thing about the graph rag now um to achieve this with the graph rag you want to customize it with the tuning and the tuning that we see here it's prompt prompt tuning. Some call it P tuning. And remember, we talked earlier about few short prompting where we give the model some examples so that it can learn. With prompt tuning, we are giving it so many examples so that it can refine or it can improve uh in the accuracy of how it uh it responds so for example here uh we see our samples here uh the human the the prompt might be which material has the most suppliers and then we have this um cipher query here which helps us retrieve the information and we are going to have lots of of examples of these cipher queries you can use gen ai to generate some of these ciphers then modify them to make sure that they are accurate and feed them as samples to help improve how your model response is when you prompt it so we talked about how we use semantic search and we talked about how we do we use cipher to search for information in our graph so once we are able to do those two, it's going to enrich our responses. And in our results and findings, we discovered that the graph lag has enormous potential to transform track and trace in manufacturing. An example we see on the right where we are prompting with our graph lag i'm just going to provide a prompt and say i'm feeling very thirsty and on the beach which smoothie should i have and how many suppliers are used to make this product so what we can see is uh based on the semantic search our model is able to recommend the watermelon smoothie because in its description it says it it's watery and it's good for hydrating uh on a hot day that's why it recommended um the watermelon smoothie also it tells us that this product has four suppliers and that's achieved with the cipher queries that we saw earlier the to to to achieve something like this the there is of course a lot of experimentation that is required um particularly with the prompting and also with the cipher and also the mapping of the the knowledge graph requires deep domain expertise because we want to make sure that our nodes and relationships are capturing the domain rather than just connecting things randomly the other thing is capabilities of AI what we found out they are evolving very fast the technology is evolving very fast and what you think is not impossible today it may change tomorrow and that presents a challenge in terms of um how you keep up in terms of IT infrastructure in terms of um learning uh the new technology that's involving so it's kind of like um very challenging to keep up with how AI is evolving because it's evolving very fast and then the last thing in our findings is it's very challenging to quantify the return on investment for these tools we all know that we want to improve productivity but quantifying it it's always very challenging now um despite all these good things there are a few primary ethical considerations and here i have just four uh the first one is these models they tend to lack explainability and interpretability that's why i'm a big fan of traditional AI ML. There are explainability and interpretability tools there, like Lime and Sharp. But with the large language models, it's a little bit challenging. It's evolving. Some solutions are beginning to come up, but they are not that good. Then the other challenge or ethical consideration beginning to come up, but it's the they are not that good. Then the other challenge or ethical consideration is data provenance. So once you expose your data to so that it can be used in the RUG architecture, you want to make sure that the data has integrity and there are no challenges in terms of missingness and so forth because your model is not going to know that then the other ethical consideration is a sensitive information disclosure you want to make sure that the data you want your large language models to access it's kind of like isolated and has been curated if i may say because if you just expose it to the entire databases it may pick up personal identifiable information or some other sensitive information and you don't want that and then the last ethical consideration is the future of work is changing there's been concerns around worker displacement and displacement and that's just going to be an ongoing ethical consideration given how these um AI models are evolving so that's pretty much about it if you want to have a hands-on exercise on what I've just gone through I have a resource for you here you can scan this qr code and then you can use it you can play with the code and experiment or with the example that's on this link and then lastly thank you all for joining me on this journey through the world of ai today we have explored not only the powerful potential of AI, but also the responsibilities that come with it. As we look to the future, AI is not just a tool, it's a transformative force that can redefine industries, enhance human capabilities, and address some of the world's most pressing challenges. Thank you. With that, I can take any questions.
NODES 2024 Best Of: GraphRAG
Re-Run of two great sessions from NODES 2024 in November! 
Watch all NODES recordings: https://neo4j.com/video/nodes-2024/

Session1: Measuring the Performance of VectorRAG vs GraphRAG

Join this session for an in-depth exploration of evaluating Retrieval Augmented Generation (RAG) pipelines, which are crucial for enhancing the performance of large language models (LLMs). As RAG becomes increasingly prevalent, understanding its evaluation is vital to identifying and improving weak points. This session delves into the benefits of integrating knowledge graphs for better grounding and explainability, contrasting it with traditional vector-based retrievers. We will discuss the latest LLM-based tools for automated RAG evaluation, and showcase applications using Neo4j-backed RAG pipelines and the RAG Automated Assessment (RAGAS) framework. You will gain valuable insights into advanced RAG evaluation techniques, helping you optimise your own RAG implementations.

Nathalie Charbel, Makbule Gulcin Ozsoy, Estelle Scifo


Session2: GraphRAG for Increased Trust and Confidence in GenAI for Manufacturing Processes

Disparate data systems often obstruct a comprehensive, end-to-end view of processes, hindering digital transformation efforts. In this session, Clive will explore how to overcome these challenges by modeling knowledge graphs for manufacturing processes. You will learn how knowledge graphs combined with generative AI can integrate and enrich data across various functions, providing deeper insights and an enhanced prompting experience.

Clive Ntuli


Get certified with GraphAcademy: https://dev.neo4j.com/learngraph
Neo4j AuraDB https://dev.neo4j.com/auradb
Knowledge Graph Builder https://dev.neo4j.com/KGBuilder
Neo4j GenAI https://dev.neo4j.com/graphrag