 right we have to wait until the countdown is done okay can you just let me know when to start it yeah i will we like you do introducing me or how did it go say i'll say a few one sentence and then i take it over all right welcome everybody we are kicking it off with tomas bratanic who is a graph data science expert author and all over GNI mastermind at Neo4j with his session building knowledge graphs with LLMs take it away Tomasz. Okay so thanks for the nice introduction so as Alex mentioned today we'll be talking about building knowledge graphs with LLMs and as he mentioned we're also writing a book with Oscar about Knowledge Graph Rec but enough of that. So funnily enough I was looking at the previous session for like five minutes and I saw kind of nice continuation how we're gonna continue with this session, because first we're not going to jump straight into Knowledge Graph building, but we're going to look at when text embedding approach fails and then how to overcome it. And then the overcoming will be using Knowledge Graph, as mentioned by the title. But let's just go into what are the limitations of text embedding approach. In a typical REC pipeline, I've probably seen this so many times now, but basically you take a couple of PDFs, you chunk them up, you index them using any text embedding models, and then at query time you use the same embedding model, fetch the top four or five or ten most relevant documents, pass them to an LLM, you hope that any relevant information is in that document, and then the LLM generates the final answer. But there are some use cases for domains where this approach doesn't really work well, because this approach really works well for documentation, but for example for other domains like legal, for some questions this might not be the best approach as we'll see. And here I have a couple of questions outlined that you can ask. So for example, what are company policies on the remote work? But also you can ask other questions like what's the total value of contracts with Neo4j or whichever company? We'll see that different types of questions require different types of approaches. For example, when you're asking about the company policies on remote work, that's like a documentation type of question where basically you just need to find the top relevant text chunks that talk about remote work, pass it to an LLM and you get your answer. So that's all good. But what are payment terms for specific contracts. So in this case if you took a naive approach and just return top four chunks from your vector index, what might happen is that you can get four or here have three text chunks from different contracts which might not be related. The LLM could be quite confused because, let's say, you get three different payment terms for three different contracts. In this case, just using Naive vector similarity search might already fail. Here in the title, I can say this can be overcome, this can be solved by using metadata filtering. So if your vector text chunks have some structured data and you can filter by contract ID or company name or something like that, then you narrow down your results, and then you don't get text chunks from different contracts because you used metadata filtering. That's one scenario where basically you can use structured data to enhance your results. But then if you ask questions like what's the total value of our contracts with ACME or how many contracts expired this month, because this is a question that you might ask if you have a legal chatbot. In this case, basically, naive approach, even if you add metadata filtering, completely fails because the vector index, like the REC approach just returns for different text chunks. Maybe, hopefully, in ideal world, you would get full text chunks with contracts about with ACME that have like the value in it and then like you would rely on LLM basically doing some arithmetic and adding up the total value but what if you have more than four contracts, if you have 10 contracts with Acme.Ink, then in that case it completely fails. If you ask how many contracts expire this month, again, basically the four chunks that you get from a vector index doesn't really help in answering that question. Sometimes what we see is that you might get one or two contracts that might expire this month and then LLM will be okay, these two contracts expire this month, but that might become just a partial answer because maybe 200 contracts will expire this month, but LLM doesn't know because it gets you just for top-k text chunks and you can get inaccurate or partial results. As mentioned, certain questions that require filtering, sorting, counting, and aggregating cannot be answered with text embeddings only because basically you need structured data to perform these sorts of aggregations. And then there's one which I didn't really mention before, but sometimes when you have information spread across multiple documents, the vector approach might also fail because, let's say, the top K is free, but all the information is in five documents. Then even if your vector similarity search is perfect and it gets the exact information that you want, it's still really hard to define the exact top K because it might be dynamic. Sometimes you want like five documents, sometimes you need 15 documents, and this is mostly like unsolved problem at the moment. So, uh, structuring this data can also help with overcoming the top-k issue. This is my first memory, I guess. Basically, when people think or talk about the REC pipelines, mostly they just think about PDFs and documentation and just text. But there's this whole beautiful world of structured data that is overlooked a lot and it can help you answer these types of questions, whether you need to do structured operations or for more complex questions, like multi-hop questions. So here knowledge graphs come into play. So probably you've heard a lot about knowledge graphs today, so we won't go too much into depth about them. But basically knowledge graphs are used as data representing structured data. So this is probably the most used image to show basically what knowledge graphs are about. So you can have entities and relationships between them. But this, I think, is now slightly outdated because in the world of LLMs and RAC pipelines. Basically what you want most of the time, your data sources contain structured information and also unstructured information, because here we only have structured information, but now we want to also bring in additional unstructured information. And the nice thing about knowledge graphs is that you can have both of these types of informations in a single database. For example, here we have a contract and it's linked to two companies and these companies are located in whatever cities. But then, for example, if you chunk your contracts by clauses, then you can have each clause linked to the contract, and you could embed clauses separately and have them linked to the structured part of the graph. Basically, you could think about clauses or your text chunks with quite complex metadata, right, because the structured part of the graph can be thought about as metadata related to the unstructured part of the graph, right. So basically how I see it now that in the words of LLMs, the nice thing is that you can have both of these types of information together. And that's the power of knowledge graphs. So now let's finally talk about building the actual knowledge graphs with LLMs. Because that's kind of a nice thing everybody is funny, but in LLM world like 90% of the time the input is a PDF or a document right so we need some sort of way to build a knowledge graph to select that information from a PDF and this is what we'll be talking about. Basically, you can have, as I've shown before, multiple documents and then you build a knowledge graph based on the information in those knowledge graphs. The nice thing about knowledge graphs is that they're very good at combining information from multiple documents. Here we have four documents that represent this information, but now if we ask a question like previous OpenAI employees, which companies they founded, this would turn into a very simple Cypipher query instead of us having to look through multiple documents and then maybe one document contains that Dario and Daniela worked at OpenAI and then another document contains that they found it Anthropix. So this is kind of the multi-hop types of questions where they have to look to do multiple documents to find the whole truth. And this is also like a very nice use case for knowledge graphs of combining that information in a nice and compact way. And building knowledge graphs from text used to be called information extraction and it used to be like very complicated you need multiple models and each model in the in the pipeline was very custom and domain specific so mostly academics did that or like billion dollar companies and nobody in between but now basically you just ask an LLM, LLM please extract information in a structured way, mostly the structured way is a JSON and that's it right and so it the extracting of structured information from text has become much more mainstream and that also opens up the possibility of building knowledge graphs from text. Extracting structured information from text is so frequent use cases with LLM that basically now you have like JSON modes or like OpenAI also has like structured output mode and like you can also kind of hack the tools and use them to extract structured output so basically extracting structured information with LLM is such a frequent use case that there are lots of tools built around LLMs to make it easier for the developers to get like a consistent output. And as mentioned, basically the whole magic of information extraction is that you take some text. Some magic happens and you get a nice JSON. So basically you get a nice structured information. And how I see it basically, the information extraction pipeline, there's a spectrum of approaches, how to do that, and how I see it is like there's the spectrum lies from the most generic extraction where you have no idea what's in the text, what to extract, you just say LLM extract as much information as possible and then on the other side you have very domain specific where you know exactly what's in the text, for example, legal documents, and you can define each property in the JSON, what it should look like and what should be expected. Now let's look at the spectrum of approaches and how they actually work in practice. The first one is about the most generic approach where you have no idea about documents, what's in them, because a very typical scenario that I see is basically your boss gives you 10 000 pdfs and just says build the chatbot and you have no idea what's in them. In that case, you could use the most generic approach, you just say extract nodes which have ID and labels and extract the relationships between those nodes, and then you hope for the best of that. This is such a frequent scenario as well that you will see a lot of frameworks supporting the generic approach where you don't have to define any schema. Here I have the code for Langchain because I like Langchain because I implemented that, so I like to show off. So basically, the most generic approach with Langchain just I implemented that, so I like to show off. Basically, the most generic approach with Langchain just looks like this. You decide which LLM you want to use, and then you press the document and build the knowledge ref. Here I have one example of a contract between Mortgage Logic and TudorLate. So here I ran the extraction on the same contract two times, and then you can compare the extraction. So now because we didn't define any schema what we want to extract, you will get some relationships might be the same, but some relationships might be the same, but some relationships might be different. And like the knowledge graph, the extract information can be quite different depending on different runs, right? So for example, here we have TudorLink hosts a website and then MortgageLogic uses it. Andic uses it and in the first one on the left side we don't have that information. Here we have that MortgageLogic provides credit data and client content and here we have credit data and client content. So some things might stay the same and some things might change but what we see is that like the schema is very inconsistent and also the information that's being extracted and yeah so basically extraction can be noisy because RLM decides what's to extract and there's no schema consistency right so If you run this extraction on multiple contracts, there is no guarantee that node labels or relationship types will stay the same. Then I see there's a middle ground approach where you can define the node labels and relationship types you want to extract. This requires some upfront work because you need to define what you want to extract. But the nice thing is that you can use LLMs to help you with defining the schema. Because you can just sample a couple of documents as the LLM, what types of nodes, node labels, and relationship types did you find in the text, and you just fit that into an LLM. Again, you can use Langchain, but it's the Vodanka code, but basically what you would do, you would define the types of labels and the types of relationship types. Here you would have allowed nodes and allowed relationships, but somehow I missed that. Then your schema. Here again, I ran the extraction on the same contract two times in a row, so now the extraction is much more consistent between two runs because we guide the LLM what do we actually want to extract, we'll get more consistent outputs. There's still some discrepancy between the depth of information, between extracted information, between different calls, because that's just the nature of LLMs. Sometimes they want to extract more information, sometimes they want to do less. But the nice thing is that we also have better retriever options because now that your schema is more defined, that also gives you better options for the retriever because you kind of know what information to expect in the knowledge graph, you also know how to then retrieve it in better ways or more ways. Then the last approach that I like the most is the domain specific approach. So now basically you define every, if like you're extracting a JSON, you define every key in the JSON and you give it description, you can give it like available options, like for example dates, you define what's the format that should be used to write. So this type of information extraction requires the most work, but obviously it gives the best results. Here's one example that you could use to extract information from a contract. You can have a type with predefined types and parties of organizations, dates, scopes, amounts, and since this is more like a low level extraction, there's not a specific tool you can use, but you can use frameworks like LankChain, Instructors, Lama Index, or OpenAI Structured Outputs. You define the information you want to extract using pidentic objects. With this approach, because you define what you want and you're very descriptive and constrained about the information that needs to be extracted you will get the best results right so here I have basically one example and then it's basically we have some information as notes but then some information is stored as properties as well right so for example this agreement has effective date like a summarized context scope and stuff like that. So basically, when you can... So basically, this should be domain-specific approach, but basically, the LLM is guided exactly on the type of information it should extract, and the schema is fixed because it's not actually as before. We said to an LLM, extract notes and relationships, but now we basically say to an LLM, extract contract type, extract effective date. So the LLM doesn't actually do the graph modeling. The graph modeling is a separate step. In this case, the graph schema is fixed as it's separated from the LLM. Because we can guide and give instructions to an LLM exactly what we want extracted, we'll get the best results. And since the schema is fixed, you can create all sorts of tools for the retriever. So basically because you know exactly what is in the graph, what's the format of properties and all of that, you get the best options to build a rack application that can do all sorts of aggregations, filtering, multi-hop questions, all of that. That's the three types of approaches that I've seen in practice. Now some observations using that in actual practice. Obviously, anytime you're going to be building knowledge graphs and text, you'll have to do some cleaning up afterwards. The most frequent one is the entity resolution. You can have multiple nodes representing a single relevant entity. So here in this example, we have a UTI asset management company, and there are three nodes that represent the same node, the same entity. So you want to have some post-processing steps where you identify these duplicates and merge them. That way, your graph will have much better structural integrity. Some observations from papers. This is the paper from Microsoft where they're building knowledge graphs. So like when you instruct the LLM to extract or let's say all people and organizations from text, will the LLM actually extract all people and organizations from the text? And the answer is mostly not right. So when working with telelamps nothing is really perfect. And they also observed that the extraction is quite dependent on the chunk size that you use, so the smaller chunk sizes that you use the more information you will extract overall. Another technique that they've introduced is that you can do multiple passes on a single text chunk and then extract more information. So if you say that on a single pass we will not extract all the information, maybe let's do two passes or three passes and try to get as much information as possible. Then this one is also quite interesting, as we've seen before. This is how consistent is the selection. Even with a fully defined schema, like let's say domain-specific approach, how consistent is selection and it's valid depends on the types of documents, but the less ambiguous are the documents, the better the consistency will be. Here in this example, series are least ambiguous, So here in this example series, I'll let them list ambiguous so they'll have the most consistency. And then it's funny, but scientific articles and websites are kind of similar. But we can see that scientific articles are still more consistent because when you're doing science, you shouldn't be as much ambiguous as you can be when you're basically doing websites. One thing that's also quite interesting to see is that nobody like this is like an evolving space and nobody really knows what's the best approach to extracting knowledge graphs or like structured information. So like then you have so many options how to do like you can use JSON modes with LLMs, you can use tools, functions, you can just do regular prompts and define the example with fuchsia. And basically, this is already model-specific. So it's really hard to say what works best because it really depends on the model and not just the model, let's say, because even GPT-4.0 from January behaves differently than GPT-4.0 from April. But this is also something to think about when you're building knowledge graphs. And that's it for me, so just on time. And now we have two minutes for questions. And basically also because this is like a theoretical talk, but if you want some code so you can try it out, you can go and look at my blog post or code examples and test it on your own. So yeah, now I don't know if we have time for questions. I think I've already squeezed one in. Thank you, Tomas. The one with the most votes comes from raya and they are how does knowledge graph update internally when i add new document to the graph yeah i mean so that's basically up to you uh but uh build like adding additional information is not uh is like not a big problem because it's just adding additional information is not a big problem because it's just adding additional nodes and relationships. But the only thing you have to be careful is that you do entity resolution. So if an entity is mentioned in the text that's already in the graph, you kind of want to use the same name because as mentioned, sometimes not exactly the same name is used. So then the only problem is dealing with this. But this could also be like as post-processing. You update your knowledge graph and then try to search for duplicates. But other than that, it should be very straightforward. Yeah, cool. And then there's one one more maybe we can squeeze it in as well from david um and we'll keep on answering this one and then we close it but if you want to jump to the next session you can do so uh what is your recommendation to perform graph disambiguation merging duplicated nodes or different nodes that are actually the same thing? Yeah, so what you see in practice is that it really depends on the types of nodes you want to merge together. So it's very different when you're merging people than when you're merging organizations or locations. But for the most part, what we've seen is that you use text embeddings to find potential candidates that could be merged, and then you can use some sort of word distance or you can have an LLM as a judge deciding if two entities should be merged. So the generic approaches is that you use a combination of text embeddings and then additional logic to give the final decision if two entities should be merged or not. Super. Cool. With that, I think we're at the end thank you very much tomas thank you for watching uh keep keep continuing with the notes uh presentations throughout the day and uh yeah see you at the next uh next session uh and uh somewhere in the future bye see you later bye
NODES 2024 - Building Knowledge Graphs With LLMs
Tomaz will share lessons learned from building knowledge graphs with LLMs, from using various text chunk sizes to constrained schemas and more.

Register to NODES 2024: https://bit.ly/47VqQkp

Get certified with GraphAcademy: https://dev.neo4j.com/learngraph
Neo4j AuraDB https://dev.neo4j.com/auradb
Knowledge Graph Builder https://dev.neo4j.com/KGBuilder
Neo4j GenAI https://dev.neo4j.com/graphrag


#nodes2024 #neo4j #graphdatabase