 Hello everybody, good morning, good evening, good afternoon and thank you too and well happy 2024 I guess I should also include to this one. Hi Jesus, how's it going? It's going very very well thank you happy 24 to you too. Thank you very much yeah good to see you too it's it's been a while it's I must say I'm just re-emerging out of this time, you know, between Christmas, New Year, what day is it actually? But yeah, it's been good. And I'm happy to do this very first live stream of the year with you together, Jesus. I know we kind of discussed whether it would be a bit too early in the year, so I'm sure we're catching someone still with one leg in vacation, one leg coming back. What better thing to do, even if you're on vacation, to watch an episode of Going Meta, right? Yeah, totally. Just a little service update. It's Thursday today, the 4th of January so just in case you're wondering that's not a variation i mean but we still are the the webcast that runs on the second no the first tuesday of the month right yeah some minor exceptions i mean it would have been a bit you know really really too early in the year right yeah yeah yeah that would have that would have been would have been a tough sell i think but yeah um but yeah today uh the fourth so i'm hoping everybody had a good a good new year's had had some some some nice days off if you had them or you know if you work through it all then obviously um welcome nevertheless to to day's episode um it's already episode 24 so we are we are well in into uh into the higher numbers now almost and we have yet another topic it's like a little mini series within the series i would say so i think let me quickly look at this yeah the thing yeah we started rich and so so timely that it's difficult to get out of it, right? So I'm sure we'll take a break. But we're kind of connecting it to something we have done in the past, right? So we're staying on the topic of knowledge grabs and LLMs and RAG. But we're connecting it to ontologies and to kind of the essence of our series, right? And we're going to hopefully go meta today. So we're, again, building on what we've done over the last two, three episodes, and bringing in some additional elements. So that's the plan today. Sounds good. Sounds great. I'm looking forward to it. If you haven't watched the previous episodes, don't worry. Everything is on our episodes, don't worry. Everything is on our overview page. If you click on YouTube, you'll find the playlist there as well. I'll post a link in the chat in a second so you can catch up on the episodes of the past. But you don't really have to. You have to, obviously. But you don't have to do it now. You can stick around and you can watch this one. As Jesus said, I think we're going to tie together a little bit to the previous ones, obviously. But I think it should be okay. Sure. I'll open with a brief recap of the previous episode and and yeah like you say anyone who wants to dive deeper you know all this all the assets all the recordings are available so feel free to go and and consume them at a time that works for you should I um actually I'm going to share my desktop as usual. Let me do my entire screen here. Let me know when you can see it. Actually, quite conveniently, because after choosing and preparing the session for today, I got tagged in this note by Dan and on something that uh jason had shared today and it's you know this time of year when everyone publishes you know the predictions for the year right yes it was very interesting right because they it's totally the topic that we've been working on for the last few episodes and that we're talking about today right and one of the predictions and it's actually the one that Jason mentions, is all about how RAG will evolve from semantic as in vector search to something richer, to something like based on graph networks, right? A lot of what we've been doing. So it was another, you know, confirmation of what we're doing is perceived as really something that's valuable, and it's the direction that this space is going to take. So it was quite nice to get that note from both Jason and Dan. So, yeah, I was going to say that, as usual, I'm going to open with a Yeah, I was going to say that as usual, I'm going to open with a couple of slides to set the context and then jump into some code. So I hope it's all visible and readable. But yeah, the idea is talk about still RAG, still integration of knowledge graphs and large language models. But we're bringing in ontologies. And let me explain how that happens. Again again two minutes on on what we've been doing so it's all about rag i mean for those who have been uh who have missed the previous episode so have been living under a row under a rock uh rag is this uh pattern this architectural approach if you want of uh this integration that for interacting with large language models that involves an external source of information, right? So we know that large language models are great at the creative, generative capabilities, the language capabilities, they can do analogies, they can do reasoning, they can do generalizations, they can generate language, but sometimes the knowledge of the world is limited and it's even more limited in uh your enterprise specific knowledge right so they're they're not being trained they've not been trained on that so we have to inject that knowledge somehow and rag is is the kind of most widely accepted approach to do that so when we ask a question to an llm first we go out to an external source, Neo4j Knowledge Graph in this case, we retrieve all the information that we believe, that we think it's going to be relevant to answer our question, and we pass it to the LLM as context so that the answer generated is based on this information. So we kind of take control over that. And that's the idea of RAG. I mean, like I said, we've been talking about it for weeks, so I won't spend much more time on it. What was interesting is recently in the last episode, we focused on what we call advanced RAG patterns. And these are the ones that actually combine two elements. One is, I mean, basically the two types of semantic search, vector search and graph-based search. And it's really when it becomes interesting, right? When you can combine the two. And in an attempt at explaining how that works, I've created this visual, which I hope is going to be useful for this. So we have a knowledge graph, right? A knowledge graph that captures information about your organization, about your problem, about your domain, right? And your knowledge graph will contain notes and relationships, obviously. But some of these notes will contain properties that may be fragments of text. Maybe because that comes, I mean, that's human-generated, human-entered information. Maybe because it's the result of parsing unstructured documents. There can be many reasons, but there may be fragments of natural language in there. We want to make use of that. The best way to process and to do semantic search on natural language is to create embeddings and index them in a vector index. The way I see that is somehow like a projection of these nodes in a different search space. I mean, I put it like a plane there. It's not a plane because it's multi-dimensional. I mean, we cannot even draw it. But the idea is we pick some of the information in these nodes, we vectorize it, and we project it over a different search space, which is this vector search space. And what we can do there is we have like, let's say a question or a string or an image or an audio or whatever element, and we can do a vector based search that basically will scan that the index and return some results. So when we do a search we find some I mean search is based on on on distance right on on this distance in a back in a vector space so that will be typically Euclidean distant cosine a distant but the idea is that you find the closest vectors to the one that your your search element right and for those vectors what we're going to do is we have to get back to the graph because these vectors are stored in nodes and that's where we do this kind of second step where we enrich the results so we not only uh uh stay in this vector search but we once we have the vectors we navigate back to the to the graph and we can expand we can because we're aware of the structure in the graph we can enrich these results and keep in mind that all this process of going to the server to the vector search to the vector index and navigating the graph is what happens in this intermediate step in the rag interaction so we are in the process of retrieving all the information that's potentially relevant to answer a question or to generate some content, to provide an answer to something. And that's the information that I'm going to pass to the LLM. So I'm interested in having the highest precision, the highest accuracy in this information. So I want to make sure that I combine in the most powerful way this vector search with the graph search. And that's what we've been doing. You know, I don't know if this sort of visual helps or confuses more, but that was my attempt at describing what we've been doing before we go into the code. Right. So, yeah, I think it's helpful. It's helpful. I think it's helpful. It's always especially with I mean, you notice we talk about the graph space since many years now. And I think now, I think we come at a time where people understand when we talk about graph, we don't talk about charts. So at some point, people understand this. now we bring in the vector comes around it's like a completely new concept for many many people it's again very very abstract very unlike graphs very abstract very very difficult to grasp what it is and then you know i think this is a good visual to understand okay this is this there's the graph there's the vector there's the the points in this you know exactly crazy grid and then combining and somehow that that also explains these these kind of prediction i mean it's not a prediction because it's more a reality than anything else that we saw on linkedin before so yeah you could you could get some valid results with just vector search but you would only be doing half of the half of the job right so you could deal with the with the simple the simplest cases where you kind of just chunk unstructured data and you put it in a vector but when you combine the two cases where you kind of just chunk and structure data and you put it in a vector. But when you combine the two is when you can be really precise, really rich, and really accurate. And that's what we're gonna be. We've shown it in the past, and we're going to reiterate that idea in today's episode. So that's RAC. That's what we call advanced RAC because it combines vector search with graph exploration and um and the example that we used uh in in last uh in last episode was um uh if you remember that of uh it was a legal document it was like some legislation from ireland where we have a just a pdf right or even an html version but it was a text document it was there was no structure i mean there was some structure in it because the document has kind of parts, and the parts had sections, and the sections had definitions. So what we did, and I'm not going to go into that today, is kind of try to create a graph representation that somehow captures the structure of this document, and that's what we did. And the thing is once we have it in the graph this graph reflects this idea right so we have the legislation as you can see the orange note that can this is just one of the one of the paths that takes you to one of the definitions but it will be a kind of a tree uh there will be multiple of those in the documents so you have the legislation with parts with sections, and clauses inside the definition. And the thing is, while breaking down in small chunks is good in a way, because it will give you much higher precision when you do vector search, sometimes it can be a problem because if you get just a portion of a definition, because if you get just a portion of a definition, let's say you have a high match, a high proximity on a vector search on this portion, you might use these as the relevant fragment of information for your answer. But the interesting thing is that these should be taken into context. And this is only a clause inside a bigger definition. And this idea would be lost if we don't do the graph exploration and that's exactly what we what we did last time and the way we did it uh um and and you can also look at the other slides from previous uh episode was by uh and we use langchain right we're gonna go into the code we're gonna use python and langchain is is super convenient to to kind of orchestrate the interaction with the knowledge graph and with the LLM. And basically what we did is we have this kind of vector index handler where we can do vector search and it had this additional parameter, which was the retrieval query, where we could inject any random cipher, cipher is a query language for Neo4j, of course, that would basically implement this navigation in the graph that we saw visually before so to the results of your vector search when you do a vector search in on this handler every node return every match return you would apply this uh contextualizing query so you would run a cipher on each of the results and that's it so that was that was what we did. And that was great. It's super powerful, but it had some limitations. And I mark it there. So it's a bit rigid. Why? Because these queries had to be specific of your model. Remember? So of your model, of your graph, basically. So this cipher has to be prepared to explore, basically know the kind of information that this vector search is going to return. And based on these return nodes, know how to navigate that. And that's entirely graph dependent. And while that can be acceptable, what we're going to be exploring today is how can we make this a bit more flexible, a bit more dynamic? And that's where ontologies come into play. Make sense? Yeah, that sounds cool. And the idea is pretty simple because we've done it before. I mean, I put just a brief pause here just to, I mean, that's just animation. But you probably remember in episode 15 where we built, again, it was another ontology-driven, I mean, we call it semantic data, but it was a streamlit application that was entirely driven by an ontology. So the idea there was, okay, we're going to build an entirely generic streamlit application that probably showed some elements in the graph, but there's no cipher hardcoded in there. So I cannot anticipate what the shape of my graph is going to be, what the schema of my graph is going to be. So my application, what it's going to do is it expects some ontology, some schema definition on the graph side. We're going to read it, and this is going to drive the behavior of my application, and that's going to drive the kind of queries that i have to to create the idea again if you remember was instead of having bigger and heavier application logics which means a lot of a lot of code on this side maybe thinner application logic and more information on the data side so we put a bigger description in the graph and we'll see that an ontology is also a graph and my application is going to be dynamic, entirely dynamic. It's going to go read your ontology and drive its behavior from there. So that's what we did in this Streamlit application. And what we're going to be doing here is exactly the same. Basically, instead of giving you a hard-coded cipher query here, what I'm going to do is go read your graph, get the ontology, and based on the ontology that you share with me, I will derive the behavior that I want to implement. So it's going to be entirely dynamic and driven by what I put on the graph side using ontologies. Does it make sense? Yep, that makes lots of sense. Sounds good. Perfect. OK, so that was that was the the idea in that episode, which I think it's kind of relevant for these for this one. And well, from there, let's go and. See some code. So, OK, as usual, I've created a sandbox that i will destroy at the end of this session it's an empty one and i'm going to be driving from this notebook i hope you can uh it's readable is it big enough alex is it maybe make it a little little bigger okay let's try yeah i think that should be good i hope the um yeah the text fits in the in the page but well initially we just uh import i mean we're writing python we're going to be using langchain uh open ai i mean gpt uh as the as the uh um llm and neo4j as the knowledge graph for RAG. So here I'm just connecting, I've pasted the IP. So when you, this of course, like always is gonna be available in the GitHub repo in at the end of this session. And basically what I do is, you know that when you access your sandbox, you have all the connection details, user password and the URLs. So I've copied this and i'm um oops i'm here uh these should be my credentials so i can now connect uh from my code so one of the things that i can do from langchain is connect to the graph uh in in a standard way like using the the driver and running some cipher against it. And I'm going to use that to populate the graph. And I'm going to rebuild the graph with the legislation that we used last session. So I'm connecting to my graph. I'm creating an EFJ graph handler with my credentials. And I'm going to run these three steps, which basically um the legislation graph into my instance so hopefully in a second here we go we should see all the elements loaded here just as a way to uh to check that everything looks as expected we can run this query this query and we should get the expected results basically what I'm doing is I'm getting the path the tree that gets from the root element which is the legislation all the way to the definition of the term unavailable deposit can you zoom in here as well a little bit? Let me make this a bit bigger. Yeah, perfect. So, yeah, I'll show the cipher again. But basically what I'm doing is, okay, I follow, in this case, we created intentionally a pretty simple graph using just a contains relationship. That's something that can be, you know, modeling whichever way you prefer. And basically the idea of today's session is that there will be elements that change and they have different names in the graph and we want to be able to have our rag application dealing with with those in a dynamic and flexible fashion so basically i'm starting on a root that contains the asterisk is a variable number of hops. So there will be a chain of contains that take you all the way to a definition. And then the definition I'm saying that is that of the term unavailable deposit. And the definition will have clauses, right? And these are the three last elements. So we can see here that the coloring reflects that. So the blue ones are closest, the green is the definition, and the rest are sections and subsections inside the chain. So it looks like it's loaded correctly. So we have the tree that represents that portion of the legislation. And then what we did here is, you probably remember, is we are going to vectorize some of the content so if you remember there is um these definitions uh contain fragments of text right for example this is a definition of a deposit that has some text and that we might want to to run some vector search on it so i'm going to take these definitions along with the term and um and i'm going to um to vectorize that my strategy to do that and you can do it in many different ways i mean because uh when you use uh langchain here what you have to pass is the a node label so which the which node types you want to embed in which of their properties so you specify an enumeration of the properties that contain a text that you want to embed you want to vectorize an index so my strategy here because there are nodes of different types because we have like we saw we have definitions we have clauses we have all the types of elements, the ones that I want embedded, I gave them an additional tag, which I call embeddable. So that's my way of, I added to the closers, I added to the definitions and to all the elements that contain some properties that I want embedded. And like this, it's easy. I mean, I have a secondary, an additional tag, an additional label that then I can use to say, go and get all the embeddable nodes. And in particular, these three properties and go create the embeddings. So this is using the OpenAI embedding service. You can use your preferred one. So there is not prescriptive here. LanChain supports integration with a large majority of providers. So you can use Vertex, you can use Bedrock, you can use even local, you can use Ollama if you have a local LLM. So there's many options there. So what this has done is it has created a vector index that I've called vector index, not very original, and it's embedded these uh these values so if we go now to a graph and i do schema i should see a vector index which is of type vector that has been populated for vectors of type embeddable right and every node at the same time you will see that uh i'm going to have to rerun this query to see not this one the previous one and we will see that uh i'm gonna have to rerun this query to see not this one the previous one and uh we will see that uh definitions now have not only the definition before or closest but also a representation a vector representation of the embedding right and that has been persisted as a property good perfect so we are all set to start running some queries. So back to our code. The first thing we're going to do is let's get a question, the one that we used last time. Let's get the first one. Our question is, what is an unavailable deposit and how does it become unavailable? That was exactly the definition that we saw a minute ago right so we see that an unavailable deposit that's the definition means a deposit that is due blah blah blah blah and then it further defines it by saying uh it can be either and there are three clauses when the competent authority blah blah when a court has made for reasons which are directed i mean an enumeration of clauses that that further specify the definition of a of an unavailable deposit it would have been a couple of lines paragraph lines different you know different paragraphs in the in the actual pdf oh exactly exactly but but the point here is that you have to take the whole thing in in context because otherwise you would get just a fragment of it and that's what we saw here right so Exactly. But the point here is that you have to take the whole thing in context, because otherwise you would get just a fragment of it. And that's what we saw here, right? So if I take this question, I can do a vector search. So I have a vector handler now. That's what I, this created the vector, created the vectors, the embeddings, and populated index, but it gives me a handler that I can use to run some queries. And here I'm doing a vector search. I'm doing a similarity search with the question that I've selected. And this returns a number of matches. So this is in the visual that we used before, we're in the projected space, and we're finding the vectors, the fragments of text in our graph, in our legislation, that are closer to our question and the question remember is what is an unavailable deposit and how does it become unavailable and luckily enough yeah one of them is uh it refers to to an unavailable deposit right and and but it you know it's fragments of them some might be closer some might be uh incomplete so there you go so that's the the kind of results when you do a vector search now because these three components these three clauses make sense in context and all together what we need to do oh and let's well before i go into that we build that this vector search in what's called a rack chain right right? So we, yeah, lang chain gives us the possibility of creating a retrieval Q and A chain, which basically implements the rack chain that we saw before. So it will go and do the vector search that we just saw here. And with the results, they will be passed to the LLM and the LLM will return an answer. So basically these orchestrate the whole process. with the results they will be passed to the llm and the llm will return an answer so basically these orchestrate the orchestrates the whole process so if i run if i create this handler and run the question uh it will tell us you know in natural language that the definition that mentions uh oh sorry i why did i that's That must, it looks like it seems like it's taking a, oh, I hadn't run this one. Did I not run that one? Oh, that makes a lot more sense. Unavailable deposit. Yeah, I was getting, I think I didn't click on this one and it was getting the results from a previous execution. Anyway, when we have an unavailable deposit, the closest answers are, yeah, the definition of an unavailable deposit, but then there's the closest and some other fragments sorry maybe people were a bit confused because the results were not making a lot of sense but anyway these are the vector results that match this search and then when we run uh the the whole chain the rack chain that does the vector search passes it to the lm then what we get as an answer there you go is an unavailable deposit refers to a deposit that it's due and playable so it looks like it's picking something right probably the first result which is a good part of the definition but we know that it's not complete so that's the kind of experience that we're going to get if we go just with vector search so that's exactly what we wanted to implement these advanced patterns. And that's what we're gonna do. I mean, I'm not going to repeat exactly what we did last week, but what we did is we implemented, okay, we search and then we, on the results, we navigate to complete the picture and provide a consistent context. Now, what we're gonna do is I'm going to define, this is a bit large, I don't know if I'm going to, I mean it's not very large, it's just two, it's a function, a couple of functions, but basically what this function is going to do, it's going to generate dynamically the context query, because further down, as we're going to see, I'm going to create this, as I showed in the slide, I'm going to create a contextualized vector store that is, again, another handler on the same vector store, but I will inject this retrieval query at the end. So I'm going to say, do the same thing that we've done before, but to your results, apply this contextualization. And this is not hard-coded anymore. This is going to be run dynamically. And this function, getDynamicContextQuery for a particular pattern, is the function that I was showing here. And I'm going to try to go into a bit of detail here. So the first thing, it's going to go and get the ontology from the graph. So in order to have that happen, and we see that there are resources, there are subproperty odds, we'll make sense of that in a minute, we need to have an ontology in the graph. And we don't have it at the moment. We only have the graph that we've created. So let's probably go and populate that. To do that, we're going to use you can use your favorite ontology editor I'm going to use protege which I've used in the past that's the local I mean the desktop edition of the the web protege that we've used also in other episodes and what I'm going to do is I'm going to basically annotate my ontology so in my ontology describes the content of a graph right it's a kind of a an explicit description of the graph in a way we could we get something close to an ontology when we do this DB schema visualization so if I do schema visualization wow that's that's not very good that's why an anthology is much better i mean what we're doing here and yeah that makes sense because because of my use of of the related to uh and um and the contains relationship so essentially we have um um this is clause, this is the legislation, so this shows the different entities in my graph, and then the possible relationships between them. This is of course not formally described by anyone, this is just learned from the graph, and because we use the same two relationships, related to and contains, you will find them across every possible pair of nodes which is a bit confusing and not very helpful in this case so here's where you will probably want to create a real ontology with a formal definition i mean pretty much the same thing which are the entities and how they relate to each other and eventually annotate them as well which is what's going to be useful for us what's going to be useful for us so let's do that so i go back to protege and i'm going to say okay i'll create my ontology and i'm going to give it a uri for example i'm going to call it going live and this is my ontology uh let's call it it's going meta24 and this is the legislation. Whatever, I mean, you just pick a name. For example, this one. And I'm going to import an ontology and I put it here because I've created a code ontology for rag patterns that just defines a couple of patterns that I've implemented. These are the ones that I've implemented. So these will be extended, of course. This is a turtle definition, but basically it describes there's a further detailed relationship. So I'm going to describe that there are some relationships in my graph that are useful to provide further details. That's the idea of the definition before. So when you have a definition, the definition can have sub clauses. So this relationship, whatever it's called, contains in my my graph but it could be called whatever you want in yours i'm going to annotate it and say this is a relationship that is of type further detail relationship because these relationships are the ones that my rag application are going to take to use and expand the result so that's what i try to describe here further detail relationship is a relationship that connects an entity to related entities that provide further details about it, right? I'm also defining another one, which is the one that goes backwards. It can be seen as a synonym. I don't know if it's exactly the same. Is this readable? Should I make it a bit bigger? But I call it inverse contextualizing relationship. So what happens if I get a portion of a relationship, sorry, a portion of a definition or a portion of a document? Sometimes I want to put it in context and I want to navigate to the parent. And that's what I think Thomas Bratenich and others have called the parent pattern. So it's a way of going to a container and then retrieve all the siblings to get not only the exact match to your vector search, but all the additional elements that are connected to the same parent. So it's a way of bringing, again, additional context that's potentially relevant for the LLM to generate the answer. So these are the two relationships that I've defined. And I've implemented the associated behavior in my RAG application. fun and and i've implemented the associated behavior in my rack application so these are the two and i'm going to import this uh these ontology in in protege because i'm going to extend it so i can import ontologies here i'm going to say that i am going to import an ontology that's located on the web and i have a i've added a bookmark here so this is my ontology again all this is is public and we'll add it to the repository but uh if i continue uh it will import it finish and it's all done so this ontology has been imported uh and now if we go to my entity view there's no classes defined that's what i'm going to do now and uh and there's no object properties well there's the two that we defined there's the further detailed relationship and the inverse contextualization relationship okay so let's I don't know let's create a class for example if we go to our graph you probably remember that there were definitions and definitions contain clauses right the way to do to represent these in an ontology would be something like okay there is a class I'm going to add a subclass that is going to be called a definition good so I create this entity and there's another entity which is a close I'm not going to create the whole ontology but you see what I'm going we've done these in other episodes I believe in this one with streamlit so you have the whole condition of the ontology but basically you describe the entities in your the types of nodes in your in your graph and then you can define the relationships between them and in particular here i'm going to create the relationship that exists between these two which is called contains it happens to be the same across all the graph it's probably not the case in in any other real we've called it like that here in particular here let's go back to my project, contains is a, let's call it is an object property. An object property is a relationship. I mean, in in RDF in OLL, which is the language that we're using here, because the definition between object properties and data type properties in a Neo4j graph in the property graph, an object property is a relationship. And a data type property is an attribute as a property of a of a node. So basically, attribute there's a property of a of a node so basically those that have a literal value that can be a numeric a string a boolean an array so in this case we're defining the relationship contains so contains is a relationship but and it's also a relationship we can say that um uh it has a domain and a range it will remember that domain and range is the definition of what entities does it connect so the domain of this relationship will be it starts from a definition and it ends the range ends in a clause ends the range ends in a clause okay so basically what we're saying is that the contains relationship it's it's not entirely true because we're using it for other cases but in this the fragment that i'm interested in is the one that connects the definition to the clause and i want to specify that my contains relationship is a further detailed relationship is a relationship that actually i want my rag to use to to further specify to further contextualize the results of the of the vector search so that my way of specifying that is saying that this is um a sub property of further detailed relationships so if i do, it will naturally put it in the hierarchy under its parent. And that's, I mean, I'll stop here. That's enough for me for now. But you see how we could define the elements in my ontology and annotate them. So what I've done here is I've said, okay, there is a relationship called contains that connects definitions to clauses. And this relationship is very particular because it's a further detailed relationship according to the RAG ontology. What my RAG general purpose engine is going to do is going to go and find that there is a further relationship existing in my ontology and it's going to generate cipher based on that. So let's first save this. So if I do file, save, and we're going to generate cipher based on that. So let's first save this. So if I do file, save, and we're going to choose. I mean, we could pick any syntax. I like turtle. I think it's particularly readable. So if I save this, I'm going to give it a name. For example, I don't know, something like going meta24 onto legislation. And I save it locally perfect and now I'm going to show you what that looks like now it's in my local drive and this legislation if I open it with my text editor, it looks something like this. Let's make it a bit bigger. And again, I don't wanna go into the details of the syntax, but you see the elements defined. So we've defined, I mean, we've created an ontology, that imports the rag ontology, which is a public one that we have created, but it's public. And it defines uh one object property the the properties contains and it's an object property that's a sub property of a further detailed relationship and it's a property that connects a definition the domain is definition and the range is a clause which are themselves classes so this is all right so we have an ontology definition. It took us two minutes. It's a limited one. It's a small one. Now I'm going to put that in my graph because this is what my RAG application is going to go read. And based on that, it's going to build the relevant cipher. So how do I do that? Well, I can go to my graph and i can import ontologies using near semantics so it's i'm in sandbox so it's already installed i can do something like pull antennas onto import actually before we input let's uh let's preview let's just let's see what it looks like and um actually i'm i mean i could fetch it from a file but I'm going to literally copy paste it and use the inline flavor of the method so if I do that and say that it's turtle all I'm going to do is like I said paste my ontology here I'm literally copying it from the file that I just showed you and pasting the content here. So these ontology import ontology preview, they have the two flavors, the fetch where I can say, go read this file and take the content and the file can be local, can be HTTP, can be FTP, whatever. Or I can just inline the ontology. And here what I've done, like I said, is I pasted the file that I just showed you. It's really copy-paste, right? So I'm not loading it yet. I'm just previewing it. And if there's no errors here, that should show me, well, more or less what I was expecting, right? So there's a class, sorry, a relationship, right, as you can see, called contains. let's make it a bit bigger so contains is a relationship you can see it here that has a domain definition and a range clause right so it's a relationship that's established between definitions and clauses and it's also a sub property of a sub relationship of a standard one called further detailed relationship so that's exactly what i wanted to do make sense yeah so it tells me basically that this is part of of a of a you know like graph which which is not not a you know how do you say it say, it's part of a tree, if you will, and then you can, when you look at it, you should be looking also at the nodes and the relationship that come before it, so it knows it's not stuck on this individual point. So this, in a way, this is kind of a separate graph. So once we import it, and we can run it right now, because this a preview let me let me run instead of preview let's run the import so now we've loaded it so there were six triples there we've imported them and they've been persisted in in the graph so now we have this these uh loaded in our in our um in our neo4j instance so we have it here persistent and it contains if we expand it here, persistent, and it contains, if we expand it, we have the domain and range. So you're right, this is a graph. It's another graph, but it's kind of a metadata graph. So it's kind of separate. We have our legislation with all our nodes and relationships, and separately, we have a separate graph that describes my graph, right? My instance graph. So it's somehow as if I had persisted as nodes and relationship the result of my call db schema visualization right so uh in this case i am i am explicitly stating things rather than learning it from the database so i'm saying there's a contains relationship that is established between definitions and clauses. And it's a sub property as a sub relationship of these. And there could be more and I could actually, I mean, we could perfectly well come, come to my, maybe in my graph, there is, there's no others, but we could add all the relationships saying, Hey, this is also a further detailed relationship. And it's built in a taxonomy. I mean, in this case, there's a one level taxonomy, but that's exactly what my what my rag application is going to search for. It's going to go read this kind of descriptive graph, the metadata graph that describes what am I going to find here? And it's going to build a query based on that. So that's the ontology driven part. So it's going to build the query based on that so that's the ontology driven part so it's going to go read the ontology and based on that it's going to generate the relevant query what that means is that the query itself is not hard-coded in my in my rag application the the query is dynamically generated entirely based on what i put in my graph so okay that's why my application becomes way more flexible and I take control of its behavior from the graph, from the ontology definition. Make sense? Yeah, that makes a lot of sense. Yeah, I think that was helpful for everybody to understand that you take the graph, whatever it looks like, and you have an explanation layer basically on top of it, that then a system like LLM can understand and read and knows exactly what's individually happening in each branch. So let's say here it contains, but there could be other relationships. There could be, I guess, node labels as well and stuff like that. And that would then be helpful for the system to understand, okay, how the world, the graph basically is connected, what the individual parts of the graph are and how it then can interpret that and make work. This is exactly what we did. I mean, sometimes it's called like a semantic layer, like a description, a semantic description of what's in the data that can be consumed by any application. And if you go, any of the people attending wanna go look at episode 15, we did exactly that. So we created the definition and then my Streamlit application would go read the ontology and dynamically generate the queries that would consume. And it's also a way of defining views over your data, because maybe you want to expose part of it and not all. And maybe instead of having an application that behaves dynamically based on the ontology, you want to do some reasoning and the reasoning. And we did that also in episode, I believe it was four. You probably rememberlex well we decided we expressed something like uh uh a director or what was it or an artist is someone who performs in a in a play right so we defined and then because you performed or i performed in in going meta we were annotate we were tagged we were marked in the graph as artist or whatever. So basically some behavior is dynamically driven based on what our definition is in the ontology. So this approach of formally putting some meaning, some semantics explicit in the graph and having some software in the more general term, it can be a rag, it can be a stream data, it can be a reasoner, that does something based on that is this idea of the explicit semantics of the semantic layer or however we wanna call it. Yeah, yeah. So if we go back to the, I mean, the whole reason why we've done this is because now we can go back to our rag application and this is looking exactly for that. So we know how ontologies look like. I mean, the sub property of relationship and let me go look like. I mean, the property of relationship. And let me go down here. I mean, I think I've run this code. But you will see how we can test. So there's two types of queries, further detail or the other one, which is inverse contextualizing. So if I run this query now, you see that it's generating something that contains, that relationship contains. Why? Because it's gone to the ontology. It's read that contains is a further detail type relationship. And it's found it and populates the graph like that. Just to show very quickly how this is dynamic. You'd like to reload. yes, let's do that. Yeah, I created, I mean, this, so let's say I'm gonna create another one. Let's say there's another relationship that's, it doesn't exist in the graph, right? Hang on. Not this, I'm going to use this. Another relationship, it's non-existent, it's just so that you can see how the query is built dynamically. Let's say includes. Let's say that there's another portion in the graph, in the legislation that something includes something, that that's also there. So I'm annotating it also as a further detailed relationship, right? In this case, I'm not even specifying the domain and range. So I'm going to do this and I'm going to save it. So if I save it it i will have my uh um ontology here let me you know you have to i have to reopen it because the text editor doesn't get the refresh so if i open with with text edit there we go let's make it a bit bigger we see that now this contains and this includes right so if I take again the whole content and do exactly what we did before if we not here here. So if I go to my ontology imports and replace the previous one with this new version and I'm doing a quick and dirty copy paste. Now it's eight instead of six, as we saw before. So if I rerun this, bang, now we have two, we have includes and contains. So we have two we have includes and contains so we have two relationships that i can use in my rag application to you to do further detail expansion right so if i go back to my uh my notebook look at this query before it was just contains if i rerun it now it's finding includes and contains because it's found two types of relationships that are worth exploring. So this is entirely dynamic. The moment you change things on your graph, the behavior of your RAG application is going to change. Why? Because you're being explicit and saying this relationship has this behavior. And this sort of engine, if you want, reasoning engine, whatever you want to call it, interprets that as this is the way you want to use it. So long explanation, I'm conscious of time, but basically what I'm doing here in this step is if I run it, I'm going to basically create a contextualized vector index handler, basically on a dynamically generated cypher query so what happens if i ask the question now and everything goes well as expected look at that so now the the result from scanning is not fragments is getting the whole package is getting an unavailable deposit is the data and it has the three sub clauses so it's fumbling it all together basically it's running this query that I showed you so when we have a node resulting that contains or includes something take these these queries entirely generic so it's taking all the properties in the node and generating them so I don't care what's there it's entirely dynamic that's the whole point and returns these results so So this is what this is returning So we have all the information and if I pass that if I build another Q&A chain on these contextualize Handler it's now going to produce hopefully a much richer definition of what a Unavailable deposit is. There you go. So now we have an unavailable deposit is blah, blah, blah, and it has a totally accurate description of one, two, three. So I've made sure that the LLM gets all the relevant context perfectly bundled so that it can generate the rich response that's accurate and complete. Right? Yeah, yeah, super cool. This is one pattern. But we I mean, like I mentioned before, we have other examples when it goes the other way. And here, the sky's the limit. I mean, you decide which are your patterns and what's the best way to chunk your document. So that's what I came up with these two ideas. So there was the other pattern where I I was talking about reverse contextualizing. So it's kind of the opposite. Instead of getting a definition and adding all these children, I'm doing the parent approach. And that's the other idea. So let's say that in my ontology, and I'm going to do this quickly, let's say yes, I'm going to create, indicate that that contains it's actually not only a further detailed relationship but it's also of the other type it's a contextualizing relationship so i can use it to navigate the other way so if i persist that and it was file save save and I reopen my document with the text editor. There we go. Now we have... did it reopen it or contains? No, it didn't. I have to close it. I'm opening it with the text editor which is dynamic. So now it basically looks both ways exactly so well now it's defined i mean we can choose how to i mean this is too small let's make it a bit bigger but you see that contains now is defined as both a little bit smaller further detail but it's also if inverse inverse contextualizing so it's a relationship that I can navigate in both directions right so again let me do it I'm doing it this manually of course this will be part of a of an automated kind of ETL chain of some sort but anyway for this session I think it's good to do it like that I'm literally copy pasting again. Now the newer version of the ontology, boom. Now it has another one. And if I rerun this query, now we have, there you go. So we have further detail and contains is both further detail, but also inverse context, right? Nice. Yeah. So again, I'm enriching the ontology. I keep adding elements and remember that contains also had the domain and range right so there's there's additional elements so i can keep adding explicit fragments of knowledge to my uh semantic layer to my ontology and that will further drive the behavior of my rag application so if i go down here now i'm going to do something similar so uh instead of the this pattern i'm going to use the other one and of course you will be wanting to use both in combination here i'm use them use them using them separately but let me uh create another one uh and uh basically this is the the opposite right so if you get a detailed fragment it will navigate to the this is the opposite, right? So if you get a detailed fragment, it will navigate to the pattern. It's the pattern template. And for that, I'm going to use a different query. Sorry, I'm accelerating a little bit in the interest of time. But for example, what definition mentions section 17 of the Building Societies Act? Boom, just one question. So if I run this question, look at this. So like we did before, you will get vectors that closely relate to this one. For example, there's the building society. There's the section 17. There's a 17 here. I don't know if that's the reason why it brings it up. So there's things that are similar to it. Again, they're probably being used in isolation so there you go section 17 up so this is probably the the the best and the right match but it's incomplete right because it's a close of a bigger definition and what i'm interested in what i'm asking actually is what definition mentions that because i don't know and if if i get only that and i build the rack based on this fragmented information it's going to tell me that the definition is the definition of a building society under close b which is it is incomplete right because uh let's let's see what it's like actually let me um so uh how can i do that so i can search for a close as we saw that. Let me see. That's and and dot definition starts with something like that so if I run that there you go so we get these why is it so big so we get we get the close but if I expand it I probably get what I'm after so this is is contained in the definition of uh definition embedding the term authorized so what's an authorized I don't know whatever authorized means in this context in relation to a credit institution so in the definition of authorized if i expand it we have a mention of these uh regulations section 17 of the building societies so what i'm up to really is this this definition and what it's telling me as an answer is just based on this fragment which is incomplete as an answer is just based on this fragment which is incomplete right yeah so that's what i'm getting now in this case is a perfect k perfect scenario for this navigating backwards i've got into something that's very detailed and i want to put it in context by navigating to the parent and that's what i've done here so i've created uh um not a further detail, but the inverse context. So if I run this, I can now, I suppose we have the question loaded. Now the context, look at that. Perfect. The defined term. So instead of getting me fragments, now it gets me the whole thing because it's applied the contextualizing query that hasn't been generated dynamically again. So we have the defined term, which is authorized. And then the definition is all these sub nodes that we saw before. So this is much richer and much more accurate context to pass to the LLM. And if I pass it like we did before, I would expect to get a more accurate and richer. Oh, that was concise, but they're perfect, right? So it's the definition of Oprah. It's going definition of. Perfect answer, very short and concise, but there you go. I'll stop here because I see it's the top of the hour. But I know it's a pretty dense session because it's one of those where we really go meta, but people get the message. What we're doing here is we are making the rag part more dynamic so that it's driven from the graph side. So I decide what to put in my ontology, and that drives the behavior of my rag. At the moment, my rag application, as you're going to see when you have a look at it in GitHub, it only implements these two behaviors, these two patterns, the further detail and the inverse contextualizing. But you can think of adding several more. I mean thomas has published things like uh what was it the synthetic questions the summary so you can do multiple patterns and and have kind of micro reasoners on the rack side that can work on any graph and that's the powerful thing of having it data driven ontology driven semantic layer-driven. I'll pause here and see if we have some questions because it's, like I said, I know it's been a dense one. So I'll give people a break. Totally. But I think it's very cool. I mean, last episode basically was already very interesting. But now basically this is making it more and more usable, I would say. I mean, not that the other one wasn't usable, but the bigger your data pool is, the bigger your graph is, the more you don't want to type it in yourself. You want to have some kind of layer in between that takes some workload away from you and gives you... It's a good point that you bring there, Alex, because some people might say, hey, I don't need that level of complexity and flexibility there. I mean, I'd rather hard code my query because it's only going to work on this graph that I'm working perfectly fine. Maybe you want to... But in a way, you're kind of reinventing the wheel every time. So having these kind of repeatable patterns that are generally useful so this idea of navigating to the pattern to the parent to contextualize or further uh specifying something i mean these are kind of it's like you're building a library of tools that you can then reuse and and and this works across graphs i mean one of the things that i was thinking if i was maybe taking that and show how it works in a different graph. All I have to do is layer the ontology on top and it will work exactly in the same way. It's too much for a single session, but yes, I mean, it's yet another tool for people to think, do I want to make my logic a bit more flexible, more dynamic and drive from the ontology? Maybe not. Maybe I just happy hard coding and some people might go down that route. So it's both are fine. Yeah, totally. We had a couple of questions. So let's try if we can make it work. One or two actually from Ultra Eagle. They say, would it make sense to track the queries being run for traceability purposes? Absolutely. Yeah. And I didn't show it. And you're totally right. And one of the biggest problems of these RAG applications is traceability. What's going on? Because you see that in this single line of code, when you build the chain, there's a lot going on because you're going to the external store, to the graph store, you're doing a graph search, you're enriching the results, you're passing it to the LLM. so you want to really track what and for that there are tools for example langchain has uh can't remember the name of the name of of the tool they offer some i don't know if it's commercial or open source but yes i mean you could build an element of that some logging in the code that i've created but there's there are tools that that do that for you but But I totally agree. And when you're working and debugging and trying to find where things go wrong, this traceability is critical. Yeah. So spot on. I totally agree. Cool. Then they follow up with, does it make sense to create virtual nodes from the ontology and then generate the query? I mean, you generate virtual notes when you're kind of doing the preview, but you have to persist them at some point, because if you think of it, the rag part will go and query that. So the ontology has to be persisted in the Neo4j graph so that I can query it and get the description in the ontology. So the virtual node is also useful when you are kind of exploring the ontology before you import, but it has to eventually be persisted. I mean, think of these multiple, even multiple rag applications. So you need to have it persisted and centralized it. So I don't think maybe unless i'm missing something the virtual nodes i mean the visualization is one part but you have to persist it eventually so that these elements um yeah work uh work correctly so oh lang smith exactly that's that's uh thanks roy so there's um that's that's one option but the lang chain offers these these um these tool to to give you all all traceabilityability on what's going on in the chain. Yeah. And there was one more from Skander. You created the ontology file manually. Do you think it can be generated automatically based on a document? I mean, you can, but that's a different problem. I mean, you can, but that's a different problem. I mean, so you can, yeah, if you give a document to an LLM, for example, an LLM will do entity extraction, and it will tell you these are the types of entities in your document. So you can generate an ontology out of unstructured data. But in this case, it's different. Here, I did want to take control of, because remember, that is driving the behavior of my RAC application. So to answer your question, you can do it. But in this case, I wanted to own that process and indicate this relationship is one that you might want to use to contextualize in one direction or the other. And I wanted to own that. So that's why I went down with the manual creation. But yes, absolutely, that ontologies can be, I mean, they're always going to be halfway between the automatic generation and the curation. I mean, ontologies in a way are kind of capturing experts' knowledge. So yes, you can get help with automation, but I think it's inevitable to have some element of curation and human in the loop in that and in this case yeah it just it made sense for us but yeah question yeah thank you there are a couple more questions i don't know how much uh are we looking for time i mean we can we can go over them uh briefly but because we already over um maybe we briefly touch them um one is um do you have ontology templates examples in neo4j so that's more like a general question i guess not necessarily yeah uh i mean we have created in this series a bunch of them, and you will find references to them, and we publish them normally in this URL that we use today. But they're not really, I mean, other than in our collection of assets of the going meta series, they're not very well documented. I mean, they just mentioned. So, I mean, the quick answer is no, we don't. Maybe that would be an interesting thing too. But yeah, given unlimited time, that would be something. Yeah, yeah. So, I mean, we don't have one. You will find some examples in the series. Maybe in the life sciences space, there is BioCypher, which is kind of going in that direction. So, I mean, that could be helpful. But, you know, there are as many ontologies and semantic descriptions of as many topics in the world you could look, you can think of. So it's a tricky thing. Yeah. okay, cool. And the other question, just to close it down, was from Viditya and they say, would I be able to compare two versions of a document that has changed over time using Neo4j, also more generic than... Yeah, I would say that's a slightly different type of problem. Here, yes, here we assume that we're, I mean, we didn't even touch on the versioning, on the tracking changes. I would say there is an episode where we did ontology versioning. So I would recommend you have a look at that. It's just part of it. But so yes, the answer is yes, you can do it. You have to define the scope, the depth of the change tracking that you want to build in the graph. But yeah, we didn't do it in this episode. Maybe have a look at this ontology versioning episode and that can get you maybe started. Yeah. One more, I think, because it's an interesting one, is from Angel or Angel. Tell me if I got it right. You query the ontology to get the extended cipher queries. Have you tried to use the LLM to use the to use the or take the LLM to use the ontology to query the graph directly? So have I tried to use to have the LLM use the ontology to query? I mean but how do you I mean you mean probably pass the ontology to the LLM mean the the the thing is the llm is a passive element so you can ask i mean the lm is not going to go uh query i i suppose the i mean i suppose there are different patterns here i mean the the short answer is is you know in this case we were looking at this particular pattern where we have some context and some structure in the graph that we want to leverage as part of this kind of advanced rack pattern. But I suppose you could pass the ontology and context and have the LLM generate a query, if that's what you mean. I mean, that's something that we've done and that's exactly what Langchain, I mean, that's something that we've done and that's exactly what one of the capabilities of language chain is the generation of cipher and it does something very similar. You will go query the graph, you will introspect the schema elements, and based on that it will generate queries from natural language. But that's yet another interaction pattern. So that's where you want to, you're not doing this approach of vector search and further enrichment, you're trying to generate cipher and return results. So that's possible, but different type of scenario. Different, yeah. Okay, cool. I think we close it for today. Thank you everybody for your questions. If we couldn't get to yours, sorry for that. But you can always reach out on the Discord server, the community forum as well. And on various other outlets where you can interact with us, ask questions. And we should be able to find somebody from the community to help you out or to discuss further things. And yeah, it would be always interesting to hear your questions, hear your thoughts. So if you have any other points, use the YouTube comments section, also a good one for feedback, for inspirational ideas you have. If you try this out yourself, if you want to give us feedback, then use this outlet for that. Let us know if you are struggling with something or if you tried something and had a good result of it, then obviously we would like to hear that as well. Always nice if these, you know, we show you a little bit how it could be done, but it's these, you know, we show you a little bit how it could be done, but it's just, you know, a teaser sort of an inspiration for you to see what's possible. If you then take one of these episodes and build something out of it and make it work in a full scale or proper application scenario, then that would be great to hear because that's helpful for us to see that this is something that people actually take on in a daily life scenario. And yeah, I think we are at the end. The next session that's coming up is obviously in February, so next GOING Meta is in February. is in February. Next Tuesday, though, we'll be back with another Neo4j Live session. And on Tuesday, we discussed unique graph use cases for reducing risk and improved bottom line. So it's more like a bit more classical graph database technology session, but it should be interesting. Some learnings there, some good feedback, some good use cases of how to use Graph Databases and where to use them. So looking forward to this one and yeah, thank you all for watching. Thank you for sticking around and again, happy 2024 and see you soon. See you. Thank you. Thank you. Take care. Bye bye.
Going Meta - Ep 24: KG+LLMs: Ontology driven RAG patterns
Episode 24 of Going Meta – a series on graphs, semantics and knowledge

Jesús Barrasa: https://twitter.com/BarrasaDV
Repository: https://github.com/jbarrasa/goingmeta
Knowledge Graph Book: https://bit.ly/3LaqE6b
Previous Episodes: https://neo4j.com/video/going-meta-a-series-on-graphs-semantics-and-knowledge/

0:00 Intro
5:10 Ontology driven RAG patterns
17:20 Code 
1:00:31 Q&A
1:08:20 WrapUp

Check out https://community.neo4j.com/ for questions and discussions around Neo4j 

#neo4j #graphdatabase #knowledgegraph #semantics #ontology