 All right. Hello, everybody. And good morning, good evening, good afternoon. And thank you for joining to Going Meta, episode 23, Advanced Rack Patterns with Knowledge Graphs. So this is the first, Jesus. We continue immediately after previous episode with the topic, kind of. immediately after a previous episode with the topic, kind of. It may be, yeah, that's probably right. But it's such a rich topic and such an interesting one that I wouldn't be surprised if we spend a few more on it. But yeah, you're probably right. No, I'm sure we will. And it's a great topic. But first, thank you for joining, Jesus. How are you doing today? I'm very well. I'm not as cold as you are probably. After seeing the pictures, it's pretty crazy over there, right? It was crazy. Yeah, I was at my parents' over the weekend. And basically from, I mean, it started snowing Thursday afternoon or something. And Friday evening, it was already like 20 30 centimeters of snow saturday morning we woke up and it was like almost a meter of snow everywhere we were completely snowed in uh there was no way of getting getting anywhere so until saturday sunday early afternoon or something we were we were kind of like left for us it's like the you know internet connections have not been affected. No, the internet was good. We can see you well, which is great. That's amazing, yeah. No, it's all good. It now already seems to be thawing a little bit. So at least here in Munich and back in Munich now, the temperature has gone up. But it was super cold. It was minus 15 or something so serious temperatures i don't think we've experienced that here i mean ever i would say but happy to keep things like that to be honest yeah yeah don't need don't need don't need that every uh every other day so um so for sure now i know how our colleague adriana feels like when she goes out of the house in Canada where she lives. Looks like it's chilly in Chicago as well. Oh yes. Hi Daniel. Good to see you. Yeah. There's a little a drop here and other places. So yeah, so it's good, but yeah, this is, like I said, episode 23, last episode, we discussed a little bit of RAC and then on all of that and how does it work? What does it mean? What is going on in terms of knowledge graphs and LLMs and how does the two concepts flow together or how do they combine well and how do they add things towards each other? I guess it's a good way of starting the conversation. If you missed it, don't worry. I think you can, as usual, even though this episode may be a little bit building upon what we talked about last month, but you're probably going to be up to speed quickly. And then, you know, you can always watch the episode later on. Yeah, I'll do a couple of minutes recap of what we covered last session. But yes, the recordings are up and all the assets. So it's up there for everyone to try at home and yes, it's, yeah, should be, hopefully, I mean, it's pretty dense. I mean, as usual, have loads of content. We'll see if we manage to squeeze it all in the session, but we'll see. If not, we'll have for the next session. Have space for the next one, exactly. And yeah, if you watch us live, then please use chat as you go through the content. If you have any questions, anything is unclear, any comments, even if you just want to say you like it, it's also nice to hear. But if you have any misunderstandings or anything is unclear, then please let us know. If you watch this after the fact that we streamed this today, then obviously the YouTube comment section is a great place to put something in there. Even better yet, the community on Discord or on the Neo4j community forum are places where you can ask questions and we can have a look for them there and help you out if anything is is unclear afterwards perfect so shall i go ahead and and share my screen yep let me know if you can see it i'm going to be sharing my whole screen and moving this out of here. Oh, and I see Noel joining from Argentina. Probably he is having a completely different experience there, shorts and a t-shirt, but also nice. Different type of Christmas to the one that we have here. Yeah, yeah. Good, good, so you can see my screen, right? I can see your screen, yes, here we go. Good, good. So you can see my screen, right? I can see your screen, yes. Here we go. Perfect. Good. So, well, yes, like you said, I'll take it where we left it last episode, right? And we talked about RAG. And just to recap and to refresh, for those of you who missed it or who have been under a rock, like I said, you probably haven't heard about RAG. But RAG is kind of an architecture, right? you who missed it or who have been under a rock, like I said, about a rag. But rag is kind of an architecture is like, you know, a framework and approach is a way of building applications based on large language models, but integrating in real time with some external source of facts that's curated, that's trustable, that's explainable, and that enriches the whole, I mean, the quality and traceability and explainability of the results. So that's high level what it is. And that can happen in many different ways. Of course, the one that we've been exploring here is when we use this kind of grounding partner, this external source being a knowledge graph. But the idea is like you see in these diagrams. So we have some form of front end. It can be an interactive one. We looked at chat style applications, but it can be anything. And today, actually, we're going to be looking at something that's more kind of a batch process where we generate personalized content for emailing or things like that. So anyway, some form of application that interacts and leverages the conversation, well not the conversation, the linguistic and the reasoning capabilities of an LLM, but that injects the knowledge of the world, the facts. Because as we've heard and we've discussed in the past, large language models are great at these things, at reasoning, at talking, at generating content, translating, summarizing, but sometimes they struggle with world facts. And of course, they've not been trained on your own private data, your organization's private data. So that's something that you're going to have to inject. And that's exactly what this architecture, this approach that's right solves. Right. So that that's the that's the idea. And again, last episode, we touched briefly on how rag knowledge graphs in rag, you know, improve the quality of the solutions. And I have the impression that, you know, I kind of went over it high level, like saying types of solutions where the graph captures the structure of a document. And because you're structure aware, then you can leverage that structure to generate. And I'm going to explain what that means, because you know that here we're big fans of not using just the words, but showing what we mean by them. But I talked about this kind of modeling in the graph, the structure of document, the sections, the subsections, the images, the tables, and things like that. We talked also about the idea of having a knowledge graph where we have entities, where we have relationships, and using that to enhance the result of a vector-based search by doing some semantic search, vector-based search, and then taking the results and doing some post-processing and contextualizing them, enriching them by navigating the graph. But even though I explained that, I have the impression that the demonstration, I mean, the experiment that we ran last time, look at the two approaches, the vector-based approach and the graph-based approach separately and uh and maybe some people came with the conclusion that you know you take one or the other but not the two and that's exactly what we're going to show today so how the combination of the two in these two type of examples is really really powerful so the idea is that now as you know the mvj uh supports vector-based search since the introduction back in August of the vector index. Now we can combine the two things. And that's really, really useful. And that's exactly the plan for today. So does that make sense, Alex? Yeah. Yeah. Perfect. That makes sense. Yeah, that's cool. That's cool. That's interesting. I think that's a good point that we showed previously, the two different approaches and that we now go and combining them, I think, is an interesting story. Yeah, so the first one focuses, and they're quite different from each other, right? The first one focuses on, like I said, unstructured document. And it's something that we used a few weeks ago in the summit here in London. And it went down quite well and people liked it. So I thought I'd reuse it here, at least a part of it. And here the idea is that we have a legal document in this particular case. It's something that my colleague Michael was asked to do by a customer. And there's this PDF and you have the URL down there. It's some financial regulation in Ireland and in the document it's a PDF if you go to that URL you'll see that it's described and it contains a number of parts each part contains sections and and yeah it's natural language and then sections contain things like definitions. I intentionally put that fragment. I don't know how readable it is. But this block there contains the definition, right? So if you look closely... Let me just go to the little bit. It's a bit small. Well, anyway. You know what a definition is, I guess. Well, we're going to see that in the graph in a minute. But basically, that describes what an unavailable deposit is, right? And when and how does it become unavailable and these sort of things. And you see that it has different, you know, clauses in the definition. There's A and there's a section, B, C, and it happens to be across different pages, which can also be a problem. Because if we look at the way we traditionally would take a document like this one and put it in a vector index, it would be that by doing what we call chunking, right? You take fragments of it, you know, but you could, of course, vectorize the whole document, but then the precision of the search would not be very, very good. So what we normally do is chunk the document in small portions, create entries in the vector index, and then do searches on that. That's exactly what we're going to do in this experiment. But the main difference is that what you see there, to me, and I'm sure to many of you joining this session, it looks like a graph, right? Because you have a regulation that has parts, the parts have sections, the sections have... I already start to see a tree and that's exactly the case. So that's, we're going to see that in a minute. But if we model that in a graph, and I will not cover so much the construction, because I want to focus on the rag part. But you see how that, and actually the LLMs do help. And that's the part that Michael did in the graph summit. So you can also take that PDF and ask the LLM to do the structuring of the document, the structure extraction of the sections and the different parts in it and generating a JSON based XML based whatever. And that's something that you can then ingest into the graph. So the construction is also something that LLMs will help with. But let's start from the point where we have that already. So we have a node representing the legislation. It contains a number of parts, as we saw in the previous screen. The parts contain sections, the sections ultimately contain definitions, and the definitions will contain clauses. So this is a standard graph. Now the thing is, some of these nodes will contain fragments of natural language text. For example, the definition like we saw before will have a piece of text, same with the clauses. And this is the kind of thing that I will want to first vectorize and then index in the vector index. So that's where the demonstration is going to begin. So how am I gonna do that? Because I wanted to jump from the slides to my notebook. So before I go into the into the code let me um yes i thought of that actually because i have a number of things going here good because what i have is i have an instance an on our instance that has the graph that i've just described already loaded okay i have a notebook that i'm going to be sharing as usual in the a notebook that I'm going to be sharing as usual in the GitHub repository. And yeah, my plan is to run it step by step. And maybe it's worth, because I'm going to use this time, langchain. And langchain, some of you may know, is kind of an orchestration package. It's a component in Python that really, really simpl the the construction of applications that interact with lms and do things like rag and stuff like that so uh yeah like like i was saying let me go back to the to the slide so i have this graph already and what i want to do is first take a bunch of the properties in these uh in these nodes and put them in, index them, vectorize them and index them. So I'm going to do that with the help of, it can be done in a number of ways, but I'm going to do it using langchain. Sorry, jumping back. And I'll show this fragment and feel free to not dive into all the detail. But basically what I'm doing here is doing a single call to this method called from existing graph, which basically is going to do the job of vectorizing certain properties and adding them to a vector index. And the way to pass the configuration is pretty intuitive. First, I pass, which is the embedding service. And in this case, I'm using OpenAI. I'm giving the credentials and the access point to my knowledge graph. And that's my Neo4j instance with my username and password. Then I'm giving them the name of the index. Every index in Neo4j has a name and I'm calling this legislation. And what I'm gonna do is tell this function, what you have to do is go take all nodes of a particular type or with a particular label and certain properties within those nodes. That's the way I select the properties, the text properties in my graph to populate that vector index. Then I store the vector itself in a new property that I'm going to call embedding. So all this is going to happen in a single call. So I don't need to care. And that's that's how this language chain is great, because this rather complex multi step process is going to become very straightforward. Now, some of you might be asking, hey, what's that embeddable label, I have not seen that before. And you're totally right. So there's a number of ways of i mean actually i can show you what um um we have the graph here so if if i look at the clothes for example there you go we have some of them here or even even let's look at the definition okay so if i take a definition uh um i see that it's it's labeled also as embeddable but i can expand that and if i can uh let's make that a bit more readable so um this is the section that contains the definition uh the the definition the section will be itself well it will contain all the definitions that that was not a very good actually maybe from the notebook because I have a nice query here that's going to return something a lot more readable but what I what I wanted to show let me go back to my legal one so here I'm taking the the one that we saw in the slides the definition of an unavailable deposit and that takes the whole chain and makes it look quite nice so we have the the legislation of the root we have the part actually maybe to make it a bit more readable i can make that bigger yeah that's good so we have the the legislation at the root then we have the the part the section subsection and then we have the legislation at the root, then we have the part, the section, subsection, and then we have the definitions and the clauses in them. Now, what I've done is, OK, I have to select which are the nodes that contain textual information that I might want to embed. And my strategy has been, OK, I'm going to add an extra label to my definitions. to add an extra label to my definitions this is a definition that i've also tagged as embeddable and the closest i've also tagged them as embeddable so now this embeddable is kind of an extra tag that i put on certain things to indicate my vector index okay i will call my vector index my lan chain go and take all these things that are tagged as embeddable and they can be of different types and uh pick the properties that i'll tell you and use them to to create the vectors and that's exactly what we were doing in in this section so that's how i i um i was uh passing this embeddable tag which is an additional one that i've created as my way of marking the types of nodes that i want uh indexed so that's that's what this uh is gonna do this is is the magic of off-line chain, that it does the whole thing. And once I have that, I have a handler here, this vector index, that I can use to run semantic search. So that's what this does. But let's jump now into the notebook and let's run it for real and we'll see what it looks like. So, again, I've run the input, so I initialize my environment variables with my access point to the instance, to the Neo4j Aura instance, credentials, etc., etc., my OpenAI key. And just as a test, because, you know, the whole thing thing here and I'm going to show that in a in a mini application also in Streamlit is okay I can I mean I could ask directly an LLM kind of skip the rag step and say okay I can ask this question of what is an unavailable deposit directly to the to the LLM. So if I go back to this slide, I could say, okay, my application can ask directly the LLM and I will get some results. Might be good, it might be bad. In this case, it will give a better, I mean, I wouldn't call this a hallucination because this, as you can see, this is returning what's an unavailable deposit. And it's probably based on general knowledge. But now that's probably not the answer that I want in the context of my regulation, my legislation, sorry, my private information. So I use that as a kind of like a baseline. That's where we start. But the next thing we're going to do is, okay, we have some information vectorized in the graph. Actually, I'm going to write it here. This is the fragment that I showed a minute ago. So all the credentials, I'm going to run this. And it should populate the graph, sorry, the vector index. And now we have a handler. That's this vector index. What I can run are semantic searches, as in vector-based searches. And the syntax is pretty similar. I can just invoke on this object, vector index that has been created out of this, a similarity search based on a question. And the question, as you've seen here, I've defined it, what is an unavailable deposit and how does it become unavailable? It's interesting because I've run this in a couple of other events and it's amazing how you can run it in other languages and still semantic similarity works. We're going to see that later. But that's the reason why you find it here in French and even in Spanish I've run it. But anyway, so we have a question, a question in natural language. If I pass that question to this vector index similarity search, that is going to return the best match, and that's where the K1, I'm returning the closest neighbor to that question on my vector index. And that's returning effectively the node that's closer to the definition of an unavailable deposit, which is the definition node, the one that we saw before. Fair enough. That's kind of something to expect, right? The problem is, as you saw before, is that this definition is incomplete because that's just the first part, because the definition contains three clauses that completed the actual definition of an unavailable deposit. So if I build a whole rack chain and that's something that again LandChain helps us with and I believe that's the content in the next slide although I might not go back to it again but But essentially, that's the construction of the QNA, the Retrieval Augmented Generation chain, where we pass the vector that we just created as a retriever. That's all. And of course, the LLM that we're going to be interacting with. That's what we're doing here. So if I create this chain, I can then run exactly the same question. And instead of getting this result as output, which is what's happened when I just query the vector index, what's happening there is it's building the whole chain. It's querying the vector index by using the vector index here. But then it's passing it to LLLM, which is this LLLM, chat GPT. And it's going to return the answer based on this context. So that's what's going to happen. So if I run that, that's going to give me a more elaborated answer based on this definition. And that's a lot better because that's already one step in the right direction. But still not happy with this because as we saw before, we know that a definition may potentially contain clauses. So here's where I'm going to combine the vector search with the graph exploration. Maybe my match returns something like this because it's effectively the closest match, but I'm going to say for any match that you get, go and try to navigate the contains relationship. And if it contains any clauses and reach the result and produce a more a more contextualized and more enricher kind of answer so that's exactly what happens here and uh i'm going to go back to my slides another i don't know if that's going to make people a bit that's okay anyway the idea is pretty simple. I mean, before we were calling from existing graph, and that would take the graph and populate the vector index at the same time. If the vector index already exists, I don't need to recreate it. Now I can say, okay, give me a handler on this vector index from an existing one, and I pass the name. The name is legislation. It already exists, and I'm going to use it. Same credentials, my user, my password, where my Neo4j instance is, and the additional element and where the magic is going to happen is this additional configuration component. I'm saying there's a retrieval query and this query is some random cipher that I'm going to configure. I'm going to show you what it looks like and that's what i'm going to say for any match that the vector search returns go and run this cipher because that's going to enrich and reach the result and that's exactly what we are doing in this next step so here we go so that's the code and uh you see that's uh the fragment that I just showed, and this is the query because the graph is pretty- Say that again? That's the cipher magic. Exactly, that's the cipher magic. And in this case, it's pretty simple because the graph, I've tried to build it in a pretty consistent way where there's a contains relationship. It was a tree and the legislation contains parts, the parts contain sections, the sections contain definitions, the definitions contain clauses. So if you just navigate the contains parts, the parts contain sections, the sections contain definitions, the definitions contain clauses. So if you just navigate the contains relationship, you could even make it like a variable length one and get as many child elements as you could. And we'll discuss how making that generic can be powerful. But we have now this contextualized vector store, which is, again, it's a pointer. It's a handler on my vector index, but now it's enriched with this query. When I run a question pretty much like I did before, the same question and the same limit, I should get something different. If we've added, there you go. We have the definition that we had before, but now we get the three clauses following it. If I pass that to my rag chain and I'm creating here a new rag chain now using these enrich contextualized vector index I can rerun my query and print the output and hopefully hopefully I'm pretty convinced unless things go wrong but this is going to look a lot better because now what we're doing is, OK, we're asking a question and we're going to the graph. There we go. And that's a much richer, a much more complete definition. And unavailable deposit is a deposit that's supposed to be paid, blah, blah, blah. And it can become unavailable in three possible ways. And basically it's literally listing the three clauses that we had in the in the graph right so so that's how this combination of vector search that finds the closest to whatever my question in natural language is after can get enriched by exploring uh the the knowledge graph and now just to show the whole thing in in action i mean that's because you know because this is fine for techies that are watching us, which is probably most of them. But the kind of thing that you're after is something like that. Oh, and it's already on. Damn it. I wanted to. Anyway, let me start from scratch. So I've created this basic stream lead up, which basically has exactly the same code behind. The only thing is it creates an interface on top. And what we do is I enter my question in natural language, and I have the three interactions that we show. That's what happens if I ask directly without any rag. If I go straight to the LLM, that will give me potentially a valid null hallucination, but completely no context. I mean, at least not the context that I'm interested in, which is the fact that I'm talking about this particular, and think of this legislation as any enterprise private data, data that has not been used for the training of the large language model, and therefore, you know, no context is available on that side so that's the kind of answer that i would get which is plausible but not what i'm after the second one is a bit better but it's incomplete right second step we just do the vector search and get the best match but then the third one it's actually the richest one and it's the one that i'm really after because it uh it returns uh following the whole process all the context around the vector index results. Okay, I'll take a quick pause here, short pause here. Any questions? Does that make sense? Yeah, no, that's very cool. I think especially now that you showed us side by side, I think it makes it very clear that the one after the other or one or the other result is much richer, much more information, much more contextualized, as you said. So I think that's important because we say this, we use the knowledge graph to ground stuff to contextualize your results. But what does it actually mean it's it's it's it's a good very good demo exactly exactly and you know it's also a kind of a demonstration of how vector search only it's not as rich as vector plus graph plus knowledge graph because you know sometimes and some people think that maybe a vector database is enough. And it might be for some cases. But if we have a proper rich context capture in the knowledge graph, that really enables for much richer and more precise results. So that's the idea. Yeah. results. So that's the idea. Yeah. I mean, with the vector, you kind of get to the right, let's say, if you imagine it's like being an area, like a space of the graph, you get to your point where you want to go in a very quick way, but then it stops there. It doesn't give you more information about it. It's like if you were gaining directions of saying, hey, I need to go in a very quick way but then it stops there it doesn't give you give you more information about it it's like if you were gaining directions of saying hey i need to go to city hall and somebody says yeah it's at i don't know the city city hall one square and then you get there but then you they don't tell you anything what's what's around and i say okay yeah here's uh i don't know some other locations around it and around it or that needs to be informed about the whole, that makes the square, makes the information about it. Absolutely, yeah. And it goes beyond that. I mean, it's probably not something that we'll have time to show here, but if you think of it, you can do a lot more there. I mean, you can apply access control. I mean, the moment you have the graph, you can be more fine grained. If you hear I'm not modeling any kind of user configuration, but if if if my graph is user aware, I can even return personalized results based on who you are, because I have control over the results plus the navigation that comes that comes after. So, yeah, absolutely. I think the richness of the context return is significantly improved. So that was the first. Interesting questions from Gopesh about what's needed for this to work in such a way. And he says, it seems like a contextualized rack needs a highly enriched knowledge graph. Would you agree with that? Or what would you say? Yeah, I mean, it requires a knowledge graph. I mean, the thing is, and that's probably something that we'll cover again in future episodes. But you definitely need a graph. You need some some kind of in this case, some awareness of the structure. But I wouldn't say it's highly I mean, it will be enriched over time. But this, I would say, is a pretty straightforward type of structure of a document. Right. If you think of it in the context, it can be a documentation. I think that's similar to what Adam has built in our Graph Academy, right? So you can have a different type of structure for courses or even for web pages. So I think the construction of this should not be, I don't know if his question is more in the direction, it requires a massive effort. I mean, it requires definitely some awareness of the context and of the structure, but I don't think it's a huge and significant effort up front. I mean, it's probably bigger if we get into something that we covered in previous session, like entity extraction, where you have to build richer structures. But this one is, I would say, pretty approachable and something like I said, that will grow organically over time and will keep enriching. Because it's true that in this case, I've created sort of a custom query that follows the, you see that it's the contains relationship that exists all over the graph, right? You can go meta there, and that's the plan relationship that that exists all over the graph right yeah you can you can go meta there and that's the plan for for next next episode and and actually be more dynamic about that so maybe i'm not going to pass the set the cipher there i'm going to configure that on the fly with applying ontologies applying some annotation of my graph so we'll go into that but but you know to do to to the question, yes, you need a graph. I don't think this is a massive effort. It's relatively straightforward to build a graph. What you need, though, and I think Kopech then adds this in a chat message later, is having domain knowledge, obviously. So if you set this up and if you like when you went through and you put the the notes uh with embeddable as a second tag on it as a second label then yeah you know you understood okay what's important what what what what do I flag um as the expert the domain owner or the domain yeah that's a fair point although again in this in this this is the first example that focuses more less less on the on the domain knowledge and more on the structure of the of the of the input so here it's more about hey there's sections there's parts there's definitions and and yes in general i understand that definitions are important and the kind of things that i'm going to be looking for but uh yes, I mean, of course, you know, the richer, because you know, the value proposition here is that this knowledge graph is not just a means to enrich RAC. The building knowledge in the knowledge graph is a great investment of effort. I mean, it's going to deliver value in many different fronts, right? So I think it's valuable in many ways. But in this particular case, it's true that I focused more on the structure of the document than in the domain knowledge. But that's exactly what we're going to cover in the second example and what we saw in the previous episode, for example. So yes, building domain knowledge in the second example and what we saw in the previous episode, for example. So, yes, building knowledge, building domain knowledge in the graph is definitely an effort that's worth it and that you should definitely invest time on. Absolutely. Yeah, sit down with your experts, sit down with the team. It's the usual. It's very approachable for everybody, even if you have no ideas of what a graph database or even what a LLM or RAG means. If you show this to somebody from your legal department and say, hey, what's the important part here? Then they will be able to tell you what should be flagged and what shouldn't. Maybe before we go to the next example, because it's an interesting question from Peter here. You use an embeddable label, but could you have just used the text node type of node instead? The text node type. So, I mean, you can do it in many ways. I mean, what you need to populate the vector index is define a pair of label and property or collection of properties so I've seen it done in different ways some people what they do is they have the node and they create an external separate node that contains the actual unstructured text and that might be that might be good for a number of reasons I mean depending on how the graph is is being used maybe it's true that having uh the the a large chunk of text in a separate node for for i mean to enable all the types of of uh of accesses to the graph so that could be another option like having these nodes definitions linked to a text or to a vector and then you say okay that uh that node that you can label as as text or as vector is the embeddable one so you you can you can do that in in many different different ways so so if that's what peter means absolutely so you can do it in i mean that's that's my way of doing it because i i you know i kind of like the sort of compact graph that this creates so i just you know add an additional tag tag to to these notes and and it's all kind of like the sort of compact graph that this creates. So I just add an additional tag to these nodes, and it's all kind of bundled in the same node. I could have extracted it to a separate one, and that would be absolutely fine. Cool. Thank you. Good. So good. So that's experiment one. Now, the second one is more, it means less about the structure of the document, and it's more, let me go back to the slides. Let's start that. One second. And, yeah, so this is covered. So here's the second one. So we have a – oops, wrong. I don't know where that came from. Anyway, so yeah, this is – I'm going to try and create an art gallery assistant. But this is not going to be kind of a conversational one. This is the one that's going to generate some personalized emails in this case. But the starting point is I've looked at the tape modern here in London. So they have the catalog online and the catalog looks like these pages that you see on the left. So there's an artwork and there's the information about the author. There's lots of textual information and some subjects that are touched by the this artwork. So I've loaded that in a graph, in a very simple graph, and basically the attributes of the artwork are built into a node that represents the artwork, which is what you see on the right-hand side, things like the dimensions, the materials, the medium as they call it, like the materials that are being used, the date, some description, the kind of display caption, what you see in the museum if you go and read the label next to it. Then in addition to that, there's a note representing the author, and then another collection of notes, the blue ones, that are the subjects that this touches on. So this particular one is about figures, dynamism, geometry, abstraction. So these type of concepts that describe the artwork. So that's the graph that we have. So here there's no notion of structure of the document. What we have is standard type of graph and what i what i want to do here is something oops before i go into the code because i mean the objective here is slightly different so i'm going to again same same idea i'm going to pass to my llm information that i control and um basically i'm going to give the facts for the LLM to generate a personalized, yeah, let's call it an email that's, let's say Alex is going to visit the Tate Modern and you enter a topic of interest and then the agent is going to look up the catalog and give a recommendation of these are some artworks that you should go visit and that is why. So instead of again, delegating that to the LLM, I'm going to take ownership of the facts. Again, this is of course the take modernist public data, but think of these as your own enterprise private data. So the LLM is not aware of this information. So what you have to do is find the relevant, the potentially relevant data, that's the idea of RAG, and pass it on to the LLM in context for it to generate the response. And the, yes, let's go into the code. It's slightly more cumbersome, still using lan chain, but the idea is similar in concept. What we're doing here is we're defining here a sort of, let's call it an application. In principle, what we're doing is here, I wonder if I should show them the demo first. It's going to make it more understandable. But anyway, so effectively what I'm passing is, well, the customer name. And this is just, of course, not just because I want my email personalized, but it's a way of indicating that I could pass some identification of the customer of the user so that you can even further like I said before filter personalizing the results so that's that's one thing that this example doesn't do but could be done and then a search prompt which is a some form of search string a question like we saw before like before I was asking what was the definition of an invalid deposit. Here I'm going to ask what's, you know, how does Andy Warhol address the topic of, you know, happiness or whatever. And then you'll take that to generate this sort of guide. So these are the two inputs. And these are the two parameters that I'm going to be passing as input. And these are the two parameters that I'm going to be passing as input. And this is going to be used on this function that I call Recommendations App that you will not be surprised to see that it's exactly the same thing we saw before. So this is going to be a vector index handler like the one we saw before. And this is the similarity search that we run in the previous example. So no difference. I'm going to pass the input. And in this case, I'm returning the top three results and just joining them into a chunk of text to return. So this is just wrapping the same processing that we were doing before, but in a function, because then I'm passing that function here to this kind of chain. I mean, I won't go too much into the details of how this syntax, which is a bit, you know, weird, but basically think of it, this is the input. Then we, I'm gonna show the prompt, how we use this input in a prompt. We pass that to the LLM and then we stream out the output. So it's kind of a declarative sort of syntax to indicate how the chain happens. And the other part that I want to show is what this prompt looks like. Because the prompt is critical here. Because what I'm saying is, okay, I'm going to pass some context, which is the output of the vector search plus the graph navigation. But look at what I'm asking the asking uh the llm i'm saying hey you're a personal assistant for an art gallery and you have to write an email to the customer name that i'll pass you who's a member of the of the of the gallery and and and he or she is planning their next visit so the aim the email should summarize you know the artworks uh that they have reached they have search with the search string and uh i suggest okay why not introduce kind of a joke uh how it does and um but i insist please only use uh the artworks or the context that i'll be passing in context so again don't use your knowledge of the world just use the knowledge that I'll be passing and that will be the result of my retrieval from the knowledge graph. And so on and so forth. I'll pass with them the URL, I want the URL to be also visible in the output, and that's pretty much it. So that's the instruction, the prompt sort of definition, and I will be passing a list of relevant artworks from my graph based on the search that I will get from the user. And again, that's just standard a long chain syntax. You'll see that in the in the in the notebook. And once I have this this prompt application, I'll invoke it and you will see the results. So before I go into the next section, let's jump to yet another, I have to reconnect now. But okay, let's bear with me. I'll do the input. But again, you will see some some pretty familiar elements. So still same thing. Now I'm connecting. And don't worry about you seeing the user and password. This is a sandbox that I will after this session. Anybody views, it's not a problem. So I'm going to be perfect. So that's already been installed. I can get rid of that. So I define my environment where my instance is. This is another one that I have. Of course, it's gone sleeping now, but it's here. So the graph that I showed a minute ago is already in my sandbox and uh should be visible here anytime there we go so we have the artists the artworks and the subjects and so on and so forth so you can see how there you go so this is an artwork the the topics and and so on so that's that's the one i showed so back to back to my notebook. So here I'm doing exactly the same as in the previous example. So I'm creating in this case, I'm not populating the index, it's already been done. And what I'm saying is just like I did before. In this case, take the artworks and use the title, the display caption, and the artist. These are the fields, sorry, the properties that I want to, I want the index to capture in the vector and to add to the index. So that's, I won't repeat it, it's exactly the same thing we did before. I'm storing the embedding in the embedding property. It's already been installed and the index is gonna be called artwork. So that's, I think it's done. There you go. And now I have a handler and i'm in a ready to do exactly the same thing as before so i can ask a question and i can do a semantic search brazil for example i'm interested in works by constable featuring buildings natural language question search string if i run that and again do a simple vector index search and get three results this is going to return uh three nodes in the graph uh that here three results. This is going to return three nodes in the graph. Here we go. It's going to return the title, the display caption, and the artist, which are the three that I embedded. These are the three closest matches based on these fields. To my question, works by constable featuring buildings. So that's my context. But in this case, again, like I did before, I don't want to pass all this textual representation to the LLM. I want to pick portions of information that I want to be used for the recommendation. And that's what I do with my uh retrieval query like i did before in this case what i'm doing is i take the artwork and navigate to the subjects you remember how every artwork contain a number of subjects so here i'm aggregating them the blue notes if you remember exactly same same same logic as before i connect to the to the index but i pass this retrieval query and the retrieval query means every time you find a match and reach it with whatever this query returns. And this is exactly what this handler does for me. And I put it in a function, just like I showed before. I'm just wrapping it in a function because that's what I'm going to pass in the second step. Just to test it, I could call it with my search string of the constable paintings. So if I run it, this is the kind of output that it produces. It gives me the title, the author, the medium, and the features. So I ignore all the blah, blah, blah, all the text, and I want to pass that to the LLM so that it produces the recommendation or the personalized email exclusively based on that. And then the URL in case you want to click on this. So this is my top three, I'm limiting it to three. And now pretty much the same way, but now using the prompt instead of the direct rag that I showed at the beginning, I create my prompt definition and the application like I saw before, like you saw before because i i should i'm just running the code that i showed on the slides and from here all i have to do is call it and of course i'm going to call it with you alex i'm going to say i'm customer alex and i'm interested and the two parameters that i need to pass is my search string which is the one that we saw before the constable paintings and uh and that's my name alex so if i run this it should now chain the whole thing so it's going to uh take the search string exactly do the vector search return probably these elements uh then pass them to the llm apply the prompt and pass the prompt and and the context and this should generate uh something that looks like an email that i will be that my agent, the agent will be sending you on your mobile app when you go to the tape modern saying Alex, that's what you have to do. And I'm really curious to see how they do with the Joe. Yeah, that's probably why it's taking so long. The good one. that's probably why it's taking so long it has to come over with a good one something like that so you see uh subject well subject your next visit to the gallery constable masterpieces away so dear alex uh i hope this might blah blah blah all this thing and uh uh incredible art artworks by john constable but so all looking great right so there's and of course you know as expected this is one thing that you should go see and and that's why. And here, I mean, in this case, it's being, I mean, that's surprising because one of the things with when you invoke on an LLM is that the results can look different at each time. And here it's somehow been very, very kind of not very verbose, right? It's literally taking the elements and putting them in a kind of you see as a list of features and it's true that this one is kind of build the phrase out of it rather than just put the list but yes it's decided to do it in this way and then it says click here to view the artwork blah blah blah and now here comes the art pun for you, there you go Constable works are truly building a legacy in the art world well, pretty bad and actually I don't like the fact that it says here's the pun for you, I would have liked it to kind of subtly blend it into the text but anyway, it's not perfect but you see the idea, right, so in this case it's built,, but you see the idea, right? So in this case, it's built. And again, I've done something similar to what we did before. And I'm going to say the same, right? So I'm going to ask Alex. And this time, I'm interested in, I have an idea here. Why not? Because that's what Alex asks himself when he visits the Tate Modern. I'm interested in artworks that focus on geometric forms. Actually, I like those. You know, I was doing the research. I came across this one and I found it beautiful. I just loved it. I mean, I hadn't heard before about this artist, John Wells, but I can't wait. The good thing is this is just around the corner from the office. So I'm going to go I mean, I hadn't heard before about this artist, John Wells, but I can't wait. The good thing is this is just around the corner from the office. So I'm going to go have a look, but I really love it. That's a great example. So that gives the, sorry, the application. There we go. So we have the, and that's what it would look like, the final product. So it would produce something that looks like an email. I hope this email finds you well, blah, blah, blah. And you see all these things are it's different. So now it's gone with a more kind of standard kind of paragraph style rather than a sort of a bullet point. But the good thing is that these things are clickable. So now you can see spiral movement and you can go in and see actually the the page um and uh let's see with the pun here you have preference to that it's just skipped it right he didn't didn't do a joke didn't do it yeah oh yeah and remember when it comes here it's all about finding the right angle there you go okay that's better anyway well that that's, I like, you know, because sometimes we tend to focus too much on the Q&A type of and they're interesting, of course. And that's great for, you know, I mean, for example, we mentioned before the graph Academy, you know, when in a kind of a training environment, being able to ask question is great. But sometimes, you know, it's really powerful to use LLMs to generate highly personalized content. And this is a great example of that, where I leverage the LLM for the generation of this kind of natural language content, but I give the LLM exactly the information that I want it to use. And I kind of take ownership and control over the actual facts. And again, back to the idea of the explainability, I give the provenance in a way by giving the URL and the clickable link. But yeah, like I said before, that can go even further. So here I'm passing the fact that I'm Alex. If this happens in a session, I would have your user credential. I could use that to even further personalize. I know who you are i know what you're interested in like you know without you telling me that you like geometric forms maybe you just need to say hey um uh i know i'm i feel happy today uh tell me what should i go see and because you know it knows and it accumulates information about you it can provide uh uh something something relevant based on on the knowledge that are uh that's stored about you in the in the knowledge graph so yeah or like another imagine you have a have a user i mean the the the the examples that come to mind is like board game geek or goodreads or something like that where you have your your your reading list or your board games you play and the the ratings and all of that and you can say, okay, I have a user. I want to send them a personalized email. I know what they like. I know what they played recently. I know what genres they go for. And then I can combine the two. Absolutely. So I hope that's given kind of an overview of, well, these two examples of how the graph can can play the role of you know this kind of brain that that knows all the facts and and and gives us control over them and and how we can decide what's you know how we enrich how we expand how we contextualize these facts and then we pass it to the llm because the llm is great at you know at reasoning at generalizing at doing analogies at generating uh text as we've seen, as having conversations. But we keep control of the facts, which is extremely, extremely important. So that was the idea. And to your point, Alex, I wanted to conclude, and I'm conscious of time, with a final slide with where to go next, because there you go. So that's that's some content that Dan Bukowski from one of our colleagues in the US has been creating over the last few months. And it's I think it was presented, I don't know if I mentioned it last time. But it's it's super interesting, because it goes in exactly in that direction that you said, what he talks about is how the graph can not only contain the the actual facts the the knowledge about the domain but it can also store the conversations i mean this interaction that we've had with the crowd when we ask about what's an unavailable deposit in the in the legal example well it can be stored and it can uh persist what your question was what the answer was which chunks of of which bits of the graph were used as context to generate that content, right? And like this, it's building again, a history of all the interactions and all the knowledge that you've been, you know, exchanging with the LLM that again can feed future responses and build a more personalized experience. So that's definitely something that people should go look at. There's a bunch of blog posts and there's even a YouTube session pre notes where, where him and Alex went over the, you know, what they had built with when was that? I think that was, yeah, that was the road to notes. The road to notes is like a workshop training, a little bit, a little bit longer content, but more hands-on more tutorial style um if you want to do it yourself you want to learn how to do it if you want to get some some step-by-step guiding then this is definitely um worth worth looking at yeah yeah so that's that and for us what's next like i mentioned before because you you know, one thing that I, you know how we like to go meta here, right, Alex? So like I mentioned before, I don't want to have to be explicit about my cipher. So how to, depending on the question, how to expand it. I want to be able to control that by annotating my graph. So that's what we're going to be covering next, because I think it's it's an interesting topic. It aligns with our kind of general topics of semantics and ontologies, because, you know, the way I see it is I'm aware of my knowledge graph and I can say, hey, this is a property, this is a node that can and should be expanded, should be enriched. And the way to do it is by using these relationships, providing these type of annotations so that the hey, this is a property, this is a node that can and should be expanded, should be enriched. And the way to do is by using these relationships, providing these type of annotations so that the retrieval part of the rag will be able to leverage that to do something in a more dynamic way. Because now, the way I built it, if I want to change my logic, I have to go and change the cipher and then get a different behavior. But so that's the plan for next. Probably, I mean, unless we think of something else. Come up with something else. But yeah, that sounds great. Sounds good. Very interesting. So that's what I have for today. I mean, I'm really happy. I hope it was, you know, all the code will be there. I might have gone a bit too fast. If you are not familiar with langchain, it might be a bit too much code on the screen, but hopefully it was at least understandable at a high level. And then when the different components, the notebooks and all that are in GitHub, you're of course free to go play with it yourself and it will become a bit more clear, hopefully. Yeah, let us know if that was the case play with it yourself and it will become a bit more clear. Hopefully. Yeah. Let us know if that was the case or if not, that was the pace was great. And then also let us know about that. Yeah. Mike Miguel asks in chat, when will be notebook and the graph available in the repository? I think probably it is already. Well, not this time, sorry. But it will happen, you will happen in the coming hours. I mean, tonight for sure. Later today, latest tomorrow. Check it out. The GitHub repository and the links are in the video description, either on YouTube or on our landing page. So check it out there and you'll find the information on the code and all the snippets from there. There was one more comment from Daniel I wanted to read out. Daniel says, so many benefits from the knowledge graph, both on the human and on the machine sides. Seems downright foolish not to apply them broadly, especially in the case of a Gen AI application. I could not agree more. It becomes more and more clear that there are a perfect combination, right? Like the brain, as in the, I know I've said it a hundred times, but I'll say it a hundred more, like the brain, the facts, the explainable, the trustable knowledge, and then the reasoning, the creative capabilities of LLM. And the combination of these two are really, really powerful. I mean, these, of course, I understand these are sort of like everything we build in this series, the toy examples, but I guess, you know, the concept is pretty easy to get and pretty easy to extrapolate to really large scale applications. And we're talking about really relatively easy to build solutions. So yeah, totally agree with Daniel. It's a super strong combination. Yeah, and even like you said, even though they are toy examples, I think we'd like to encourage you to try these yourselves, obviously in a toy environment, but you know, you can take it on to the next level and take a snippet of your data. I mean, if you're working with something where you have a use case that goes into this direction, you don't have to go all in right away. You can say, okay, I want to try this out. I want to see if the improvement is there right there and take a sample set of your data and see for yourself how it benefits and how it improves the results. And, you know, looking at this today and then on all the other sessions we did before and on other streams and video and written content that is out there and like what Daniel said, I think you will immediately see there is a benefit. It was very evident today with the, with the written PDF use case. But, you know, if, if you imagine you have your own data, then give it a, give it a go for sure. And then there's one more question from Orolesco. Do we support Haystack as well? Oh, good question. I'm not. with Haystack. I need to find out. I'm not familiar with Haystack to be honest. But I'll have to get back to you on that so framework to build production re applications uh i've i can't give you an answer right now let's let's take that offline and we'll get back to him so i haven't looks like it's a framework to build uh um applications but i i don't know i mean i wouldn't be surprised if it's based on on on python.net or any of the languages that we support with our drivers. That should be pretty straightforward. But yeah, we'll do a bit of research and get back to you on that. Cool. All right. I think that brings us to the end of today's session. Thank you very much, everybody, for watching. Thank you very much, Jesus, for yet another great hands-on demo together with slides and great background information so this was there was this was super good and I hope you know we can continue on this exciting journey forward exploring more rag exploring more LLM chat knowledge graph combination the world the world there is I think very, very much interested in this and you know, like everybody else. The next one will be in 2024 already. The next one will be in 2024, exactly. So, new year. Don't be scared, but it's exactly true. So we are looking at January of next year for the next episode. And yeah, before we wrap, I'd like to say thank you all for watching. Thank you for being part of Going Meta through 23 episodes now with me, Jesus. And thank you all for watching. Some of you have been watching from episode one, so it's been really great fun for you to accompany us through this journey, through the various topics around knowledge graphs. And I think we've got a couple more, or many more, hopefully, in our back of mind of what other topics we can cover in this series. So it's not going to go away. So we're very happy to have you as viewers, as watchers, as commenters afterwards. And we'd like to keep going. So it's great. It's good fun. Excellent. Well, see you all. Have a great end of the year. And see you in the the year and see you in the next one. See you in the next one. Exactly. If you are interested in Rack and all that stuff, we are doing a Neo4j Live or I am doing that together with Tomasz and Leanne on powering advanced streamlit chatbots with Gen.ai. So very similar topic to what we have today. So if you want to know more, if this topic is interesting, tomorrow at the same time, same place, basically 5pm Central European time, we're going to do that. So tune in tomorrow and otherwise we'll see you in January. Take care everybody. Bye.
Going Meta - Ep 23: Advanced RAG patterns with Knowledge Graphs
Episode 23 of Going Meta – a series on graphs, semantics and knowledge Jesús Barrasa: https://twitter.com/BarrasaDV

Links:
Vector Search: https://bit.ly/4cuZoeS
Grounding an LLM: https://medium.com/@bukowski.daniel/the-practical-benefits-to-grounding-an-llm-in-a-knowledge-graph-919918eb493

Repository: https://github.com/jbarrasa/goingmeta
Knowledge Graph Book: https://bit.ly/3LaqE6b

0:00 Intro
5:07 Advanced RAG with Knowledge Graphs
13:10 Example 1: Legal Text Chatbot
22:25 Code with Graph post-processing
26:09 Chatbot 
36:53 Example 2: Art Gallery Assistant
44:00 Code for personalized Emails
50:55 Email Creator 
54:24 Next Steps 
1:01:34 WrapUp

Check out https://community.neo4j.com/ for questions and discussions around Neo4j

#neo4j #graphdatabase #knowledgegraphs #knowledgegraph #semantic #rag #chatbot #llm