 Hello everybody, good morning, good evening, good afternoon and thank you for joining our episode number 27 of Going Meta, a series on graph semantics and knowledge. So very welcome to everybody who's joined freshly, newly to this series, everybody who's been around for a while, thank you for coming back, thank you for giving us the reasons to go on and to continue with this lovely series here on on you know YouTube or LinkedIn or X or wherever you're watching this right now or if you're watching on on demand and you know welcome break up to you as well for later in the time and specifically you know today we have as usual a great great person with me and that great person is jesus hi jesus how is it going hi alex yeah it's going great yeah very excited to be here again it looks like you know it gets shorter and shorter between episodes i don't know it feels like really really excited and i think we're having a very interesting topic today so yeah looking forward to it it It does look like that. Even though March has not been a short one, maybe even this year, we don't even have an excuse. I mean, February was a long one this year. So we don't have anything to say about that. Just a busy time, maybe a lot going on. Yeah. Especially, I mean, I guess for some, Yeah. Yeah. Yeah. Yeah. Especially, I mean, I guess for some, it has been a long weekend at least here in Europe, it had been a nice, a nice long weekend with a couple of days off. So, so that was, that was nice. So maybe, maybe that is also why, you know, it feels like sometimes things are, are moving quickly and we're into April already. I think this is probably as early as we can be in the month as well so maybe this is another reason we are doing these every on the first tuesday of the month so i mean being today the second it couldn't be any almost any earlier than the first could have been a tuesday and obviously that's that but you know uh we are we are very um very regular i would say with with this date. We try to keep it as much as possible. So if you are looking forward to the next episode already, then the next one will be the first Tuesday of the month, probably again. We'll confirm it soon after. But yeah, I'm looking forward to today. It should be a fun one. It should be an interesting one. We are talking about building reflection agents with Langraph. So Langraph is without taking anything away from the actual thing you're going to say, because I'm pretty sure you will explain what a reflection agent is. But Langraph is a part of the Langchain library that is used pretty frequently nowadays in the Gen AI ecosystem. Lots of things are orchestrated through LanChain, specifically with the GraphRack things we talked about in a couple of these episodes on this session, but also in the wider Gen AI world. I think you probably stumbled across this. And LanGraph is a, yeah, how would you say, is a sub-part of, or a sub-implementation? Yeah, LandChain, define it as an API. It's a library kind of that's built on top of LandChain. And that helps, you know, creating applications, agents based on LLMs. And what's interesting about it is that you actually build a graph, like a flow chart of how your application reasons. And I think that's a very interesting thing. I'll spend a couple of slides explaining how I got to this idea and why, and where we're going to take it from here, because I think that brings a bit of context. But yeah, that's it. I mean, it's one of the most most i think it's by far the most popular kind of orchestration uh library in in python for building llm applications and and it's amazing the work that they're doing for me it's kind of a these these sessions are are an excuse to you know to dive deeper in some of these technologies that are moving so fast and and and, and to, you know, to, to learn. So I hope I'm going to be today sharing what I've, what I've learned over the last few, few days, few weeks. And yeah, Landgraf is really, really powerful and really interesting. So I think it's going to be an interesting one. Yeah. Yeah. I'm, I'm I think that's, that's, that's really good. I, I, you know, I'm, I'm also writing the Neo4j newsletter that comes out on a weekly schedule. And this week, I'm featuring an interesting website, and I'd like to share it with you today. I think it's a good fit we have a neo4j labs page on the genii ecosystem which collects all the interesting bits and all the interesting known libraries demos examples training courses interesting blog posts all this stuff in uh in around the genii world and lots of langchain implementations as well and i put the link in chat and i'll put it in the video description on YouTube as well. So if you are in this space, interesting in it, this is a great, great resource with many, many links that gets updated almost daily, but not really. But, you know, very, very frequently when something new comes out and we put it up there. So it's a good resource to check back on. Yeah, I think with that, I'm happy to dive in. Let's get... Yeah. Let's put this on. Perfect. As usual, yeah, well, let's use a couple of slides to give some context and to set the scene for the more hands-on part that we'll use in the second part. Yeah, the idea is to, we're going to get to building a reflection agent and we'll explain what that means and we're going to use LandGraph. But like I was saying before, I mean, the initial idea was slightly different and it started with this tag that I'm, I suppose we'll be sharing, I mean, the initial idea was slightly different and it started with this tag that I'm... I suppose we'll be sharing, I mean, you can probably copy the link down there, but this is from, again, from the the Langchain team, from Lance Martin, where he was kind of discussing or reflecting on, starting from this this tweet uh where when he was kind of uh asked whether the rag was dead when you know the the context windows were growing and and and he was practically unlimited the amount of information that we can pass to llms and you know spoiler i mean the conclusion is it's very much not dead it's it's uh it's more alive than ever and and what's interesting and i totally recommend you go to watch this deck. And I think there's a YouTube video that I think it's linked in some of the pages or several ones where he explores some of the ways in which RAG is actually evolving. a very very rich uh kind of uh um approach to to to to to integrating llms and and an external you know grounding grounding databases or data data knowledge bases if you want and uh so my initial idea was because if you think of it alex what we've done always okay we we show in a pretty linear way how uh if you do rag based on a knowledge graph, it improves significantly by providing context, the accuracy and the quality of the data that you would retrieve if you were looking at, for example, just a vector database. But the reality is that RAG, the retrieval step is becoming way richer. It's not necessarily a linear process where you go retrieve and whatever you get, filter or pre-filter, post-filter, you pass it on to the LLM. And what Lance was describing in this deck is that there are very rich patterns to do some reasoning. Like things like, for example, I retrieve, I evaluate the quality of what I've retrieved. Is that information that I have retrieved from my knowledge graph, from my vector database, whatever my grounding database is, is it relevant to answer the question that I'm trying, or to fulfill the task that I'm trying to DLLM to fulfill, or is it not? And depending on that, I might decide to do several things. if he talks about maybe you want to go and and search the web for example or use any other tool or or reiterate or refine or come back and do multiple iterations of retrieval anyway that was i was thinking that there's something really interesting there and i i wanted to to explore that but um I thought I'd put this idea on hold when I, again, Dan Klosky created trouble and posted again. He's a great source of super relevant and up-to-date information on everything grabs, but also on Gen AI, of course. And he talked about this idea of agents. I mean, he basically referred to this presentation by Andrew Eng. And at the same event, there was actually a presentation by Harrison Chase from Langchain. And they were both talking about agents. And the idea of agents is how not only the retrieval part, but how the interaction with LLMs is becoming richer, right? It's not like, again, a linear thing, which we might have shown in simpler examples, and how it becomes, you know, a lot more interesting. And in particular, Andrew, and again, these two, I think the two links are in the comments. So if you want to share them, Alex, or we can do it after. So he talks about how we can build. He refers to that as agentic behaviors or agentic patterns. So does our application behave as an agent or just as a non-agentic workload? Does it just do a thing, starts and ends, the example that you see on the left, or does it do something more like iterative something? Because that's the way we actually work. And that's what made me change my mind because that connected very well with episode minus two. That was 20, well, 27. It was 25, right? Well, we did like this model generation, automating model generation, right? We had a data set of flattened or denormalized data set. And we tried the LLM to generate a graph model. And then we moved the data from this tabular format into Neo4j. And it was fully automated. But we took this approach like, okay, we get the metadata of the data set, we produce a model, and boom, we generate the cipher inputs and the data is migrated. And if you think of it, it's never like that. I mean, in a real world scenario, we don't model and that's it. I mean, you typically come up with a first model, you share it with someone else. So your coworkworker, with your colleagues, and you iterate, right? So if you're working as a team, you iterate on that until you come up with a version of it that you're comfortable with, that you think is right. And that's exactly what he describes as one of the patterns, because he describes a number of them uh but that's that's more realistic and and and he was talking about the amazing results that can be achieved by following these these these approach right so so i thought that that was probably something worth exploring and it's somehow connected to the to the previous set which will just spark for now and and we'll talk more which is this idea of of the kind of applied the same idea of reflection on on on the retrieval step on a rug application so i retrieve but then i do some reasoning and in terms of evaluating how good it is and whether that's uh that requires uh additional iterations so well like said, that's something that we'll do in future episodes. But this was sort of a low-hanging fruit. And I thought I'd dive into it. And what I, of course, found is- MARTIN SPLITTINGER. Oh, yeah. OK. Yeah. I just, you know, I haven't seen your slides. So believe me if I say this, I just wanted to say this now makes interacting with an LLM even more human-like, because you do something, you think about it, and you change a little bit of what you do, and then you reiterate on this. And this is your slide here, right there. Absolutely, yeah. And this is the version from, you have the URL there from the lang chain, or what they define, reflection agents. But in the presentation that I was showing before, I don't know if it's readable here, but he kind of describes the same idea, Andrew Ang. So the idea of reflection is you ask the LLM to generate some code, and this is this person to the left. But then you have another actor in your application that acts as a reviewer. It might be the same LLM. It's the same LLM that you used to generate the code in the first place. But if you ask it to act as a critic, as a reviewer, to highlight points of improvement or issues with whatever has been generated, you create this cycle, highlight points of improvement or issues with whatever has been generated, you kind of create this cycle, this iterative process that kind of simulates, like you were saying, the human process of building solutions. And that's exactly the way the Langchain team describe it. And here, the basic example, which is the one that I'll be implementing today, is you have some user requests, in our case it's gonna be generate a model out of this dataset and we will do an initial generation and then we will reflect on it. And we will reflect it, I mean, the LLM will reflect on its creation and it will produce some reflections that will kind of maybe trigger the generation of a new version of the initial model, and so on and so forth. And you can repeat that process many times until you consider that the quality is the one that is expected, or the requirements are met, whatever the termination condition is and then you return the result but it's not like a linear process anymore it's kind of a more more richer more complex and in this case an iterative and cyclic exercise so that's uh that's the idea of a reflective agent it's just it reflects on its own uh production and you use that reflection to kind of iterate and and make it eventually obviously uh better right so that's uh that's the idea so does it make sense does it align with with your thinking as well yeah yeah that makes that makes perfect sense and i think that's that's that's a really interesting journey so to speak where we we came from right i mean we we if you remember going back a couple of months when when uh when everything was about chat GPT and it's going to be the new savior and and of everything and you just put in something there and then you know your answers on any kind are created back to you in in almost magically then people realize oh actually you need contents context you need there's hallucination involved there is there's problems there if you don't if you don't ground your lm that that's like you know first step but now here next step almost is like okay yeah but sure you you need all the context but still you kind of need to iterate a little bit about it you need to think about what what the the first shot will not is usually not not the best you know rally i mean if your genius maybe it is but if you're not even in the first draft or something is it's good okay but then you know you iterate on it you do some thinking you tweaks here and there and then you improve over it and then you at some point you say oh yeah that's not good enough let's let's let's ship it basically absolutely yeah yeah yeah totally and and that's that's exactly you know what uh one of the things that lang chain in general but landGraph helps us with. And again, there's a couple of links there that I invite you to take a look at. And these are the tools that we're going to be using in today's session. And again, the problem that we're going to be experimenting with is the one from episode 25. If you remember it, what we did is we took a data set from Kaggle, I think we used, and that was particularly convenient because Kaggle, as we're going to see, probably remember it, offers kind of an endpoint where you can just look at the metadata of the dataset. So which are the fields, which are the data types, is there a description? So you can very easily query that. And that's all we need for the look at the metadata of the dataset. So which are the fields, which are the data types? Is there a description? So you can very easily query that. And that's all we need for the generation. If you remember what we did is we get the metadata out of any public dataset. And with that metadata, we're gonna prompt our large language model, except that it's not gonna be a linear thing like here, one, two, three, three. So it's gonna be a more iterative one, but the process is pretty much the same. So we asked the LLM to produce a model. And then in that episode, we took the model and we did the data ingestion. In this case, we're going to focus on this initial step of creating a model because that will be not a one hop or one step, but more of an interactive, iterative, if you want. Interactive as well, because we're going to see that this opens the door to introduce a human in the loop. So we can think of entirely automating it, like getting an LLM to talk to itself, like, hey, LLM, produce a model. Hey, LLM, criticize the model that you have produced and suggest improvements. And then it will regenerate and can have a conversation with itself until you come up with a condition for it to terminate or you can uh instead of having the llm uh you know critic commenting or providing feedback on the model you can introduce a human there or you can combine the two and say give me your feedback i'll mark the ones that I approve and the ones that I want to ignore and I introduce my own. So by creating this type of more rich and more sophisticated applications, we're building something that's closer to the reality of how we work. So, yeah, so that's the plan. So I hope that gives enough context. And now we're going to hopefully see it in action. Very cool. Sounds like a very interesting topic. I'm looking forward to it. If you have any questions, any comments, anything while we do it, then please don't be shy. Use chat. We can see your messages so we will try to answer them as we go through. So it's all going to be in a in a notebook that we will share at the end of the session so we'll start with some imports uh the usual i think we need to zoom in a little bit into this one let me make that a bit bigger yeah something like that yes perfect thank you good stuff okay and then the usual, the imports, and then I'm going to be working with OpenAI. So I'm going to use my key, which is already stored in my Colab environment, but you'll use your own. And of course here, in this particular exercise, I'm not using any specific integration with OpenAI. I mean, you could use any, and I have run some basic experiments with open source NLMs running locally through Olamo. And maybe that's something we can explore in another session. But in this version today, we're going to be using OpenAI. So, right. So we said that LandGraph is a way of, is a library for building sort of, they call it stateful and multi-actor applications with LLMs. So I use this analogy and this terminology to call the different parts actors. And in our exercise, we're going to have three actors. We could simplify it to two, but we're going to have three of them. The first one is a modeling expert. And that's where I kind of imported some of what we did in episode 25. So this modeling expert is a system that's capable. I mean, you can see the configuration here that I provide. It's an expert in creating entity relationship models, I mean, which is equivalent to a graph model in our example. And I've given it some kind of basic principles, right? I said, don't over-normalize. I mean, you know, we are going to be starting from a totally denormalized flat table that we get from Kaggle, don't overdo it. Because sometimes in some of the experiments, you might go too granular and break it all down in small entities. So don't over normalize. And some general guidance. So don't use the same relationship name for multiple relationships. Don't make sure that all the features in the data set are used. Don't ignore any one of them. And also have one-to-one mapping. used don't ignore any one of them and also have one-to-one mapping i don't want to have kind of complex ctl process where you combine two fields or transform them so let's keep it simple so it's pretty much you know general guidance it's nothing super sophisticated so we create this uh this first actor which is the modeling expert which is simply uh you know uh uh we're gonna be prompting the the you know gpt4 which is the modeling expert, which is simply, you know, we're going to be prompting the, you know, GPT-4, which is the one that we're using. And all it does is, you know, we're going to get a prompt that I'm going to show in a minute, and we use these LLMs. So that's the first actor. And in order to prompt it, because we're going to be using data coming from Kaggle, I've created this utility function that will, given a dataset name, it's going to go and retrieve the metadata. That's the endpoint that I mentioned before that it exposes. It's called Croissant format. I think we talked about it. So if you're interested, you can go to episode 25 and it's in JSON-LD format. So all we do is, OK, you pass me your data set name and i'm gonna go to this url which is the the metadata description and basically i'll i'll put it in a textual format i'll put you know all the field names in a string so that's my function that's what it does and um and the prompt again is the one that we used in episode 25. So I'm giving you a list of features. I don't expect you to read the whole thing, but I'm giving you a list of features, which is the columns in the data set. And with that, you need to create an entity relationship model, a graph model. And things like attributes don't need to be named exactly after the others, but the usual thing, so you can rename it. I added something interesting that for some of our followers can be of interest because I said, in addition to that, to generating this model and I asked you to serialize it as JSON, find what's the schema.org term, the closest one that you can find for each of the elements in the model. And we'll see what it comes up with. But it's a way of providing an annotated data set. And if they will want to publish it, you know, using schema.org, that can be useful. And I tell it what the shape of the output is. Basically, it's a JSON document that contains the listing of the entities with the attributes, etc, and then the relationship. I say, well, don't give me any text or comments, just the JSON. Pretty straightforward. Then all I'm passing is the dataset as we retrieve it from Kaggle. I'm calling the function that I just show a minute ago and we put it all together. And that creates a very nice prompt that we're gonna be passing to the expert that we've just created. So let's give it a try. So this is what we did last week. This is the first part, the model creation. And I have a couple of datasets. So there's one for supply chain data analysis, which let's find it. So probably Kaggle. Let's see if it finds it. Whoops. What have I done? Yeah, I didn't type it correctly. Let's see if it finds it. Oops. What have I done? Yeah, I didn't type it correctly. Anyway, I can use that, which should be on my history. There we go. So that's that. So if you search for it, I think I went to the trending ones, the most popular ones, and grabbed one or two from that. What i looked at is something that's rich enough and and if you look at this one it has a file which has 53 columns so it's rich enough so it contains potentially a number of entities so i think that would that would be a a good example so this is the one that i'll be using for now i'm not'm not importing the data, just the metadata, just the description of the columns that I get from this. This is what I was mentioning, the metadata as croissant format. So that's what I'm doing. And I pass the name of the data set, and it will go use my function, retrieve the metadata, generate the prompt using the function that we've just seen. We're going to call our model generate actor. I call it model generate that you have it here, remember? That's what I call our experts capable of generating models. Here we go. I'm going to call this and if I run this, it might take a few seconds because what it's doing is first going to interesting. I have something unexpected. So, yeah, we're retrieving the metadata from Kaggle. We are generating the prompt saying, hey, with this data set or this structured data set, generate a graph model, taking into account the guidelines that we described both in the prompt and in the configuration. So it's done it. And if I, you know, I've stored it in, we call it model as text. If I print it, that's the sort of structure that we saw also in episode 25. But remember what it was like, right? So we have a collection of entities. So we come up with a portion of a So we have a collection of entities. So we've come up with a portion of a customer and then in a collection of attributes. What's interesting and what's new is that we are adding also the schema.org term, which is a pretty, I mean, we're not gonna use it today, but it's something that we can use in future episodes to kind of create APIs, because we have a model that's already mapped to schema.org terms. So we have a model that it's come up with. Now, one thing that I did, because at the moment this is just code and it's not very visual, but remember that we had, this was the data set. So we have things like in our data set, we can look at the columns here. We have days of shipping, delivery status category looks like there's customer information uh product information city location there's plenty of stuff in there so there's a number of entities and that's exactly what this has uncovered now if i close this textual view i'm going to try to show it visually and for that i'm going to use graph viz which is this And for that, I'm going to use GraphViz, which is this old applications that you can use to create simple graph representations. And there we go. So that's the model that it came up with. It's a bit large, but we'll try to make it fit. But it said, OK, it looks like there are several entities there. The first one is a customer, the notion of a customer. And it's annotating it with the schema or element. But basically, I see that there's a customer and the attributes of the customer would be the city, the country, the email, the first name, last name. So it's taken a subset of the columns in the in the data set and said there's a notion of a customer. So there's a kind of a node type in Neo. And customers are connected through the makes order relationship to uh to another entity which is an order the notion of an order perfect that makes a lot of sense an order has a city a country and orders or the dates the state status blah blah blah and then the order contains products so pretty cool i think that's a decent model it's a basic one but it's uh it's good enough and like this we have an an idea of what the model has come up with. Perfect. So we have our first kind of actor in our application. And we're going to create a second one, which is the model reviewer. And it's pretty similar in terms of structure is identical. It's the same LLM. It's's the same open ai except that now we're saying hey you are a data modeling expert that can analyze an entity relationship model a graph model and suggest changes so you're not supposed to generate a new model just provide suggestions so i really want you to be the critic of of the model that we've created and look at things for example like other other elements that are under normalized or over normalized like am I still creating like fat nodes or that I should be you know refactoring and extracting entities from so this is the kind of things that I'm asking and expecting from this critic from this reviewer to receive maybe renaming things that might not be great. Well, this is the definition of this second actor, which is the reviewer. Exactly like we did before, we're going to have a function that creates the prompt, and it's very similar to much more simple. In this case, we provide the model that we've just created in the JSON format serialization, and we said analyze this model and provide two or three changes. I mean, don't come up with a list of 25. Let's go small. And as usual, no preamble, no comments. Just give me your recommendations. So that's what I've created. And now we're going to test it like we did before. So are we going to invoke the. And now we're going to test it like we did before. So are we going to invoke the model reflect as we call it, what the reflection actually takes place. And we're going to store the output in a variable that we call critic. So that's the feedback that we get from the expert from our first model. And it's done and i can print it out and it's saying hey that's why don't you extract the address as a new entity from uh both the customer and the order entities and that makes kind of a lot of sense right so it's a good idea yeah exactly because you see hey the the the customer it has the city the country the you know the state the street so maybe that could be in a separate entity, in a separate node. It makes a lot of sense. And the same thing happens, I believe, in the order. The order has some geographical information. So very valid feedback, right? So that's good stuff. Then the attribute segment in customer might be confusing. Why don't you rename it to customer category? Well, great. I mean, it looks like very, very interesting stuff. It's the same LLM, but it's looking at it from a different angle. So really, really valuable feedback. And there's some others. The relationship contains product, could be renamed like this. Fantastic. So we have a second actor that's an expert in analyzing what the first actor produces. And then we have a third one, which is the one that I was saying could be combined with the first actor produces. And then we have a third one, which is the one that I was saying could be combined with the first one, which is the executor, the one that will take the feedback and apply it to the model. So I call it the model changer or the model editor. And basically, it's your data modeling expert that can modify an entity relationship model that's serialized as JSON, and you apply to it the changes that you're given. And you see where this is going, right? So this you apply to it the changes that you're given and you see where this is going right so this is going to be the expert that's going to take the feedback and the first model and it's going to modify it to create a new iteration so similar structure so it's a simple invocation of of chat gpt uh sorry gpt4 in this case and we have a prompt that's applied the following changes to the model and uh i'm going to give a list of changes pretty much in the same way as the critique or the reviewer has produced them so um i'm going to test it here so that's the prompt i forgot to put the comment here but this is the uh let's test it so Test it. And what this is going to do is, like we did before, it's going to call the model change agent or actor, as we call it here, and it's going to pass the message, which is the initial model and the feedback provided by the second actor. So if we run this and we store the output as a new model, it will produce a new version of the model that we had initially, but with hopefully the changes applied as recommended by the reviewer. So once it's done, we can print it as text, which is not going to be very helpful. I mean, it is because we see that it hopefully follows the same structure, but it will be a JSON document, and then we'll be able to visualize it. And I was thinking a bit of a longer time, which is not an issue. This is a modeling expert, which is a task that's meant to be like brain 2, rather than.. Anyway, let's let it finish. And we're going to see how... There you go. It's done. So if we print it out, we have a new version of the model, again in JSON. And if we look at the visual of it, that should look something as before. Yeah. Yeah. Interesting. Interesting. But we see that it has factored out of the person the uh the entity it's connected it through the lives at which also was part of the feedback i didn't i didn't look in in the whole detail but if you look at the first line uh extract any relationship and it's saying uh connected to the customer in order with a lead that from customer and uh delivered at so it's given me you know a very rich description of the chain. So it does a very, very good job at doing this reflection, which is the whole point of our episode today. So it's implemented that. So it's extracted the customer address, and now we have a new type of node. It's done the same with the order. The order is delivered at an address, makes the order stays the same. And this, I suppose, stays unchanged. I mean, I think there was a couple of properties or attributes renamed. But that's pretty much it. So it's done a good job, right? So we've basically shown the whole process. So what we're going to do is, so we have now the building blocks. Now we're going to go and get a land graph to orchestrate the whole thing and that's what we're going to do in this in this final step so i'm going to define the note i did in our graph i was going to draw a diagram but i didn't get to do it in the end so but we have clearly a node per actor right so we have a this node uh the model generation node is the start node. It's the one that will take the input, which Kaggle dataset we want to work on, and it will generate a first model. So we call it model generating. It's run only once in the first iteration. The second one is the model reflection. And again, I'm calling the functions that we've just seen and the actors that we've just seen. I'm just replicating. Once you get access to the code, you'll see that it's exactly the same functions that we used in the test. We're just putting it into these functions that define the nodes. So the reflection node it will do, it will take the model that has been generated by the model generation node, it will analyze it and it will produce some feedback. And I have two versions of this because I don't have time to to to look at it. But my first version is I call it human in the loop falls. So by default, we go with a fully automated version, which is I get a model generated, I get the element to review it and act as the as the the actor that provides the feedback and then I modify and then I ask the element. But at review it and act as the actor that provides the feedback. And then I modify and then I ask the LLM. But at some point I might say, hey, I don't want the LLM to do that job. I want a human to enter some input. And as a very rudimentary form of doing that, I said, well, let's call that human in the loop. And if we go human in the loop, set it to true, instead of going and calling the reviewer actor, Instead of going and calling the reviewer actor, it will just ask me to enter some input via text. So I would do something like, as an expert, go and refactor these or that. So I will enter by just to show how this same infrastructure could be used to combine the two approaches. So this is the second node, the the reflection node where we analyze what we produce and then the third node is the one where we apply the changes and um yeah so that's that's the third one where we basically pass the the original model the changes and it will produce a new version of the model so how do we build a graph so So it's a very simple definition. And this is the LandGraph API. So we create a message graph. And we said, well, add a node, which is the first one, the model generation node that we've created here. And we give them names, the model reflection and the model change. And the starting point is the model generate. So my graph starts at the model generate. So that's my entry point. And uh i have to define the edges so before i show this function the edges is okay from the model generation you go to the model reflection so that's my my flow uh of of uh in the application so generate and then reflect. Then based on the reflection, you either iterate and generate a new version or you terminate. It might sound a bit counterintuitive, but my initial thinking was more the human in the loop version. I wanted to take control and say, if I provide no feedback or an empty string feedback, it means I'm happy with the model and the process terminates. That's why I put the conditional in that direction after the reflection. Now, it might sound a bit, I mean, but as you can see, it's very simple. We can tune that and change that depending on how we want to do it. But basically, we'll take the feedback, we will decide if we want to run another iteration or if we wanna end. And once we apply a change, we go and reflect again. So that's systematic. Every time we have a new version, reflect on it and then decide if I want to iterate or not. And then the condition on whether I should iterate or I should terminate is this function that I'm defining here. So I'm saying if I'm in a human in the loop mode, if I enter an empty string, so if the feedback that I'm providing, the length of feedback is greater than zero, so it's not a non-empty string, then you go to model change, but if not, you terminate. Now if we are in a fully automated mode, which is the one that we're testing now, at the moment I'm setting the limit after a number of iterations. So I say when the state is smaller than six, you keep iterating. But then when you get to six, stop it. Right? Okay. Of course, this could be more sophisticated and maybe we could go in future episodes on how to evaluate the quality of the model. When we decide, rather than saying, you know, after three iterations, you're done. What are the characteristics of the model that says it's good enough? Stop iterating. So that would be, again, you see that this is a very simple function to show here. But you could build in some logic there. I mean, this is just for demo purposes, I say. function to show here, but you could build in some logic there and that would determine. I mean, this is just for demo purposes, I say. And if you just prove a point almost, but if anybody wants to take this on, obviously the code will be available in the repo afterwards and then you can iterate on it and and you know make it more a little bit more sophisticated exactly because you know it's it's an additional step in the in the reflection so you can say hey before you well maybe as as output of the reflection you can either uh maybe uh yeah we could even add to the prompt if you're happy with the model if the model is good enough you know return an empty string and if that returns an empty string then this logic would work i mean but you know you can you can tune that to determine when uh when the model is uh is is finalized and we want to end so that there you go so we we build the graph now we compile it so if i run this code it creates the the application in the form of a graph and Now we just need to test it. All we're going to do is we create an initial prompt. It's not even a prompt, it's just we pass the dataset and the prompt was created by this function that we showed before. We're going to use the supply chain dataset that we were looking at. I'm going with the completely automated process. So I'm setting it by default to no human in the loop. So it's going to iterate up until the state is, I mean, I didn't mention it, but this graph shares is stateful and it shares a chain of messages that are exchanged between the different steps. And basically, when that state gets to length six, it stops iterating. So that's what it's doing. So, yeah, let's run it and let's see what happens. Because basically what this is going to do is kick off the process. So go get that data set and start the process. Generate a model, reflect on it and suggest changes, apply the changes and then go back again to reflection and eventually enter a cycle and get to a termination, which in this case would be after, I believe three iterations so I might take a few a few seconds but I hope the idea is is it's clear and and what gonna be interesting is to analyze what the process where the process to cut yeah these are completely you know delegated version of it. And what's interesting about these is that you can introduce the land graph API, the libraries is really powerful because we're starting to get results. So you can set checkpoints and travel back in time. So for example, if at some point in this iteration, things go wrong, you can always travel back to that point and then iterate from there. Very interesting thing. Because we are in a, so there you go. So it's starting to, you know, it's calling asynchronously, passing an initial message, which is the dataset. And what it's doing is as events come out of these asynchronous invocation, we're just printing them. And the first one, it's generating the model. The model is in the JSON form that we saw before. And then we are getting some feedback. The customer, it's probably similar to what we saw before when we tested. Yeah, I just wanted to say. Yeah, sounds like the same suggestions. So it will go in that process a number of times, a couple of times, and it will terminate there. And we're using the same models, right, as before. So again, like every reflection is like two to three suggestions there and then incorporating those and then starting again. Exactly, exactly. So there you go. So now it's regenerated the second version of the model after applying these changes. Now it's probably going to provide new suggestions. There you go. So under normalized attributes, so in the customer, it's saying customer F name, map to two features, can be extracted to separate attributes. Well, there's a risk element here because we don't know what it is that come up with. But once it I mean, once it terminates, what we're going to do is we're going to try to analyze the results produced. And again, LandGraph has a number of elements in the library that helps analyzing what's being produced and printing it in a nice way. I mean, another thing that we're not exploring today is the use of langsmith, which is a kind of an observability element that also the Langchain team offers. And that gives you a lot of visibility on each individual step and might be another interesting one to explore in another session. There you go. So we have a new version of the model. And with the new version of the model, probably, there you go. So it's got, it produces a third set of suggestions, but because it got to size 1, 2, 3, 4, 5, 6, it decided to terminate. Then the last node in the graph, the termination node, it contains all the information. We can say, if we print the output, we see all the different steps, like initial prompt, first version of the model, feedback, second version of the model, prompt uh first version of the model feedback second version of the model feedback third version of the model and feedback and then we decided to ignore the last step because of the way we defined the the determination conditions but you know we can look at what the the model looked like and and we can uh for example look at the first uh and it's pretty similar to what we saw before, right? So we see the person, the order, and the product. We see that once that was done in the second iteration, it probably factored out slightly different, right? So in this case, wow, look at this. So we had the product. Actually, I print out the feedback that we got and we can see how it implemented it so we see a new version of it slightly different to the one we saw before and there's even a uh um a third one where we got this feedback and then it came up with uh with the final version of the model so it looks like it ended up with an order of containing products the customer, the address. So, you know, it's not miles away, but it looks like it converged into a version of it that it was considering good enough for. In this case, after three steps, that was it. So that's the idea. I mean, what's super powerful about this is that we can, you know, I'm going to, I suppose, pause here. But before pausing it, I'm going to just take another example. And this one is some crime data. I don't know why I came up with that one. I think it was a rich data set rather than something that I'm particularly interested in. But anyway, we have another 28-column data set with obviously information about crime data and uh so if i'm gonna run that because it's exactly the same logic but just apply to a different uh data set and we're going to see how it behaves on on a second model and and it should be pretty straightforward pretty similar but it's um yeah i think this is a simple but but really powerful implementation of of this idea of reflection where where like i say we generate in multiple steps and and providing getting feedback from the llm uh until we converge into uh into uh a model i didn't show the the human in the loop version but it's if we have time after this we can we can show how it goes. I'll pause here because I'm conscious of time and with the minutes that we lost in the problem with the key. Thank you again because I was not expecting that. Yeah, so let's leave that running. We will eventually spit out the results, and we can analyze them. But yeah, let's look at, do we have any questions, any comments, or even maybe you, Alex, anything that you want to? Yeah, I think that's very cool. I am always amazed how this goes and how well that kind of stuff works nowadays. So it's really interesting. I think this iterative process is very, very interesting. I think that gives a lot of opportunities and a lot of possibilities to how to work with data. I mean, you you can like we said automate it a little bit and then you know uh give get some get some already iterated version then you you put in some comments yourself and then let it iterate again to to fine-tune basically your your desired outcomes a little bit more you don't have to do everything manually uh it's very good to to have this this human in the loop sort of option there. I think that gives you a lot of possibilities. Imagine, you are the person who is developing the application in the end, and you have a data modeling expert sitting in your office, and you can talk to them and say, hey, this is what we initially came up with and give me some feedback. And then you pipe that back into it and then you develop it further. It's really good. It's a very powerful combination. Yeah, to add to what you said. I mean, exactly. Because here we're taking kind of a neutral approach as in when we create this initial model, we're kind of applying the general practices and in our training, some modeling trainings, we tend to say, well, normally nodes are our nouns and verbs can be the relationships and it's a way of analyzing your requirements, your domain. And but then it's true that, you know, when you start iterating in because that's the way we work when we come up with graph model, we not only bring this initial approach, but we say, this is the way I'm going to be analyzing or querying my graph. So we can provide additional input a human in the loop version, like, hey, that's great. And you've got to a reasonable version. But what if I'm going to query my graph with this type of workload? That's the kind of analysis. Would that suggest any kind of optimization or any kind of denormalization, normalization? So, yeah, it can really, really replicate the way we work. And ultimately, if you remember, our objective, and this kind of goes even one step further in that direction is, okay, in the simplest case, we have a dataset, any dataset in Kaggle, and after an intuitive process, we have a reasonably good model and I can move the data into a graph database and start doing my analysis without having to type a single line of code or being an expert, and just relying on the expertise that's built into the into the lm here we go so this is this has finished now let's do a quick quick analysis and uh i'll go straight to the model so if we go to the first version oh why is this vibration happening it's very interesting do you see see that or is that just on my screen? MARTIN SPLITTINGER. I think it's just on your screen. I don't see if I've, oh, yeah, I see it. Yes. And I, well, yeah. MARTIN SPLITTINGER. It's very strange. It might be my, anyway, well, now it seems to have calmed down a bit. Anyway, it comes back. But if you look at the crime data, there was information, about area so geographical information information about the victim information about the weapon status location etc etc so the first model it came up with sorry about the vibration is we have a notion of a crime is a a weapon uh and and there's the victim of a crime and then it's a location so you know very decent starting point i don't know what vibration comes from but anyway so what uh in the second second step if i run uh the analysis the feedback was uh the attribute status in the crime seems to be over normalized. It could be merged with status description. The weapon entity could be merged into the crime. So look, in this case, it went in the opposite direction. It said, why don't you denormalize? So the weapon that used to be a separate entity, now it's been brought back into the crime. Well, maybe a good idea, maybe not. I kind of like sort of the previous version more. But anyway, you know, this is, I guess, a full, and it came back to a, yeah, it denormalized it again. So it renormalized. That's an interesting point. And I'm happy that this happened because in this simple version, I'm only taking the current version of the model. Now, if we pass the LLM, this is the current version and these are the previous version. So don't travel back in time, don't suggest changes that undo the changes that suggested in the previous iteration. So it can get more sophisticated and build even more of the way we normally reason about it. But just as a simple example and with three iterations, it's a pretty cool example that I would definitely encourage people to try and to improve and to test in their environment. Because I was impressed with with uh how how good it was and uh so um it's very impressive yeah i think it's pretty pretty cool and we we didn't we didn't really add much in terms of uh of of the the graph true features that come to it so i mean you you you put it in a graph and then you get some additional insights out of it by maybe reuse similarity, maybe use community detection, maybe add vector search to it, all of it. And that could give you some insights that could then be bounced back to it and then you find even some more. Yeah, totally. And you know how we you know we like we like going meta here so this this was kind of my like episode one in this because i want to go deeper in this so here totally delegating to the llm on on how to how to um how to you know modify the model, create the model. But what if I capture in my knowledge graph my preferences in terms of my knowledge about modeling? In that case, I can start interacting with and control the way the LLM behaves. That's where I want to go, but you're right. This initial step, it just shows more the framework, what this kind of reflection type of agent can look like. But yes, we'll bring the graphs into the process as a way to kind of, again, own the way this process kind of happens. Yeah. Oh, that's very cool. So this is a great teaser, a great cliffhanger to end on episode number 27. So you can look forward to the next one then. I think that's going to be very exciting and very, very interesting to see how this develops and how we can, you know, learn and add new features across the coming episodes until we have like a super AI graph modeling, you know, expert. I don't know if we mentioned it on 25, but I mean, the way I see this, I mean, we now produce a kind of a Jason model and I think we show, yeah, we did show how it could automate the creation of the cipher to do the import, but you know, why not? For example, the integrations like, you know, with Google Cloud, what was the name of the framework? It's escaped me now. Gemini? No, but the kind of the sort of ETL, well, it's not really ETL, but the 25, it was Dataflow. Dataflow is, I actually have a slide, but yeah, the idea is, you know, once we have a model, why do we not use it to maybe to generate, let me put that slide up there. How do I show it? Let me window. That's not the one I want to share, is it? That's not the one I want to share, is it? I don't see it on my side. Oh, here we go. Well, yeah, so Dataflow is a framework or a tool from Google Cloud to move data from BigQuery into move database in the file transformations. So why could we not generate these type of mapping documents that describe how these data, because that's what we're doing here. In BigQuery, we have basically a big fat table, you know, that can be pretty much the same that we've seen from Kaggle. So we can use this process to generate the configuration and to automate the ingestion from BigQuery into Neo4j. So that's one example, but the moment we kind of have come up with a model, we can automate and not only come up with a model, but map it. Remember that the model that we generate, it maps to the sources, to the original file, to the original columns in the file. So we can basically translate that into a sort of an ETL process to to to move the data in the same way we did it using just cipher, but in using the data flow framework here. So that was another idea. Too many ideas to... Indeed, too many ideas. All right. But that's cool. That I think should round us up out very well to the end of this episode and gives us a nice like I said earlier, like a nice cliffhanger to what to look forward to in episode 28 and onwards. Next, or do you want to say something? No, I was going to say because I was reading a question from Peter on whether trying it with different models behave in the same way. I've tried only with not the commercial ones, but the open source ones on whether the trying it with different models behave in the same way. I've tried only with not the commercial ones but the open source ones and the difference was substantial. So it was the quality was much better with OpenAI but I have not tried it you know with Anthropic or any of the of these you know Gemini but that's definitely something that I want to give a try. So that OpenAI performed way better than for example which one Mistral is the one that I tested through Lama and Zephyr. I tried a couple of them and the quality was not as good. So that's my kind of data points that I have, but it's not really an exhaustive or a really serious analysis. My first impression is that the quality was much better with OpenAI. But yeah, worth exploring with others. Absolutely. I mean, there are always other options. And if you try it out with something else, let us know. Give us a shout out on the YouTube comments maybe or in the community forum. And let us know if you have some experiences with trying it out with And let us know if you have some experiences with trying it out with different models and what your results are. So before I let you go, next stream is happening on the 18th of April. So a little break next week. No session. But afterwards, I'm coming back with Tomas as a guest. He wrote a book on graph algorithms for data science. And we will introduce this book with a little bit of an in-depth look at the chapters, the contents, and what he wrote there. I think Tomas is a well-known figure in the graph data science, Gen AI world around Neo4j. He writes very, very many blog articles and gives interesting, has a great knowledge on the topic. So I'm looking forward to having him as a guest to talk about his book in more detail in two weeks time. And between now and then, what's next? What can you do? Join the Neo4j community, join our Neo4j Discord server. And for learning, you can check out the Graph Academy with courses. My colleague Martin or our colleague Martin just released something on vector search. So if that's interesting to you, then you can learn there. And obviously, there is a wide range of sessions around Gen AI as well these days, but also beginners courses for Neo4j newbies. So yeah, with that, I think we are at the end of today's episode. Looking forward to next month's edition. Wishing you all a great rest of your day, rest of your week, and hope to see you soon in one of these uh stream streams or at the community sites um you know let us know reach out and uh you know speak uh later take care everybody and thank you jesus for the great demo today it was really interesting yeah thank you see you next time bye next time
Going Meta - Ep 27: Building a Reflection Agent with LangGraph
Episode 27 of Going Meta – a series on graphs, semantics and knowledge

Jesús Barrasa: https://twitter.com/BarrasaDV
Repository: https://github.com/jbarrasa/goingmeta
Knowledge Graph Book: https://bit.ly/3LaqE6b
Previous Episodes: https://neo4j.com/video/going-meta-a-series-on-graphs-semantics-and-knowledge/

Links:
GenAI Ecosystem: ttps://bit.ly/4cpAY6D
Is RAG dead? https://docs.google.com/presentation/d/1mJUiPBdtf58NfuSEQ7pVSEQ2Oqmek7F1i4gBwR6JDss/edit#slide=id.g26c0cb8dc66_0_131
What’s next for AI agentic workflows (Andrew Ng): https://www.youtube.com/watch?v=sal78ACtGTc
Reflection Agents: https://blog.langchain.dev/reflection-agents/
LangGraph: https://python.langchain.com/docs/langgraph
Supply Chain Dataset: https://www.kaggle.com/datasets/shashwatwork/dataco-smart-supply-chain-for-big-data-analysis
Crime Dataset: https://www.kaggle.com/datasets/sahirmaharajj/crime-data-from-2020-to-present-updated-monthly
Dataflow BigQuery: https://neo4j.com/docs/dataflow-bigquery/

0:00 Intro 
18:30 Reflection Agents with SupplyChain Data in a Notebook
35:00 Orchestrate with LangGraph
47:24 Same Logic with Crime Data 
56:00 Summary and Outlook
1:00:49 Wrap-Up

#neo4j #graphdatabase #knowledgegraph #langchain #llm #genai #rag #graphrag