 Hello, everybody. Good morning, good evening, good afternoon, and thank you for joining this Neo4j Road to Nodes workshop in the final weeks leading to Nodes 2024. I'm happy to do the workshop together with my dear colleagues Alex, Thomas, and Will Tai. So I'm looking forward to this cool session. And what's it called? It's called Mastering Retrieval Augmented Generation with the Neo4j Gen AI Python Package. Another very long name, but I'm sure you'll see that it's gonna be very interesting, particularly mastering graph reg, or reg rather, in this case, with the Neo4j Python Package. Should be very very interesting it's a very hot topic very very interesting for everybody at this point in time so i'm i'm very much looking forward to today's session before we dive in i'd like to let you all know that we will record this session so you if you you know need to drop off at some point or if you want to watch something again you know we missed something or maybe something was too a little bit too quick for you you can always re-watch um materials like slides and anything else we will share with you also in a follow-up email so don't don't worry about this um it's good to see um to see already in chat a little bit of activity here. I see people from Germany, from Austria, from Bangalore. Great to see you all. Let us know where you're from and get the ball rolling a little bit. Make this interactive as well. So if you have any questions, any comments, please don't be shy. Comments and anything else, things you think are cool or interesting, please put them in chat. If you any questions in big marker you see a little q a icon where you can click on and add um any other you know questions you might have anything that's unclear and we'll have a look at it um we are uh also in the chat for for any other questions obviously and there will be probably some time uh some little break times where i can also take some question on on screen so um just you know if anything is unclear we can we can make it public um yeah i see more people italy belgium london france barcelona china stockholm malaysia australia sicily nice That's probably better than here where it's raining. Bedford, UK, probably also not very nice weather. Sorry. Another Germany, another Delhi, that's probably also very good weather. It's good to see you all, but let's dive in. Alex, Will, I'll hand it over to you and we can get going. Brilliant, thank you, Alex. I'm just gonna try sharing my screen just um could uh could you let me know if it if um you can see yeah not yet let's see um here it comes yep there we go you should all see now a web page with a um with a graph academy course on it brilliant let's begin then um so yeah thank you everyone um for uh coming to this course so um the name of this this workshop today is um mastering uh retrieval augmented generation with the Neo4j GenAI Python package. Now, just up front, I should probably say that we have added a rename of the Python package in the last couple of weeks. So the Python package is now called the Neo4j Graphrag package. You can still use the old name. So if you are using any projects and you're pipping, installing Neo4j-GenAI, that still should work, but you will be limited to version 0.5. So if you are going to build any projects off of the back of this workshop, it's the Neo4j-Graphrag package that you'll want. The purpose of this course today is to give you guys a little bit of an overview of retrieval or augmented generation. I'm going to go through some of the sort of core concepts that lie behind that, things like vector search, semantic similarity, full text search, and knowledge graph creation. And then I'm going to take you through kind of the, how you can build, use those core concepts to build kind of full retrieval, augmented generation apps using the Python package. So this course is going to be run in roughly two halves. The first half I'll be covering, but we'll be going through kind of core concepts. So semantic search, vector indexes, embeddings, full text search, and creating knowledge graphs from unstructured data. Then in the second half, Will Tai, who's also here, my colleague, is going to take over and is going to sort of take you through how we can kind of draw that together and build full apps using those core concepts. So this course is going to consist of several sort of lessons where I will sort of describe core concepts to you and several coding challenges. So as we go through, there'll be bits where you guys can get involved and do bits of coding. I really encourage you to try stuff out. None of the coding stuff here is particularly difficult. It's not going to require a huge amount of brain power for you but hopefully it should give you a chance to test the concepts that we're going to discuss today um so i think the first thing to do is to post a link to this course for you guys to sign up so this is a course on graph academy which is our uh training platform here at Neo4j. So I'll post a link in the chat, and then you guys can sign up and work through this course at the same time with me. So that should now be in the chat for you guys. Let's begin. Kim, I've just seen your comment. Yeah, there will be a recording of this talk available later. My apologies. Okay, here we go. So like I was saying, the first half of this course, the purpose is to kind of take you guys through some of the course concepts that kind of lie at the base of retrieval or augmented generation and at the base of quite a lot of AI and Gen AI applications. So in this first part of the course, we're going to cover vectors, using vectors to search for similar pieces of data. We're going to cover embeddings and vector indexes. And we're going to build a knowledge graph from unstructured data using Python and the Neo4j GraphRag package. All right, let's go so for this course we're going to be using um primarily uh neo4j the neo4j browser as well as um a git repo that we put together with some code snippets in it so i will post the uh the git this is the uh the python the uh the git repo we're gonna use. So I'll post that in the chat. You can use, you can of course just run these code snippets on your local machine if you want to clone it there, or you can use Gitpod workspace. So you need a, I believe you need a GitHub or a GitLab account to access this. This could give you essentially a Visual Studio Code instance in your browser if you don't want to clone this repo to your local machine, and you can run through the examples there. So let's open Gitpod workspace. So. Here we go. What this is doing is this is going to clone the Git repo for us and give us a workspace with all of the dependencies already installed and all the code already there. Yep. So here we go. Just take a while. I think it's just installing dependencies in the background. There we are. Yeah. So just give us a little time to install dependencies. So, So maybe I'll just go through this repo while these dependencies are installing. So, like I said, this is a Python repo with some of the coding challenges that we're going to be going through in the course today, split into two sections, which roughly mirror the two sections of the course. So this is the first section. So we're going to be going through first today. As you can see, there's two coding challenges. Each one has got places where you'll put your code in. If you ever get completely stuck, though, there are solutions here for each of these. So if you want to cheat a little bit, you can. But yes, we'll put some instructions in the chat there for those of you who want to run on a local machine. And Manjunath, you don't need a GPU to run this locally. It should run fine on There we are. Okay. So there is one thing we need to do in order to set up this code environment to be ready to submit solutions, is we need to create a copy of this.env example file. So this is just going to export some environment variables that we'll use for connecting to OpenAI and connecting to the Neo4j database. So we create a copy of that and we just call it.env this time. And then what we need to do is we need to fill in with our various, well, first with our OpenAI key. So I believe, Will, have you posted the OpenAI key to the chat? If you have your own OpenAI key to use for this course, please use it. If not, we have a key that will be available just for these two hours, and then we'll be removing it. So you can use that. And then we also need to post here the Neo4j database connection strings. So with this course we will be provided with a sandbox Neo4j instance so we just need to take the login details here and paste them the end file. Cool. Let's go back to the course then. So is anybody... Oh, I should say as well, you can run the test environment file, which should tell you if your environment is set up correctly. So if you've properly filled in all of these details in your.env file, you should get okay. Is there anyone having trouble setting that up? Is there anyone who needs me to go over there anything again? Or if not, we can move on with the course. I think we can maybe give it a second or two, but I don't see any particular question on this. But we could maybe we could, I don't know if you want to, it's a little detour, but we could look at this. This might be an interesting question from Bjorge. And he asks, what are the considerations between vector indexes and graph indexes? I'm not sure what he means by a graph index here. Yeah, I think probably he means why should we add a vector in, well, maybe you can elaborate a little bit, Bjerg, I'm not sure either. Do you mean the difference between vector databases and graph databases or yeah maybe maybe um yeah let us know what what you mean by that actually you're right if you mean um if you mean where where can kind of graph where the graphs sort of do better than like vector indexes on their own? We will cover that later in the course. So we're going to dive into kind of the advantages of using a knowledge graph along with a vector index. So that's going to be in the second half of the course, if people are curious about that. And this first half is going to be sort of core concept covering. Yep. Super. Cool. Super. Cool. I'll give it another like 30 seconds because there's some people in the chat saying that they're just waiting for Gitpod to launch. So I'll give you guys a little bit of time to catch up there, but I wouldn't worry too much because the first couple of lessons here aren't gonna be interactive. It's just going to be me describing a few core concepts. So you can sort of leave Gitpod in the background to load as I go through this stuff. Yeah. Okay. Yeah. Let's move on then for now. If anybody is really having trouble setting up GitPub, if it doesn't load in the next minute or so, stick another comment in the chat and we can go back if needs be. Okay. So in this first section, we're going to cover vectors and embeddings. So two core concepts in retrieval augmented generation or RAG, I'm just going to call it RAG from now on. So firstly, what are vectors? So vectors are simply a list of numbers. They can be used to mathematically represent many things in the real world. For example, the three-dimensional vector here represents a point in space. Vectors can also be used to represent many different types of data, including text, images, audio. And we typically use vectors with very high dimensionalities, often in machine learning and RAG applications. So embeddings. When we talk about vectors in the context of machine learning in AI we typically call them embeddings. Embeddings are vectors which represent complex pieces of information such as text and images. For example here we have an embedding that represents the word Apple. Typically in AI applications, we use specialized ML models called embedding models to create embeddings from our data. There are also versions of these models that work with images or text in languages other than English. And text embedding models, for example, which are the ones we're going to deal with mostly in this course, are trained on large amounts of text data. And these models learn to represent each word or phrase in a given bit of textual data with a vector that kind of captures the meaning of that piece of text or phrase. So semantic search. So semantic search is a very common component in many RAG applications. Semantic search allows someone to search a database for text which is semantically similar to their query. So for example, the phrase big boats is semantically similar to large ships. Even though both of these phrases contain completely different sets of words, they mean pretty much the same thing. So they are semantically similar. The way semantic search is done in RAG applications is using embeddings. So each piece of text in a database indexed with an embedding, which captures its meaning. And then when a user searches, their query is converted into an embedding, which is sent to the database. The database then compares this vector with its index of vectors and finds vectors which are close to it mathematically in some way. And you can see in there's a section just below this which kind of talks about that mathematical comparison process. I'm not going to go through it here because it's a little bit too in-depth, but if you want to know more about the mathematics, there is a little section after this. The database then uses its index after it's found these vectors that are similar to the user query vector to return the text associated with those close vectors. So like I said below, we have some sort of mathematical detail here about this. I don't think it's too interesting to go into here, so I will skip it for now. So are there any questions? So just to check here. Björg, I heard there's something called graph index. Oh yeah I'm not sure what you mean by that maybe it'll become clear as the course goes on. Yeah, probably. Yeah. I have one question from Sebastian. What's the difference between embeddings and indexes? Embedding and indexes. So the embeddings are just vectors, so lists of lists of like numbers that represent, say, text. Indexes here are the kind of data structures within Neo4j, within generally a database that are used to sort of search those vectors and to return appropriate data. So an embedding vector, long list of numbers, index, a data structure within the database to kind of search through those long lists of numbers and search through the text associated with those long lists of numbers. Cool. And one more maybe we can cover publicly here from Marie. Does the Graphic Python package support other LLMs besides OpenAI? We'll already answered, but just to make sure. Yeah, absolutely. So Mistral, Azure, Vertex, Cohere, I believe, as well now. So, there's a fair few others. Yeah, definitely not just over the night. Yeah, super. Cool. I'm hoping everyone's got Gitpod set up now because we're going to go through the next section, which is the first interactive coding challenge. So we're now going to try to semantically search a Neo4j database containing information about movies. We're going to semantically search the database. We can see some examples of the movies here. So if you look, we're running this Cypher query, and we can see we have some movies in our database. We have Toy Story. We have their plot lines, description of the movie, Jumanji, Grimpy Old Men. So we're going to be using this database for the rest of the course. We can also look at examples of individual movies. So if you want to investigate that. So here we're just looking at Toy Story specifically. Feel free to change this to another movie if you want to see the plot line and the title. The embeddings have already been created and indexed for this challenge. We will see in the next challenge how we can create indexes and embeddings, but for this one, we already have them here. And we can see the embedding we have here for the plotline of the movie Toy Story. So these embeddings have been created on the plotlines of the movies in order to kind of capture the meaning of the plot line so you can see here it's a very very big list of numbers like i was saying um this is the one for toy story plot line so this this embedding here represents this bit of text here So if we want to find other movies in the database with similar plot lines to Toy Story, we can compare this embedding here with other embeddings in the database using the next snippet. So this is what this snippet does. It takes the embedding from Toy Story and compares it with the other embeddings in the database and returns embeddings that returns the movies and the titles of, returns the titles and the plots of movies with embeddings that are similar to this. So we can see here the first movie, which is most similar to Toy Story, is of course Toy Story, because it's the same movie. We also have The Little Rascals, which is apparently a similar plot to Toy Story, The Neverending Story, and Drop Dead Fred. The scores here, so this is the third thing here in each result, are similarity scores. So what these tell you is how close the plot, the meaning, how close semantically the plot of Toy Story is to other plots in the database. So you see here for Toy Story, it's the higher this number, the closer the plot is to Toy Story. So the close to one, the higher, the more semantically similar that plot is to the plot of Toy Story. So it's 1.0 for Toy Story because Toy Story is the same plot. For the Little Rascals, it's giving a score of 9.92. So this embeddings model thinks that the Little Rascals plot is very close to the plot of Toy Story. I should say the plot description to be more accurate. So it's this text in particular that we're comparing, not the plot in some abstract sense. So again, feel free to try these snippets with your own movie. So you can compare Jumanji to other plots, compare Goodfellas, see which other movies in the database are similar to the plot of that movie based on the right-hand side to start up. So if you're having trouble running any of these Cypher queries, just refresh your browser and then wait a little bit. It can be a little bit slow. Okay, so on to the next challenge. So in the last challenge, we used our database of embeddings to compare movie plots to one another and to find those which were most semantically similar to the plot of the movie Toy Story. In this challenge, we're going to find out how we can create our own embeddings. As I mentioned a few lessons ago, in AI, we typically use pre-trained ML models in order to create embeddings, which is what we'll do here. And we're going to use a model from OpenAI. So have, again, that OpenAI key that was posted in the chat a few moments ago ready, as we'll need it here. So we can use the genai.vector.encode function from within Neo4j to create embeddings for pieces of text. So if we open those up in the browser, we need to replace this SK here with our OpenAI key. And then we can, so then we're going to create an embedding for this piece of text here, which is just text to create embeddings for. Feel free to change this to something else if you want to see what an embedding would look like for your own piece of text. We should be able to run this. And we can see we have an embedding here. But like I said, we can change this to something else. And you should see the embedding change. Now, creating embeddings is cool, but what we really want to do is we want to create our own embeddings and use them to search our database. So we can do this by combining the genai.vectorEncode function with the db.index.vector.queryNodes function that we saw in the last challenge. So once again, we need to input our token here. So we're going to see in our, this, what this is going to do, this is going to create an embedding for this piece of text. It's then going to compare that embedding with the plots of the movies in the database. And it's going to return the titles, the plots and the scores of the movies that are most similar to this query. So a materious spaceship lands on Earth. Hopefully we'll see some sci-fi movies up here. So yeah, 2001, Space Odyssey, Alien. Makes sense. It's talking about spaceships. These movies are kind of sci-fi and about spaceships. Okay, so on to the next challenge, are there any questions? I'll stop there just before we go on. In such a high dimensional environment the scores can always be quite high. No, I guess it depends, I mean the shouldn't necessarily affect it. The scores should capture the semantic similarity between two embeddings. So if you can test it out yourself, try, if you compare something completely unrelated the vector of something completely unrelated with the plot of toy story you should get a very low score you know to like talk about like i don't know the geography or something like that it will it will give you a low score hopefully with the plot of toy story it shouldn't it shouldn't always be 0.9 or above. There was another, I mean, Will is very active in Q&A, so I don't want to read out too many here, but maybe this could be interesting is for the embeddings, does it include multi-model embeddings? So image, voice, and videos. So we'll answer this with yes. So just so you know, you can create an embedding depending on the model, it is flexible. Yeah, and we'll see this now. So we're gonna create some embeddings for movie posters. Yeah, cool. So I'll continue then. I'll let Will, if there's any specific questions that you'd like me to answer, Alex, I'll let you jump in. Okay, so we've seen how we can create embeddings and use them to search a vector index in a Neo4j database, but what if we want to create our own index? So we already have a vector index for the Neo4j database, but what if we want to create our own index? So we already have a vector index for the plots of our movies, but as I mentioned before, we can also use embeddings to represent images as well as text. So let's try creating an index for the posters of the movies in the database. So we do have posters available. So let's take a look at an example first. So Toy Story here. So you'll be returned within the database the link to the image of the poster. So you click on it. When you run these examples, you can see the posters. Cool. So luckily, we already have embeddings created for these posters. We'll see soon how we can create our own embeddings. Well, we saw in the last challenge how we can do that. We'll see how we can create embeddings a little bit later using Python and upload them to nodes. These embeddings can be used to compare images for similarity. So in the last couple of challenges, we had embeddings on text and we were comparing these embeddings in order to gauge how similar two pieces of text were semantically, how similar they were in meaning. In this situation, these embeddings represent how similar two images are in kind of look. So two images that look roughly the same should have beddings that are mathematically quite close. So let's first take a look at the poster embedding. So like I said, this embedding here represents this image here. So now let's create our index on these embeddings. So to create an index within Neo4j, we could use the create, to create a vector index, we can use the create vector index command. With this command, we just need to include a name for our index. So we're gonna call it movie posters here. We need to let the command know which type of node we want to create the index on. So here we're creating them on movie nodes. And we need to let the command know which property on those nodes we want to create the index for. So we're creating them on the poster embeddings here. We also need to let it know the dimension of the vectors we're going to be using. So these vectors have 512 numbers in them. And then we need to also let the index know what mathematical function we're going to be using to compare these vectors. So I wouldn't worry too much about this part. I can go into more detail, maybe at the end if you'd like, but there is some detail, like I said, a couple of slides ago that goes into kind of the mathematics of this, but it's a little bit more for you guys to read in your own time. So let's run the second code snippet. What model did we use to create the poster embeddings? Oh, yeah. Apologies. It should be up here somewhere. We're using the clip model. Oh, maybe it's not listed here. Potentially. Yeah, we're using the clip model. The exact one I can list for you. Yes, opening our clip model. Okay, cool. There we go. I thought it for you. Yes, open an ad clip model. OK, cool. So it's in here. There we go. I thought it was listed there somewhere. Yep, so let's create our index. So we can run this code over here. And then we can see if it's been created here. Movie plots. We have our movie plots, vector index. We also have our movie posters index here. All good, it's online. So don't worry too much about the ID. As long as it's online, it should be good. So let's now use this index to search for some similar posters. So we're gonna look at the movie Babe and we're gonna compare the embedding for the poster of the movie babe with other embeddings in the database i'm going to see posters that are similar to the poster for movie babe so we run it here first let's see this is the poster for the movie babe and we can see the poster which is most similar to it is for the movie Lassie which I guess is kind of similar to the movie Babe so we can see them all here they're all kind of animals on fields so I think this is working it's a field, sort of rural kind of look. Okay. You can, so again, a sort of challenge for you guys, if you have time, is to sort of try this again with other movies and kind of compare, see if you can compare them with other posters in the database, see which movies have similar posters to the movie you select. Right, onto the next challenge. I made a quick one because it came up two times now. When comparing images, does the image size have to be the same to be able to compare or is this not so important no um just the we're comparing so we're not directly comparing the images we're comparing the embeddings for the images um as long as you're using the same set of embeddings it shouldn't matter okay yeah sense. Okay, so if you guys remember from a few lessons ago, we were using semantic similarity to compare two pieces of text to see if they had similar meanings. For example, the two sentences, the child is playing with a toy and the kid is having fun with a play thing, are semantically similar as they share a similar meaning even if they don't share many of the same words. Semantic search uses semantic similarity to find text with similar meanings to a user's query. However, semantic similarity isn't always the best measure to use to return the most relevant pieces of text for a user's query. So for example, if we are searching using domain-specific terms, semantic search can fall apart. For example, semantic search may fail to return anything useful if you ask for an unusual name of a character or a place in a movie, as it would have no way of knowing what this term means outside of the context. An example of this could be like if you were searching for something like what is the what is a movie set on the planet Tatooine. This query only has meaning if you know what Tatooine means which is a movie from the planet movie from the series sorry a planet from the movie Star Wars. If the embedding model that has been trained on this textual data doesn't have any knowledge of what Tatooine means and hasn't seen it linked to Star Wars in any way, it's not going to be able to semantically match embeddings for Tatooine with Star Wars plot lines, for example. So the embedding model needs to know these specific terms for it to be effective. So if you're searching with anything really specific, you can't always expect Semantic Search to perform well. Another example of where Semantic Search falls down is when a user query includes specific strings, such as names and dates, that need to be matched exactly for accurate results. So for example, with the query Oscar winners 2019, semantic search might interpret the query broadly and return conceptually related results, so such as films, Oscar films, or discussions about Oscars in general. But potentially, it might not return, it might return results of different years when what we really want is only information about 2019. So we need an alternative which allows us to deal with these issues. Full text search is an alternative which matches queries on exact wording including variations in spelling or grammar rather than on semantic meaning so full text search works by splitting a user's query in and into the individual words and comparing them with the individual words contained on a specific property on a specific type of node in our neo4j database and the database will return the nodes uh the nodes that can that who specified text property has the biggest overlap with the user's query so basically the results that have the greatest number of words in them that are also in the user's query. So this means for a query like Oscar winners 2019, a full text search would return results containing 2019. And as such, it might be a bit more likely to return relevant results than would happen with semantic search. And then finally, there are a range of search methods which combine semantic search and full-text search in the hopes of kind of harnessing the best of both these techniques. And we'll see an example of this later in the course. So now we're going to try and create a full text index on our movie plots and querying it, and we can see the sort of difference of how full text search works compared to semantic search. So creating a full text index is very similar to creating a vector index. We just use the create full text index command and feed it the name for the index we want to create, a type of node again, and a property that we want to create the index on. So again, here we're going to create our index on the plot of the movies. So we can run this. And now let's see if our index has been created. And we're good. It's online. We have our plot full text, full text index. So now let's try querying the index to see how it's different from querying a vector index. So again, this is a similar process to querying a vector index. We just use the db.index.fulltext.query nodes rather than the db.index.vector.query nodes function here. So what we're doing here with our query, we're querying the plot full text index. And this second string here is our query. We're querying the plot full text index, and this second string here is our query. So you can see here we're querying for 1375 Imperial China, and we have found a movie, the first movie here, Mr. The Warrior, which includes 1375 Imperial and China. So it makes sense. This is the most relevant movie. It contains all of the words in our query. As an extra challenge to anyone who wants to do it, try running this query with the vector index as well and see how it changes, see which different movies you returned with the vector index and with the full text index and see how see how they differ. And have a think about why that is. Why is this returning this movie here? And why might it not return this movie with the vector index? Alex, did you have a question? Yeah, I just wanted to read out a question, which I think is good for what you're showing here right now. It's from Rohan, and he asks, why create an index on the plot and not the plot embedding? Because this is a full text index, not a vector index. So with full text, we don't use embeddings at all. This is just using, this is just splitting the text on the plot line into its component words and we're comparing the component words of our query with the component words of the text in the plot line and we're seeing we're returning nodes which have the greatest kind of overlap so there's no embeddings involved at all. This is just directly comparing the words. So like I said, this has some advantages over using embeddings. For example, here we're searching for 1375 Imperial China. This search is not trying to capture the meaning of this query in any way. It's just seeing which plot lines in our database contain 1375 or contain imperial or contain China and returning them, or even sort of slight variations in spelling. So for example, if I accidentally included an additional character, this search should find words that are similar to imperial with a Z, for example. So this is not involving beddings at all on a full-text index. And the advantage of this type of index is that it will return things with these specific terms in them so um like i said a challenge is to to uh to try this query with a vector index and see which movies it returns and maybe ever think about why that is not maybe not returning this movie in particular um do we have any other questions or is that it? There was another question in regards to the score. What kind of score gives that back or how does it calculate the score? Because it cannot be, you know, as before. Yeah, it's quite a complicated process. I won't go into the exact exact details of it here i can send some uh um send some uh some links with the with the exact but it is quite involved um but yeah the score is the score is similar in meaning to the semantic similarity score so the higher the score the better the match basically okay And then maybe we do one more from Simon. And he asks, what speed difference do we see when comparing full text and semantic search? Embedding the search phrase probably takes some time. Yeah, this is a lot quicker in general, because you don't have that OpenAI request that you need to make. You don't have to send the data to OpenAI, you don't have to create the embedding, return it, and then send that to the database and then to compare that. So in general, the full text is quicker. Cool. Thank you. All right. On to the next challenge. So now I want to, for the sort of last kind of 10 minutes, I want to talk a little bit about unstructured data. So unstructured data refers to kind of information which doesn't fit neatly into predefined structures and types. For example, text files, emails, social media posts, videos, photos, audio files, web pages. And unstructured data is often more difficult to analyze than structured data. So we're going to talk in the next 10 minutes about how we can use Neo4j to kind of analyze unstructured data. So a little bit about vectors and graphs. So we've already seen that embeddings are good at representing unstructured data such as text. However, graphs are also a powerful tool for representing unstructured data. Graphs can be used to represent the entities and relationships within long pieces of text. For example, a graph could be used to represent the relationships between different products and customer feedback in a customer feedback system. Or a graph could be used to represent various laws and regulations inside a legal document. We call these types of graphs knowledge graphs and we'll also see later how combining vector indexes and graphs can help us build really advanced search processes which go a lot further than semantic search on its own. So for the next few lessons I'm going to talk about how we can build graphs from structured data and upload them to Neo4j. So chunking is often the first step when processing large amounts of unstructured data. Chunking involves breaking a piece of text down into smaller chunks, which can be more easily processed. This is a very common process when whatever database you're using, when you're importing data into a RAG application. Text data is broken down into chunks. After this, the usual next step is to create embeddings for these chunks. So we've already seen how we can create embeddings. We're going to do it here for some text data. So like I said, we'll see more how graphs can be used in RAG applications in the second half. But for now, I'm going to go through the process of constructing a graph from unstructured data using the Neo4j GraphRAG library. So first up in this challenge, we're going to use the Neo4j GraphRAG Python package to chunk a piece of text and create embeddings from those chunks. So we're going to go over to Git Pod now, and we're going to open the create embeddings chunk file in our code. And so here we go. It seems like it's falling asleep on me. I try refreshing it a little bit., can be a little bit slow. Maybe we can do one question while it's loading. From Danny, which score categorized as good? One is good and zero is bad or how does it work? Yeah, for semantic search zero is zero is these two pieces of text are not at all semantically related, and one is they're exactly the same piece of text. So the closer to one, the closer in meaning they are. For full text search, the scores can go above one, but basically they are, the higher the score, the more overlap there are in kind of words and characters between the two bits of text. So the higher the score, the better for full text search. Cool. We have our first coding example open. So we're gonna use the first paragraph of the Wikipedia description of London, because it's where I live, it's the city I like. But if you want to use another piece of text, feel free, just post it in here and you can chunk that up and create embeddings for it just in the same way. So first, we're going to chunk the text. So what we need is we're going to use a text splitter from the Neo4j Graphrag package. This splitter simply splits text into chunks, which are less than or equal to the chunk size parameter in number of characters. So this is the maximum number of characters we can create for a chunk of text. And which we also allow a little bit of overlap between the chunks. So we have a 10 character overlap here. This is to help avoid sentences being kind of split in the middle between two chunks. So each complete sentence will hopefully be contained in at least one chunk. It can also help when constructing the graph, as it can help to ensure each chunk has enough content to extract the right entities and relationships when we're actually constructing the graph from each chunk. So let's run this piece of code and see how this splitting process works. So we copy and paste it in here. We save. And then we run the createon bed chunks. So we should see now, you can see here this is split into different chunks. So we have the first chunk, London is the capital and largest city in both England and the United Kingdom with a population of, and then we can see the next few chunks, settlement, city of Westminster, the national government. So we can play around with these chunk size, we can have a bigger chunk size if we want. You can see here, this is the first chunk, the second chunk is here, the third chunk, and so on. Okay. So as you can see, we've split each chunk into sort of 200 character bits of text here. Each chunk also contains a little bit of the last chunk. So you see the first chunk here ends with a population. The next chunk starts with population of. So we have a little bit of overlap between the chunks. Just to add as well, there are many different techniques for chunking. So this is a very simple one here that we're using in this library, which just splits the text based on characters. But there are many different ways to chunk text. You can use embeddings to find the best way to split a document into chunks. You can split based on paragraphs rather than just on characters. So we can split this into two paragraphs, for example. We can split based on bullet points. So if you're doing this in the future, if you're constructing a RAG app, this is something you definitely want to experiment with, is your chunking technique. So back to the course. The next step we're going to take is to create embeddings for each chunk. Alex, did you have a question? Yeah, there was a question, I think an interesting comment. It would be interesting to hear what you think of this from Audrey. With the now large context window of the L the other lens is there really still a need to consider applied chunking um generally speaking it sort of depends on how big your uh your data is so and also in terms of like even with the large contact winds that windows um you know if you're sending a 50 page pdf to the llm it's going to take a lot longer to process and it's going to cost you a lot more so in general i would say yes particularly for bigger pieces of data you only want to really provide the llm with the uh the most relevant context rather than just a giant document. So this is definitely still a relevant thing for RAG apps. Super, thank you. Okay, so now we're going to create some embeddings for each chunk. So as I said, we want to use semantic search with our data. If we want to use semantic search with our data, we need embeddings. Again, we're going to use OpenAI models for this. I'm going to use the text embedding three large model from OpenAI. So we copy our code here. We copy it into the second browser window here. And let's see what our own embeddings look like for these chunks. You can see we've got a rather large embedding here for our chunk. Again, feel free to try this with your own data and see how it varies, see how changing the chunk size changes the embeddings, things like that. Okay, on to the next section. So before we continue creating our graph, I just want to go into a bit more detail about knowledge graphs. As I said earlier, graphs are essential to many AI machine learning applications. Knowledge graphs are a great way to represent the information contained in a particular domain. They provide a structured way to represent entities, their attributes, and their relationships, allowing for a comprehensive and interconnected understanding of the information within that domain. And we can tailor knowledge graphs for semantic search, data retrieval, and reasoning. And in the next challenge, we're going to see how we can build a knowledge graph tailored for a RAG application. So you can see an example a little bit here of a knowledge graph that represents Neo4j. And finally to add, large language models can also be used to create knowledge graphs. So, you know, for example, if we wanted to create a knowledge graph from a piece of unstructured text, unstructured data, such as a long piece of text, we can first chunk this text up. Text chunks can then be fed to an LLM, which can then extract entities, relationships, and properties from it. And then these can then be used to populate nodes and edges in a knowledge graph. So LLMs are very good. This is one of the things they're very good at is creating knowledge graphs from unstructured pieces of text. I'm going to see how this is done in the last challenge of this section. So we're going to open the, we're going to go back to the Git pod and we're going to open this file, the build graph file in our editor. So you can see here, once again, we have a few sections where we're going to fill in our code. If you first want to scroll to the very bottom of this page, you'll see the text that we're going to use to create our knowledge graph from. It's just a few sentences about characters from Dune. We're big fans of Dune in the team. But again, if you guys want to try your own text here, it can be fun to create a knowledge graph from, I don't know, description of your family or whatever you guys would like. Okay, scrolling up to the first snippet now. So as I said, the first text, the first step often in creating a knowledge graph is to chunk up our text. So we're gonna do that here. So once again, I'm gonna use the fixed ties text splitter class to do this. So we take our code here and we stick it in the first section. Next, we wanna create embeddings for these chunks. Again, we're going to use the same code we used in the last challenge for this. We're going to use the text embeddings-3 large model from OpenAI to create an embedding for each of our chunks To help guide the LLM, we want to list the types of entities and relationships we're looking for in our instruction text. So this helps, this means that the LLM won't just extract any old information from our text, but it will look for specific types of information and specific types of relationships between those entities so in our in our piece of text here we're looking for person we're looking for a house we're looking for a planet looking for organizations we're looking for son of relationships heir of relationships rules and member of relationships and we're looking for, what we're also saying, person can have a relationship with another person, whether a son of a person. We're saying a person can be an heir of a house. We're saying a house can rule a planet. We're saying a person can be a member of an organization. So if you go back down, if we go back down to the text here, we can sort of see why that is. We're looking for persons, so we're looking for people like Lady Jessica, Paul. We're looking for house, so we're looking for house Atreides. We're looking for a planet, so planet Caledon, hopefully. The LLM will pick up. We're looking for organization, so we're looking for the benny jenner's jeseret organization um so we want to we want to provide these this sort of um schema to the llm to help guide it to ensure that it extracts the right entities and relationships it otherwise it can it can just extract anything it likes and you can end up with quite a messy knowledge graph so this is really useful to help guide the LLM. So we copy this over to our notebook. Finally, so next, sorry, we want to add an LLM to this. So we're going to use GPT-4-0 here with a few model parameters set, which should help the LLM work a little bit better here. But again, this should work with any LLM. And as I said, we support Vertex AI, we support Mistral, we support other LLM providers here. Okay. Next thing we want to do is we want to create our knowledge graph creation pipeline. So this is the class the various processes in knowledge graph creation together. So the text chunking, the embedding, the LLM extracting, this class will do it under the hood for us and chain these processes together to correct struct our full knowledge graph. So we need to provide this class with our Neo4j driver to access our database, the text splitter we want to use, the embedder we want to use, the entities, relationships, schema, as well as the LLM. And then finally, we have a few uh a few other parameters so on error this just says ignore any errors that are generated and continue and from pdf so we can still use we can use this uh this pipeline to uh extract stuff directly from pdfs but um but here we're just going to use the text directly so finally we want to run the pipeline. So like I said we're not going to use, we're not going to extract from PDF, we're just going to feed text directly to this. So we're going to feed the text about June and we're going to run the pipeline to see how our graph looks. So we save. And we run. So you can see here, it's going through the process of chunking the text, creating the embeddings. And we should see in a second, okay, we're done in five seconds. That was very quick. So now we can run this command to see what our knowledge graph looks like. It should look a little bit like this. So it's a little bit messy. I think we might have duplicated ourselves a little bit here but you can sort of see we have we have chunks which contain which contain our text so we have the first chunk the son of Duke Leo Atreides and Lady Jessica Paul is the heir in the house and we can see pointing to these chunks we have entities of persons like Paul we have Duke Leo Atreides, which has been extracted from this chunk. We have House Atreides. So we said we were looking for house entities and we found House Atreides here, which is pointed to chunk zero. We were looking for planets. We found planets. And we've also found relationships between people. So we can see here Paul has a relationship here, son of, which points to Leo Atreides and which points to Lady Jessica. And we can also see we have relationships between the chunks. So zero to one. So we have our first chunk. So these are indexed from zero. And we can see this is the first chunk of text but we also have our second chunk of text so we can see that the relationship between the chunks as well here with this next chunk relationship so apologies this is i think i might have run this earlier and this is uh this has doubled up a little bit but this is a cleaner version of what this this should look like um you have like i said you have your chunk uh nodes have, like I said, you have your chunk nodes, you have your planet node, you have your house node. You kind of see how from this piece of text, the LLM has kind of extracted what we're looking for. It's extracted our knowledge graph. We have our chunks, we have our house, we have our person nodes, we have our house node, and we have our person nodes, we have our house node, and we have our planet node. So again, I encourage you all to try this with your own piece of text, see what kind of knowledge graphs you get. We have a couple of questions. Sorry, Alex, go ahead. Yeah, yeah, I'll just finish up here because this is the last bit. So there's a few bonus challenges as well here for you guys in your own time. once you have a knowledge graph um try creating um you know something try creating a vector index on the embedding nodes for these uh on the chunk nodes for this graph and try using uh you know the vector index uh db.vec indexvector.queryNodes procedure to query this property and try the same with full text indexes. Yeah, sorry, so this is the end of my section. Do we have any questions? Yeah, a couple here for this last bit I think I'd like to cover here quickly. Artul mentions here text chunks can be fed to LLM, and they might create one kind of relationship in one another type for the same chunk. So I think how to get consistency here is a question that comes up a couple of times. I think earlier you provided sort of labels and relationship types to to help help guide the LLM by saying okay this is this is what you should be using this is a this is how exactly this one here so this gives gives one this is one way of doing it a couple of people said can we use an ontology or can we use some some other guiding system to feed the LLM or to provide the LLM as a pillar to say, okay, this is how I already have defined my data. Can you please adhere to this one? Yeah, this is essentially what we're trying to do here. So I wouldn't call this an ontology, it's more like a sort of schema, but what we have here are like entities relations and the kind of relationships between them. So what we're saying have here are like entities relations and the kind of relationships between between them so what we're saying here for example is you know a person node can have a relationship with another person node that is sunoff a person node can have a relationship with a house node which is air of and what this does is in our in our prompt to the llm we say look for these sorts of nodes look for these sorts of nodes, look for these sorts of relationships, and don't return anything which doesn't match these sorts of nodes and relationships. So you can edit this and try it with something else, something else that's not in the data, and you'll see that it won't return those sorts of relationships. So if you added added i don't know like a dog node or something like that here entity there's no dog mentioned in in in the text so it won't return dog nodes so this is quite this is a very good way of like ensuring consistency this this really helps it's not perfect and we we we do want to do i mean in the future we're hoping to add some extra bits of type checking to the package which will really ensure that you only get what you want but this is a this is a very good way of ensuring that you you you're only extracting the sorts of entities relationships there's the sorts of you're only constructing the sort of knowledge graph that you want from the data um is to provide this kind of schema to it when you are constructing the knowledge graph yeah and i'm just gonna share this in in chat as well we have jesus and i have a series called uh going meta where we recently covered exactly this in a bit more detail using an ontology adding an ontology to give structure to a genai graph rack application so if you're interested in this more in more detail and you know maybe watch the the last of the previous episode to to find out a little bit more how this can be done yeah cool um i'm a bit conscious of time um so yeah thank you for thank you for um for uh paying attention hopefully that was useful um i'm now going to hand you over to will who's going to hopefully draw some of these concepts together um and take you through how you can use some of the stuff you learned in this section to really build like RAG applications. So I'll stop sharing my screen now and over to Will. Thanks, Alex. Apologies, let me... There we go. I think that's worked now. Hey, Will, over to you. Thank you. I'm trying to share my screen. Can you see this? I think it's working. I cannot see your screen yet. Okay. Now, here we go. Okay, there we go. Yeah. All right. Yeah, that was quite a lot of content for the first half. Hope you're enjoying so far. Just take a deep breath a bit, just relax, stretch out a bit. And yeah, I think if you're ready, we can continue with the second half, which hopefully ties things together. It's going to be a bit more hands-on, but we are going to take a look at the Python package a bit closer. We're going to look at the different retrievers. So these retrievers are different types of retrieval objects that will help you with your Rack application. And finally, we are going to build a multimodal application at the end by combining textual and visual data. So if everyone's ready, let's go. So just to recap what RAG is, it combines retrieval-based methods with genitive language models to enhance response quality and accuracy. So what does that mean? So the keyword here is, as Alex has mentioned many times, search. And imagine when you prompt your LLM to give an answer. Before it gives you the answer, it searches over the database and then finally returns the answer to you. So by doing this, you can generate responses that are contextually relevant and factually correct. So what does that mean? So there are two main benefits here. One is your LLM is actually frozen. You don't have to train your LLM. You only need to search over your database. It's quite like a neat alternative to fine tuning your data, for example. The second benefit is your data stays private here so your LLM does not have had to seen your training data because you can keep it in your database. So yeah now I'm going to show you how you can use our open source python package Neo4j for graph regack, for Rack. Let's move on. Okay, and just wanted to double check here that everyone is, everyone has their Git pods set up, because we're going to use it a lot for this section. If not, yeah, maybe Alex can try and help you maybe in the chat. So, so here, okay. Yeah, so the first retriever we're going to look at today is the vector retriever. So we have many different types of retrievers in our package. And you can select each different types of retriever for your different use case. The vector retriever here is the most basic one. And it allows you to search over your vector index. So remember the vector index that you built in the previous section. You can use this Python object to search over your vector index. And what does that mean? That means you can search over the different vector embeddings that you store in the database. So how does that work? So let's say there's a, so yeah, you've stored all the vector embeddings in a database, you've built your vector index on top of them. Now, let's say you have an incoming query text, for example. So what you would do is you convert this query text using the same vector embedding method that you created with your stored vector embeddings. You use the same method, convert your query text into a query embedding. And then now you search for the most similar vector embeddings to your query text. So what this means is, so it's an approximate nearest neighbor algorithm that's used for this retriever. And what do most similar vector embeddings mean? If in embedding space they're very close to each other, that means it's very close in meaning. So that will correspond to the text being close in meaning or semantically. And you can not only create embeddings from text, like I mentioned before, you can also use audio images, any kind of unstructured data you can convert into vector embeddings. So let's open up GitPod now. We'll take a look at the first example, Factor Retriever. So don't forget you need to have your.m set up. Make sure you have your OpenAI API key and the Neo4j credentials as well. Okay, so I'm going to go through this. So the first slide here is really mostly setting up, connecting to Neo4j using the Python driver. The first thing we need to do is initialize our embedder. So this is the embedder that creates the vector embeddings. Here we specify this model, text embedding ADA2. And as mentioned before in the question and answers, you have to use the same embedding method for all your embeddings. Otherwise, it doesn't make sense to search over a different embedding space. So we are going to copy this. Oh, sorry, this is incorrect, actually. It's OpenAI embedding, it's not sentence transformers. We're going to copy this, paste it here. The next step, so now we have an embedder. Next step is we are going to initialize our vector retriever. And I'm going to go through each of these arguments, what do they mean? So copy this, paste here. So yeah, there are a couple things that go into this vector retriever. The two most important ones are the driver. So this is the Python driver we'll be using. We're connecting to this database. The index name you want to search over. So if you have gone through, if you ran the cipher statements in the previous section, you would now have movie plots in your sandbox database. So this is the vector index. This is the vector index we're going to be searching over. Here I provided an embedder, which is from here. So it specifies like, OK, whenever we have a new query text, we're going to use this embedder to convert the query text into a query embedding and here this argument return properties be it is something we provide to allow you to sort of return the properties that you want. Okay, and then finally to put everything together, we will copy this part. Paste it here, and yeah, this is how we use the Retriever. So, we provide this GraphRack object, and this GraphRack object requires a a retriever and an LLM. So retriever for the retrieval part, the LLM for the generation part. And the LLM here you can specify any supported LLM. So here I'm using the OpenAI LL lm and so this is different from the embedder this is the lm that will so the embedder so the retriever retrieves the context and then finally the lm is the one that you chat with so um the return result will be generated using an LLM, which the context is provided to. So now let's run this and see, run this. So the query text here is give me three films where hero goes on a journey. And I call rag.search here, where I provide the query text and I provide top K, so top K5. So what that means is, what is happening here is the retriever tries to retrieve the top five most relevant or most similar results to the query text, and then provides it as context to this llm and finally returns this result so as you can see here it provides three movies the last action hero never ending story resolve wizard of ours nice so that was the vector retriever. Hope it's clear so far. So yeah, here's some important tips that I recommend. Yeah, just going to reiterate, it's important that both the query embeddings and the stored embeddings are generated using the same embedding model. So in our case, remember it was text embedding ADA2. And yeah, it only makes sense to search within the same embedding space. Second step is you can utilize arguments such as return properties. And there's also another one called result formats, I think, to specify which node properties you want return. Or even you can transform the return Neo4j records. And lastly, there's an important parameter here, top K. Given this is, so what is this is retrieves like the top K most similar records in your database. And given this is an approximate nearest neighbor algorithm, the retriever, what it does is attempts to find the top k most approximately similar to your query vector. This is different from an exact nearest neighbor search. So sometimes I recommend setting your top k to be a bit higher than the number that you want, just three here. So sometimes it's better to set top k to higher values just so you can make sure you retrieve the relevant results. And yeah, so that's the vector retriever. I'm going to move on to... Is it important to set the temperature to zero? Or is that something you would... I mean, it's done here, so I was just wondering if that is another tip. It depends on your application. I'd say temperature for the OpenAI models is something that determines... It just makes it a little bit more deterministic. If, yeah, the higher your temperature, the more varied your output gets. I think, yeah, temperature is zero here mainly to, so that I can reproduce the result, but it depends on your application, I think. Okay, thank you. Yeah, good question. Actually here you can specify the different model params as well that goes into your OpenAI model. Because yeah, I think like if you've ever used the OpenAI API, you can set other parameters, there's temperature, there are quite a few others, you can specify them here actually, it's quite flexible. And yeah, that was the vector retriever we're gonna see how we're going to we're going to do something similar with another retriever so you might be asking now where does the graph come in and the answer is here this is the second retriever the vector cipher retriever. It's basically very similar to the vector retriever, except you can leverage the graph structure. So what does this mean? I'm going to show you this cipher command. I'm just going to run this. Actually, let's go through this first. If anyone here is familiar with Cypher, or not familiar with Cypher, Cypher is the query language for Neo4j. And what we're doing here is we're trying to find movie notes with the title Jumanji. And then we have small m here. So capital M here means like it's a movie node. And then we can set a variable here, small m. And we say, oh, we also want all the actor nodes that have this relationship with the movie nodes. So this means our actors that have acted in the movie Jumanji. And then we finally, yeah, we want to collect these actors. So if I run this, you can try running this as well. You see on the side here, yeah, it returns. So this is what is currently in the database. Jumanji, and then you can see, okay, there are four actors here. Nice. So how do we use this for RAC? The answer is you can use it with the Vector Cypher Retriever. So it's basically, the Vector Cypher Retriever is kind of, you can think of it as basically the vector cipher retriever is kind of, you can think of it as like the vector retriever, but you can add cipher statements to it to query things. So how is this good? We can, this means we can combine the knowledge of our graph with vector retrieval. And when would you do this? I'd say you typically do this if you understand your graph structure and you want to leverage and some kind of structure in your graph and like fully utilize your knowledge of this graph. So yeah I'm going to show it here. Here it's the same thing as before. So okay, now let's move on to the second example. I'm gonna close this. So we're in a vector-stackle retrieval example now. I'm going to copy this. So it uses the same embedding model as before. So that's the embedder. Now, okay, we have this. So I'm just going to copy this. I'm going to go through this quickly. So, okay, if you look at the vector cipher retriever, it takes very similar arguments to before. So you have the driver, which is the Python driver, Neo4j Python driver. You have your index name. So this is what we're searching over, the vector index. We have embedded again. Now we have this new argument, retrieval query. And I'm going to go through this. I'll zoom in a bit as well. So this is very similar to the Cypher statement we're looking at before. And you might wonder, okay, where does this node variable come from? So node is sort of like used internally in this retriever. And and it first does it first retrieves the it first retrieves the nodes that are most similar to your query text that you see later on and then you can find the actors that have acted in the movies. And then you can return these properties. So you return the movie title, movie plot, and actors. And this will eventually be provided as context to LLM. So let's try running this. Okay, so we can run this here. We need this last section which is kind of the same as before except I just changed the query text. I've copied that. I'll paste it here. I'm gonna use the retriever now. So yeah, as you can see this is the same LLM. We pass retriever and LLM to the GraphRack object. Now I ask here, who were the actors in the movie about the magic jungle ball game? So I'm going to run this. All right, nice. So it says the actors in the movie Jumanji, which is about the Magic Jungle board game, are Robin Williams, Bradley Pierce, Kirsten Dunst, and Jonathan Hyde. Nice. It actually managed to use the four actors from here. And then I think you can see the benefits. So for example, if you try to run the same query text with the previous one that didn't use the Cypher statement, let's see what this gets. So here, yeah, you can see it actually listed four actors. Bonnie Hunt was not in the database, although I think she was in the film. But yeah, it was probably like information stored by the LLM. But because we wanted to use data from a database, if you run it using a vector staff retriever, in the database, it was only these four actors. And this is what the LLM output. So it sort of grounds its knowledge according to your database. And I think now you can start to see, OK, why this can be very useful because oh yeah once you start I guess this is where like the benefit of RAC comes in the output of the LLM will be based off data in your database not not data that the LLM has seen on its own during training. So yeah, that's for the vector cipher retriever. Yeah, I think like the tips I recommend are similar to before, make sure you use the consistent embeddings. Make sure you utilize result formatter here so you can maybe transform the result, the Neo4j records or result that you want. And for this retriever specifically, you can really leverage cipher proficiency. So, for example, if I really understand how your graph database is structured, you can fully utilize this retriever. Is anything unclear before we move on? Because I think these two retrievers are, once you understand these two, it's much easier to understand the others. I'm just going to see if there's anything. Yeah. gonna see there's anything yeah um there was another a couple of questions alex answered most of those i think interesting one is um probably from why um they want to retrieve small similar documents from let's say 2023 or something in index but the index contains documents from many many years can you um can you define like a filter that then says um only show documents uh from from this specific year or with this specific um you know range or let's say you know value depending on uh on your query and could could you do this before pre-retrieval or post-retrieval or both? So I think the answer is you can definitely do it for both. Yep. And for example, if you know how to do that in Cypher, it's a query language, so you can use it as a filter. You can specify, oh, I only want movies from this year. And then, yeah, your retriever will only retrieve those notes. You can definitely also do it in post-filtering, although you probably have to think about, oh, yeah, how would I? Yeah, whether that's cost-efficient, things like that. Because at that point, how would I, yeah, whether that's like cost-efficient, things like that. Because at that point, you would have searched across the entire movie database. And if you only do it post-filtering, it's probably easier, but may not be the most efficient thing. So yeah, there's definitely a lot of things you can consider if you want to do filtering. Yeah. Do you have any any validation ideas um you can think of validation meaning so when when something is like here in the the the the actors for this movie do you could we do some kind of validation of or some you know way of proving sort of where where this where this knowledge comes from so that you can say okay with some kind of validation of or some way of proving sort of where this knowledge comes from so that you can say, OK, on this, I decide no because of this and that. VALIDATION. OK, I think, yeah. I think it will all be provided as context to that. So yeah, I think you can be sure it comes from the F4G database. Yeah. Yeah. Cool. OK, cool. I'm going to move on. So let's move on. That were the two retrievers. The retriever we're going to look at is the hybrid retriever. So if you remember from before, there were two indexes we talked about. So first one's the vector index, second one's the full text index. And we can make that useful here. So yeah, if you recall the full text index, it leverages keyword matching. And yeah, so why not we search over both of these so we can actually do that so this hybrid retriever leverages this not only the vector index but also so see in vector search let's zoom in event so you can leverage not only the vector index, but also the full text index. And what the hybrid retriever does is it searches over both indexes, and then it has a merging step at the end. So it ranks the retrieved results across both indexes, and finally returns the top key nodes here. So yeah, I'm just going to cover this quickly. Yeah, I mentioned this specter index, we search over a full text index and find us like a merging ranking step. And when would you want to use this? So when your data is diverse, for example, like in the 1375 movie from before, in some cases it doesn't make sense to only use like semantic search, but you also want to have a keyword search. So in those kind of use cases, you can consider using the hybrid retriever. Let's look at a practical example. So here, we'll close this. So we'll get a Hybrid Retriever. So copy and paste this. You only need to initialize the embedded for the vector index. Same as before. So now here we're going to copy paste the code for the hybrid retriever. Here we look at the arguments again. So yeah it's the Python driver so we're connecting to this database. There's the vector index we specified. We'll be plots. We're using the plot full text index from before. So I assume you've created the index in Alexis section. And there's the embedder, which we provide. And like before as well, you can specify which node properties you want returned. Let's run this. This looks the same as before. Now to protect this. What is the name of the movie set in 1375 in Imperial China? So I'm going to run this. OK. We're going to issue. No full text index. OK, I didn't create a full text index, it looks like. So I'm going to go back to the previous section and run this. So now I can show full text index. I want to make sure it's there. Okay, plot full text is there. I'll go back here. Run this again. Okay, turning to an issue here. Okay, live demos. Okay, ideally if you set this up correctly, it should work. I'm just going to double check here, if it's here. Okay, let's skip over this part perhaps. I think, Will, it might be an issue with the full text index name. I think the capital letter might be wrong. Oh, yes. That's a good catch. So thanks Alex. So here, it was the wrong full text index name. So, run this. There we go. So it says the name of the movie set in 1975 in Imperial China is Musa the Warrior. And I think if you go back to try this with the vector retriever, you can see how it compares. It says, okay, the provided context does not include a movie set in 1375 in Imperial China because remember the vector index was built over the plot descriptions. So maybe, you know, the term 1375 or Imperial China somehow just isn't in the plot. And here you can, you manage like leverage like a keyword search kind of thing basically. Okay, so hope that's clear. I'm gonna move on to the next retriever. Actually, yeah, just before that I'll, yeah, reiterate. Reiterate on some tips. Oh, yeah, reiterate. Reiterate on some tips. So yeah, make sure you use your consistent embeddings. Make sure you use the correct full text index name. And yeah, to effectively, to build effective full text indexes, make sure you create your full text indexes on relevant properties so that you can leverage like keyword search. And yeah, I'm gonna move on to the next one. So the next retriever we're gonna look at is the hybrid cipher retriever. And this is very similar to previous retrievers. So basically for this retriever, you can also leverage the graph structure for your hybrid retriever. And if, yeah, let's take a look, let's take a close look at this set for command this is very similar to one before so you want movies with this title movie set a warrior bracket musa and then you want all the actors that have acted in this movie so here in the round brackets it's the nodes square brackets is the relationship and you can also like specify the relationship direction. Let's run this. So yeah, you see like, okay, there's these four actors here. And yeah, ideally you want to like give this as context to our hybrid retriever. So yeah, like this diagram is very similar to the one before. You have, you search over both indexes, you have a ranking step, and then, and then finally, what is the final return one is you have an additional cipher query that can, you basically do a graph traversal according to the Cypher query and then what gets returned is, yeah, according to that Cypher query. So yeah, typically when you use this as similar to before you have like you have you have um quite quite complex data so like you want to search over like uh some something that gives like semantic meaning something that has like keyword search and then also you want to like traverse the graph so let's take a look at this close this. So now on the hybrid cipher retrieval example, we define our embedder, same as before. So now we, okay I'm gonna copy this, we define the inputs into our hybrid cipher retriever. So same as before, we have our driver, vector index, full text index, this is the correct one. No capital T this time. Now we specify our retrieval query. So this is very similar to before, our cipher query. We want to take the movie nodes, find the actors that have acted in this, like the retrieved movies, and then return movie title, movie plot actors. Use this as context for the LM. Okay, wonderful. So I need to copy this last bit. This is very similar to the hybrid retriever question, except now I want to know what are the names of the actors in the movie set in 1375 in Imperial China. This is a little bit different from the one before. Before we only asked what is the name of the movie, now we want to know what the names of the actors in the movie now we want to know what the names of the actors in the movie in the movie is so let's run this and nice okay so it's specified that, okay, this is the movie you're looking for. These are the actors that have acted in this movie. And how does the other them know this? It's because it's taking a look at this database, your Neo4j database, and then use it as context. So, great news. And yeah, here are some tips that I recommend to use this effectively, same as before. Have consistent embeddings. Use your full text index wisely. Use it on data that you can leverage like keyword search. And with this retriever, so the hybrid cipher retriever, you can really leverage your cipher proficiency. So same as the vector cipher retriever. Meaning, okay, if you have cipher scales, you can really encode it into this retriever. Okay, so we're going to look at the final retriever now. Something quite different, but it's still a retriever. And it's called the text cipher retriever now. Something quite different, but it's still retriever and it's called the text the cipher retriever. And what this does is say you don't want to manually write out your cipher statements. So before you for the vector cipher retriever and hybrid cipher retriever, you need it to you know, understand your graph structure and then you know write out cipher retriever, you need it to you know understand your graph structure and then you know write out cipher. Let's say you don't have to manually write that out. What you can do is get an LLM to do it for you. So in this example you have your natural text here on the left so yeah use English here to describe what you want and then on the right is the output of the LLM. So what comes out is the cipher statement. And this is what the Texas Cypher Retriever uses under the hood. So the Texas Cypher Retriever would allow users to interact with the database using just plain language. So you know it makes it easier to use, it's a bit more accessible. Use it with care though, because with any LLMs, the output can be non-deterministic. So if you do not set it up right, you can still run into LLM errors where the LLM generates invalid cipher. So okay, I'm just going to run this cipher command. So here what we're looking at is we want to find actors with the name Hugo Weaving, so the actor Hugo Weaving and the movie, all the movies that Hugo Weaving has acted in. So here you can see its return is eight films and we want to be able to get the LLM to use this as context. I'm gonna go straight to the example here. Copy this. I'm gonna go to getPot. I'm gonna go to git pod, I'm gonna close the others. Let's open the text-asypher retriever, copy paste this. So okay let's focus on what's going on here right now. There's a lot of things here, so here we import a text-asypher retriever, we use the LLM that the text-asyp things here. So here we import a Texas Cypher Retriever. We use the LLM that the Texas Cypher uses. So if you look at the inputs to the Texas Cypher Retriever, there's the driver, there's the LLM. There are a lot of additional things you can provide as context to this LLM, just to make things a bit more robust, make sure it outputs the right cipher query. You can provide the Neo4j schema, for example, here. Okay, like if you already understand some kind of structure in the database, you can provide it as a prompt to the LLM. You can provide examples as well. So that's, I think I find this very helpful. So yeah, like all these are used as like context to this LLM to generate the correct cipher that can be used as a retrieval query. And finally, so now that we've defined the retrieval, we're going to use it. So this is the same thing. So yeah, the same thing here. We plug it into, we plug retriever and LLM into a GraphRack object. And then now we have this query text, which movies did Hugo Weaving star in? So ideally the LLM should use the eight films that we saw earlier. Going to run this now. Eagle weaving start in the following movies. And yeah, I think these are correct. So if you take a look at yeah, yeah I think that was correct. So I'll text the cipher retriever managed to generate its own cipher and retrieve the relevant movies that Hugo Weaving has started. So actually, if you want to contrast this, if you try to ask this question to the vector retriever, so the vector retriever doesn't use cipher, it just does it off similarities such as the vector index. Here, it does not know the context. So that's very helpful with the text-asset for retrieval. And yeah, here are some tips that I recommend. Yeah, make sure you provide clear inputs into your text asset for retriever. So whatever you can, like schema, example queries. And be prepared to handle errors as well, because it's using an LLM under the hood. It's non-deterministic, so the generated query could still be incorrect. Nice, just to recap a bit, we've covered five different retrievers and now we're going to look at a more practical example. So here we're going to use the VectorCypherRetriever, the second one, the one that does similarity search with a graph traversal. We're going to build a multi-model application here now. So okay let's take a look at the data that we want to use. Well, I'm just going to run this quickly just to show you. Here, if you look, I'm searching for movies with the title Homeward Bound, The Incredible Journey. Return the actors that have acted in this movie as well. So yeah, here you see like four actors. And if you click on the movie node here, capital movie, you can see, okay, there's a plot description. There's an embedding of the plot description that we created earlier. There's also a poster, a link to the poster. There's also a poster a link to the poster there's also a poster embedding so this was the image embedding that we created from the poster image before and we're going to try and search over this poster embedding actually so let's take a look. So here we initialize the embedder. Close this. Okay, first we initialize the embedder. So the embedder we are using is this clip vitv32 embedding model that was created using the one from sentence transformer. And remember I mentioned like, okay, you have to use consistent embeddings. It's because this post embedding here was created with this model. So whenever we want to have new query text later, we have to make sure we use this correct one. So I'm going to copy paste this. And now, OK, the retrieval bit. So close this. We can take a look. Yeah, here. We are going to search over the movie posters embeddings. So the poster index name is movie posters. We're going to use this retrieval query here. So after we retrieve, after we search like, okay, what is the most relevant images? We want to return the title of that movie, the plot, and also the poster itself. So okay, we have our retrieval query, we have our post index, now we can like properly use our I'm going to copy paste all this here. Got a bit of code. Yeah, most importantly, you look at the inputs into the vector separate retriever. You have our driver, index name, the retrieval query specified. There's this result formatter, which I didn't show before. But it basically transforms the return result from the initially the initial return result from the retriever and yeah it transformed and tries to return something in a more structured way so that we can use now the query text I was I'm trying to get is, find a movie where in the poster there are only animals without people. And let's try and see if this works. Here it is. So they should print out the content, the score, which is like the similarity score, and a link to the poster image. So yeah, I asked, okay, give me movies where there are only animals in the poster and no people. Okay we got three movies here Homeward Bound, The Incredible Journey, Lassie and Babe. So we also have to return the plots and then these images so if you click on the link okay this one all animals no people it's very cool. Let's open this as well. Okay, it's all animals. Oh, there's one person there. So not exactly correct, but hopefully mostly correct. This one as well. Bunch of animals. Okay, vaguely some people in the middle there. So maybe you can forgive the embedding here. But I'd say it was pretty close. So yeah, that was the multimodal search application that we built. And yeah, I recommend like... So here we're using image embeddings instead of text embeddings. Make sure you use the correct embedding model for your query text and your stored data. So, you know, if possible you can even like utilize the text index somehow, you know, if you try to be clever with your cipher a bit. And yeah, you can see like how you can integrate across like different multi-modalities of data. So yeah, that was it, gonna wrap up now. Congratulations, you successfully cover a lot from this workshop. You learned about five different retrievers. You learned about, well, you managed to build a multi-modal app. And you managed to figure out how to use the Neo4j GraphRack package. So yeah, go and build something cool with it and let us know how it goes. And yeah, as to finish off, like here's like a table comparing the different retrievers. Yeah, feel free to take a look at this as well. So I'm gonna, yeah, that's it for me. I'm gonna hand it back to Alex. Super, thank you very much, Will. Thank you also, Alex. Great session today, great workshop. Thank you all for watching. And sorry, we couldn't answer all your questions. I think we have a couple of open ones, but I think it was really good. We tried to answer most, but I think we were able to cover the most important bits here. And it was really good um we we tried to answer most but i think we we were able to to cover the most important bits here and it was great to you know especially this this table i think is a is a great finishing point here with all these retrievers we covered a lot um even if it uh time went by quickly uh so we had six different five different retrievers uh covered in this session and we also explained all the basics. So really, really good workshop. So yeah, I'd like to thank Will and Alex again for presenting, for preparing the workshop and for guiding us through. Thank you all for watching. I mean, like I said in the beginning, if you are wondering where to find the material and everything, there's going to be something sent to you with the recording, with the link to this Graph Academy course in the coming days. So you can also catch up on those later. If you are eager for more, then we have another Road to Nodes workshop happening next week. This time we cover Neo4j Bloom, which is Neo4j's visualization platform. So you can learn how to visualize a graph in there and how to make sense of your data in a more visual aspect. And then we're running this very workshop again in two weeks' time on the 24th of October, but in a different time slot. So if you want to come again, you can obviously join us again in two weeks' time. And yeah, this all leads into our big conference, online conference, Nodes 2024 on 7th of November. So we'd really like to join us there with lots of workshops, lots of sessions. We have a dedicated AI track with many, many presentations from Neo4j staff, but also from our many community members so if you're interested in that join join it and and have a listen to all the the interesting presentations you can check out the agenda already and see what what's interesting to you and yeah if you are you know wondering what to do next uh join our community at the forum or on our discord server you know look at other graph academy courses there is just one recently being published on building knowledge graphs there's another one how to build a chatbot with neo4j so i think that's a it's a great add-on sort of to this one so if you are now you know know how to work with neo4j and build graphic applications i think you could potentially build a chatbot with this knowledge and and create interesting applications with gen ai and yeah if you want to do more with neo4j beyond the learning platform check out aura or a free which is our graph database as a service we can just try for free and have a database running just for you and yeah i think with that we are at the end, right on time, a little bit over, but not too much. Again, thank you all for watching. Thank you, Alex and Will, for this cool, cool presentation, the cool workshop today. I hope you had fun, everybody, and you learned something, and yeah, see you around the world of graphs i would say take care and
Road to NODES: Mastering Retrieval-Augmented Generation with the GraphRAG Python Package
In this workshop, we use the neo4j-genai-python open source package for Retrieval-Augmented Generation (RAG) applications. Learn to integrate Neo4j with generative AI models to enhance graph-powered applications and AI solutions.

GraphAcademy Course: https://bit.ly/4dKPgyg
Repo: https://github.com/neo4j-graphacademy/genai-workshop-graphrag

#neo4j #graphdatabase #graphrag #llm #genai #workshop #python #pythonprogramming