 Hello community! You ask me, hey, is it possible to connect multiple agents to a knowledge graph? Yes, of course we can! And I will show you today here a new methodology that stands for Adaptive Generative Engine for Task-based Interaction and Graphical Representation Agent Interaction Graph and it is a novel platform to design that is designed to bridge the gap between llms and knowledge graph for multiple agents beautiful this is here the paper october 15 2024 university of tokyo university bielefeld duke medical school universidad de alicante yale university xiamen university canal medicine and hinan university. Beautiful. Eight institutions, eight famous institutions. Now, we will use our old friends, our tools that you know, chain of thought reasoning, a React framework will be implemented in our agents. And we use in-context learning, where we will apply few-shot learning. And this is the way we will build our prompts for the LLMs. Now here we have seven agents, aren't they beautiful? So let's have a look, just look at the name and you might understand what we are talking about. The user intent interpretation agent, the key concept extraction agent, the task planning agent, the knowledge graph interaction agent, we say, ah, here it's happening, the reasoning agent, the response generation agent, and the dynamic knowledge integration agent even want to feed new nodes and edges back to our knowledge graph. So, interesting, eight institutions, seven new agents. Let's have a look. Let's say I have a user query. I say, hey, I want to understand the relationship between quantum entanglement and teleportation. What foundational concepts should I learn first? The user intent interpretation agent identifies this now as a particular request, and there is a predefined class. This here is called a prerequisite prediction, and I will show you the different other classes in a minute. So we have predefined structures that we use. The key concepts extraction agent says okay I extract here the object's entanglement, teleportation as key concepts. The planning agent has now to do here to determine here the foundational concepts leading to or in the environment of quantum entanglement and teleportation and map out a learning path to answer the query. The knowledge graph interaction, you get it. It generates here some SPACO queries or cipher queries or whatever to retrieve concepts from the knowledge graph. And the reasoning agent analyzes to retrieve data to identify the most relevant foundational topics. And the response generation agent crafts the response. And then the dynamic knowledge integration agent updates the knowledge graph if we discovered some new relations that are not established in the knowledge graph. And here we have the official documentation visualization. We have a user input. We have a multi-agent framework that interacts with a knowledge graph. And an output is generated. And this is here a rather simple representation of a particular workflow. So we have here the task definition, and then we have agent 1 to 7. And then somehow all those agents interact in a specific way with a knowledge graph, and this is it. Well, we have to have a closer look now. And you might say, wait a second, this sounds familiar, no? And you say, hey, look at this video where you showed us here the interaction between an LLM and a knowledge graph for the new GIF methodology. And over there, you showed us here Sync on Graph, deep and responsible reasoning of large language models on knowledge graph. And although this is already here for March 2024, we had a look at this and we discovered that instead of just simply fetching here facts from a knowledge graph with a simple SPARQL query, the LLM could here actually sink its way through the knowledge graph and explore different reasoning paths through the knowledge graph like a detective gathering the clues. And this is what I told you that Sync-on graph is rather beautiful for. And you remember I showed you this. We had just an LLM plus a knowledge graph. And then we looked here at Sync-on graph. This is an LLM tensor product knowledge graph so this means a stronger interconnect with the knowledge graph plus was simple we generate a sparkle query or any other query that you like from a graph database you have a question you got a response but if there is a link missing if your knowledge graph is incomplete, and most cases knowledge graphs are incomplete, you have a problem here. But not here, because as I told you, here on the example of Canberra, Australia and Antony, here we have now that we built here subgraphs here of the epsilon environment here of our particular topics. And then we were able to deduct the information from the query. And you remember the mechanism that I told you how Sync on Graph works, that it couples here the LLM with a knowledge graph using the beam search algorithm and this was the beauty that we do not just go with a greedy algorithm but beam search allowed us here the llm to explore multiple possible reasoning paths at once in parallel and then choose here depending on the width of beam, to choose the top three or top five causal reasoning structure. Now, this here is not the case. So this is a step back, because now we just have simply a greedy query. So you see, I ask myself reading this, wait a minute. So I understand perfectly if you have an LLM and a knowledge graph and you have simply your SPOGL query. But how do the authors here in this new paper go now to a closer interrelation between the seven agents and the knowledge graph. What is the mechanism? Because it is not, as I showed you here on Sync on Graphs, a beam search. It must be something different. So what is it? Now, we have to dive, have now a deep dive into every agent in the agenti graph. And I ask here our little strawberry here to provide a simple example. After I had some exchange, we have strawberry on the topic of this paper. So let's have a look. But yeah, let's make it a little bit bigger here. First detail we have. How many points we have? You have 7 points, points seven agents so the first three agents what are they doing now exactly and the query is hey how does photosynthesis relate to cellular respiration and what concept should i understand first you see very clear predefined question what a coincidence so let's have a look. So the user intent interpretation agent determines that the user is asking for both the relationship between two concepts, photosynthesis and cellular respiration, and any prerequisite knowledge. And now you might say, ah, now I know where the epsilon surrounding of our topic are coming from. So we have here the output is relationship judgment and prerequisite prediction. This is here our interpretation what the user wants from the AI. Then we have the key concept extraction agent. Now it is happening. We extract now the entities from the user query and you see it is photosynthesis and cellular respiration. And it also extracts here the relation between those two entities is relate to, exactly as in the user text. So now that we identified what it is really all about, we already used up two agents. Okay, let's go with this. And then comes another interesting part, the task planning agent. So whenever we have some intelligence that is invested here in the planning of the strategy of the whole system, now it gets interesting. The task one and two, find the relationship between photosynthesis and cellular respiration. Yes, this is exactly the question. And identify prerequisite concepts for understanding these processes. So we are now looking, if you want, for additional information explaining, maybe in the simplest case, what is photosynthesis? Where is here connection to, I don't know, animals or to plants or to whatever we have. What is cellular respiration? What is a cell? What is respiration? How is this combined? What is the latest research on the topic? So anything that has a prerequisite concept, we're looking for. So those are now these two tasks that we are looking for. And now the AI comes and says, now the knowledge graph interaction agents comes and says, hey, I have here a knowledge graph for me. Structured knowledge is available for me. So it builds now, it constructs now and executes here a database query, a graph database query, a cipher query, or a Sparkle query, whatever you have. It doesn't matter what system you're using, for both tasks, one and two. You remember my, hey, wait a minute, wait a minute. Here, task one, task two. Okay. So, cipher queries for both tasks. So, now we understand where we get the structured knowledge from. And and then retrieves the relationship and prerequisite concepts from the knowledge graph now this is a little bit vague because how does it do this now let's just follow along the reasoning agent analyzed the result determined that photosynthesis and cellular respiration are complementary processes that identified at the prerequisite in our case here identified by the ai given the knowledge graph that was available included basic chemistry cell biology energy transfer and in the end and then we have a response generation agent and a dynamic knowledge integration agent great yeah they implemented this here with a graph database on Neo4j, but whatever, you don't have to use SciFag, you can go with whatever you like. But you see, now it is interesting that we just have here a graph database query. But where does this relationship and the prerequisite concepts from the knowledge graph come from for the LLM to understand and integrate. Because where we are is more or less we are here now. We have an LLM plus a knowledge graph. And we have here a Spark query or a Cypher query or whatever you have. And if there is a missing link, you have no way to find an alternative route between those two entities. So how does it do this, that it is, you would say, equivalent here to our more interlinked representation of an LLM in a knowledge graph, like in SyncOnGraph. Now at this point here, I looked here at explicitly the agent and here the user intent interpretation agent and I used and I looked here at the prompt this is the absolute concrete prompt that the authors used so you see here this is what defines what an agent is doing one you are an expert NLP task classifier specializing in knowledge graph interactions, and analyze the given query and classify it into the following categories. So we do not go with the complete free search space, but we say, hey, either relation judgment, prerequisite prediction, path searching, concept clustering, subgraph completion, idea hamster, or then everything else, a freestyle question. And then they provide examples, beautiful, in-context learning is happening, and then you see, great, and then please answer in the following JSON format. Okay, this is point number one. Now it gets interesting, because now we have the key concept extraction prompt. And this is the official real prompt. And they say, now as you are an advanced concept extractor, whatever this means, your task is to identify and extract key concept entities and relationships from a given user query, using here our old friends here the named entity recognition and the relation extraction techniques that we know now for dozens of years and you will then map these to the knowledge graph schema using bird derived vector representations for semantic similarity and i give here examples and beautiful And this is exactly where I also said, wait a minute. You will then map these to the knowledge graph schema using BERT-derived vector representation for semantic similarities. I have three immediate ideas how I do the mapping. So which kind of mapping is it in detail? What are you talking about here? And unfortunately they are rather vague here in their detailed explanation. It is just this sentence. And you see here a screenshot and it tells us here, yep, maps extracted entities to the knowledge graph by semantic similarity with BERT-derived vector representation. So we have to have now a two-step process suddenly within the extraction agent where we have new objects. So suddenly we don't have seven agents and a knowledge graph, but suddenly we have to build here, if you want here, a birth model, kind of a sentence transformer. What I would do, and this is only one option how we do the mapping, is this one. Let's say, and I'll show an example to make it really clear, how does global warming influence sea ice melting and what are the main factors involved? So you see, we need some predefined knowledge here. What are those factors? So we do not just have a relation between global warming and sea ice melting with the object influence, but we want to exactly know how and why and what is in the thematic clustering here of those terms. So this is my idea how it is could be done, but if this is the exact case, or maybe there are three, four, five other mapping technologies, I don't know, but I want to explain this here, and this is the simplest case if you want. So step one, we extract the entities and the relations from the query. Easy, no? Global warming, sea ice melting, beautiful. And the relation we have is influence and maybe you go even with main factors involved. Then if we use BERT, we have embeddings, we have vector representation in a mathematical vector space. We now generate embeddings for the query entities and the relation that we just found. So using now a BERT system, but hey, stop, we have to build a BERT system, no? We have to build here a complete vector space where our semantic similarities in the linguistic text is now encoded in a close-by relationship in a mathematical vector space. This is the whole sense of using BERT. So we have now to embed here each and every term. So we have the BERT embedding of global warming, BERT of CI smelting, and you get the idea. So we have one set of vectors if you want. But how do we get the knowledge graph embeddings? Now, either you buy those here, let's say from OpenAI, or you use your own BERT that you created for your domain-specific knowledge. So let's have a look. And it is in the knowledge graph. Let's say that in your knowledge graph somebody built those system and they are a little bit different terms technical terms you have to label climate change and not global warming and you have ice cap melting instead of sea ice melting and you get the idea so we now extract here from a knowledge graph embeddings, and there's a simple methodology that we can do this, not only the entities, but also the relations. And then, but only then, if we have the embeddings for the query and the embeddings in the knowledge graph, then we can compute the similarities. So you see the global warming here. Here we have the global warming. And the climate change that is here in the knowledge graph, they have now a similarity in the constructed bird system that we built out of thousands and thousands and thousands of documents. They are now really semantically similar and if we go with sea ice melting and ice cap melting we also get here the highest correlation between those two terms and sea ice melting is of course the term that we used in the query and ice cap melting is the term that we have available in the knowledge graph So if you say now there's a mapping between those, this is the easiest way I can imagine to perform the mapping. But I have multiple other ideas how to do this mapping. So you see, it is not really specific. Or if you found here the hint how they do it, please leave a reply. Step five, mapping to the knowledge graph. And here it's happening. So we have the global warming as I told you here from the query. We map this now to the node in our knowledge graph and the best node is climate change. And for the sea ice melting our semantically most close by node is ice cap melting in the knowledge graph. So now we have now an allocation of the query to the embeddings in the knowledge graph. My goodness. Yeah, impact on knowledge graph query. Now we can have, for example, here with Cypher. We formulate now a query in Cypher to find a relationship between climate change and ice cap melting through the cause relationship. But you see, this is quite some step, no? Because either, okay, you buy here the embeddings from some company that you like or wherever you are, or from your LLMs. But normally I build my own bird systems because they are highly domain specific. Only for theoretical physics, I have 12 BERT systems. So you see, you have to have those terms in your knowledge graph and you have to have those terms in your embedding structure. Great. This was agent number two. Then we have the task planning prompt. This is here really the prompt that generates you the agent. And we say, hey, as the task planning agent, your role is to decompose the identified user intent into a logical sequence of executable tasks for the knowledge graph interaction. Create an optimal plan, considering now that we know the task dependencies and the execution order. Here's an example. and you provide in context learning look do this then do this beautiful now agent number four you know the knowledge graph interaction you're not going to believe it here as a knowledge graph interaction again your task is to translate high level task into executable graph queries we now have our sci-fi queries, our SPARQL queries, utilize Vue short learning and React framework to generate and define queries dynamically. Here are the examples, beautiful example, beautiful in-context learning, Vue short learning. Then we have the reasoning agent prompt, apply logical inference to the raw query results, leveraging contextual understanding now from the LLM and the reasoning capabilities from the LLM. The raw query results come from the knowledge graph. So now you bring them together and you use now the intelligence of the knowledge graph for causal reasoning. Examples, beautiful. And then response generation prompt. Yeah, I guess it is clear now to us how this works. Dynamic knowledge integration prompt. And those were seven agents. And the prompt that generate those agents and define here the name, the scope, the role, the example, and the task and the output former, the schema, how they exchange information between seven ei agents and one knowledge graph or maybe you have multiple knowledge graph then i would put in another eight agents to do the coordination here of the information flow to multiple knowledge graphs and the authors tell us here our agenda graph has been rigorously evaluated and the authors tell us here the performance matrix is achieved 95% accuracy and task classification and 90% success rate in task execution. Okay, beautiful. And they tell us here, hey, we have two mode. Either you can chat here with the the llm or you can have here an exploration mode that you discover here new relations beautiful so you see a new way with seven agents here that work together interact here in a very specific way i showed you here the specific prompts of each and every agent and this is how the system is built up. Okay you might say now but we know that in certain aspects we could optimize here this particular structure because if we go back let's go back here to the general form now here we know for some agents there are already optimization routines available like a beam search mechanism or how we integrate here the knowledge graph in a more refined way yeah but this is as always so some beautiful research work presented to us here by eight beautiful institutions and they show us here their way plus we immediately know and if you are a subscriber of this channel you immediately know where are the points where you can say hey i just learned from the other videos that we can do further optimization. And you see, this is the beauty of currently AI research. Reading all this beautiful publication by all the AI researcher, you understand that each and every task that you might have, there are hundreds of people that already tried to optimize this particular task. And in the end, you can get maybe a real optimization of your complete methodology. So there we are. This is exactly what I set out to do. I hope you enjoyed it. I hope you had a little bit of fun. Maybe it was informative and it would be great to see you in my next video.
Seven AI Agents and a Knowledge-Graph: AGENTiGraph
AGENTiGraph introduces a new framework that integrates Large Language Models (LLMs) with Knowledge Graphs (KGs) through a multi-agent system, addressing key limitations in current AI models for complex, domain-specific tasks. Explore multi-agent dynamics with a KG (knowledge-graph). 

The system employs specialized agents - each leveraging LLMs - to interpret user intents, extract key concepts, plan tasks, interact with the KG, perform reasoning, generate responses, and dynamically integrate new knowledge. By decomposing user queries into manageable tasks and leveraging the strengths of both LLMs and KGs, AGENTiGraph enhances factual consistency, reasoning capabilities, and adaptability in handling intricate queries.

A central method in AGENTiGraph is the semantic mapping of extracted entities and relations from user queries to the KG using BERT-derived embeddings. The Key Concept Extraction Agent performs Named Entity Recognition (NER) and Relation Extraction (RE) to identify relevant concepts and relationships in the query. Both the extracted elements and the KG's entities and relations are embedded into the same semantic space using the same BERT model. By computing cosine similarities between these embeddings, the system accurately maps user inputs to corresponding nodes and edges in the KG, even when terminology differs, thus enabling precise information retrieval and interaction with the knowledge graph. 

all right w/ authors:
AGENTiGraph: An Interactive Knowledge Graph Platform for
LLM-based Chatbots Utilizing Private Data
https://arxiv.org/pdf/2410.11531

00:00 Multi AI agents and a Knowledge Graph 
01:08 7 Agents with specific functions
01:43 Simple example and all 7 agents
03:03 Official AgentiGraph Workflow
03:51 Open questions and Think on Graph (ToG)
07:02 Every Agent explored with its PROMPT 
14:39 Map entities to knowledge graph embeddings
22:05 Performance Metrics of Agentigraph

#airesearch 
#ai 
#aiagents 
#knowledgegraphs